Aborting Incomplete S3 Multipart Uploads with MinIO Client
Multipart Uploads allows you to upload large files (up to 5 TB) to the Object Storage platform in multiple parts. This allows faster, more flexible uploads.
If you do not complete a multipart upload, all the uploaded parts will still be stored and counted as part of your storage usage. Multipart uploads can be aborted manually via the API and CLI or automatically using a Lifecycle rule.
If you use the API or the AWS CLI, you will have to abort each incomplete multipart upload independently. However there is an easier and faster way to abort multipart uploads, using the open-source S3-compatible client mc, from MinIO. In this tutorial, we show you how to use mc to abort and clean up all your incomplete multipart uploads at once.
You may need certain IAM permissions to carry out some actions described on this page. This means:
Follow the instructions given in the official MinIO documentation here to install the MinIO client (
mc) for your OS.
For example, on Linux:wget https://dl.min.io/client/mc/release/linux-amd64/mcchmod +x mcNote:
Make sure that you download the binary somewhere that is in your
mc --helpto check it has installed.
From the same page as before, follow the instructions to add a cloud storage service:mc alias set <ALIAS> <YOUR-S3-ENDPOINT> <YOUR-ACCESS-KEY> <YOUR-SECRET-KEY> --api <API-SIGNATURE>
For example, if your Scaleway Object Storage endpoint is in the fr-par region, you can use the following command (replace the two fields in pointy brackets with your access key and secret key):mc alias set s3 https://s3.fr-par.scw.cloud <YOUR-ACCESS-KEY> <YOUR-SECRET-KEY> --api S3v4
You should now be able to run some basic commands, eg:
List all your buckets at the endpoint you configured in the previous step, via:mc ls s3
List all objects in a given bucket, via:mc ls s3/<your-bucket-name>
Remove an object from a bucket, via:mc rm s3/<your-bucket-name>/1.pdf
…plus all other commands detailed via
- Use the basic
mc lscommand we saw above but with the added
-I(incomplete) flag to list all incomplete multipart uploads:mc ls s3/<mybucketname> -I
- Use the basic
mc rmcommand we saw above, but with the added
--forceflags to abort and clean up all incomplete multipart uploads:mc rm s3/<mybucketname> -I -r --force
Your incomplete multipart uploads are now aborted and all the parts cleaned up, in one simple step!