Aborting Incomplete S3 Multipart Uploads with MinIO Client

• minio
• multipart-uploads

S3 Object Storage - Multipart Upload Overview

Multipart Uploads allows you to upload large files (up to 5 TB) to the Object Storage platform in multiple parts. This allows faster, more flexible uploads.

If you do not complete a multipart upload, all the uploaded parts will still be stored and counted as part of your storage usage. Multipart uploads can be aborted manually via the API and CLI or automatically using a Lifecycle rule.

If you use the API or the AWS CLI, you will have to abort each incomplete multipart upload independently. However, there is an easier and faster way to abort multipart uploads, using the open-source S3-compatible client mc, from MinIO. In this tutorial, we show you how to use mc to abort and clean up all your incomplete multipart uploads at once.

Before you start

To complete the actions presented below, you must have:

• A Scaleway account logged into the console
• Owner status or IAM permissions allowing you to perform actions in the intended Organization
• Generated your API keys
• Created an Object Storage bucket
• Started a multipart upload

Installing MinIO Client (mc)

1. Follow the instructions given in the official MinIO documentation here to install the MinIO client (mc) for your OS.

   For example, on Linux:

   wget https://dl.min.io/client/mc/release/linux-amd64/mc
   chmod +x mc
   

   Note

   Make sure that you download the binary somewhere that is in your $PATH

2. Run mc --help to check it has been installed.

3. From the same page as before, follow the instructions to add a cloud storage service:

   mc alias set <ALIAS> <YOUR-S3-ENDPOINT> <YOUR-ACCESS-KEY> <YOUR-SECRET-KEY> --api <API-SIGNATURE>
   

   For example, if your Scaleway Object Storage endpoint is in the fr-par region, you can use the following command (replace the two fields in pointy brackets with your access key and secret key):

   mc alias set s3 https://s3.fr-par.scw.cloud <YOUR-ACCESS-KEY> <YOUR-SECRET-KEY> --api S3v4
   

4. You should now be able to run some basic commands, eg:

   • List all your buckets at the endpoint you configured in the previous step, via:

     mc ls s3
     

   • List all objects in a given bucket, via:

     mc ls s3/<your-bucket-name>
     

   • Remove an object from a bucket, via:

     mc rm s3/<your-bucket-name>/1.pdf
     

   …plus all other commands detailed via mc --help.

Aborting incomplete multipart uploads with MinIO client (mc)

1. Use the basic mc ls command we saw above but with the added -I (incomplete) flag to list all incomplete multipart uploads:

   mc ls s3/<mybucketname> -I
   

2. Use the basic mc rm command we saw above, but with the added I (incomplete), r (recursive), and --force flags to abort and clean up all incomplete multipart uploads:

   mc rm s3/<mybucketname> -I -r --force
   

Your incomplete multipart uploads are now aborted and all the parts are cleaned up, in one simple step!