Jump toUpdate content

Managing multipart uploads

Reviewed on 15 June 2021Published on 19 April 2019

Multipart uploads allow you to upload large files to the Object Storage platform in multiple parts. It is possible to store objects up to 5TB per object on the platform.

A multipart upload consists of three steps:

  • The client initiates the upload to a specific bucket
  • The different parts are uploaded
  • Object Storage constructs the object from the uploaded parts

Once the object has been constructed, it can be accessed as any other object in the bucket.

The following technical specifications apply to multipart uploads:

  • 1 to 1000 parts per object
  • 5 MB to 5 GB per part (except for the last one)
  • Each object stores up to 5TB

Initiating a multipart upload

When initiating a multipart upload, the Object storage platform generates a unique identifier for the multipart upload. This identifier must be included whenever parts are uploaded, listed, the upload completed or to abort an upload.

To initiate a multipart upload, the client sends a POST request as follows:

<bucketname>.s3.<region>.scw.cloud/<object_key>?uploads

Sample request

POST /large-file.tar.gz?uploads HTTP/1.1
Host: bucketname.s3.nl-ams.scw.cloudx-amz-content-sha256: 4cf002db03f94b9ab876f38ad65863c987af96c6526673cfee22fad443fee9cbx-amz-date: 20190517T125417ZAuthorization: AWS4-HMAC-SHA256 Credential=SCWGJ3BW6Q22HCEB16GJ/20190517/nl-ams/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=4e6b0dea0badf41e2aafca17017d479138ce1d34e1c3fcf2df00193ae3c68c1c

Sample response

HTTP/1.1 200 OKx-amz-id-2: tx0db175df07644b858daab-005cdeaef9Content-Length: 265x-amz-request-id: tx0db175df07644b858daab-005cdeaef9Date: Fri, 17 May 2019 12:54:17 GMT
<?xml version="1.0" encoding="UTF-8"?>  <InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01">    <Bucket>bucketname</Bucket>    <Key>large-file.tar.gz</Key>    <UploadId>Y2VhZDUxYjAtNjVjMC00NTIxLWEwNWUtNWM3NDAyOTQ4ZWYz</UploadId>  </InitiateMultipartUploadResult>'

Uploading a part

Once a multipart upload has been initiated Object Storage keeps all parts of the upload until it either completes or is being aborted. When uploading a part, a part number must be specified in addition to the upload ID returned when initiation the multipart upload. The part number can be any number between 1 and 1000 and identifies uniquely the part and its position in the object being uploaded.

During the lifecycle of a multipart upload, invoicing is taking place for all storage and bandwidth usage for its associated parts. If a multipart upload is aborted, all parts already uploaded are deleted and billing for these resources stops.

To upload a part of an object in an initiated multipart upload, send a POST request as following:

<bucketname>.s3.<region>.scw.cloud/<object_key>partNumber=<part_number>&uploadId=<upload_id>.
  • <part_number specifies the position of the part uploaded inside the object.
  • <upload_id> is the unique identifier returned during the initialization of the multi-part upload.

Sample request

PUT /large-file.tar.gz?partNumber=1&uploadId=Y2VhZDUxYjAtNjVjMC00NTIxLWEwNWUtNWM3NDAyOTQ4ZWYz HTTP/1.1host: bucketname.s3.nl-ams.scw.cloudx-amz-content-sha256: 4cf002db03f94b9ab876f38ad65863c987af96c6526673cfee22fad443fee9cbx-amz-date: 20190517T131738ZAuthorization': AWS4-HMAC-SHA256 Credential=SCWGJ3BW6Q22HCEB16GJ/20190517/nl-ams/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date

Sample response

Content-Length: 0x-amz-id-2: txc9ed83e20cd84f56b826c-005cdeb472Date: Fri, 17 May 2019 13:17:38 GMTETag: "d41d8cd98f00b204e9800998ecf8427e"x-amz-request-id: txc9ed83e20cd84f56b826c-005cdeb472

Completing a multipart upload

To complete a multipart upload session and to generate the final object, a POST request as following is required. <upload_id> corresponds to the unique identifier returned during the initialization of the multi-part upload:

<bucketname>.s3.<region>.scw.cloud/<object_key>?uploadId=<object_key>

The request requires that an XML element named CompleteMultipartUpload containing information about each part of the multipart upload is present. The element requires the following contents:

  • Part: Information about each individual part of a multipart upload.
  • PartNumber: A sequential identifier specifying the part of the object.
  • ETag: The entity tag (Etag) containing an MD5 hash of the objects part.

Sample request

POST /large-file.tar.gz?uploadId=Y2VhZDUxYjAtNjVjMC00NTIxLWEwNWUtNWM3NDAyOTQ4ZWYz HTTP/1.1
X-Amz-Date: 20190517T140506ZX-Amz-Content-SHA256: 4cf002db03f94b9ab876f38ad65863c987af96c6526673cfee22fad443fee9cbAuthorization: AWS4-HMAC-SHA256 Credential=SCWGJ3BW6Q22HCEB16GJ/20190517/nl-ams/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=2ab2471b862268c3509aad01974153fc44455f7f6998c0e3e7fbdfaf8f4f5efdContent-Length: 0
<CompleteMultipartUpload>  <Part>    <PartNumber>1</PartNumber>    <ETag>"4cb0d956600bab287520b442e83035a6"</ETag>  </Part>  <Part>    <PartNumber>2</PartNumber>    <ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag>  </Part></CompleteMultipartUpload>

Sample response

HTTP/1.1 200 OK
x-amz-id-2: tx321faae63fb14971a8d8a-005cdec63eDate: Fri, 17 May 2019 14:05:06 GMTx-amz-request-id: tx321faae63fb14971a8d8a-005cdec63eConnection: close
<?xml version="1.0" encoding="UTF-8"?>  <CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">    <Location>bucketname.s3.nl-ams.scw.cloud</Location>    <Bucket>bucketname</Bucket>    <Key>large-file.tar.gz</Key>    <ETag>6eb393dac21d595a813f0c8bf4e9bc23</ETag>  </CompleteMultipartUploadResult>

Listing multipart uploads

This command lists in-progress multipart uploads. An in-progress multipart upload is an active multipart upload request that is still ongoing and has not yet been completed or aborted.

The response of this command includes a list of all ongoing multipart uploads, with a maximum of 1000 in-progress multipart uploads, ordered by their key.

To get a list of in-progress multipart uploads, send a GET request to the URI of the multipart-uploads subresource of the bucket. Use the max-uploads attribute to limit the number of returns in the output.

<bucketname>.s3.<region>.scw.cloud/?uploads

Sample request

GET /?uploads&max-uploads=2 HTTP/1.1Host: bucketname.s3.nl-ams.scw.cloudDate: FThu, 03 Sep 2020 13:29:39 GMTAuthorization: authorization string

Sample response

x-amz-id-2: txa0875396ef3e47e9ac144-005f50efc4Content-Length: 1609x-amz-request-id: txa0875396ef3e47e9ac144-005f50efc4Content-Type: application/xmlDate: Thu, 03 Sep 2020 13:29:40 GMT

<?xml version='1.0' encoding='UTF-8'?><ListMultipartUploadsResult    xmlns="http://s3.amazonaws.com/doc/2006-03-01/">    <Bucket>bucketname</Bucket>    <KeyMarker></KeyMarker>    <UploadIdMarker></UploadIdMarker>    <NextKeyMarker>multipart/02</NextKeyMarker>    <NextUploadIdMarker>MzdjNzJlM2QtYjc5NC00YzI4LTk2MjItM2M1NjQ2YmYwMjlm</NextUploadIdMarker>    <MaxUploads>2</MaxUploads>    <IsTruncated>false</IsTruncated>    <Upload>        <Key>multipart/01</Key>        <UploadId>Y2RmMTk4OTgtYzZjYS00OTgzLWIzZWYtYWNiYjVhMTUzNWNm</UploadId>        <Initiator>            <ID>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</ID>            <DisplayName>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</DisplayName>        </Initiator>        <Owner>            <ID>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</ID>            <DisplayName>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</DisplayName>        </Owner>        <StorageClass>STANDARD</StorageClass>        <Initiated>2020-09-03T13:14:44.000Z</Initiated>    </Upload>    <Upload>        <Key>multipart/02</Key>        <UploadId>MzdjNzJlM2QtYjc5NC00YzI4LTk2MjItM2M1NjQ2YmYwMjlm</UploadId>        <Initiator>            <ID>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</ID>            <DisplayName>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</DisplayName>        </Initiator>        <Owner>            <ID>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</ID>            <DisplayName>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</DisplayName>        </Owner>        <StorageClass>STANDARD</StorageClass>        <Initiated>2020-09-03T13:15:40.000Z</Initiated>    </Upload></ListMultipartUploads>

CLI command

In this example we use a 3GB .mp4 file.

$ aws s3api list-parts --bucket mybucket --upload-id "OWZiZTA4YzUtODExZC00ZjE5LTkyMjUtZGVmNjcwNjBiYWQ1" --key video.mp4{    "Parts": [        {            "PartNumber": 34,            "LastModified": "2020-12-14T12:59:42.000Z",            "ETag": "\"3ed6157a7f0b59ba8b2acfcd7e7f9f5f\"",            "Size": 10485760        },        {            "PartNumber": 85,            "LastModified": "2020-12-14T12:59:42.000Z",            "ETag": "\"ef553a411c2194543155f6f1e711fcee\"",            "Size": 10485760        }    ],    "Initiator": {        "ID": "0457951d-0186-4e23-897c-1895b192844c:0457951d-0186-4e23-897c-1895b192844c",        "DisplayName": "0457951d-0186-4e23-897c-1895b192844c:0457951d-0186-4e23-897c-1895b192844c"    },    "Owner": {        "DisplayName": "0457951d-0186-4e23-897c-1895b192844c:0457951d-0186-4e23-897c-1895b192844c",        "ID": "0457951d-0186-4e23-897c-1895b192844c:0457951d-0186-4e23-897c-1895b192844c"    },    "StorageClass": "STANDARD"}

Aborting a multipart upload

After initiating a multipart upload and beginning to upload parts, Object Storage stores these parts, but the final object is only assembled from the parts upon request once the multipart upload has been completed.

If the multipart upload is not completed, the parts will not be assembled and no object will be created. The parts remain on the object storage platform and stored parts are subject to billing.

In case these parts are no longer needed, it is possible to clear them by aborting the multipart upload session.

To abort an active multipart upload session a DELETE request as following shall be sent to the platform:

<bucketname>.<region>.scw.cloud/<object_key>?uploadId=<object_key>

Sample request

X-Amz-Date: 20190517T145721ZX-Amz-Content-SHA256: 4cf002db03f94b9ab876f38ad65863c987af96c6526673cfee22fad443fee9cbAuthorization: AWS4-HMAC-SHA256 Credential=SCWGJ3BW6Q22HCEB16GJ/20190517/nl-ams/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=96eff1cd4072eec6cb805037e1819c8c1a39666aa1a7d49d9a994e8b287f24b6'

Sample response

x-amz-id-2: tx43fe9acf22a7494493750-005cdecbd1x-amz-request-id: 'tx43fe9acf22a7494493750-005cdecbd1Content-Length: 0Date: Fri, 17 May 2019 14:57:21 GMT

CLI command

Important:

The file size threshold for regular uploads done through the awscli is 8MB. When uploading a file larger than 8MB, the awscli automatically starts an ongoing Multipart Upload.

To abort the upload and clear the parts, run the following commands:

$ aws s3api abort-multipart-upload --bucket mybucket --upload-id "OWZiZTA4YzUtODExZC00ZjE5LTkyMjUtZGVmNjcwNjBiYWQ1" --key <my-file>$ aws s3api list-multipart-uploads --bucket mybucket{  "Uploads": []}
See Also