S3 Object Storage - Multipart Upload

Object Storage - Multipart Upload

Multipart Upload allows you to upload large files to the Object Storage platform in multiple parts. It is possible to store objects up to 5TB per object on the platform.

A multipart upload consists of three steps:

  • The client initiate the upload to a specific bucket
  • The different parts are uploaded
  • Object Storage constructs the object from the uploaded parts

Once the object has been constructed, it can be accessed as any other object in the bucket.

The following technical specifications apply to multipart uploads:

  • 1 to 1000 parts per object
  • 5 MB to 5 GB per part (except for the last one)
  • Each object stores up to 5TB

Initiating a Multipart Upload

When initiating a multipart upload, the Object storage platform generates an unique identifier for the multipart upload. This identifier must be included whenever parts are uploaded, listed, the upload completed or to abort an upload.

To initiate a multipart upload, the client sends a POST request as follows:

<bucketname>.s3.<region>.scw.cloud/<object_key>?uploads

Example Request:

POST /large-file.tar.gz?uploads HTTP/1.1

Host: bucketname.s3.nl-ams.scw.cloud
x-amz-content-sha256: 4cf002db03f94b9ab876f38ad65863c987af96c6526673cfee22fad443fee9cb
x-amz-date: 20190517T125417Z
Authorization: AWS4-HMAC-SHA256 Credential=SCWGJ3BW6Q22HCEB16GJ/20190517/nl-ams/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=4e6b0dea0badf41e2aafca17017d479138ce1d34e1c3fcf2df00193ae3c68c1c

Example Output:

HTTP/1.1 200 OK
x-amz-id-2: tx0db175df07644b858daab-005cdeaef9
Content-Length: 265
x-amz-request-id: tx0db175df07644b858daab-005cdeaef9
Date: Fri, 17 May 2019 12:54:17 GMT

<?xml version="1.0" encoding="UTF-8"?>
  <InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01">
    <Bucket>bucketname</Bucket>
    <Key>large-file.tar.gz</Key>
    <UploadId>Y2VhZDUxYjAtNjVjMC00NTIxLWEwNWUtNWM3NDAyOTQ4ZWYz</UploadId>
  </InitiateMultipartUploadResult>'

Uploading a Part

Once a multipart upload has been initiated Object Storage keeps all parts of the upload until it either completes or is being aborted. When uploading a part, a part number must be specified in addition to the upload ID returned when initiation the multipart upload. The bard number can be any number between 1 and 1000 and identifies uniquely the part and its position in the object being uploaded.

During the lifecycle of a multipart upload, invoicing is taking place for all storage and bandwidth usage for its associated parts. If a multipart upload is aborted, all parts already uploaded are deleted and billing for these resources stops.

To upload a part of an object in an initiated multipart upload, send a POST request as following:

<bucketname>.s3.<region>.scw.cloud/<object_key>partNumber=<part_number>&uploadId=<upload_id>.
  • <part_number specifies the position of the part uploaded inside the object.
  • <upload_id> is the unique identifier returned during the initialization of the multi-part upload.

Example Request:

PUT /large-file.tar.gz?partNumber=1&uploadId=Y2VhZDUxYjAtNjVjMC00NTIxLWEwNWUtNWM3NDAyOTQ4ZWYz HTTP/1.1
host: bucketname.s3.nl-ams.scw.cloud
x-amz-content-sha256: 4cf002db03f94b9ab876f38ad65863c987af96c6526673cfee22fad443fee9cb
x-amz-date: 20190517T131738Z
Authorization': AWS4-HMAC-SHA256 Credential=SCWGJ3BW6Q22HCEB16GJ/20190517/nl-ams/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date

Example Output:

Content-Length: 0
x-amz-id-2: txc9ed83e20cd84f56b826c-005cdeb472
Date: Fri, 17 May 2019 13:17:38 GMT
ETag: "d41d8cd98f00b204e9800998ecf8427e"
x-amz-request-id: txc9ed83e20cd84f56b826c-005cdeb472

Completing a Multipart Upload

To complete a multipart upload session and to generate the final object, a POST request as following is required. <upload_id> corresponds to the unique identifier returned during the initialization of the multi-part upload:

<bucketname>.<region>.scw.cloud/<object_key>?uploadId=<object_key>

The request requires that an XML element named CompleteMultipartUpload containing information about each part of the multipart upload is present. The element requires the following contents:

  • Part: Information about each individual part of a multipart upload.
  • PartNumber: A sequential identifier specifying the part of the object.
  • ETag: The entity tag (Etag) containing an MD5 hash of the objects part.

Sample Request:

POST /large-file.tar.gz?uploadId=Y2VhZDUxYjAtNjVjMC00NTIxLWEwNWUtNWM3NDAyOTQ4ZWYz HTTP/1.1

X-Amz-Date: 20190517T140506Z
X-Amz-Content-SHA256: 4cf002db03f94b9ab876f38ad65863c987af96c6526673cfee22fad443fee9cb
Authorization: AWS4-HMAC-SHA256 Credential=SCWGJ3BW6Q22HCEB16GJ/20190517/nl-ams/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=2ab2471b862268c3509aad01974153fc44455f7f6998c0e3e7fbdfaf8f4f5efd
Content-Length: 0

<CompleteMultipartUpload>
  <Part>
    <PartNumber>1</PartNumber>
    <ETag>"4cb0d956600bab287520b442e83035a6"</ETag>
  </Part>
  <Part>
    <PartNumber>2</PartNumber>
    <ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag>
  </Part>
</CompleteMultipartUpload>

Sample Output:

HTTP/1.1 200 OK

x-amz-id-2: tx321faae63fb14971a8d8a-005cdec63e
Date: Fri, 17 May 2019 14:05:06 GMT
x-amz-request-id: tx321faae63fb14971a8d8a-005cdec63e
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
  <CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Location>bucketname.s3.nl-ams.scw.cloud</Location>
    <Bucket>bucketname</Bucket>
    <Key>large-file.tar.gz</Key>
    <ETag>6eb393dac21d595a813f0c8bf4e9bc23</ETag>
  </CompleteMultipartUploadResult>

Aborting a Multipart Upload

After initiating a multipart upload and beginning to upload parts, Object Storage stores these parts, but the final object is only assembled from the parts upon request once the multipart upload has been completed.

If the multipart upload is not completed, the parts will not be assembled and no object will be created. The parts remain on the object storage platform and stored parts are subject to billing.

In case these parts are no longer needed, it is possible to abort the multipart upload session.

To abort an active multipart upload session a DELETE request as following shall be sent to the platform:

<bucketname>.<region>.scw.cloud/<object_key>?uploadId=<object_key>

Example Request:

X-Amz-Date: 20190517T145721Z
X-Amz-Content-SHA256: 4cf002db03f94b9ab876f38ad65863c987af96c6526673cfee22fad443fee9cb
Authorization: AWS4-HMAC-SHA256 Credential=SCWGJ3BW6Q22HCEB16GJ/20190517/nl-ams/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=96eff1cd4072eec6cb805037e1819c8c1a39666aa1a7d49d9a994e8b287f24b6'

Example Output:

No output.

x-amz-id-2: tx43fe9acf22a7494493750-005cdecbd1
x-amz-request-id: 'tx43fe9acf22a7494493750-005cdecbd1
Content-Length: 0
Date: Fri, 17 May 2019 14:57:21 GMT

Listing Parts

It is possible to retrieve a list of the parts of a multipart upload.

The following GET request returns a XML element named ListPartsResult containing information about the multi-part upload. <upload_id> corresponds to the unique identifier returned when initializing the session:

<bucketname>.<region>.scw.cloud/<object_key>?uploadId=<upload_id>

Sample Request:

GET /large-file.tar.gz?uploadId=Y2VhZDUxYjAtNjVjMC00NTIxLWEwNWUtNWM3NDAyOTQ4ZWYz HTTP/1.1
host: bucketname.s3.nl-ams.scw.cloud
x-amz-content-sha256:4cf002db03f94b9ab876f38ad65863c987af96c6526673cfee22fad443fee9cb
x-amz-date:20190517T133608Z

Sample Output:

HTTP/1.1 200 OK

x-amz-id-2: tx0c4d0853b19b49a8b64ed-005cdeb8c8
Content-Length: 1304
x-amz-request-id: tx0c4d0853b19b49a8b64ed-005cdeb8c8
Date: Fri, 17 May 2019 13:36:08 GMT


<?xml version="1.0" encoding="UTF-8"?>
  <ListPartsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Bucket>benetest</Bucket>
    <Key>Dockerfile</Key>
    <UploadId>Y2VhZDUxYjAtNjVjMC00NTIxLWEwNWUtNWM3NDAyOTQ4ZWYz</UploadId>
    <Initiator>
      <ID>04dcf44f-a6ca-4e69-a74c-f0c557d87d79:04dcf44f-a6ca-4e69-a74c-f0c557d87d79</ID>
      <DisplayName>04dcf44f-a6ca-4e69-a74c-f0c557d87d79:04dcf44f-a6ca-4e69-a74c-f0c557d87d79</DisplayName>
    </Initiator>
    <Owner>
      <ID>04dcf44f-a6ca-4e69-a74c-f0c557d87d79:04dcf44f-a6ca-4e69-a74c-f0c557d87d79</ID>
      <DisplayName>04dcf44f-a6ca-4e69-a74c-f0c557d87d79:04dcf44f-a6ca-4e69-a74c-f0c557d87d79</DisplayName>
    </Owner
    ><StorageClass>STANDARD</StorageClass>
    <PartNumberMarker>0</PartNumberMarker>
    <NextPartNumberMarker>2</NextPartNumberMarker>
    <MaxParts>1000</MaxParts>
    <IsTruncated>false</IsTruncated>
    <Part>
      <PartNumber>1</PartNumber>
      <LastModified>2019-05-17T13:17:38.000Z</LastModified>
      <ETag>"4cb0d956600bab287520b442e83035a6"</ETag>
      <Size>5242880</Size>
    </Part>
    <Part>
      <PartNumber>2</PartNumber>
      <LastModified>2019-05-17T13:54:09.000Z</LastModified>
      <ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag>
      <Size>5242880</Size>
    </Part>
  </ListPartsResult>

Discover a New Cloud Experience

Deploy SSD Cloud Servers in seconds.