Object Storage - C14 Cold Storage Class

Scaleway Elements C14 Cold Storage Overview

Scaleway C14 Cold Storage, also known as S3 GLacier, is a secure, durable, and extremely attractive Object Storage class for data archiving and long term backup. Fully integrated within the lifecycle bucket API, in order to lower your storage costs of large amounts of data. In this document, we will describe what is the storage class GLACIER, how to use it and why. Find below several general principles about GLACIER:

  • GLACIER is cold storage. That means it is mainly used for backups and archives. You cannot directly download data from GLACIER, it needs to be restored first in an Object Storage bucket. As a result, applications such as CDNs are not suited for GLACIER usage.

  • In order to read the content of an object stored in GLACIER, it must be restored first in the STANDARD storage class. The object can then be accessed with the usual APIs.

  • Restoration is not instantaneous; indeed, restoring an object could take from minutes to several hours, depending on the charge on the service at that time.

  • You can either put data in GLACIER directly or automatically with lifecycle rules.

Requirements:

Scaleway Storage Classes

  • STANDARD: The Standard class for any upload; suitable for on-demand content like streaming or CDN.
  • GLACIER: Archived storage; prices are lower, but it needs to be restored first to be accessed.

Operation Procedures

Uploading Data to Glacier from the Scaleway Console

You can send objects directly to the GLACIER storage class by using the Scaleway console.

1 . Log in to the Scaleway console.

2 . Click on Object Storage in the storage section of the side menu. The list of your object storage buckets displays.

3 . Click on the bucket name which contains the objects that you want to transfer.

4 . Click next to the object you want to transfer to glacier to display the object management menu.

5 . Click Send to S3 Glacier to send the object to the C14 Cold Storage class:

6 . Confirm the change of storage class by clicking Send to S3 Glacier:

The object is being sent to Glacier, the new storage class displays next to the object:

Managing Direct PUT through the API

One can directly upload data to the GLACIER storage class simply by putting the HTTP header x-amz-storage-class to GLACIER in a PUT request. Here’s an example with aws-cli:

$> aws s3 cp object s3://bucket/ --storage-class GLACIER
upload: object to s3://bucket/object

$> aws s3api head-object --bucket bucket --key object
{
    "AcceptRanges": "bytes",
    "LastModified": "Wed, 09 Oct 2019 14:22:41 GMT",
    "ContentLength": 1644,
    "ETag": "\"ba30b81c13b9c6d7c4938fefa3a491b6\"",
    "VersionId": "1570630961054836",
    "ContentType": "application/octet-stream",
    "Metadata": {},
    "StorageClass": "GLACIER"
}

The first command uploaded an object directly to GLACIER, and the second command helped retrieve some information on the uploaded object. In this example, the StorageClass is rightly returned as GLACIER, which means one cannot directly GET the object:

$> aws s3 cp s3://bucket/object .
warning: Skipping file s3://bucket/object. Object is of storage class GLACIER.
Unable to perform download operations on GLACIER objects. You must restore the
object to be able to perform the operation. See aws s3 download help for
additional parameter options to ignore or force these transfers.

Managing Lifecycle Options through the API

Instead of manually putting objects in GLACIER, it is possible to set an Expiration lifecycle rule to execute it automatically:

$> cat rule.json
{
    "Rules": [
        {
            "Filter": {
                "Prefix": "logs/"
            },
            "Status": "Enabled",
            "Transitions": [{
                "StorageClass": "GLACIER",
                "Days": 30,
            }],
            "Expiration": {
                "Days": 730
            },
            "ID": "LogArchive"
        }
}

$> aws s3api put-bucket-lifecycle-configuration --bucket bucket --lifecycle-configuration file://rule.json

In this example lifecycle configuration, any object in the prefix logs/ is determined to be automatically transferred to GLACIER 30 days after creation, and deleted after 2 years. The transfer between STANDARD and GLACIER happens automatically at midnight, on the server-side.

Temporary Restore from the Scaleway Console

You can restore your data directly from the GLACIER storage class by using the Scaleway console.

1 . Connect yourself to the Scaleway console.

2 . Click on Object Storage in the storage section of the side-menu. The list of your object storage buckets displays.

3 . Click on the bucket name which contains the objects that you want to restore.

4 . Click next to the object you want to restore from glacier to display the object management menu.

5 . Click Restore to restore the object in STANDARD storage class:

6 . Choose the availability period of the object in STANDARD storage class. Once the period is over, the object will be send back to GLACIER:

Note: Restoring objects may take up to 6 hours.

7 . Once restored, the object displays with STANDARD storage class in the list of objects. You can now launch additional actions on it, if required:

Temporary Restore through the API

You can temporarily restore an object from GLACIER to STANDARD to access it:

$> aws s3api restore-object --bucket bucket --key object --restore-request Days=7

In this example, the object will be restored in the STANDARD storage class for 7 days, and will be automatically transferred back in GLACIER after that period. It is up to you to access that object before it is archived again. Since the object is restored in the STANDARD storage class during 7 days, you will not pay the GLACIER price during that period. Only STANDARD fees apply.

Managing Definitive Restore through the API

You can restore an object from GLACIER to STANDARD permanently without specifying any days in the restoration process:

$> aws s3api restore-object --bucket bucket --key object

Important: If the object was transferred in GLACIER with a lifecycle rule and that same rule is still active when the object is restored, the object will be transferred-back to GLACIER the same day at midnight. It is up to you to modify the bucket lifecycle configuration in order to avoid such cases.

Restoration through the API

As specified earlier, restoration can take up to several hours. The backend keeps a flag in the object in order to inform the user of the ongoing restoration:

$> aws s3api restore-object --bucket bucket --key object
$> aws s3api head-object --bucket bucket --key object
{
    "AcceptRanges": "bytes",
    "Restore": "ongoing-request=\"true\"",
    "LastModified": "Wed, 09 Oct 2019 17:22:59 GMT",
    "ContentLength": 1644,
    "ETag": "\"ba30b81c13b9c6d7c4938fefa3a491b6\"",
    "VersionId": "1570641779502690",
    "ContentType": "application/octet-stream",
    "Metadata": {},
    "StorageClass": "GLACIER"
}
[After some time...]
$> aws s3api head-object --bucket bucket --key object
{
    "AcceptRanges": "bytes",
    "LastModified": "Wed, 09 Oct 2019 17:24:25 GMT",
    "ContentLength": 1644,
    "ETag": "\"ba30b81c13b9c6d7c4938fefa3a491b6\"",
    "VersionId": "1570641779502690",
    "ContentType": "application/octet-stream",
    "Metadata": {}
}

The state of the restoration is indicated by the HTTP header x-amz-restore (here translated as Restore by aws-cli). In case of a definitive restore, the header will go away after the object has been successfully transferred to the STANDARD storage class. However, for a temporary restore, the behavior is different:

$> aws s3api restore-object --bucket bucket --key object --restore-request Days=30
$> aws s3api head-object --bucket bucket --key object
{
    "AcceptRanges": "bytes",
    "Restore": "ongoing-request=\"true\"",
    "LastModified": "Wed, 09 Oct 2019 17:29:44 GMT",
    "ContentLength": 1644,
    "ETag": "\"ba30b81c13b9c6d7c4938fefa3a491b6\"",
    "VersionId": "1570642184893299",
    "ContentType": "application/octet-stream",
    "Metadata": {},
    "StorageClass": "GLACIER"
}
[After some time...]
$> aws s3api head-object --bucket bucket --key object
{
    "AcceptRanges": "bytes",
    "Restore": "ongoing-request=\"false\", expiry-date=\"Fri, 8 Nov 2019 00:00:00 UTC\"",
    "LastModified": "Wed, 09 Oct 2019 17:30:47 GMT",
    "ContentLength": 1644,
    "ETag": "\"ba30b81c13b9c6d7c4938fefa3a491b6\"",
    "VersionId": "1570642184893299",
    "ContentType": "application/octet-stream",
    "Metadata": {}
}

On a temporary restore, the Restore header will stay until the object goes back automatically in GLACIER, here, after 30 days. The content of the header indicates that there’s no on-going restore request, and when the object will be transferred back into GLACIER.

Pricing

For GLACIER, the pricing model is as follows:

  • Like hot storage, you will be billed for 1Gb/hour for every data stored in GLACIER. Of course, that price is much lower on GLACIER than on STANDARD
  • Transfer from STANDARD to GLACIER (archiving) is free of charge.
  • Transfer from GLACIER to STANDARD (restoration) is free of charge.
  • At any time, you are billed for the storage class that you are using. If an object is 7 days in STANDARD and 21 days in GLACIER you will pay 7 days of STANDARD storage and 21 days of GLACIER storage.
  • Note that restoration is simply a transfer from GLACIER to STANDARD; you will stop paying GLACIER and be billed for STANDARD fees.
  • There aren’t different tiers for restoration; you cannot pay more in order to restore your data faster.

Going Further

Discover a New Cloud Experience

Deploy SSD Cloud Servers in seconds.