Object Storage - Cold storage

Scaleway Cold Storage is a secure, durable, and extremely attractive Object Storage class for data archiving and long term backup. It is integrated within the lifecycle bucket API, in order to ease your costs on unneeded objects. In this document, we will describe what is the storage class GLACIER, how to use it and what it’s for.

General Principles

  • GLACIER is cold storage. That means it is mainly used for backups and archives. You cannot directly get data from GLACIER, it needs to be restored first. Thus, applications like CDNs are not suited for GLACIER usage.

  • In order to read the content of an object stored in GLACIER, it must be restored first in the STANDARD storage class. The object can then be accessed with the usual APIs.

  • Restoration is not instantaneous; indeed, restoring an object could take from minutes to several hours, depending on the charge on the service at that time.

  • You can either put data in GLACIER directly or automatically with lifecycle rules.

Storage Classes

  • STANDARD: The Standard class for any upload; suitable for on-demand content like streaming or CDN.
  • GLACIER: Archived storage; prices are lower, but it needs to be restored first to be accessed.

Putting data in GLACIER

Direct PUT

One can directly put data in the GLACIER storage class simply by putting the HTTP header x-amz-storage-class to GLACIER in a PUT request. Here’s an example with aws-cli:

$> aws s3 cp object s3://bucket/ --storage-class GLACIER
upload: object to s3://bucket/object

$> aws s3api head-object --bucket bucket --key object
{
    "AcceptRanges": "bytes",
    "LastModified": "Wed, 09 Oct 2019 14:22:41 GMT",
    "ContentLength": 1644,
    "ETag": "\"ba30b81c13b9c6d7c4938fefa3a491b6\"",
    "VersionId": "1570630961054836",
    "ContentType": "application/octet-stream",
    "Metadata": {},
    "StorageClass": "GLACIER"
}

The first command did upload an object directly in GLACIER, and the second one was simply to get some information on the object we just uploaded. In this example, the StorageClass is rightly returned as GLACIER, which means one cannot directly GET the object:

$> aws s3 cp s3://bucket/object .
warning: Skipping file s3://bucket/object. Object is of storage class GLACIER.
Unable to perform download operations on GLACIER objects. You must restore the
object to be able to perform the operation. See aws s3 download help for
additional parameter options to ignore or force these transfers.

Lifecycle

Instead of manually putting objects in GLACIER, one can set an Expiration lifecycle rule to do it automatically:

$> cat rule.json
{
    "Rules": [
        {
            "Filter": {
                "Prefix": "logs/"
            },
            "Status": "Enabled",
            "Transitions": [{
                "StorageClass": "GLACIER",
                "Days": 30,
            }],
            "Expiration": {
                "Days": 730
            },
            "ID": "LogArchive"
        }
}

$> aws s3api put-bucket-lifecycle-configuration --bucket bucket --lifecycle-configuration file://rule.json

In this example lifecycle configuration, any object in the prefix logs/ is determined to be automatically transferred to GLACIER 30 days after creation, and deleted after 2 years. The transfer between STANDARD and GLACIER happens automatically at midnight, on the server-side.

Getting data from GLACIER

Temporary restore

One can temporarily restore an object from GLACIER to STANDARD in order to access it:

$> aws s3api restore-object --bucket bucket --key object --restore-request Days=7

In this example, the object will be restored in the STANDARD storage class for 7 days, and will be automatically transferred back in GLACIER after that time. It is up to the user to access that object before it is archived again. Note that since you transfer an object from GLACIER to STANDARD for 7 days, you will not pay the GLACIER price for that time, but STANDARD fees.

Definitive restore

One can definitively restore an object from GLACIER to STANDARD simply without specifying any days in the restoration operation:

$> aws s3api restore-object --bucket bucket --key object

Important: If the object was transferred in GLACIER with a lifecycle rule and that same rule is still active when the object is definitively restored, the object will be transferred-back in GLACIER the same day at midnight. It is up to the user to modify the bucket lifecycle configuration in order to avoid such cases.

Restoration

As specified earlier, restoration can take up to several hours. The backend keeps a flag in the object in order to inform the user of the ongoing restoration:

$> aws s3api restore-object --bucket bucket --key object
$> aws s3api head-object --bucket bucket --key object
{
    "AcceptRanges": "bytes",
    "Restore": "ongoing-request=\"true\"",
    "LastModified": "Wed, 09 Oct 2019 17:22:59 GMT",
    "ContentLength": 1644,
    "ETag": "\"ba30b81c13b9c6d7c4938fefa3a491b6\"",
    "VersionId": "1570641779502690",
    "ContentType": "application/octet-stream",
    "Metadata": {},
    "StorageClass": "GLACIER"
}
[After some time...]
$> aws s3api head-object --bucket bucket --key object
{
    "AcceptRanges": "bytes",
    "LastModified": "Wed, 09 Oct 2019 17:24:25 GMT",
    "ContentLength": 1644,
    "ETag": "\"ba30b81c13b9c6d7c4938fefa3a491b6\"",
    "VersionId": "1570641779502690",
    "ContentType": "application/octet-stream",
    "Metadata": {}
}

The state of the restoration is indicated by the HTTP header x-amz-restore (Here translated as Restore by aws-cli). In case of a definitive restore, the header will go away after the object has been successfully transferred to the STANDARD storage class. However, for a temporary restore, the behavior is different:

$> aws s3api restore-object --bucket bucket --key object --restore-request Days=30
$> aws s3api head-object --bucket bucket --key object
{
    "AcceptRanges": "bytes",
    "Restore": "ongoing-request=\"true\"",
    "LastModified": "Wed, 09 Oct 2019 17:29:44 GMT",
    "ContentLength": 1644,
    "ETag": "\"ba30b81c13b9c6d7c4938fefa3a491b6\"",
    "VersionId": "1570642184893299",
    "ContentType": "application/octet-stream",
    "Metadata": {},
    "StorageClass": "GLACIER"
}
[After some time...]
$> aws s3api head-object --bucket bucket --key object
{
    "AcceptRanges": "bytes",
    "Restore": "ongoing-request=\"false\", expiry-date=\"Fri, 8 Nov 2019 00:00:00 UTC\"",
    "LastModified": "Wed, 09 Oct 2019 17:30:47 GMT",
    "ContentLength": 1644,
    "ETag": "\"ba30b81c13b9c6d7c4938fefa3a491b6\"",
    "VersionId": "1570642184893299",
    "ContentType": "application/octet-stream",
    "Metadata": {}
}

On a temporary restore, the Restore header will stay until the object goes back automatically in GLACIER, here, after 30 days. The content of the header indicates that there’s no on-going restore request, and when the object will be transferred back into GLACIER.

Pricing

For GLACIER, the pricing model is as follows:

  • Like hot storage, you will be billed for 1Gb/hour for every data stored in GLACIER. Of course, that price is much lower on GLACIER than on STANDARD
  • Transfer from STANDARD to GLACIER (archiving) is free of charge.
  • Transfer from GLACIER to STANDARD (restoration) is billed by the amount of data restored.
  • At any time, you are billed for the storage class that you are using. If an object is 7 days in STANDARD and 21 days in GLACIER you will pay 7 days of STANDARD storage and 21 days of GLACIER storage.
  • Note that restoration is simply a transfer from GLACIER to STANDARD; you will stop paying GLACIER and be billed for STANDARD fees.
  • There aren’t different tiers for restoration; you cannot pay more in order to restore your data faster.

The above does not take into account Free Tiering or the pricing during the BETA. Please refer to the Object Storage pricing page for information on said prices.

Discover a New Cloud Experience

Deploy SSD Cloud Servers in seconds.