Managing the lifecycle of objects

It is possible to configure a lifecycle of a group of objects so that their storage costs can be effectively managed. A lifecycle configuration is a set of rules that defines actions applied to a group of objects stored on Object Storage.

Currently, the expiration, transition and incomplete multipart uploads actions are supported on the platform:

• Expiration defines when objects expire, to delete objects automatically on your behalf.
• Transition defines when objects change their storage class
• Incomplete Multipart Uploads allows you to stop uploads automatically if they are not completed.

There might, for example, be a need to store log files for a week or a month, after which they become obsolete. It is possible to set a lifecycle rule to delete them automatically when they become obsolete. If you consider that a 3-month-old object is rarely used but still has a value, you might want to configure a rule to send it automatically to Scaleway Glacier, for example.

Lifecycle management on Object Storage is available on every AWS S3 compliant tool (sdk, aws-cli, boto, etc), as well as from the Scaleway console.

Lifecycle specification

A lifecycle configuration is an XML file, containing a set of rules with predefined actions that you want Object Storage to perform on objects during their lifetime:

<LifecycleConfiguration>
  <Rule>
  ...
  </Rule>
  <Rule>
  ...
  </Rule>
  ...
</LifecycleConfiguration>Copy

Each Rule consists of the following:

• Filter identifying a subset of objects to which the rule applies.
• Status whether the rule is in effect (a rule can be present but Disabled)
• Lifecycle transition(s) to perform on the selected objects

Available XML tokens

Rule

Filter

And

Sample

<And>
  <Prefix>myprefix</Prefix>
  <Tag>
    <Key>mytagkey1</Key>
    <Value>mytagvalue1</Value>
  </Tag>
</And>Copy

This applies the Prefix rule and the Tag.

Prefix

Tag

Status

Transition

Days

StorageClass

Expiration

ID

Configuration example

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>id-1</ID>
      <Expiration>
        <Days>0</Days>
      </Expiration>
    <Filter>
      <And>
        <Prefix>myprefix</Prefix>
      <Tag>
        <Key>mytagkey1</Key>
        <Value>mytagvalue1</Value>
      </Tag>
      <Tag>
        <Key>mytagkey2</Key>
        <Value>mytagvalue2</Value>
      </Tag>
      </And>
    </Filter>
    <Status>Enabled</Status>
  </Rule>
</LifecycleConfiguration>Copy

The lifecycle configuration above translates to the following effective rules:

• on the rule named id-1, we set the objects with the myprefix Prefix
• the tags mytagkey1, mytagkey2 sets to mytagvalue1
• ‘mytagvalue2’ is set to expire after one day.

Setting rules for incomplete multipart uploads

When using the awscli to perform a multipart upload, it is possible to set a bucket lifecycle rule that allows you to automate the stoppage of uploads if they are not completed. You can define a deadline for the expected completion with a lifecycle rule. If the multipart upload is incomplete by the end of the defined timeframe, it will be stopped and the parts associated with it will be deleted.

You can automate the removal of incomplete multipart uploads by adding or uploading a lifecycle configuration to your bucket. The set-up can be done via an API call or an AWS-CLI command.

Send a PutBucketLifecycle request with the following XML payload:

<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
    <ID>Remove uncompleted uploads</ID>
    <Filter>
       <Prefix></Prefix>
    </Filter>
    <Status>Enabled</Status>
    <AbortIncompleteMultipartUpload>
         <DaysAfterInitiation>1</DaysAfterInitiation>
    </AbortIncompleteMultipartUpload>
</Rule>
</LifecycleConfiguration>Copy

CLI command

• Create a file called incomplete-upload-lifecycle-policy.json with the following content.

{
  "Rules": [{
    "ID": "Remove uncompleted uploads",
    "Status": "Enabled",
    "Filter": {
      "Prefix": ""
    },
    "AbortIncompleteMultipartUpload": {
      "DaysAfterInitiation": 1
    }
  }]
}Copy

• Run the command:

aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://incomplete-upload-lifecycle-policy.jsonCopy