S3 Object Storage - Lifecycle Management

Object Storage - Object Lifecycle Management

It is possible to configure a lifecycle of a group of objects so that their storage costs can effectively be managed. A lifecycle configuration is a set of rules that defines actions applied to a group of objects stored on Object Storage.

Currently the expiration action is supported on the platform. It defines when objects expire, to delete objects automatically on your behalf.

There might be for example a need to store log files for a week or a month, after which they become obsolete. It is possible to set a lifecyle rule to delete them automatically when they become obsolete.

Currently Lifecyle Management on Object Storage available on every AWS S3 compliant tool (sdk, aws-cli, boto, etc), but currently not yet available on the Scaleway management console.

Lifecycle Specification

A lifecycle configuration is an XML file, containing a set of rules with predefined actions that you want Object Storage to perform on objects during their lifetime:

<LifecycleConfiguration>
  <Rule>
  ...
  </Rule>
  <Rule>
  ...
  </Rule>
  ...
</LifecycleConfiguration>

Each Rule consists of the following:

• Filter identifying a subset of objects to which the rule applies.
• Status whether the rule is in effect (A rule can be present but Disabled)
• Lifecycle transition(s) to perform on the selected objects

Available XML Tokens

• Rule: Child of LifecycleConfiguration

  Container for a lifecycle rule. There is a limit of thousand (1000) rules per bucket.

• Filter: Child of Rule

  Container for elements that describe that identifies the subset of object the rule applies for. If the content of the tag is empty, the rule applies to all objects in the bucket.

• And: Child of Rule

  Container for chaining rule filters. For example:

   <And>
            <Prefix>myprefix</Prefix>
            <Tag>
                <Key>mytagkey1</Key>
                <Value>mytagvalue1</Value>
            </Tag>
    </And>
  

  Which applies the Prefix rule and the Tag

• Prefix: Child of Filter or And;

  Object key prefix identifying one or more objects to which the rule applies. Object Storage does limit this token to 1 per Filter Rule.

• Tag: Child of Filter or And;

  Container for specifying a Key and a Value

• Status: Child of Rule;

  Describes whether or not a rule is to be enabled. Values: Enabled,Disabled

• Transition: Child of Rule;

  Specifies a period and a destination for an object lifetime

• Days: Child of Transition and / or Expiration;

  Specifies the number of days after object creation when the rule takes effect.

• StorageClass: Child of Transition;

  Specifies the Storage Class to which the object shall be transfered to.

• Expiration: Child of Rule;

  Describe a period in the objet’s lifetime for a deletion. If versioning is enabled, this rule shall only delete the current version of an object.

• ID: Child of Rule;

  Unique identifier for the rule. This value is limited to 255 characters.

Configuration Examples

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>id-1</ID>
      <Expiration>
        <Days>1</Days>
      </Expiration>
    <Filter>
      <And>
        <Prefix>myprefix</Prefix>
      <Tag>
        <Key>mytagkey1</Key>
        <Value>mytagvalue1</Value>
      </Tag>
      <Tag>
        <Key>mytagkey2</Key>
        <Value>mytagvalue2</Value>
      </Tag>
      </And>
    </Filter>
    <Status>Enabled</Status>
  </Rule>
</LifecycleConfiguration>

The lifecycle configuration above translates to the following effective rules:

On this rule named “id-1”, we set the objects with the ‘myprefix’ Prefix and the tags ‘mytagkey1’, ‘mytagkey2’ sets to ‘mytagvalue1’ and ‘mytagvalue2’ to expire after one day.

PUT Bucket lifecycle

Sample Request:

PUT /?lifecycle HTTP/1.1
Host: bucketname.s3.nl-ams.scw.cloud
Content-MD5: e0d656aa8bed46fe766420c6d21537d5
Authorization: authorization string
<LifecycleConfiguration>
    <Rule>
        <ID>ExampleRule</ID>
        <Filter>
           <Prefix>documents/</Prefix>
        </Filter>
        <Status>Enabled</Status>
        <Expiration>
             <Days>30</Days>
        </Expiration>
    </Rule>
</LifecycleConfiguration>

Sample Output:

HTTP/1.1 200 OK

GET Bucket lifecycle

Sample Request:

GET /?lifecycle HTTP/1.1
Host: bucketname.s3.nl-ams.scw.cloud
Authorization: authorization string

Sample Output:

HTTP/1.1 200 OK
x-amz-id-2: txda274593e2dc4974a82cc-005cdbe130
x-amz-request-id: txda274593e2dc4974a82cc-005cdbe130
Date: Wed, 15 May 2019 09:51:44 GMT
Content-Length: 238

<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
    <ID>ExampleRule</ID>
    <Filter>
       <Prefix>documents/</Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Expiration>
         <Days>30</Days>
    </Expiration>
</Rule>
</LifecycleConfiguration>

DELETE Bucket lifecycle

Sample Request:

DELETE /?lifecycle HTTP/1.1
Host: bucketname.s3.nl-ams.scw.cloud
Authorization: authorization string

Sample Output:

204 No Content
Content-Length: 0
Date: Wed, 15 May 2019 09:59:29 GMT