Managing the lifecycle of objects

Reviewed on December 24, 2024

Before you start

To complete the actions presented below, you must have:

Owner status or IAM permissions allowing you to perform actions in the intended Organization
An Object Storage bucket

You can configure the lifecycle of a group of objects so that their storage costs can be effectively managed. A lifecycle configuration is a set of rules that defines actions applied to a group of objects stored on Object Storage.

Currently, the expiration, transition, and incomplete multipart uploads actions are supported on the platform:

Expiration defines when objects expire. This allows for the automatic deletion of objects on your behalf.
Transition defines when objects change storage classes.
Incomplete Multipart Uploads allows you to stop uploads automatically if they are not completed.

Important

Lifecycle rules only allow the following transitions:

Standard -> Onezone IA
Standard -> Glacier
Onezone IA -> Glacier

Refer to our dedicated documentation to manually change the storage class of your objects.

There might, for example, be a need to store log files for a week or a month, after which they become obsolete. It is possible to set a lifecycle rule to delete them automatically when they become obsolete. If you consider that a 3-month-old object is rarely used but still has a value, you might want to configure a rule to send it automatically to Scaleway Glacier, for example.

Lifecycle management on Object Storage is available on every Amazon S3 compliant tool (sdk, aws-cli, boto, etc), as well as from the Scaleway console.

Lifecycle specification

A lifecycle configuration is an XML file, containing a set of rules with predefined actions that you want Object Storage to perform on objects during their lifetime:

<LifecycleConfiguration>
  <Rule>
  ...
  </Rule>
  <Rule>
  ...
  </Rule>
  ...
</LifecycleConfiguration>

Each Rule consists of the following:

Filter identifying a subset of objects to which the rule applies.
Status whether the rule is in effect (a rule can be present but Disabled)
Lifecycle transition(s) to perform on the selected objects

Important

Lifecycle rules are limited to 500,000 jobs per day. Beyond this limit, you can transition or delete your objects manually.

Available XML tokens

Rule

Description: Container for a lifecycle rule. There is a limit of one thousand (1000) rules per bucket.
Parent: LifecycleConfiguration

Filter

Description: Container for elements that describes the subset of the object the rule applies for. If the content of the tag is empty, the rule applies to all objects in the bucket.
Parent: Rule

And

Description: Container for chaining rule filters.
Parent: Rule

Sample

<And>
  <Prefix>myprefix</Prefix>
  <Tag>
    <Key>mytagkey1</Key>
    <Value>mytagvalue1</Value>
  </Tag>
</And>

This applies the Prefix rule and the Tag.

Prefix

Description: Object key prefix identifying one or more objects to which the rule applies. Object Storage limits this token to 1 per Filter Rule.
Parent: Filter or And

Status

Description: Describes whether or not a rule is enabled.
Parent: Rule
Values: Enabled, Disabled

Transition

Description: Specifies a period and a destination for an object's lifetime
Parent: Rule

Day

Description: Specifies the number of days after object creation when the rule takes effect.
Parent: Transition and/or Expiration

StorageClass

Description: Specifies the storage class to which the object shall be transferred. On the Scaleway platform the storage classes STANDARD, GLACIER and ONEZONE_IA are available.
Parent: Transition

Expiration

Description: Describes the expiration of the object lifetime. If versioning is enabled, this rule only deletes the current version of an object.
Parent: Rule

ID

Description: Unique identifier for the rule. This value is limited to 255 characters.
Parent: Rule

Configuration example

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>id-1</ID>
      <Expiration>
        <Days>0</Days>
      </Expiration>
    <Filter>
      <And>
        <Prefix>myprefix</Prefix>
      <Tag>
        <Key>mytagkey1</Key>
        <Value>mytagvalue1</Value>
      </Tag>
      <Tag>
        <Key>mytagkey2</Key>
        <Value>mytagvalue2</Value>
      </Tag>
      </And>
    </Filter>
    <Status>Enabled</Status>
  </Rule>
</LifecycleConfiguration>

The lifecycle configuration above translates to the following effective rules:

on the rule named id-1, we set the objects with the myprefix Prefix
the tags mytagkey1, mytagkey2 sets to mytagvalue1
mytagvalue2 is set to expire after one day.

Note

To see examples of lifecycle rules API requests, refer to the Bucket Operations documentation page.

Setting rules for incomplete multipart uploads

When using the awscli to perform a multipart upload, it is possible to set a bucket lifecycle rule that allows you to automate the stoppage of uploads if they are not completed. You can define a deadline for the expected completion with a lifecycle rule. If the multipart upload is incomplete by the end of the defined timeframe, it will be stopped and the parts associated with it will be deleted.

You can automate the removal of incomplete multipart uploads by adding or uploading a lifecycle configuration to your bucket. The setup can be done via an API call or an AWS-CLI command.

Send a putbucketlifecycleconfiguration request with the following XML payload:

<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
    <ID>Remove uncompleted uploads</ID>
    <Filter>
       <Prefix></Prefix>
    </Filter>
    <Status>Enabled</Status>
    <AbortIncompleteMultipartUpload>
         <DaysAfterInitiation>1</DaysAfterInitiation>
    </AbortIncompleteMultipartUpload>
</Rule>
</LifecycleConfiguration>

Note

In this example, the multipart upload timeframe is of one day. You may change the number of DaysAfterInitiation according to your preference by replacing 1 in the XML payload.

CLI command

Create a file called incomplete-upload-lifecycle-policy.json with the following content.

{
  "Rules": [{
    "ID": "Remove uncompleted uploads",
    "Status": "Enabled",
    "Filter": {
      "Prefix": ""
    },
    "AbortIncompleteMultipartUpload": {
      "DaysAfterInitiation": 1
    }
  }]
}

Note

In this example, the multipart upload timeframe is one day. You may change the number of DaysAfterInitiation according to your preference by replacing 1 in the .json file body.

In a terminal, run the command below to apply your lifecycle configuration to the desired bucket:

aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://incomplete-upload-lifecycle-policy.json

Still need help?

Create a support ticket