Managing the lifecycle of objects
You may need certain IAM permissions to carry out some actions described on this page. This means:
- you are the Owner of the Scaleway Organization in which the actions will be carried out, or
- you are an IAM user of the Organization, with a policy granting you the necessary permission sets
It is possible to configure a lifecycle of a group of objects so that their storage costs can be effectively managed. A lifecycle configuration is a set of rules that defines actions applied to a group of objects stored on Object Storage.
Currently, the expiration, transition and incomplete multipart uploads actions are supported on the platform:
- Expiration defines when objects expire, to delete objects automatically on your behalf.
- Transition defines when objects change their storage class
- Incomplete Multipart Uploads allows you to stop uploads automatically if they are not completed.
There might, for example, be a need to store log files for a week or a month, after which they become obsolete. It is possible to set a lifecycle rule to delete them automatically when they become obsolete. If you consider that a 3-month-old object is rarely used but still has a value, you might want to configure a rule to send it automatically to Scaleway Glacier, for example.
Lifecycle management on Object Storage is available on every AWS S3 compliant tool (sdk, aws-cli, boto, etc), as well as from the Scaleway console.
Lifecycle specification
A lifecycle configuration is an XML file, containing a set of rules with predefined actions that you want Object Storage to perform on objects during their lifetime:
<LifecycleConfiguration> <Rule> ... </Rule> <Rule> ... </Rule> ...</LifecycleConfiguration>
Each Rule
consists of the following:
Filter
identifying a subset of objects to which the rule applies.Status
whether the rule is in effect (a rule can be present butDisabled
)- Lifecycle transition(s) to perform on the selected objects
Available XML tokens
Rule
- Description
- Container for a lifecycle rule. There is a limit of one thousand (1000) rules per bucket.
- Parent
LifecycleConfiguration
Filter
- Description
- Container for elements that describes the subset of the object the rule applies for. If the content of the tag is empty, the rule applies to all objects in the bucket.
- Parent
Rule
And
- Description
- Container for chaining rule filters.
- Parent
Rule
Sample
<And> <Prefix>myprefix</Prefix> <Tag> <Key>mytagkey1</Key> <Value>mytagvalue1</Value> </Tag></And>
This applies the Prefix
rule and the Tag
.
Prefix
- Description
- Object key prefix identifying one or more objects to which the rule applies. Object Storage limits this token to 1 per
Filter
Rule. - Parent
Filter
orAnd
Tag
- Description
- Container for specifying a
Key
and aValue
. - Parent
Filter
orAnd
Status
- Description
- Describes whether or not a rule is enabled.
- Parent
Rule
- Values
Enabled
,Disabled
Transition
- Description
- Specifies a period and a destination for an object lifetime
- Parent
Rule
Days
- Description
- Specifies the number of days after object creation when the rule takes effect.
- Parent
Transition
and / orExpiration
If you specify the days’ value as 0, the objects are eligible to be sent to Glacier at midnight UTC following its creation.
StorageClass
- Description
- Specifies the storage class to which the object shall be transferred. On the Scaleway platform the storage classes
STANDARD
,GLACIER
andONEZONE_IA
are available. - Parent
Transition
Expiration
- Description
- Describes the expiration of the object lifetime. If versioning is enabled, this rule only deletes the current version of an object.
- Parent
Rule
ID
- Description
- Unique identifier for the rule. This value is limited to 255 characters.
- Parent
Rule
Configuration example
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>id-1</ID> <Expiration> <Days>0</Days> </Expiration> <Filter> <And> <Prefix>myprefix</Prefix> <Tag> <Key>mytagkey1</Key> <Value>mytagvalue1</Value> </Tag> <Tag> <Key>mytagkey2</Key> <Value>mytagvalue2</Value> </Tag> </And> </Filter> <Status>Enabled</Status> </Rule></LifecycleConfiguration>
The lifecycle configuration above translates to the following effective rules:
- on the rule named
id-1
, we set the objects with themyprefix
Prefix - the tags
mytagkey1
,mytagkey2
sets tomytagvalue1
- ‘mytagvalue2’ is set to expire after one day.
To see examples of lifecycle rules API requests, refer to the Bucket Operations documentation page.
Setting rules for incomplete multipart uploads
When using the awscli
to perform a multipart upload, it is possible to set a bucket lifecycle rule that allows you to automate the stoppage of uploads if they are not completed. You can define a deadline for the expected completion with a lifecycle rule. If the multipart upload is incomplete by the end of the defined timeframe, it will be stopped and the parts associated with it will be deleted.
You can automate the removal of incomplete multipart uploads by adding or uploading a lifecycle configuration to your bucket. The set-up can be done via an API call or an AWS-CLI command.
Send a PutBucketLifecycle request with the following XML payload:
<?xml version="1.0" encoding="UTF-8"?><LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule> <ID>Remove uncompleted uploads</ID> <Filter> <Prefix></Prefix> </Filter> <Status>Enabled</Status> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload></Rule></LifecycleConfiguration>
In this example, the multipart upload timeframe is of one day. You may change the number of DaysAfterInitiation
according to your preference by replacing 1
in the XML payload.
CLI command
- Create a file called
incomplete-upload-lifecycle-policy.json
with the following content.
In this example, the multipart upload timeframe is of one day. You may change the number of DaysAfterInitiation
according to your preference by replacing 1
in the .json
file body.
{ "Rules": [{ "ID": "Remove uncompleted uploads", "Status": "Enabled", "Filter": { "Prefix": "" }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } }]}
- Run the command:
aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://incomplete-upload-lifecycle-policy.json