It is possible to configure a lifecycle of a group of objects so that their storage costs can effectively be managed. A lifecycle configuration is a set of rules that defines actions applied to a group of objects stored on Object Storage.
Currently the expiration, transition and incomplete multipart uploads actions are supported on the platform:
There might be for example a need to store log files for a week or a month, after which they become obsolete. It is possible to set a lifecyle rule to delete them automatically when they become obsolete. Also, if you consider that a 3-month old object is rarely used but still has a value, you might want configure a rule to send it automatically to C14 Cold Storage, for example.
Lifecyle Management on Object Storage is available on every AWS S3 compliant tool (sdk, aws-cli, boto, etc), as well as from the Scaleway management console.
A lifecycle configuration is an XML file, containing a set of rules with predefined actions that you want Object Storage to perform on objects during their lifetime:
<LifecycleConfiguration>
<Rule>
...
</Rule>
<Rule>
...
</Rule>
...
</LifecycleConfiguration>
Each Rule
consists of the following:
Filter
identifying a subset of objects to which the rule applies.Status
whether the rule is in effect (A rule can be present but Disabled
)Rule
: Child of LifecycleConfiguration
Container for a lifecycle rule. There is a limit of thousand (1000) rules per bucket.
Filter
: Child of Rule
Container for elements that describe that identifies the subset of object the rule applies for. If the content of the tag is empty, the rule applies to all objects in the bucket.
And
: Child of Rule
Container for chaining rule filters. For example:
<And>
<Prefix>myprefix</Prefix>
<Tag>
<Key>mytagkey1</Key>
<Value>mytagvalue1</Value>
</Tag>
</And>
Which applies the Prefix
rule and the Tag
Prefix
: Child of Filter
or And
;
Object key prefix identifying one or more objects to which the rule applies. Object Storage does limit this token to 1 per Filter
Rule.
Tag
: Child of Filter
or And
;
Container for specifying a Key
and a Value
Status
: Child of Rule
;
Describes whether or not a rule is to be enabled. Values: Enabled,Disabled
Transition
: Child of Rule
;
Specifies a period and a destination for an object lifetime
Days
: Child of Transition
and / or Expiration
;
Specifies the number of days after object creation when the rule takes effect.
StorageClass
: Child of Transition
;
Specifies the Storage Class to which the object shall be transferred to.
On the Scaleway Elements platform the storage classes STANDARD
and GLACIER
are available.
Expiration
: Child of Rule
;
Describe a period in the objet’s lifetime for a deletion. If versioning is enabled, this rule shall only delete the current version of an object.
ID
: Child of Rule
;
Unique identifier for the rule. This value is limited to 255 characters.
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>id-1</ID>
<Expiration>
<Days>1</Days>
</Expiration>
<Filter>
<And>
<Prefix>myprefix</Prefix>
<Tag>
<Key>mytagkey1</Key>
<Value>mytagvalue1</Value>
</Tag>
<Tag>
<Key>mytagkey2</Key>
<Value>mytagvalue2</Value>
</Tag>
</And>
</Filter>
<Status>Enabled</Status>
</Rule>
</LifecycleConfiguration>
The lifecycle configuration above translates to the following effective rules:
On this rule named “id-1”, we set the objects with the ‘myprefix’ Prefix and the tags ‘mytagkey1’, ‘mytagkey2’ sets to ‘mytagvalue1’ and ‘mytagvalue2’ to expire after one day.
Sample Request:
PUT /?lifecycle HTTP/1.1
Host: bucketname.s3.nl-ams.scw.cloud
Content-MD5: e0d656aa8bed46fe766420c6d21537d5
Authorization: authorization string
<LifecycleConfiguration>
<Rule>
<ID>ExampleRule</ID>
<Filter>
<Prefix>documents/</Prefix>
</Filter>
<Status>Enabled</Status>
<Expiration>
<Days>30</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
Sample Output:
HTTP/1.1 200 OK
Sample Request:
GET /?lifecycle HTTP/1.1
Host: bucketname.s3.nl-ams.scw.cloud
Authorization: authorization string
Sample Output:
HTTP/1.1 200 OK
x-amz-id-2: txda274593e2dc4974a82cc-005cdbe130
x-amz-request-id: txda274593e2dc4974a82cc-005cdbe130
Date: Wed, 15 May 2019 09:51:44 GMT
Content-Length: 238
<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>ExampleRule</ID>
<Filter>
<Prefix>documents/</Prefix>
</Filter>
<Status>Enabled</Status>
<Expiration>
<Days>30</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
Sample Request:
DELETE /?lifecycle HTTP/1.1
Host: bucketname.s3.nl-ams.scw.cloud
Authorization: authorization string
Sample Output:
204 No Content
Content-Length: 0
Date: Wed, 15 May 2019 09:59:29 GMT
When using the awscli
to perform a Multipart Upload it is possible to set up a bucket lifecycle rule that allows you to automate the stoppage of uploads if they are not completed. You can define a deadline for the expected completion with a lifecycle rule. If the multipart upload is incomplete by the end of the defined timeframe, it will be stopped and the parts associated with it are deleted.
You can automate the removal of incomplete multipart uploads by adding or uploading a lifecycle policy to your bucket. The set-up can be done via an API call or an AWS-CLI command.
Send a PUT Bucket Lifecycle request with the following XML payload.
Note: In this example, the multipart upload timeframe is of one day. You may change the number of
DaysAfterInitiation
according to your preference by replacing1
in the XML payload.
<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>Remove uncompleted uploads</ID>
<Filter>
<Prefix></Prefix>
</Filter>
<Status>Enabled</Status>
<AbortIncompleteMultipartUpload>
<DaysAfterInitiation>1</DaysAfterInitiation>
</AbortIncompleteMultipartUpload>
</Rule>
</LifecycleConfiguration>
1 . Create a file called incomplete-upload-lifecycle-policy.json
with the following content.
Note: In this example, the multipart upload timeframe is of one day. You may change the number of
DaysAfterInitiation
according to your preference by replacing1
in the .json file body.
{
"Rules": [{
"ID": "Remove uncompleted uploads",
"Status": "Enabled",
"Filter": {
"Prefix": ""
},
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 1
}
}]
}
2 . Run the command:
$ aws s3api put-bucket-lifecycle-configuration --bucket mybucket --lifecycle-configuration file://incomplete-upload-lifecycle-policy.json
Via the Scaleway Console, you can either transfer objects to GLACIER
class [manually] (https://www.scaleway.com/en/docs/object-storage-glacier/#-Uploading-Data-to-Glacier-from-the-Scaleway-Console) or automatically with lifecycle rules. These rules also allow users to set expiration dates for objects and object versions.
1 . Access your Object Storage bucket of choice on the Scaleway Console, click on the bucket name and then on Lifecycle Rules.
2 . Click on + Create Lifecycle Rule to add a rule.
3 . Choose a name for the rule.
4 . Filter the objects you want the rule to be applied to by one or both of the following criteria:
Prefix - this criteria is based on the object name. The defined rule will be applied to all objects with the same name prefix.
Tags - comprised of a « Key » and a « Value » associated to the Key. These tags should be defined prior to the rule creation, for either a bucket or an object. You can add one tag or more to your filter.
If you wish to change the storage class of two or more objects at a time, you can set up a transition rule that will automate the process.
5 . Tick the box if you wish to transfer the current version of the objects to GLACIER
a number of days after its creation. Click on - or + to ajust the timeframe, or type in the exact amount.
Expiration rules work in similar fashion, objects can be set to be permanently deleted after a given period of time.
7 . Tick or leave boxes empty for each case according to your preference.
A summary of your rule preferences displays on the bottom of the Expiration Rules step page, as such:
8 . Check that all settings are correct and click on Create Lifecycle Rule to conclude.
A list of all bucket lifecycle rules displays.
1 . If you wish to disable a rule, open … on the right hand side of the rule in question and click Disable.
2 . A confirmation window pops up. Enter Disable Rule if you wish to pursue the action.
Once disabled, a bucket’s lifecycle rule remains listed and can be enabled once more by clicking on … and then Activate.
If you wish to permanently delete a rule, click on … and then Delete.