Object Storage allows you to store any kind of object (documents, images, videos, etc.) and retrieve them at a later time from anywhere.
For instance, you can store images and they will be accessible using HTTP. You can use the control panel to manage your storage. Some tools exist to interact with Object Storage.
Requirements
- You have an account and are logged into console.scaleway.com
- You have generated your API Key
To retrieve your credentials, refer to S3 credentials.
The AWS-CLI is an open source tool built on top of the AWS SDK for Python (Boto) that provides commands for interacting with AWS services. With minimal configuration, you can start using all of the functionality provided by the AWS Management.
To interact with AWS, aws-cli
and awscli-plugin-endpoint
need to be installed. The awscli-plugin-endpoint is a great plugin to help people more easily access third party S3 providers.
1 . Install aws-cli
and awscli-plugin
using pip
pip3 install awscli
pip3 install awscli-plugin-endpoint
2 . Create the file ~/.aws/config
by running the following command:
aws configure set plugins.endpoint awscli_plugin_endpoint
3 . Open the file in a text editor and edit it as follows:
[plugins]
endpoint = awscli_plugin_endpoint
[default]
region = nl-ams
s3 =
endpoint_url = https://s3.nl-ams.scw.cloud
signature_version = s3v4
max_concurrent_requests = 100
max_queue_size = 1000
multipart_threshold = 50MB
# Edit the multipart_chunksize value according to the file sizes that you want to upload. The present configuration allows to upload files up to 10 GB (100 requests * 10MB). For example setting it to 5GB allows you to upload files up to 5TB.
multipart_chunksize = 10MB
s3api =
endpoint_url = https://s3.nl-ams.scw.cloud
Important: Set the
endpoint_url
andregion
corresponding to the geographical region of your bucket. It can either befr-par
(Paris, France),nl-ams
(Amsterdam, The Netherlands) orpl-waw
(Warsaw, Poland).
4 . Generate a credentials file using the following command:
aws configure
5 . Open the file ~/.aws/credentials
and configure your API Key as follows:
[default]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>
6 . Test your cluster
aws s3 ls
To configure another S3 client, you can follow the instructions shown on the following page, hosted on our Object Storage infrastructure.
1 . Create a bucket
aws s3 mb s3://$BucketName
2 . Upload your files as objects in your bucket
aws s3 cp $FileName s3://$BucketName
By default, the objects takes the name of the files but they can be renamed
aws s3 cp $FileName s3://$BucketName/$ObjectName
3 . List your bucket(s)
aws s3 ls
Once enabled, the versioning feature supports the retrieval of objects that are deleted or overwritten. It’s a mean to keep multiple variants of an object in the same bucket. If the feature is enabled, you can list archived versions of an object or permanently delete an archived version. Note that, once versioning is enabled, you cannot return to an un-versioned state of your bucket.
Important: If you enable versioning, you will have several object versions. Listing your objects bucket will only display the latest version of the object. Keep in mind that each version of an object stored in your bucket takes disk space and is billed at 0.01€/Go
To enable versioning:
1 . Enabled versioning on your bucket
aws s3api put-bucket-versioning --bucket $BucketName --versioning-configuration Status=Enabled
2 . Add a test
file to test the feature out
aws cp test s3://$BucketName
3 . List the versions of the object in the bucket
aws s3api list-object-versions --bucket $BucketName
which returns
{
"Versions": [
{
"Key": "test",
"Owner": {
"ID": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480",
"DisplayName": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480"
},
"LastModified": "2018-11-26T15:12:34.000Z",
"VersionId": "1543245154846491",
"Size": 20,
"IsLatest": true,
"StorageClass": "STANDARD",
"ETag": "\"0137d789c45cf8374605b69579b93640\""
}
]
}
4 . Add a new version of the test
file in same the bucket
aws cp $FileName s3://$BucketName
5 . List the versions of the object in the bucket
aws s3api list-object-versions --bucket $BucketName
which returns
{
"Versions": [
{
"Size": 15,
"Owner": {
"ID": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480",
"DisplayName": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480"
},
"VersionId": "1543245250165373",
"ETag": "\"d0e08e00cf91fb4911aa7def0f0c9f46\"",
"IsLatest": true,
"LastModified": "2018-11-26T15:14:10.000Z",
"StorageClass": "STANDARD",
"Key": "test"
},
{
"Size": 20,
"Owner": {
"ID": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480",
"DisplayName": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480"
},
"VersionId": "1543245154.00000",
"ETag": "\"0137d789c45cf8374605b69579b93640\"",
"IsLatest": false,
"LastModified": "2018-11-26T15:14:10.000Z",
"StorageClass": "STANDARD",
"Key": "test"
}
]
}
1 . Download an object in a bucket
aws s3 cp s3://$BucketName/$ObjectName .
2 . List the object(s) of your bucket
aws s3 ls s3://$BucketName
3 . (Optional) Upload an object from a bucket to another bucket:
aws s3 cp s3://$BucketName/$ObjectName s3://$BucketCopy
4 . (Optional) Download or upload a whole bucket to host/bucket
aws s3 cp s3://$BucketName . --recursive
aws s3 cp s3://$BucketName s3://$BucketCopy --recursive
Moving a Host File to a Bucket
aws s3 mv $FileName s3://$BucketName
Moving an object from a Bucket to a Host
aws s3 mv s3://$BucketName/$ObjectName
Moving an object to a Bucket to another Bucket
aws s3 mv s3://$BucketName/$ObjectName s3://$BucketCopy
The same way than aws s3 cp
you can use --recursive
aws s3 mv s3://$BucketName . --recursive
Synchronizing a directory with a bucket
aws s3 sync . s3://$BucketName
Synchronizing two buckets together
aws s3 sync s3://$BucketName s3://$BucketCopy
Deleting an object
aws s3 rm s3://$BucketName/$ObjectName
Deleting all objects from a bucket
aws s3 rm s3://$BucketName --recursive
Deleting a bucket. To delete a bucket, it must be empty.
aws s3 rb s3://$BucketName
If the bucket is not deleted, you can use the same command with the --force
option. This command deletes all the objects from the bucket and then deletes the bucket.
aws s3 rb s3://$BucketName --force
Batch deleting all the objects and versions in a bucket.
aws s3api list-object-versions --bucket bucket_name | jq '.Versions | to_entries[] | .value | .Key, .VersionId' | xargs -n2 sh -c 'aws s3api delete-object --bucket bucket_name --key $0 --version-id $1'
1 . PUT a tag set to an bucket:
aws s3api put-bucket-tagging --bucket mybucket --tagging 'TagSet=[{Key=client,Value=scaleway}]'
2 . GET the tag set of an bucket:
aws s3api get-bucket-tagging --bucket mybucket
{
"TagSet": [
{
"Key": "client",
"Value": "scaleway"
}
]
}
3 . DELETE the tag set of an bucket:
aws s3api delete-bucket-tagging --bucket mybucket
4 . GET the tag set of a bucket without configured tagging, will return an error message:
aws s3api get-bucket-tagging --bucket mybucket
An error occurred (NoSuchTagSet) when calling the GetBucketTagging operation: There is no tag set associated with the bucket or object.
1 . PUT a tag set on an object:
aws s3api put-object-tagging --bucket mybucket --key myfile.txt --tagging 'TagSet=[{Key=client,Value=scaleway},{Key=service,Value=objectstorage}]'
2 . GET the tag set of an object:
aws s3api get-object-tagging --bucket mybucket --key myfile.txt
{
"TagSet": [
{
"Key": "client",
"Value": "scaleway"
},
{
"Key": "service",
"Value": "objectstorage"
}
],
"VersionId": "1552299728854249"
}
3 . DELETE the tag set of an object:
ws s3api delete-object-tagging --bucket mybucket --key myfile.txt
1 . Retrieve your organisation ID from the Scaleway Console under your username > account in the top right corner
2 . Add full control to all users from same organization on a bucket
aws s3api put-bucket-acl --bucket $BucketName --grant-full-control id=$ORG_ID:$ORG_ID
3 . Add full control to all users from same organization on an object
aws s3api put-object-acl --bucket $BucketName --key $ObjectName --grant-full-control id=$ORG_ID:$ORG_ID
It is possible to manage buckets in multiple regions from the same computer with aws-cli by creating different profiles.
1 . Open the configuration file ~/.aws/config
and edit it by adding a new profile for the fr-par
region:
[plugins]
endpoint = awscli_plugin_endpoint
[default]
region = nl-ams
s3 =
endpoint_url = https://s3.nl-ams.scw.cloud
signature_version = s3v4
max_concurrent_requests = 100
max_queue_size = 1000
multipart_threshold = 50MB
# Edit the multipart_chunksize value according to the file sizes that you want to upload. The present configuration allows to upload files up to 10 GB (100 requests * 10MB). For example setting it to 5GB allows you to upload files up to 5TB.
multipart_chunksize = 10MB
s3api =
endpoint_url = https://s3.nl-ams.scw.cloud
[profile fr-par]
region = fr-par
s3 =
endpoint_url = https://s3.fr-par.scw.cloud
signature_version = s3v4
max_concurrent_requests = 100
max_queue_size = 1000
multipart_threshold = 50MB
# Edit the multipart_chunksize value according to the file sizes that you want to upload. The present configuration allows to upload files up to 10 GB (100 requests * 10MB). For example setting it to 5GB allows you to upload files up to 5TB.
multipart_chunksize = 10MB
s3api =
endpoint_url = https://s3.fr-par.scw.cloud
2 . Add authentication information for the new region in the file ~/.aws/credentials
:
[default]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>
[fr-par]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>
Important:
<ACCESS_KEY>
and<SECRET_KEY>
can have the same value for both regions.
3 . Run commands with the prefix --profile fr-par
for actions to be performed in the Paris region. Otherwise the default configuration will be considered as active and commands are executed in the Amsterdam region.
Important: It is not possible to have buckets with the same name in both regions.
Cross-origin ressource sharing (CORS) is a browser security system, when two domains communicate with each other through the browser. The web browser will always send a header:
Origin: http://URL_OF_THE_SITE
The browser will then expect the opposite server replying that it authorizes the URL of the original site with a header:
Access-Control-Allow-Origin: http://URL_OF_THE_SITE
CORS allows the user to configure this and a lot of other security related options like accepting PUT
, GET
, DELETE
, specific headers and more.
1 . The CORS configuration shall be provided in a JSON file. Create the file and open a text editor:
nano cors.json
2 . Put the configuration into the file:
{
"CORSRules": [
{
"AllowedOrigins": ["http://www.example.com"],
"AllowedHeaders": ["*"],
"AllowedMethods": ["GET", "HEAD", "POST", "PUT", "DELETE"],
"MaxAgeSeconds": 3000,
"ExposeHeaders": ["Etag"]
}
]
}
3 . Save the file and send the configuration to the bucket:
aws s3api put-bucket-cors --bucket bucketname --cors-configuration file://cors.json
1 . Retrieve the CORS configration of the bucket by running the following command:
aws s3api get-bucket-cors --bucket bucketname
2 . The command will return the CORS configuration of the bucket:
{
"CORSRules": [
{
"AllowedOrigins": [
"http://www.example.com"
],
"AllowedMethods": [
"GET",
"HEAD",
"POST",
"PUT",
"DELETE"
],
"ExposeHeaders": [
"Etag"
],
"AllowedHeaders": [
"*"
],
"MaxAgeSeconds": 3000
}
]
}
In case no CORS rules are defined for the bucket, an error message will be displayed:
An error occurred (NoSuchCORSConfiguration) when calling the GetBucketCors operation: The CORS configuration does not exist
1 . It is possible to delete the CORS configuration of a bucket, by running the following command:
aws s3api delete-bucket-cors --bucket bucketname
2 . No output is returned if the operation is successful.
1 . To determine if the actual request is supported with the specific origin, HTTP method and headers, a browser can send a preflight request to the Object Storage platform: ```curl -X OPTIONS -H ‘Origin: http://www.example.com’ http://bucketname.s3.nl-ams.scw.cloud/index.html -H “Access-Control-Request-Method: GET”
2 . If everything is working as expected, the Object Storage platform will send a confirmation:
HTTP/1.1 200 OK
3 . In case the CORS request is not allowed, an error message is displayed:
CORSResponse: This CORS request is not allowed. This is usually because the evalution of Origin, request method / Access-Control-Request-Method or Access-Control-Request-Headers are not whitelisted by the resource’s CORS spec. ```
For more information, you can refer to the official AWS S3 and S3api documentation.