Help


Documentation & Tutorials

How to use Object Storage with AWS-CLI

Object Storage AWS-CLI Overview

Object Storage allows you to store any kind of object (documents, images, videos, etc.) and retrieve them at a later time from anywhere.

For instance, you can store images and they will be accessible using HTTP. You can use the control panel to manage your storage. Some tools exist to interact with Object Storage.

Requirements

Retrieving S3 Credentials

To retrieve your credentials, refer to S3 credentials.

Installing AWS-CLI

The AWS-CLI is an open source tool built on top of the AWS SDK for Python (Boto) that provides commands for interacting with AWS services. With minimal configuration, you can start using all of the functionality provided by the AWS Management.

To interact with AWS, aws-cli and awscli-plugin-endpoint need to be installed. The awscli-plugin-endpoint is a great plugin to help people more easily access third party S3 providers.

1 . Install aws-cli and awscli-plugin using pip

pip3 install awscli
pip3 install awscli-plugin-endpoint

2 . Create the file ~/.aws/config with the following content

[plugins]
endpoint = awscli_plugin_endpoint

[default]
region = nl-ams
s3 =
  endpoint_url = https://s3.nl-ams.scw.cloud
  max_concurrent_requests = 100
  max_queue_size = 1000
s3api =
  endpoint_url = https://s3.nl-ams.scw.cloud

3 . Insert your Scaleway Credentials in the ~/.aws/credentials file

[default]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>

4 . Test your cluster

aws s3 ls

To configure another S3 client, you can follow the instructions shown on the following page, hosted on our Object Storage infrastructure.

Creating a Bucket

1 . Create a bucket

aws s3 mb s3://$BucketName

2 . Upload your files as objects in your bucket

aws s3 cp $FileName s3://$BucketName

By default, the objects takes the name of the files but they can be renamed

aws s3 cp $FileName s3://$BucketName/$ObjectName

3 . List your bucket(s)

aws s3 ls

Enabling Bucket Versioning

Once enabled, the versioning feature supports the retrieval of objects that are deleted or overwritten. It’s a mean to keep multiple variants of an object in the same bucket. If the feature is enabled, you can list archived versions of an object or permanently delete an archived version. Note that, once versioning is enabled, you cannot return to an un-versioned state of your bucket.

Important: If you enable versioning, you will have several object versions. Listing your objects bucket will only display the latest version of the object. Keep in mind that each version of an object stored in your bucket takes disk space and is billed at 0.01€/Go

To enable versioning:

1 . Enabled versioning on your bucket

aws s3api put-bucket-versioning --bucket $BucketName --versioning-configuration Status=Enabled

2 . Add a test file to test the feature out

aws cp test s3://$BucketName

3 . List the versions of the object in the bucket

aws s3api list-object-versions --bucket $BucketName

which returns

{
    "Versions": [
        {
            "Key": "test",
            "Owner": {
                "ID": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480",
                "DisplayName": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480"
            },
            "LastModified": "2018-11-26T15:12:34.000Z",
            "VersionId": "1543245154846491",
            "Size": 20,
            "IsLatest": true,
            "StorageClass": "STANDARD",
            "ETag": "\"0137d789c45cf8374605b69579b93640\""
        }
    ]
}

4 . Add a new version of the test file in same the bucket

aws cp $FileName s3://$BucketName

5 . List the versions of the object in the bucket

aws s3api list-object-versions --bucket $BucketName

which returns

{
    "Versions": [
        {
            "Size": 15,
            "Owner": {
                "ID": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480",
                "DisplayName": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480"
            },
            "VersionId": "1543245250165373",
            "ETag": "\"d0e08e00cf91fb4911aa7def0f0c9f46\"",
            "IsLatest": true,
            "LastModified": "2018-11-26T15:14:10.000Z",
            "StorageClass": "STANDARD",
            "Key": "test"
        },
        {
            "Size": 20,
            "Owner": {
                "ID": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480",
                "DisplayName": "4e129ace-76cf-47ef-8894-bbaf2e8ee480:4e129ace-76cf-47ef-8894-bbaf2e8ee480"
            },
            "VersionId": "1543245154.00000",
            "ETag": "\"0137d789c45cf8374605b69579b93640\"",
            "IsLatest": false,
            "LastModified": "2018-11-26T15:14:10.000Z",
            "StorageClass": "STANDARD",
            "Key": "test"
        }
    ]
}

Managing Objects within a Bucket

1 . Download an object in a bucket

aws s3 cp s3://$BucketName/$ObjectName .

2 . List the object(s) of your bucket

aws s3 ls s3://$BucketName

3 . (Optional) Upload an object from a bucket to another bucket:

aws s3 cp s3://$BucketName/$ObjectName s3://$BucketCopy

4 . (Optional) Download or upload a whole bucket to host/bucket

aws s3 cp s3://$BucketName . --recursive
aws s3 cp s3://$BucketName s3://$BucketCopy --recursive

Moving Objects between Buckets

  • Moving a Host File to a Bucket
    aws s3 mv $FileName s3://$BucketName
    
  • Moving an object from a Bucket to a Host
    aws s3 mv s3://$BucketName/$ObjectName
    
  • Moving an object to a Bucket to another Bucket
    aws s3 mv s3://$BucketName/$ObjectName s3://$BucketCopy
    

The same way than aws s3 cp you can use --recursive

aws s3 mv s3://$BucketName . --recursive

Synchronizing Buckets

  • Synchronizing a directory with a bucket
    aws s3 sync . s3://$BucketName
    
  • Synchronizing two buckets together
    aws s3 sync s3://$BucketName s3://$BucketCopy
    

Deleting Objects and Buckets

  • Deleting an object
    aws s3 rm s3://$BucketName/$ObjectName
    
  • Deleting all objects from a bucket
    aws s3 rm s3://$BucketName --recursive
    
  • Deleting a bucket. To delete a bucket, it must be empty.
    aws s3 rb s3://$BucketName
    

If the bucket is not deleted, you can use the same command with the --force option. This command deletes all the objects from the bucket and then deletes the bucket.

aws s3 rb s3://$BucketName --force

Setting ACL

1 . Retrieve your organisation ID from the Scaleway Console under your username > account in the top right corner

2 . Add full control to all users from same organization on a bucket

aws s3api put-bucket-acl --bucket $BucketName --grant-full-control id=$ORG_ID:$ORG_ID

3 . Add full control to all users from same organization on an object

aws s3api put-object-acl --bucket $BucketName --key $ObjectName --grant-full-control id=$ORG_ID:$ORG_ID

For more information, you can refer to the official AWS S3 and S3api documentation.

Discover a New Cloud Experience

Deploy SSD Cloud Servers in seconds.