How to use Object Storage with s3cmd

Using Object Storage with s3cmd

Learn how to use s3cmd with Object Storage by following this tutorial.

Requirements

• You have an account and are logged into console.scaleway.com
• You have generated your API Token

Object Storage allows you to store any kind of objects (documents, images, videos, etc.). You can retrieve them anytime and from anywhere.

For instance, you can store images and they will be accessible using HTTP. You can use the control panel to manage your storage. As the our API is S3 compatible, a lot of tools exist to interact with the object storage.

This tutorial details how to use the Object Storage with s3cmd.

There are four steps to configure and use s3cmd:

• Retrieving your Credentials
• Downloading S3cmd
• Configuring S3cmd
• Using Object Storage with s3cmd

Retrieving your Credentials

To retrieve your credentials, refer to S3 credentials.

Downloading S3cmd

A simple solution to begin with Object Storage is to use the S3cmd tool, which is a client for S3 Object Storage.

S3cmd allows you to create, list and delete buckets, download, upload, and delete objects inside the Object Storage. Download S3cmd S3cmd on github.

If you use Linux or OS X, your package manager probably has a package for s3cmd.

Important: Make sure you have at least the version 2.0 (s3cmd --version). Upgrade s3cmd if it is not the case. Previous versions might not work properly.

Configuring S3cmd

1 . In a terminal, create a file named .s3cfg in your home directory.

2 . Replace access_key = <ACCESS_KEY TO INSERT HERE> and secret_key = <SECRET_KEY TO INSERT HERE> with your credentials.

[default]
# Object Storage Region NL-AMS
host_base = s3.nl-ams.scw.cloud
host_bucket = %(bucket)s.s3.nl-ams.scw.cloud
bucket_location = nl-ams
use_https = True

# Login credentials
access_key = <ACCESS_KEY TO INSERT HERE>
secret_key = <SECRET_KEY TO INSERT HERE>

Alternatively you can also use the configuration file generator by running s3cmd --configure to generate a configuration file.

Using Multiple Configuration Files

It is possible to configure s3cmd to use different configuration files on the same computer to manage buckets in multiple regions.

1 . Create a default configuration file as indicated in the previous step.

2 . In a terminal, create a new file named .s3cfg-fr-par in your home directory.

3 . Replace access_key = <ACCESS_KEY TO INSERT HERE> and secret_key = <SECRET_KEY TO INSERT HERE> with your credentials.

[default]
signature = s3v4
# Object Storage Region FR-PAR
bucket_location = fr-par
host_base = https://s3.fr-par.scw.cloud
host_bucket = https://s3.fr-par.scw.cloud

# Login credentials
access_key = <ACCESS_KEY TO INSERT HERE>
secret_key = <SECRET_KEY TO INSERT HERE>

4 . Use s3cmd by specifying the new configuration file in the command:

s3cmd ls -c ~/.s3cfg-fr-par

Using Object Storage with S3cmd

Creating a New Bucket

The following command creates a bucket to store objects with S3cmd. A bucket name must contain only alphanumeric and lowercase characters.

john@localhost:~$ s3cmd mb s3://mynewbucket

Bucket 's3://mynewbucket/' created

Uploading Objects into the Bucket

The following command uploads movie1.avi and photo1.jpg into the bucket mynewbucket:

john@localhost:~$ s3cmd put movie1.avi photo1.jpg s3://mynewbucket

movie1.avi -> s3://mynewbucket/movie1.avi  [1 of 2]
 0 of 0     0% in    0s     0.00 B/s  done
photo1.jpg -> s3://mynewbucket/photo1.jpg  [2 of 2]
 0 of 0     0% in    0s     0.00 B/s  done

Important: When uploading files from a Scaleway Compute Instance an error message may appear. Your file will be uploaded and you can ignore this message:

ERROR: Cannot retrieve any response status before encountering an EPIPE or ECONNRESET exception
WARNING: Upload failed: /bucket_name?partNumber=1&uploadId=ZWY5ZWRhNzItMjE3NS00ZTY4LWFkOTMtYTFhODRkYTRkNzdl ([Errno 32] Broken pipe)

We are currently working to avoid this behaviour.

Listing Objects inside a Bucket

The following command lists the objects inside a bucket:

john@localhost:~$ s3cmd ls s3://mynewbucket

2014-09-10 08:44         0   s3://mynewbucket/movie1.avi
2014-09-10 08:44         0   s3://mynewbucket/photo1.jpg

Downloading an Object from a Bucket

The following command downloads the object movie1.avi locally:

john@localhost:~$ s3cmd get s3://mynewbucket/movie1.avi movie1.avi

s3://mynewbucket/movie1.avi -> movie.avi  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

Removing Object from a Bucket

The following command removes the object movie1.avi from the bucket mynewbucket:

john@localhost:~$ s3cmd del s3://mynewbucket/movie1.avi

File s3://mynewbucket/movie1.avi deleted

Removing a Bucket

The following command removes the bucket mynewbucket:

john@localhost:~$ s3cmd rb s3://mynewbucket

Bucket 's3://mynewbucket/' removed

Configuring CORS

Setting a CORS configuration on a bucket

1 . Create a CORS configuration file and open it in a text editor:

nano cors.xml

2 . Put your configuration in the file, save it and exit the editor:

<CORSConfiguration>
  <CORSRule>
     <AllowedOrigin>http://www.example.com</AllowedOrigin>
     <AllowedMethod>PUT</AllowedMethod>
     <AllowedMethod>POST</AllowedMethod>
     <AllowedMethod>DELETE</AllowedMethod>
     <AllowedHeader>*</AllowedHeader>
     <MaxAgeSeconds>3000</MaxAgeSeconds>
     <ExposeHeader>x-server-side-encryption</ExposeHeader>
  </CORSRule>
</CORSConfiguration>

3 . Set the information on the bucket:

s3cmd setcors cors.xml s3://bucketname

Deleting the CORS configuration of a bucket

1 . To delete the CORS configuration of a bucket, run the following command:

s3cmd delcors s3://bucketname

Important: It is currently not possible to get the CORS configuration of a bucket with s3cmd. It is recommended to use aws-cli to get the CORS settings of a bucket.

For more information about the different s3cmd commands, refer to the official documentation by running s3cmd --help.