Jump toUpdate content

How to use Object Storage with s3cmd

Reviewed on 14 June 2022Published on 04 June 2018
  • Object-Storage
  • s3cmd
  • CORS
  • buckets

In this tutorial you will learn how to use s3cmd as a client for Scaleway Object Storage. S3cmd is a tool that allows you to create, list and delete buckets from the command line, as well as to download, upload, and delete objects to/from your buckets.

Requirements:

Installing S3cmd

You can install S3cmd frm the command line using your Linux package manager.

  1. Enter the following command to update and upgrade your packages:

    apt update && apt upgrade -y
  2. Enter the following command to install s3cmd:

    apt install s3cmd
Note:

At the time of writing, this installs version 2.0.2. To install later versions of s3cmd, or to install on other operating systems, download directly from GitHub or SourceForge. See the official s3cmd documentation for more help.

Configuring S3cmd

Carry out the following commands from your terminal.

  1. Create a configuration file named .s3cfg in your home directory:

    cd $HOME
    touch .s3cfg
  2. Open the file with a text editor, e.g. nano:

    nano .s3cfg
  3. Copy and paste the following text into the file, then save and close. Make sure you use the host-base, host-bucket and bucket_location details of your own bucket (visible in the Bucket Settings tab of the console). You will also need to replace <ACCESS_KEY> and <SECRET_KEY> with your own API credentials.

    [default]
    # Object Storage Region NL-AMS

    host_base = s3.nl-ams.scw.cloud
    host_bucket = %(bucket)s.s3.nl-ams.scw.cloud
    bucket_location = nl-ams
    use_https = True

    # Login credentials

    access_key = <ACCESS_KEY>
    secret_key = <SECRET_KEY>
    Tip:

    Alternatively you can use the configuration file generator by running s3cmd --configure to generate a configuration file.

Using Multiple Configuration Files

You can configure s3cmd to use different configuration files on the same computer to manage buckets in multiple regions. Here, we assume that your default configuation file was for a bucket in nl-ams, and we create a new configuration file for buckets in fr-par.

Carry out the following steps from the terminal.

  1. Create a default configuration file as indicated in the previous step.

  2. Create a new file named .s3cfg-fr-par in your home directory.

    cd $HOME
    touch .s3cfg-fr-par
  3. Copy and paste the following text into the file, then save and close. Make sure you replace <ACCESS_KEY> and <SECRET_KEY> with your own API credentials.

    [default]
    signature = s3v4

    # Object Storage Region FR-PAR

    bucket_location = fr-par
    host_base = https://s3.fr-par.scw.cloud
    host_bucket = https://s3.fr-par.scw.cloud

    # Login credentials

    access_key = <ACCESS_KEY>
    secret_key = <SECRET_KEY>
  4. Use s3cmd by specifying the new configuration file in the command:

    s3cmd ls -c ~/.s3cfg-fr-par

Using Object Storage with S3cmd

Creating a new bucket

Enter the following command to create a bucket with S3cmd. The bucket name must contain only alphanumeric and lowercase characters. Here, we call it mynewbucket:

s3cmd mb s3://mynewbucket

You should see an output similar to the following:

Bucket 's3://mynewbucket/' created

Uploading objects into the bucket

Enter the following command to upload files into a bucket. Here, we upload the files movie1.avi and photo1.jpg into mynewbucket:

s3cmd put movie1.avi photo1.jpg s3://mynewbucket

You should see an output similar to the following:

movie1.avi -> s3://mynewbucket/movie1.avi [1 of 2]
0 of 0 0% in 0s 0.00 B/s done
photo1.jpg -> s3://mynewbucket/photo1.jpg [2 of 2]
0 of 0 0% in 0s 0.00 B/s done
Important:

When uploading files from a Scaleway Instance, an error message like the following may appear. Your file will be uploaded nonetheless, and you can ignore this message:

ERROR: Cannot retrieve any response status before encountering an EPIPE or ECONNRESET exception
WARNING: Upload failed: /bucket_name?partNumber=1&uploadId=ZWY5ZWRhNzItMjE3NS00ZTY4LWFkOTMtYTFhODRkYTRkNzdl ([Errno 32] Broken pipe)

We are currently working to correct this issue.

Listing objects inside a bucket

Enter the following command to list files inside a bucket. Here, we list the files in the bucket mynewbucket:

s3cmd ls s3://mynewbucket

You should see an output similar to the following, showing the files inside your bucket:

2014-09-10 08:44 0 s3://mynewbucket/movie1.avi
2014-09-10 08:44 0 s3://mynewbucket/photo1.jpg

Downloading an object from a bucket

Enter the following command to download an object from your bucket to your local machine. Here, we download the object movie1.avi from the bucket mynewbucket:

s3cmd get s3://mynewbucket/movie1.avi movie1.avi

You should see an output similar to the following:

s3://mynewbucket/movie1.avi -> movie.avi [1 of 1]
0 of 0 0% in 0s 0.00 B/s done

Removing objects from a bucket

Enter the following command to remove an object from your bucket. Here, we remove the object movie1.avi from the bucket mynewbucket:

s3cmd del s3://mynewbucket/movie1.avi

You should see an output similar to the followng:

File s3://mynewbucket/movie1.avi deleted

Removing a bucket

Enter the following command to remove a bucket. Here, we remove the bucket mynewbucket:

john@localhost:~$ s3cmd rb s3://mynewbucket

You should see an output similar to the followng:

Bucket 's3://mynewbucket/' removed

Configuring CORS

The CORS standard provides browsers with a way to request resources from remote domains when they have permission. This lets you configure how your buckets can be accessed from browser-based applications and websites.

Setting a CORS configuration on a bucket

  1. Create a CORS configuration file and open it in a text editor:

    nano cors.xml
  2. Put your configuration in the file, save it and exit the editor:

    <CORSConfiguration>
    <CORSRule>
    <AllowedOrigin>http://www.example.com</AllowedOrigin>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <ExposeHeader>x-server-side-encryption</ExposeHeader>
    </CORSRule>
    </CORSConfiguration>
  3. Set the information on the bucket. Make sure to replace bucketname with the name of your bucket:

    s3cmd setcors cors.xml s3://bucketname

Deleting the CORS configuration of a bucket

Run the following command to delete the CORS configuration of a bucket. Replace bucketname with the name of your bucket:

s3cmd delcors s3://bucketname
Important:

It is currently not possible to get the CORS configuration of a bucket with s3cmd. It is recommended to use aws-cli to get the CORS settings of a bucket.

Going further

For more information about the different s3cmd commands, refer to the official documentation by running s3cmd --help.