NavigationContentFooter
Jump toSuggest an edit

Using Object Storage with s3cmd

Reviewed on 17 June 2024Published on 04 June 2018
  • Object-Storage
  • s3cmd
  • CORS
  • buckets

In this tutorial, you will learn how to use s3cmd as a client for Scaleway Object Storage. s3cmd is a tool that allows you to create, list and delete buckets from the command line, as well as to download, upload, and delete objects to/from your buckets.

Before you start

To complete the actions presented below, you must have:

  • A Scaleway account logged into the console
  • Owner status or IAM permissions allowing you to perform actions in the intended Organization
  • A valid API key

Installing s3cmd

You can install s3cmd from the command line using your Linux package manager.

  1. Run the following command in a terminal to update and upgrade your packages:
    apt update && apt upgrade -y
  2. Run the following command to install s3cmd:
apt install s3cmd
Note

At the time of writing, this installs version `2.4.0. To install later versions of s3cmd, or to install on other operating systems, download directly from GitHub or SourceForge. See the official s3cmd documentation for more help.

Configuring s3cmd

Carry out the following commands from your terminal.

  1. Create a configuration file named .s3cfg in your home directory:
    cd $HOME
    touch .s3cfg
  2. Open the file with a text editor, e.g. nano:
    nano .s3cfg
  3. Copy and paste the following text into the file, then save and close. Make sure you use the host-base, host-bucket, and bucket_location details of your own bucket (visible in the Bucket settings tab of the console). You will also need to replace <ACCESS_KEY> and <SECRET_KEY> with your own API credentials.
    [default]
    # Object Storage Region NL-AMS
    host_base = s3.nl-ams.scw.cloud
    host_bucket = %(bucket).s3.nl-ams.scw.cloud
    bucket_location = nl-ams
    use_https = True
    # Login credentials
    access_key = <ACCESS_KEY>
    secret_key = <SECRET_KEY>
    Tip

    Alternatively, you can use the configuration file generator by running s3cmd --configure to generate a configuration file.

Using multiple configuration files

You can configure s3cmd to use different configuration files on the same computer to manage buckets in multiple regions. Here, we assume that your default configuration file was for a bucket in nl-ams, and we create a new configuration file for buckets in fr-par.

Carry out the following steps from the terminal.

  1. Create a default configuration file as indicated in the previous step.
  2. Create a new file named .s3cfg-fr-par in your home directory.
    cd $HOME
    touch .s3cfg-fr-par
  3. Copy and paste the following text into the file, then save and close. Make sure you replace <ACCESS_KEY> and <SECRET_KEY> with your own API credentials.
    [default]
    signature = s3v4
    # Object Storage Region FR-PAR
    bucket_location = fr-par
    host_base = https://s3.fr-par.scw.cloud
    host_bucket = https://s3.fr-par.scw.cloud
    # Login credentials
    access_key = <ACCESS_KEY>
    secret_key = <SECRET_KEY>
  4. Use s3cmd by specifying the new configuration file in the command:
    s3cmd ls -c ~/.s3cfg-fr-par

Using Object Storage with s3cmd

Creating a new bucket

Enter the following command to create a bucket with s3cmd. The bucket name must contain only alphanumeric and lowercase characters. Here, we call it mynewbucket:

s3cmd mb s3://mynewbucket

An output similar to the following displays:

Bucket 's3://mynewbucket/' created

Uploading objects into the bucket

Enter the following command to upload files into a bucket. Here, we upload the files movie1.avi and photo1.jpg into mynewbucket:

s3cmd put movie1.avi photo1.jpg s3://mynewbucket

An output similar to the following displays:

movie1.avi -> s3://mynewbucket/movie1.avi [1 of 2]
0 of 0 0% in 0s 0.00 B/s done
photo1.jpg -> s3://mynewbucket/photo1.jpg [2 of 2]
0 of 0 0% in 0s 0.00 B/s done
Important

When uploading files from a Scaleway Instance, an error message like the following may appear. Your file will be uploaded nonetheless, and you can ignore this message:

ERROR: Cannot retrieve any response status before encountering an EPIPE or ECONNRESET exception
WARNING: Upload failed: /bucket_name?partNumber=1&uploadId=ZWY5ZWRhNzItMjE3NS00ZTY4LWFkOTMtYTFhODRkYTRkNzdl ([Errno 32] Broken pipe)

We are currently working to correct this issue.

Listing objects inside a bucket

Enter the following command to list files inside a bucket. Here, we list the files in the bucket mynewbucket:

s3cmd ls s3://mynewbucket

An output similar to the following displays, showing the files inside your bucket:

2014-09-10 08:44 0 s3://mynewbucket/movie1.avi
2014-09-10 08:44 0 s3://mynewbucket/photo1.jpg

Downloading an object from a bucket

Enter the following command to download an object from your bucket to your local machine. Here, we download the object movie1.avi from the bucket mynewbucket:

s3cmd get s3://mynewbucket/movie1.avi movie1.avi

An output similar to the following displays:

s3://mynewbucket/movie1.avi -> movie.avi [1 of 1]
0 of 0 0% in 0s 0.00 B/s done

Removing objects from a bucket

Enter the following command to remove an object from your bucket. Here, we remove the object movie1.avi from the bucket mynewbucket:

s3cmd del s3://mynewbucket/movie1.avi

An output similar to the following displays:

delete: 's3://mynewbucket/movie1.avi'

Removing a bucket

Enter the following command to remove a bucket. Here, we remove the bucket mynewbucket:

s3cmd rb s3://mynewbucket

You should see an output similar to the following:

Bucket 's3://mynewbucket/' removed

Configuring CORS

The CORS standard provides browsers with a way to request resources from remote domains when they have permission. This lets you configure how your buckets can be accessed from browser-based applications and websites.

Setting a CORS configuration on a bucket

  1. Create a CORS configuration file and open it in a text editor:
    nano cors.xml
  2. Put your configuration in the file, save it, and exit the editor:
    <CORSConfiguration>
    <CORSRule>
    <AllowedOrigin>http://www.example.com</AllowedOrigin>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <ExposeHeader>x-server-side-encryption</ExposeHeader>
    </CORSRule>
    </CORSConfiguration>
  3. Set the information on the bucket. Make sure to replace bucketname with the name of your bucket:
    s3cmd setcors cors.xml s3://bucketname

Deleting the CORS configuration of a bucket

Run the following command to delete the CORS configuration of a bucket. Replace bucketname with the name of your bucket:

s3cmd delcors s3://bucketname
Important

It is currently not possible to get the CORS configuration of a bucket with s3cmd. It is recommended to use aws-cli to get the CORS settings of a bucket.

Going further

For more information about the different s3cmd commands, refer to the official documentation, or run the following command in a terminal:

s3cmd --help
Docs APIScaleway consoleDedibox consoleScaleway LearningScaleway.comPricingBlogCarreer
© 2023-2024 – Scaleway