Migrating Scaleway Object Storage data with Rclone

Rclone provides a modern alternative to rsync. The tool communicates with any Amazon S3-compatible cloud storage provider as well as other storage platforms and can be used to migrate data from one bucket to another, even if those buckets are in different regions.

Before you start

To complete the actions presented below, you must have:

• A Scaleway account logged into the console
• Owner status or IAM permissions allowing you to perform actions in the intended Organization
• An SSH key
• A valid API key
• An Instance running on Ubuntu Focal Fossa
• At least 2 Object Storage buckets
• Installed and initialized the Scaleway CLI

Installing Rclone

1. Connect to your server as root via SSH.
2. Update the APT package cache and the software already installed on the Instance.
   apt update && apt upgrade -y

3. Download and install Rclone with the following sequence of commands:
   wget https://downloads.rclone.org/rclone-current-linux-amd64.zip
   apt install zip
   unzip rclone-current-linux-amd64.zip
   cd rclone*linux-amd64/
   mv rclone /usr/bin/

Configuring Rclone

The Scaleway CLI allows you to create a configuration file for Rclone based on your Scaleway CLI configuration.

Run the following command in a terminal to install a configuration file for Rclone:

scw object config install type=rclone

An output similar to the following displays:

Configuration file successfully installed at /Users/yourusername/.config/rclone/rclone.conf.

Migrating data

Two commands can be used to migrate data from one backend to another.

• The copy command copies data from source to destination.

rclone copy --progress <SOURCE_BACKEND>:<SOURCE_PATH> <DEST_BACKEND>:<DEST_PATH>

For example, the following command copies data from a bucket named my-first-bucket in the remote-sw-paris remote backend that we previously set up, to another bucket named my-second-bucket in the same remote backend. The --progress flag allows us to follow the progress of the transfer:

rclone copy --progress remote-sw-paris:my-first-bucket remote-sw-paris:my-second-bucket

• The sync command copies data from one backend to another, but also deletes objects in the destination that are not present in the source:

rclone sync --progress <SOURCE_BACKEND>:<SOURCE_PATH> <DEST_BACKEND>:<DEST_PATH>

The following command copies data from a bucket named my-first-bucket in the remote-sw-paris remote backend that we previously set up, to another bucket named my-third-bucket in a different remote backend that we configured for the nl-ams region and named remote-sw-ams. It also deletes any data present in my-third-bucket that isn't also present in my-first-bucket:

rclone sync --progress remote-sw-paris:my-first-bucket remote-sw-ams:my-third-bucket

For further information, and to discover other commands available, refer to the official RClone S3 Object Storage Documentation.

Transferring data to Scaleway Glacier

When you copy or sync, you can determine which storage class you wish to transfer your data to.

At Scaleway you can choose from two classes:

• STANDARD: The Standard class for any upload; suitable for on-demand content like streaming or CDN.
• GLACIER: Archived, long-term retention storage.

If the storage class is not specified, the data will be transferred as STANDARD by default.

To transfer data to Scaleway Glacier class, add --s3-storage-class=GLACIER to your command, as such:

rclone copy --progress --s3-storage-class=GLACIER <SOURCE_BACKEND>:<SOURCE_PATH> <DEST_BACKEND>:<DEST_PATH>

You can verify the storage class of the transferred data by accessing your bucket on the Scaleway console.

Offsite backup

The sync command from rclone should be preferred to copy because it deletes files from the destination.

First, create the rclone.conf:

[other-provider]
type = s3
provider = <PROVIDER_NAME>
env_auth = false
access_key_id = xxxx
secret_access_key = xxxx
endpoint = <OTHER_S3_PROIVER_ENDPOINT>
region = <S3_REGION>
location_constraint = <S3_REGION>

[scw-fr-par]
type = s3
provider = Scaleway
env_auth = false
access_key_id = xxxx
secret_access_key = xxxx
endpoint = https://s3.fr-par.scw.cloud
region = fr-par

[scw-nl-ams]
type = s3
provider = Scaleway
env_auth = false
access_key_id = xxxx
secret_access_key = xxxx
endpoint = https://s3.nl-ams.scw.cloud
region = nl-ams

Then sync buckets from your other Object Storage provider -> SCW and SCW -> SCW:

# sync from the other Object Storage provider to Scaleway (nl-ams)
rclone --config rclone.conf sync other-provider://my-bucket/my-key-prefix scw-nl-ams://my-bucket/my-key-prefix

# sync from Scaleway (fr-par) to Scaleway (nl-ams)
rclone --config rclone.conf sync scw-fr-par://my-bucket/my-key-prefix scw-nl-ams://my-bucket/my-key-prefix