Migrating Scaleway Object Storage data with MinIO client

The MinIO Client provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff, etc. It can communicate with any Amazon S3-compatible cloud storage provider and can be used to migrate data from one region to another.

Before you start

To complete the actions presented below, you must have:

• A Scaleway account logged into the console
• Owner status or IAM permissions allowing you to perform actions in the intended Organization
• An SSH key
• An Instance
• Object Storage buckets in two different regions

Installing the MinIO client

1. Connect to your Instance as root via SSH.
2. Update the APT package cache and the software already installed on the Instance:
   apt update && apt upgrade -y

3. Download the MinIO client:
   • For x86 (Intel/AMD):
   curl https://dl.min.io/client/mc/release/linux-amd64/mc \
     --create-dirs \
     -o $HOME/minio-binaries/mc
   
   chmod +x $HOME/minio-binaries/mc
   export PATH=$PATH:$HOME/minio-binaries/

   • For ARM64 (Apple silicon):
   curl https://dl.min.io/client/mc/release/linux-arm64/mc \
     --create-dirs \
     -o ~/minio-binaries/mc
   
   chmod +x $HOME/minio-binaries/mc
   export PATH=$PATH:$HOME/minio-binaries/

Configuring the MinIO client and migrating data

1. Configure the endpoints you wish to use:

   mc config host add scw-fr-par https://s3.fr-par.scw.cloud <ACCESS_KEY> <SECRET_KEY> --api S3v4
   mc config host add scw-nl-ams https://s3.nl-ams.scw.cloud <ACCESS_KEY> <SECRET_KEY> --api S3v4

   Note

   Replace <ACCESS_KEY> and <SECRET_KEY> with the credentials of your API key.

2. Optionally, add other providers:

   For Amazon S3-compatible storage:

   mc config host add s3 <ENDPOINT> <OTHER_ACCESS_KEY> <OTHER_SECRET_KEY> --api S3v4

   For Google Cloud Storage:

   mc config host add gcs  https://storage.googleapis.com <ACCESS_KEY> <SECRET_KEY> --api S3v2

3. Create the buckets you need on the Scaleway Object Storage:

   mc mb --region fr-par scw-fr-par/<BUCKET_NAME>
   mc mb --region nl-ams scw-nl-ams/<OTHER_BUCKET_NAME>

4. Start the migration of your data:

   mc cp --recursive s3/<BUCKET_NAME> scw-fr-par/<BUCKET_NAME>
   mc cp --recursive gcs/<OTHER_BUCKET_NAME> scw-nl-ams/<OTHER_BUCKET_NAME>

   The commands above:

   1. Migrates data from an Amazon S3-compatible Object Storage to Scaleway's fr-par Object Storage
   2. Migrates data from GCS Object Storage to Scaleway's nl-ams Object Storage
   Important

   Note that this migration can incur some cost from the Object Storage you are migrating from since they may bill egress bandwidth.

   The MinIO client displays a status bar during the transfer, allowing to observe the status of the migration:

For more information about MinIO Client refer to the official documentation.