migrate data minio client

Jump toUpdate content

Migrating Object Storage data with Minio client

Published on 20 March 2019
  • compute
  • server
  • Minio
  • object-storage
  • s3
  • data

Minio client provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff etc. It is able to communicate with any S3 compatible cloud storage provider and can be used to migrate data from one region to another.

Identity and Access Management (IAM):

If you have activated IAM, you may need certain IAM permissions to carry out some actions described on this page. This means:

  • you are the Owner of the Scaleway Organization in which the actions will be carried out, or
  • you are an IAM user of the Organization, with a policy granting you the necessary permission sets

Installing the Minio client

  1. Connect to your Instance as root via SSH.
  2. Update the APT package cache and the software already installed on the instance:
    apt update && apt upgrade -y
  3. Download the Minio Client:
    wget https://dl.minio.io/client/mc/release/linux-amd64/mc -P /usr/bin/
  4. Make the file executable:
    chmod +x /usr/bin/mc

Configuring the Minio Client and migrating data

  1. Configure the endpoints you wish to use

    mc config host add scw-fr-par https://s3.fr-par.scw.cloud <ACCESS_KEY> <SECRET_KEY> --api S3v4
    mc config host add scw-nl-ams https://s3.nl-ams.scw.cloud <ACCESS_KEY> <SECRET_KEY> --api S3v4

    Replace <ACCESS_KEY> and <SECRET_KEY> with the credentials of your API key.

  2. Optionally, add other providers:

    For S3 compatible storages:

    mc config host add s3 <ENDPOINT> <OTHER_ACCESS_KEY> <OTHER_SECRET_KEY> --api S3v4

    For google cloud storage :

    mc config host add gcs  https://storage.googleapis.com <ACCESS_KEY> <SECRET_KEY> --api S3v2
  3. Create the buckets you need on the scaleway Object Storage

    mc mb --region fr-par scw-fr-par/<BUCKET_NAME>
    mc mb --region nl-ams scw-nl-ams/<OTHER_BUCKET_NAME>
  4. Start the migration of your data:

    mc cp --recursive s3/<BUCKET_NAME> scw-fr-par/<BUCKET_NAME>
    mc cp --recursive gcs/<OTHER_BUCKET_NAME> scw-nl-ams/<OTHER_BUCKET_NAME>

    The commands above:

    1. migrates data from a S3 compatible Object Storage to Scaleway’s fr-par Object Storage
    2. migrates data from GCS Object Storage to Scaleway’s nl-ams Object Storage

    Please note that this migration can incur some cost from the Object Storage you are migrating from since they may or may not bill egress bandwitdth.

    The Minio Client displays a status bar during the transfer, allowing to observe the status of the migration: