Installing and setting up Rclone

Rclone is a command-line tool that can be used to manage your cloud storage. It communicates with any S3 compatible cloud storage provider as well as other storage platforms.

Follow the instructions given in the official rclone documentation here to install rclone.

For example, on Linux:

• Connect to your server as root via SSH.

• Then, download and install Rclone with the following sequence of commands:

  wget https://downloads.rclone.org/rclone-current-linux-amd64.zip
  apt install zip
  unzip rclone-current-linux-amd64.zip
  cd rclone*linux-amd64/
  mv rclone /usr/bin/
  Copy code

Configuring Rclone

1. Run the following command to begin the rclone configuration:

   $ rclone config
   Copy code

   If you do not have any existing remotes, the following output displays:

   2021/01/18 16:03:28 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
   No remotes found - make a new one
   n) New remote
   s) Set configuration password
   q) Quit config
   Copy code

   If you have previously configured rclone you may see a slightly different output. However, that does not affect the following steps.

2. Type n to make a new remote. You are then prompted to type a name - here we type remote-sw-paris:

   n/s/q> n
   name> remote-sw-paris
   Copy code

   The following output displays:

   Type of storage to configure.
   Enter a string value. Press Enter for the default ("").
   Choose a number from below, or type in your own value
   1 / 1Fichier
     \ "fichier"
   2 / Alias for an existing remote
     \ "alias"
   3 / Amazon Drive
     \ "amazon cloud drive"
   4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc)
     \ "s3"
   
   [...]
   Storage>
   Copy code

3. Type s3 and hit enter to confirm this storage type. The following output displays:

   Choose your S3 provider.
   Enter a string value. Press Enter for the default ("").
   Choose a number from below, or type in your own value
   1 / Amazon Web Services (AWS) S3
     \ "AWS"
   2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
     \ "Alibaba"
   3 / Ceph Object Storage
     \ "Ceph"
   4 / Digital Ocean Spaces
     \ "DigitalOcean"
   5 / Dreamhost DreamObjects
     \ "Dreamhost"
   6 / IBM COS S3
     \ "IBMCOS"
   7 / Minio Object Storage
     \ "Minio"
   8 / Netease Object Storage (NOS)
     \ "Netease"
   9 / Scaleway Object Storage
     \ "Scaleway"
   10 / StackPath Object Storage
     \ "StackPath"
   11 / Tencent Cloud Object Storage (COS)
     \ "TencentCOS"
   12 / Wasabi Object Storage
     \ "Wasabi"
   13 / Any other S3 compatible provider
     \ "Other"
   Copy code

4. Type Scaleway and hit enter to confirm this S3 provider. The following output displays:

   Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
   Only applies if access_key_id and secret_access_key is blank.
   Enter a boolean value (true or false). Press Enter for the default ("false").
   Choose a number from below, or type in your own value
   1 / Enter AWS credentials in the next step
     \ "false"
   2 / Get AWS credentials from the environment (env vars or IAM)
     \ "true"
   env_auth>
   Copy code

5. Type false and hit enter, to be able to enter your credentials in the next step. The following output displays:

   AWS Access Key ID.
   Leave blank for anonymous access or runtime credentials.
   Enter a string value. Press Enter for the default ("").
   access_key_id>
   Copy code

6. Enter your API Access Key and hit enter. The following output displays:

   AWS Secret Access Key (password)
   Leave blank for anonymous access or runtime credentials.
   Enter a string value. Press Enter for the default ("").
   secret_access_key>
   Copy code

7. Enter your API Secret Key and hit enter. The following output displays:

   Region to connect to.
   Enter a string value. Press Enter for the default ("").
   Choose a number from below, or type in your own value
   1 / Amsterdam, The Netherlands
     \ "nl-ams"
   2 / Paris, France
     \ "fr-par"
   region>
   Copy code

8. Enter your chosen region and hit enter. Here we choose fr-par. The following output displays:

   Endpoint for Scaleway Object Storage.
   Enter a string value. Press Enter for the default ("").
   Choose a number from below, or type in your own value
   1 / Amsterdam Endpoint
     \ "s3.nl-ams.scw.cloud"
   2 / Paris Endpoint
     \ "s3.fr-par.scw.cloud"
   endpoint>
   Copy code

9. Enter your chosen endpoint and hit enter. Here we choose s3.fr-par.scw.cloud. The following output displays:

   Canned ACL used when creating buckets and storing or copying objects.
   
   This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
   
   For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
   
   Note that this ACL is applied when server side copying objects as S3
   doesn't copy the ACL from the source but rather writes a fresh one.
   Enter a string value. Press Enter for the default ("").
   Choose a number from below, or type in your own value
   1 / Owner gets FULL_CONTROL. No one else has access rights (default).
     \ "private"
   2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
     \ "public-read"
     / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
   3 | Granting this on a bucket is generally not recommended.
     \ "public-read-write"
   4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access.
     \ "authenticated-read"
     / Object owner gets FULL_CONTROL. Bucket owner gets READ access.
   5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
     \ "bucket-owner-read"
     / Both the object owner and the bucket owner get FULL_CONTROL over the object.
   6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
     \ "bucket-owner-full-control"
   acl>
   Copy code

10. Enter your chosen ACL and hit enter. Here we choose private (1). The following output displays:

    The storage class to use when storing new objects in S3.
    Enter a string value. Press Enter for the default ("").
    Choose a number from below, or type in your own value
    1 / Default
      \ ""
    2 / The Standard class for any upload; suitable for on-demand content like streaming or CDN.
      \ "STANDARD"
    3 / Archived storage; prices are lower, but it needs to be restored first to be accessed.
      \ "GLACIER"
    storage_class>
    4 / One Zone Infrequent Access storage class
      \ "ONEZONE_IA"
    Copy code

11. Enter your chosen storage class and hit enter. Here we choose STANDARD (2). The following output displays:

    Edit advanced config? (y/n)
    y) Yes
    n) No (default)
    y/n>
    Copy code

12. Type n and hit enter. A summary of your config displays:

    Remote config
    --------------------
    [remote-sw-paris]
    type = s3
    provider = Scaleway
    env_auth = false
    access_key_id = <ACCESS-KEY>
    secret_access_key = <SECRET-KEY>
    region = fr-par
    endpoint = s3.fr-par.scw.cloud
    acl = private
    storage_class = STANDARD
    --------------------
    y) Yes this is OK (default)
    e) Edit this remote
    d) Delete this remote
    y/e/d>
    Copy code

13. Type y to confirm that this remote config is OK, and hit enter.

    The following output displays:

    Current remotes:
    
    Name                 Type
    ====                 ====
    tuto                 s3
    
    e) Edit existing remote
    n) New remote
    d) Delete remote
    r) Rename remote
    c) Copy remote
    s) Set configuration password
    q) Quit config
    e/n/d/r/c/s/q> q
    Copy code

14. Type q to quit the config, and hit enter.

    Note:

    If you want to be able to transfer data to or from a bucket in a different region to the one you just set up, repeat steps 1-14 again to set up a new remote in the required region. Simply enter the required region at steps 7 and 8. Similarly, you may wish to set up a new remote for a different Object Storage provider.

For further information, refer to the official RClone S3 Object Storage Documentation. Official documentation also exists for other storage backends.