Encrypting your Object Storage data using rclone

Reviewed on 26 February 2024 • Published on 10 June 2020

encyption
s3
Object-Storage
rclone

Rclone is a versatile open-source command-line tool designed to streamline file management across cloud storage platforms. Built using the Go programming language, it provides broad protocol support, enabling seamless integration with diverse storage systems.

Offering virtual backends, Rclone facilitates encryption, caching, chunking, and concatenation functionalities for both local and cloud-based file systems.

Compatible with Windows, macOS X, and various Linux distributions, Rclone addresses a wide user base seeking efficient file management solutions.

In this tutorial, we will explore the capabilities of the rclone crypt module, which empowers users to encrypt their data seamlessly before transmitting it to Scaleway Object Storage via the S3 protocol.

Before you start

To complete the actions presented below, you must have:

A Scaleway account logged into the console
Owner status or IAM permissions allowing you to perform actions in the intended Organization
An Object Storage bucket

Installing Rclone

You can install rclone on your local computer using the pre-built binary files provided. Follow the steps for your computers’ operating system:

Windows

On Windows, you can download the latest version of Rclone from the official website.

Unpack the ZIP file to launch the application.

macOS

On macOS you can install rclone using the brew package manager:

brew install rclone

Linux:

Start by downloading the current version of rclone from their website and unpack the zip file:

wget https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cd rclone-*-linux-amd64

Copy binary file in the /usr/bin directory:

sudo cp rclone /usr/bin/
sudo chown root:root /usr/bin/rclone
sudo chmod 755 /usr/bin/rclone

Install the rclone manpage for additional software documentation:

sudo mkdir -p /usr/local/share/man/man1
sudo cp rclone.1 /usr/local/share/man/man1/
sudo mandb

Configuring a S3 remote endpoint

Important

You need to have your API key ready for the rclone configuration.

Before encrypting your data, create a new remote S3 endpoint in rclone using the rclone config command:

No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> scaleway
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[...]
  4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)
  \ "s3"
[...]
Storage> s3
** See help for s3 backend at: https://rclone.org/s3/ **
Choose your S3 provider.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[...]
  10 / Any other S3 compatible provider
  \ "Other"
provider> other
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
  1 / Enter AWS credentials in the next step
    \ "false"
  2 / Get AWS credentials from the environment (env vars or IAM)
    \ "true"
env_auth> false
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> <ACCESS_KEY>   
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> <SECRET_KEY>
Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
  1 / Use this if unsure. Will use v4 signatures and an empty region.
    \ ""
  2 / Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH.
    \ "other-v2-signature"
region> fr-par
Endpoint for S3 API.
Required when using an S3 clone.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
endpoint> https://s3.fr-par.scw.cloud
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a string value. Press Enter for the default ("").
location_constraint> fr-par
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
  1 / Owner gets FULL_CONTROL. No one else has access rights (default).
  \ "private"
[...]
acl> 1
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
--------------------
[scaleway]
type = s3
provider = other
env_auth = false
access_key_id = <ACCESS_KEY>
secret_access_key = <SECRET_KEY>
endpoint = https://s3.fr-par.scw.cloud
location_constraint = fr-par
acl = private
region = fr-par
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name                 Type
====                 ====
scaleway             s3
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Configuring Rclone crypt

rclone crypt will use the previously configured endpoint to store the encrypted files. Configure it by running rclone config again.

In the config below we define the Object Storage bucket at the remote prompt. In our example, we use our S3 endpoint scaleway with the bucket myobjectstoragebucket.

Edit these values towards your configuration. A long passphrase is recommended for security reasons, or you can use a random one.

rclone config
Current remotes:
Name                 Type
====                 ====
scaleway             s3
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> secret
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[...]
10 / Encrypt/Decrypt a remote
  \ "crypt"
[...]
Storage> crypt
** See help for crypt backend at: https://rclone.org/crypt/ **
Remote to encrypt/decrypt.
Normally should contain a ':' and a path, eg "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
Enter a string value. Press Enter for the default ("").
remote> scaleway:myobjectstoragebucket
How to encrypt the filenames.
Enter a string value. Press Enter for the default ("standard").
Choose a number from below, or type in your own value
  1 / Encrypt the filenames see the docs for the details.
    \ "standard"
  2 / Very simple filename obfuscation.
    \ "obfuscate"
  3 / Don't encrypt the file names.  Adds a ".bin" extension only.
    \ "off"
filename_encryption> standard
Option to either encrypt directory names or leave them intact.
Enter a boolean value (true or false). Press Enter for the default ("true").
Choose a number from below, or type in your own value
  1 / Encrypt directory names.
    \ "true"
  2 / Do not encrypt directory names, leave them intact.
    \ "false"
directory_name_encryption> true
Password or pass phrase for encryption.
y) Yes type in my own password
g) Generate random password
y/g> g
Password strength in bits.
64 is just about memorable
128 is secure
1024 is the maximum
Bits> 1024
Your password is: <YOUR_PASSWORD>
Use this password? Please note that an obscured version of this
password (and not the password itself) will be stored under your
configuration file, so keep this generated password in a safe place.
y) Yes (default)
n) No
y/n> y
Password or pass phrase for salt. Optional but recommended.
Should be different to the previous password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> n
Remote config
--------------------
[secret]
type = crypt
remote = c14-coldstorage:c14-coldstorage
filename_encryption = standard
directory_name_encryption = true
password = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name                 Type
====                 ====
scaleway             s3
secret               crypt
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Sending encrypted data

You can send encrypted data to your Object Storage bucket using the secret endpoint which acts like a proxy, encrypting your data, before uploading it into your bucket.

For example, if you want to upload your personal photo album and want to protect your privacy, you can encrypt the files in the directory by running the following command:

rclone copy --progress --s3-chunk-size=20M /home/myuser/MyPhotoalbum secret:/encrypted/MyPhotoalbum

It will transfer the directory /home/myuser/MyPhotoalbum to the endpoint secret and upload the data in the sub-directory /encrypted/MyPhotoalbum. The flags --progress display the status of the file transfer and --s3-chunk-size=20 sets the maximum size for each part of a multipart upload to 20 MB.

You can check if the data has been encrypted when running a ls command on the scaleway endpoint. It will return a list like the following:

rclone -q ls scaleway:myobjectstoragebucket
    552 3o60qe8adn5et0u45v2tidlrps/1qam1krvtvh27lapdj6s1uqbs8
    2725 3o60qe8adn5et0u45v2tidlrps/1sourbh9g9fhed0qno6uc448sdgp4okr89hvba7tsbb72oekgc70
    8244 3o60qe8adn5et0u45v2tidlrps/47bofkrpan3ppnmapi5j724jeg
    46202 3o60qe8adn5et0u45v2tidlrps/5su2ovf4mpcpgis5sll65iloasrshulp7drbpjrlpfobhh8k8qvg
    34607 3o60qe8adn5et0u45v2tidlrps/dredhm7otgba2s25nbdq2chqtpbvcqq2oum1b404mpim24dfg5ig

If you run the same command on the secret endpoint, rclone crypt will decrypt the files and return their actual names:

rclone -q ls secret:
    504 Pictures/Picture1.png
    2677 Pictures/Picture2.png
    8196 Pictures/Picture3.png
    46154 Pictures/Picture4.png
    34559 Pictures/Picture5.png

Note that rclone crypt keeps the directory structure, but encrypts both the file and folder names.

Restoring encrypted data

Decryption of data stored using rclone copy works the same way as the encryption process:

rclone copy --progress secret:/encrypted/MyPhotoalbum /home/myuser/restored_data/MyPhotoalbum

You are now able to encrypt and decrypt your data on Scaleway Object Storage using rclone crpyt. For more information about the crypt endpoint, refer to the official Rclone documentation.