You need to have your API key ready for the rclone
configuration.
Encrypting your Scaleway Object Storage data using Rclone
- encryption
- s3
- rclone
Rclone is a versatile open-source command-line tool designed to streamline file management across cloud storage platforms. Built using the Go programming language, it provides broad protocol support, enabling seamless integration with diverse storage systems.
Offering virtual backends, Rclone facilitates encryption, caching, chunking, and concatenation functionalities for both local and cloud-based file systems.
Compatible with Windows, macOS X, and various Linux distributions, Rclone addresses a wide user base seeking efficient file management solutions.
In this tutorial, we will explore the capabilities of the Rclone crypt module, which empowers users to encrypt their data seamlessly before transmitting it to Scaleway Object Storage via the S3 protocol.
Before you start
To complete the actions presented below, you must have:
- A Scaleway account logged into the console
- Owner status or IAM permissions allowing you to perform actions in the intended Organization
- An Object Storage bucket
Installing Rclone
You can install rclone
on your local computer using the pre-built binary files provided. Follow the steps for your computer’s operating system:
Windows
On Windows, you can download the latest version of Rclone from the official website.
Unpack the ZIP file to launch the application.
macOS
On macOS, you can install rclone
using the brew package manager:
brew install rclone
Linux:
- Start by downloading the current version of
rclone
from their website and unpack the zip file:wget https://downloads.rclone.org/rclone-current-linux-amd64.zipunzip rclone-current-linux-amd64.zipcd rclone-*-linux-amd64 - Copy binary file in the
/usr/bin
directory:sudo cp rclone /usr/bin/sudo chown root:root /usr/bin/rclonesudo chmod 755 /usr/bin/rclone - Install the
rclone
manpage for additional software documentation:sudo mkdir -p /usr/local/share/man/man1sudo cp rclone.1 /usr/local/share/man/man1/sudo mandb
Configuring an S3 remote endpoint
Before encrypting your data, create a new remote S3 endpoint in Rclone using the rclone config
command:
No remotes found - make a new onen) New remotes) Set configuration passwordq) Quit confige/n/d/r/c/s/q> nname> scalewayType of storage to configure.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value[...]4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)\ "s3"[...]Storage> s3** See help for s3 backend at: https://rclone.org/s3/ **Choose your S3 provider.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value[...]10 / Any other S3 compatible provider\ "Other"provider> otherGet AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).Only applies if access_key_id and secret_access_key is blank.Enter a boolean value (true or false). Press Enter for the default ("false").Choose a number from below, or type in your own value1 / Enter AWS credentials in the next step\ "false"2 / Get AWS credentials from the environment (env vars or IAM)\ "true"env_auth> falseAWS Access Key ID.Leave blank for anonymous access or runtime credentials.Enter a string value. Press Enter for the default ("").access_key_id> <ACCESS_KEY>AWS Secret Access Key (password)Leave blank for anonymous access or runtime credentials.Enter a string value. Press Enter for the default ("").secret_access_key> <SECRET_KEY>Region to connect to.Leave blank if you are using an S3 clone and you don't have a region.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value1 / Use this if unsure. Will use v4 signatures and an empty region.\ ""2 / Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH.\ "other-v2-signature"region> fr-parEndpoint for S3 API.Required when using an S3 clone.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own valueendpoint> https://s3.fr-par.scw.cloudLocation constraint - must be set to match the Region.Leave blank if not sure. Used when creating buckets only.Enter a string value. Press Enter for the default ("").location_constraint> fr-parCanned ACL used when creating buckets and storing or copying objects.This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-aclNote that this ACL is applied when server side copying objects as S3doesn't copy the ACL from the source but rather writes a fresh one.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value1 / Owner gets FULL_CONTROL. No one else has access rights (default).\ "private"[...]acl> 1Edit advanced config? (y/n)y) Yesn) Noy/n> nRemote config--------------------[scaleway]type = s3provider = otherenv_auth = falseaccess_key_id = <ACCESS_KEY>secret_access_key = <SECRET_KEY>endpoint = https://s3.fr-par.scw.cloudlocation_constraint = fr-paracl = privateregion = fr-par--------------------y) Yes this is OKe) Edit this remoted) Delete this remotey/e/d> yCurrent remotes:Name Type==== ====scaleway s3e) Edit existing remoten) New remoted) Delete remoter) Rename remotec) Copy remotes) Set configuration passwordq) Quit confige/n/d/r/c/s/q> q
Configuring Rclone crypt
rclone crypt
will use the previously configured endpoint to store the encrypted files. Configure it by running rclone config
again.
In the config below we define the Object Storage bucket at the remote
prompt. In our example, we use our S3 endpoint scaleway
with the bucket myobjectstoragebucket
.
Edit these values towards your configuration. A long passphrase is recommended for security reasons, or you can use a random one.
rclone configCurrent remotes:Name Type==== ====scaleway s3e) Edit existing remoten) New remoted) Delete remoter) Rename remotec) Copy remotes) Set configuration passwordq) Quit confige/n/d/r/c/s/q> nname> secretType of storage to configure.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value[...]10 / Encrypt/Decrypt a remote\ "crypt"[...]Storage> crypt** See help for crypt backend at: https://rclone.org/crypt/ **Remote to encrypt/decrypt.Normally should contain a ':' and a path, eg "myremote:path/to/dir","myremote:bucket" or maybe "myremote:" (not recommended).Enter a string value. Press Enter for the default ("").remote> scaleway:myobjectstoragebucketHow to encrypt the filenames.Enter a string value. Press Enter for the default ("standard").Choose a number from below, or type in your own value1 / Encrypt the filenames see the docs for the details.\ "standard"2 / Very simple filename obfuscation.\ "obfuscate"3 / Don't encrypt the file names. Adds a ".bin" extension only.\ "off"filename_encryption> standardOption to either encrypt directory names or leave them intact.Enter a boolean value (true or false). Press Enter for the default ("true").Choose a number from below, or type in your own value1 / Encrypt directory names.\ "true"2 / Do not encrypt directory names, leave them intact.\ "false"directory_name_encryption> truePassword or pass phrase for encryption.y) Yes type in my own passwordg) Generate random passwordy/g> gPassword strength in bits.64 is just about memorable128 is secure1024 is the maximumBits> 1024Your password is: <YOUR_PASSWORD>Use this password? Please note that an obscured version of thispassword (and not the password itself) will be stored under yourconfiguration file, so keep this generated password in a safe place.y) Yes (default)n) Noy/n> yPassword or pass phrase for salt. Optional but recommended.Should be different to the previous password.y) Yes type in my own passwordg) Generate random passwordn) No leave this optional password blank (default)y/g/n> nRemote config--------------------[secret]type = cryptremote = c14-coldstorage:c14-coldstoragefilename_encryption = standarddirectory_name_encryption = truepassword = *** ENCRYPTED ***--------------------y) Yes this is OK (default)e) Edit this remoted) Delete this remotey/e/d> yCurrent remotes:Name Type==== ====scaleway s3secret crypte) Edit existing remoten) New remoted) Delete remoter) Rename remotec) Copy remotes) Set configuration passwordq) Quit confige/n/d/r/c/s/q> q
Sending encrypted data
You can send encrypted data to your Object Storage bucket using the secret
endpoint which acts like a proxy, encrypting your data, before uploading it into your bucket.
For example, if you want to upload your personal photo album and want to protect your privacy, you can encrypt the files in the directory by running the following command:
rclone copy --progress --s3-chunk-size=20M /home/myuser/MyPhotoalbum secret:/encrypted/MyPhotoalbum
It will transfer the directory /home/myuser/MyPhotoalbum
to the endpoint secret
and upload the data in the sub-directory /encrypted/MyPhotoalbum
. The flags --progress
display the status of the file transfer and --s3-chunk-size=20
sets the maximum size for each part of a multipart upload to 20 MB.
You can check if the data has been encrypted when running a ls
command on the scaleway
endpoint. It will return a list like the following:
rclone -q ls scaleway:myobjectstoragebucket552 3o60qe8adn5et0u45v2tidlrps/1qam1krvtvh27lapdj6s1uqbs82725 3o60qe8adn5et0u45v2tidlrps/1sourbh9g9fhed0qno6uc448sdgp4okr89hvba7tsbb72oekgc708244 3o60qe8adn5et0u45v2tidlrps/47bofkrpan3ppnmapi5j724jeg46202 3o60qe8adn5et0u45v2tidlrps/5su2ovf4mpcpgis5sll65iloasrshulp7drbpjrlpfobhh8k8qvg34607 3o60qe8adn5et0u45v2tidlrps/dredhm7otgba2s25nbdq2chqtpbvcqq2oum1b404mpim24dfg5ig
If you run the same command on the secret
endpoint, rclone crypt
will decrypt the files and return their actual names:
rclone -q ls secret:504 Pictures/Picture1.png2677 Pictures/Picture2.png8196 Pictures/Picture3.png46154 Pictures/Picture4.png34559 Pictures/Picture5.png
Note that rclone crypt
keeps the directory structure, but encrypts both the file and folder names.
Restoring encrypted data
The decryption of data stored using rclone copy
works the same way as the encryption process:
rclone copy --progress secret:/encrypted/MyPhotoalbum /home/myuser/restored_data/MyPhotoalbum
You are now able to encrypt and decrypt your data on Scaleway Object Storage using rclone crpyt
. For more information about the crypt
endpoint, refer to the official Rclone documentation.