Scaleway Documentationtutorials
encrypt s3 data rclone

Jump toUpdate content

How to encrypt your Object Storage data using rclone

Reviewed on 10 May 2021Published on 10 June 2020
  • encyption
  • s3
  • object
  • storage
  • rclone
  • objectstorage

Rclone crypt - Overview

Rclone is an open-source command-line tool to manage files on cloud storages. It is written in the Go programming language and supports a wide range of protocols. The tool provides virtual backends to wrap local and cloud file systems to apply encryption, caching, chunking and joining.

Rclone is available for Windows, macOS X and various Linux distributions.

In this tutorial we will have a look on the rlcone crypt module to encrypt your data before sending it to Scaleway Object Storage via the S3 protocol.

Requirements:
  • You have an account and are logged into the Scaleway Console
  • You have a Scaleway Object Storage Bucket

Installing rclone

You can install rclone on your local computer using the pre-built binary files provided. Follow the steps for your computers’ operating system:

Windows

On Windows, you can download the latest version of rclone from their website. Unpack the ZIP file to launch the application.

macOS

On MacOS you can install rclone using the brew packet manager:

brew install rclone

Linux:

  1. Start by downloading the current version of rclone from their website and unpack the zip file:

    weget https://downloads.rclone.org/rclone-current-linux-amd64.zipunzip rclone-current-linux-amd64.zipcd rclone-*-linux-amd64
  2. Copy binary file in the /usr/bin directory:

    sudo cp rclone /usr/bin/sudo chown root:root /usr/bin/rclonesudo chmod 755 /usr/bin/rclone
  3. Install the rclone manpage for additional software documentation:

    sudo mkdir -p /usr/local/share/man/man1sudo cp rclone.1 /usr/local/share/man/man1/sudo mandb

Configuring a S3 remote endpoint

Important:

You need to have your API key ready for the rclone configuration.

Before encrypting your data, create a new remote S3 endpoint in rclone using the rclone config command:

No remotes found - make a new onen) New remotes) Set configuration passwordq) Quit confige/n/d/r/c/s/q> nname> scalewayType of storage to configure.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value[...] 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)   \ "s3"[...]Storage> s3** See help for s3 backend at: https://rclone.org/s3/ **
Choose your S3 provider.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value[...]10 / Any other S3 compatible provider   \ "Other"provider> otherGet AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).Only applies if access_key_id and secret_access_key is blank.Enter a boolean value (true or false). Press Enter for the default ("false").Choose a number from below, or type in your own value 1 / Enter AWS credentials in the next step   \ "false" 2 / Get AWS credentials from the environment (env vars or IAM)   \ "true"env_auth> falseAWS Access Key ID.Leave blank for anonymous access or runtime credentials.Enter a string value. Press Enter for the default ("").access_key_id> <ACCESS_KEY>   AWS Secret Access Key (password)Leave blank for anonymous access or runtime credentials.Enter a string value. Press Enter for the default ("").secret_access_key> <SECRET_KEY>Region to connect to.Leave blank if you are using an S3 clone and you don't have a region.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value 1 / Use this if unsure. Will use v4 signatures and an empty region.   \ "" 2 / Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH.   \ "other-v2-signature"region> fr-parEndpoint for S3 API.Required when using an S3 clone.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own valueendpoint> https://s3.fr-par.scw.cloudLocation constraint - must be set to match the Region.Leave blank if not sure. Used when creating buckets only.Enter a string value. Press Enter for the default ("").location_constraint> fr-parCanned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server side copying objects as S3doesn't copy the ACL from the source but rather writes a fresh one.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value 1 / Owner gets FULL_CONTROL. No one else has access rights (default).   \ "private"[...]acl> 1Edit advanced config? (y/n)y) Yesn) Noy/n> nRemote config--------------------[scaleway]type = s3provider = otherenv_auth = falseaccess_key_id = <ACCESS_KEY>secret_access_key = <SECRET_KEY>endpoint = https://s3.fr-par.scw.cloudlocation_constraint = fr-paracl = privateregion = fr-par--------------------y) Yes this is OKe) Edit this remoted) Delete this remotey/e/d> yCurrent remotes:
Name                 Type====                 ====scaleway             s3
e) Edit existing remoten) New remoted) Delete remoter) Rename remotec) Copy remotes) Set configuration passwordq) Quit confige/n/d/r/c/s/q> q

Configuring rclone crypt

rclone crypt will use the previously configured endpoint to store the encrypted files. Configure it by running rclone config again.

In the config below we define the Object Storage bucket at the remote prompt. In our example, we use our S3 endpoint scaleway with the bucket myobjectstoragebucket.

Edit these values towards your configuration. A long passphrase is recommended for security reasons, or you can use a random one.

$ rclone configCurrent remotes:
Name                 Type====                 ====scaleway             s3

e) Edit existing remoten) New remoted) Delete remoter) Rename remotec) Copy remotes) Set configuration passwordq) Quit confige/n/d/r/c/s/q> nname> secretType of storage to configure.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value[...]10 / Encrypt/Decrypt a remote   \ "crypt"[...]Storage> crypt** See help for crypt backend at: https://rclone.org/crypt/ **
Remote to encrypt/decrypt.Normally should contain a ':' and a path, eg "myremote:path/to/dir","myremote:bucket" or maybe "myremote:" (not recommended).Enter a string value. Press Enter for the default ("").remote> scaleway:myobjectstoragebucketHow to encrypt the filenames.Enter a string value. Press Enter for the default ("standard").Choose a number from below, or type in your own value 1 / Encrypt the filenames see the docs for the details.   \ "standard" 2 / Very simple filename obfuscation.   \ "obfuscate" 3 / Don't encrypt the file names.  Adds a ".bin" extension only.   \ "off"filename_encryption> standardOption to either encrypt directory names or leave them intact.Enter a boolean value (true or false). Press Enter for the default ("true").Choose a number from below, or type in your own value 1 / Encrypt directory names.   \ "true" 2 / Don't encrypt directory names, leave them intact.   \ "false"directory_name_encryption> truePassword or pass phrase for encryption.y) Yes type in my own passwordg) Generate random passwordy/g> gPassword strength in bits.64 is just about memorable128 is secure1024 is the maximumBits> 1024Your password is: <YOUR_PASSWORD>Use this password? Please note that an obscured version of thispassword (and not the password itself) will be stored under yourconfiguration file, so keep this generated password in a safe place.y) Yes (default)n) Noy/n> yPassword or pass phrase for salt. Optional but recommended.Should be different to the previous password.y) Yes type in my own passwordg) Generate random passwordn) No leave this optional password blank (default)y/g/n> nRemote config--------------------[secret]type = cryptremote = c14-coldstorage:c14-coldstoragefilename_encryption = standarddirectory_name_encryption = truepassword = *** ENCRYPTED ***--------------------y) Yes this is OK (default)e) Edit this remoted) Delete this remotey/e/d> yCurrent remotes:
Name                 Type====                 ====scaleway             s3secret               crypt

e) Edit existing remoten) New remoted) Delete remoter) Rename remotec) Copy remotes) Set configuration passwordq) Quit confige/n/d/r/c/s/q> q

Sending encrypted data

You can send encrypted data to your Object Storage bucket using the secret endpoint which acts like a proxy, encrypting your data, before uploading it into your bucket.

For example, if you want to upload your personal photo album and want to protect your privacy, you can encrypt the files in the directory by running the following command:

$ rclone copy --progress --s3-chunk-size=20M /home/myuser/MyPhotoalbum secret:/encrypted/MyPhotoalbum

It will transfer the directory /home/myuser/MyPhotoalbum to the endpoint secret and upload the data in the sub-directory /encrypted/MyPhotoalbum. The flags --progress display the status of the file transfer and --s3-chunk-size=20 sets the maximum size for each part of a multipart upload to 20MB.

You can check if the data has been encrypted when running a ls command on the scaleway endpoint. It will return a list like the following:

$ rclone -q ls scaleway:myobjectstoragebucket      552 3o60qe8adn5et0u45v2tidlrps/1qam1krvtvh27lapdj6s1uqbs8     2725 3o60qe8adn5et0u45v2tidlrps/1sourbh9g9fhed0qno6uc448sdgp4okr89hvba7tsbb72oekgc70     8244 3o60qe8adn5et0u45v2tidlrps/47bofkrpan3ppnmapi5j724jeg    46202 3o60qe8adn5et0u45v2tidlrps/5su2ovf4mpcpgis5sll65iloasrshulp7drbpjrlpfobhh8k8qvg    34607 3o60qe8adn5et0u45v2tidlrps/dredhm7otgba2s25nbdq2chqtpbvcqq2oum1b404mpim24dfg5ig

If you run the same command on the secret endpoint, rclone crypt will decrypt the files and return their actual names:

$ rclone -q ls secret:      504 Pictures/Picture1.png     2677 Pictures/Picture2.png     8196 Pictures/Picture3.png    46154 Pictures/Picture4.png    34559 Pictures/Picture5.png

Note that rclone crypt keeps the directory structure, but encrypts both the file and folder names.

Restoring encrypted data

Decryption of data stored using rclone copy works the same way as the encryption process:

rclone copy --progress secret:/encrypted/MyPhotoalbum /home/myuser/restored_data/MyPhotoalbum

You are now able to encrypt and decrypt your data on Scaleway Object Storage using rclone crpyt. For more information about the crypt endpoint, refer to the official rclone documentation.