backup dedicated server s3 duplicity

Jump toUpdate content

How to backup your dedicated server on Object Storage with Duplicity

Reviewed on 26 April 2022Published on 13 October 2018
  • duplicity
  • backup
  • object-storage
  • gpg
  • storage
  • s3

Duplicity - Overview

In this article, we will learn how to backup our data. We will take advantages of Scaleway’s Object Storage service and Duplicity. Duplicity is a free and open-source tool that backs up folders to a remote server. Using libsync and GPG, Duplicity is able to make space-efficient encrypted backups.

Our objectives are to:

  • Make production-ready automatic backups of our data.
  • Save cost using bandwidth and space-efficient incremental backups.
  • Preserve a robust layer of security and privacy with an asymmetrical encryption.

To achieve this, we will:

  • Create a bucket in Scaleway’s Object Storage
  • Install the free and open-source backup solution: Duplicity
  • Generate a GPG Key
  • Combine Duplicity and Scaleway

You can easily backup your existing dedicated server with Duplicity. When you are using a Scaleway Dedibox or a Scaleway Elastic Metal server, the data transfer is free of charge within the same region.


Creating an Object Storage bucket

  1. Log in to the Scaleway Console

  2. Click Storage from the left side menu. The Storage page lists all your buckets. At first, your list is empty as you have not created any bucket yet.

  3. Click Create a Bucket to create a bucket that will store your objects.

  4. Name your bucket and validate your bucket creation. A bucket name must be unique and contain only alphanumeric and lowercase characters.

Installing software requirements

We are going to install Duplicity. To make sure we can generate a GPG Key, we need to create some entropy, we suggest using Haveged constantly on your sever to generate a small amount of entropy. You can find the link to the latest version of Duplicity on their website.


You may replace the link used with wget if a newer version is available.

Run the following command to update the apt package manager, upgrade the software installed already installed on the server and to download and install Duplicity:

    apt update && apt upgrade
apt install -y python3-boto python3-pip haveged gettext librsync-dev
wget https://code.launchpad.net/duplicity/0.8-series/0.8.12/+download/duplicity-
tar xaf duplicity-0.8.*.tar.gz
cd duplicity-0.8.*/
pip3 install -r requirements.txt
python3 setup.py install

Creating a GPG key

  1. To generate the GPG key, launch this command.

        gpg --full-generate-key

    Enter and remember a passphrase. You will be asked to define the characteristics of your keys. We will go with default settings:

  • What kind of key you want: (1) RSA and RSA (default)

  • What keysize do you want: (3072)

  • How long the key should be valid: 0 = key does not expire

    GPG will then ask how to call your key, an address and a description.

        $ gpg --full-generate-key

    gpg (GnuPG) 2.1.18; Copyright (C) 2017 Free Software Foundation, Inc.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.

    Please select what kind of key you want:
    (1) RSA and RSA (default)
    (2) DSA and Elgamal
    (3) DSA (sign only)
    (4) RSA (sign only)
    Your selection? 1

    RSA keys may be between 1024 and 4096 bits long.
    What keysize do you want? (3072) 3072
    Requested keysize is 3072 bits

    Please specify how long the key should be valid.
    0 = key does not expire
    <n> = key expires in n days
    <n>w = key expires in n weeks
    <n>m = key expires in n months
    <n>y = key expires in n years
    Key is valid for? (0) 0
    Key does not expire at all
    Is this correct? (y/N) y

    GnuPG needs to construct a user ID to identify your key.

    Real name: backups
    Email address: me@scaleway.com
    Comment: Scaleway Object Storage backups
    You selected this USER-ID:
    backups (Scaleway Object Storage backups) <me@scaleway.com>

    Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
    We need to generate a lot of random bytes. It is a good idea to perform
    some other action (type on the keyboard, move the mouse, utilize the
    disks) during the prime generation; this gives the random number
    generator a better chance to gain enough entropy.

    gpg: key XXXXXXXXXXXXXXXX marked as ultimately trusted
    public and secret key created and signed.
    pub rsa3072 2020-03-26 [SC]
    uid backups (Scaleway Object Storage backups) <me@scaleway.com>
    sub rsa3072 2020-03-26 [E]
  1. Use the GPG key fingerprint, which could be an 8, 16 or 40 char long hash. You can also find the fingerprint of your key with the following command:

        $ gpg --list-keys
    gpg: checking the trustdb
    gpg: marginals needed: 3 completes needed: 1 trust model: pgp
    gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u

    pub rsa3072 2020-03-26 [SC]
    uid [ultimate] backups (Scaleway Object Storage backups) <me@scaleway.com>
    sub rsa3072 2020-03-26 [E]

If you lose access to your current server, having the GPG private and public keys stored somewhere else will come in handy. Export the GPG keys with:

  gpg --armor --export backups
gpg --armor --export-secret-key backups

Combining Duplicity and Scaleway

As everything is installed and ready, we will now configure and script our interactions between our server and the cloud.

Editing the configuration

  1. Create our initial scripts files, and our log files.

        touch scw-backups.sh scw-restore.sh .scw-configrc
    chmod 700 scw-backups.sh scw-restore.sh
    chmod 600 .scw-configrc

    mkdir -p /var/log/duplicity
    touch /var/log/duplicity/logfile{.log,-recent.log}
  2. Add the following lines to .scw-configrc :

        # Scaleway credentials keys
    export SCW_BUCKET="s3://s3.fr-par.scw.cloud/<NAME OF YOUR BUCKET>"

    # GPG Key information

    # Folder to backup

    # Will keep backup up to 1 month
    export KEEP_BACKUP_TIME="1M"

    # Will make a full backup every 10 days
    export FULL_BACKUP_TIME="10D"

    # Log files
    export LOGFILE_RECENT="/var/log/duplicity/logfile-recent.log"
    export LOGFILE="/var/log/duplicity/logfile.log"

    log () {
    date=`date +%Y-%m-%d`
    hour=`date +%H:%M:%S`
    echo "$date $hour $*" >> ${LOGFILE_RECENT}
    export -f log

Our current backup policy is to make a full backup every 10 days and remove all backups older than one month.

Backup script

Using the configuration and duplicity, we automatize the backups.

  1. Copy the following script to scw-backups.sh:

    source <FULL PATH TO>/.scw-configrc

    currently_backuping=$(ps -ef | grep duplicity | grep python | wc -l)

    if [ $currently_backuping -eq 0 ]; then
    # Clear the recent log file
    cat /dev/null > ${LOGFILE_RECENT}

    log ">>> removing old backups"
    duplicity remove-older-than ${KEEP_BACKUP_TIME} ${SCW_BUCKET} >> ${LOGFILE_RECENT} 2>&1

    log ">>> creating and uploading backup to Scaleway Glacier"
    duplicity \
    incr --full-if-older-than ${FULL_BACKUP_TIME} \
    --asynchronous-upload \
    --s3-use-glacier \
    --encrypt-key=${GPG_FINGERPRINT} \
    --sign-key=${GPG_FINGERPRINT} \


    The option --s3-use-glacier will move backups into Scaleway Glacier. If you prefer to store your backups on regular Object Storage, remove the line --s3-use-glacier \ from the script.

  2. Run the script ./scw-backups.sh to make sure the configuration is correctly set. Check Scaleway’s bucket on the web interface and the logs with the following command:

    cat /var/log/duplicity/logfile-recent.log

Script the recovery of data

Duplicity also allows you to recover a backup. We will create a script to make the process easier.

  1. In scw-restore.sh, add the following:

    source <FULL PATH TO>/.scw-configrc

    if [ $# -lt 2 ]; then
    echo -e "Usage $0 <time or delta> [file to restore] <restore to>
    \t$ $0 2018-7-21 recovery/ ## recovers * from closest backup to date
    \t$ $0 0D secret data/ ## recovers most recent file nammed 'secret'";
    exit; fi

    if [ $# -eq 2 ]; then
    duplicity \
    --time $1 \
    ${SCW_BUCKET} $2

    if [ $# -eq 3 ]; then
    duplicity \
    --time $1 \
    --file-to-restore $2 \
    ${SCW_BUCKET} $3
  2. Recover the data you uploaded in the previous section with the following command:

        ./scw-restore.sh 0D /tmp/backup-recovery-test/
  3. You can also recover one specific file with the following format from a backup 5 days ago with:

        ./scw-restore.sh 5D <file> /tmp/backup-recovery-test/

Automation of the backups

  1. Use the command crontab -e to edit your crontab file and add the line to create a script that will run twice a day at 1:00 and 13:00 (1 AM and 1 PM) :

        crontab -e
  2. And after selection of your editor, write down:

    00 1,13 * * * <FULL PATH TO>/scw-backups.sh > /dev/null 2>&1


We manage to create an automatic backup using Duplicity, hosting encrypted data on Scaleway’s Object Storage. We are able to recover specific files or the entire backup itself at a given date.

To continue your implementation, you may want to consider the following:

  • Duply, giving you a front-end view of the backups.
  • Backuping more than one folder using “SOURCE=/” and the --include= and --exclude= options of duplicity.
  • Adapt the backup policy with more frequent backups, that are conserved