This migration method is not suitable for Database Instances with extensions installed. In such cases, we recommend using the automatic migration method with our script or doing a manual dump/restore.
Upgrading your Database for PostgreSQL to the latest version
- storage
- PostgreSQL
- Database
- rdb
PostgreSQL is a powerful, open-source object-relational database system with more than 30 years of active development. As major PostgreSQL releases regularly add new features, the layout of the system tables may change. However, the internal data storage format itself rarely changes. This allows you to upgrade a PostgreSQL database system without data loss using the backup and restore functions of the Database API.
In this tutorial, you will learn how to migrate your PostgreSQL databases from version 9.6 to 13. This can be achieved in two ways:
- Manually, using the console and the API. This could be a good option if you only have one database to migrate.
- Automatically, using a bash script based on the Scaleway CLIv2. This could be a good option if you have multiple databases to migrate, and/or are concerned about errors during a manual restore.
Manual migration via the Scaleway console and API
There are three steps to completing a manual migration: creating a new PostgreSQL 13 instance, backing up your existing database using the Scaleway API, and finally restoring your backup on your new PostgreSQL 13 Instance. We walk you through these three steps below.
Before you start
To complete the actions presented below, you must have:
- A Scaleway account logged into the console
- Owner status or IAM permissions allowing you to perform actions in the intended Organization
- A valid API key
- A PostgreSQL database running on version 9.6
- Installed
curl
on your local computer
Creating a PostgreSQL 13 Instance
This tutorial works with all versions of PostgreSQL. If you prefer another version than PostgreSQL 13, select your preferred version during the Instance creation.
- Open the Scaleway console in a web browser and click Managed Databases in the Storage section of the side menu. The list of your databases displays.
- Click + Create a Database Instance. The Database Instance creation wizard displays.
- Select PostgreSQL-13 from the drop-down list and fill in the required information for your new database Instance. Then click Create an Instance. Your database is created, and you are redirected to the database dashboard.
Note
Ensure your new Database Instance is located in the same geographical region as your old one.
- Click the Users tab and replicate the users of your existing Database Instance.
Backing up your existing database using the Scaleway API
Before you start
To complete the actions presented below, you must have:
- Generated your API key
- Installed
curl
on your local computer
-
Retrieve the database ID of your old Database Instance. You can find it on the Database Instance Information page of your Instance:
-
Make a backup of your logical database(s) using the API:
curl -X POST -H "Content-Type: application/json" \-H "X-Auth-Token: $SECRET_KEY" https://api.scaleway.com/rdb/v1/<region>/fr-par/backups -d '{"instance_id": "<database_instance_id>","database_name": "<my_database>","name": "<backup_name>","expires_at": "<expiry_date>"}' -
Replace the following information in the command above:
$SECRET_KEY
: Your Scaleway API secret key<region>
: The region of your Database Instance. It can either befr-par
,nl-ams
, orpl-waw
.<database_instance_id>
: The ID of your old Database Instance, for example:ad085d32-16e0-4ce6-862c-8e70c56b9ee7
.<my_database>
: The name of your PostgreSQL database, for example,customer_data
.<backup_name>
: A common name for your database backup, for example,customer_data_backup
.<expiry_date>
: The expiration date for your backup in ISO8601 format, for example:2021-06-26T13:00:00Z
.
You will receive a JSON formatted output with information about the backup:
{"created_at": "2021-06-24T12:07:17.368668Z","database_name": "customer_data","download_url": null,"download_url_expires_at": null,"expires_at": "2021-06-26T13:00:00Z","id": "7feaab67-0f22-4170-96cc-6fbb11c57256","instance_id": "ad085d32-16e0-4ce6-862c-8e70c56b9ee7","instance_name": "rdb-nostalgic-dubinsky","name": "customer_data_backup","region": "fr-par","size": null,"status": "creating","updated_at": null}NoteIf you want to migrate several logical databases, repeat the command above for each of them.
Restoring your backup on your new Database Instance
Ensure that you have replicated your database users and permissions on your new Instance before continuing with the following steps.
-
Retrieve the database ID of your new Database Instance. You can find it on the Database Instance Information page of your Instance.
-
Restore the backup of your logical database(s) using the API:
curl -X POST -H "Content-Type: application/json" \-H "X-Auth-Token: $SECRET_KEY" https://api.scaleway.com/rdb/v1/regions/<region>/backups/<backup_id>/restore -d '{"database_name": "<my_database>","instance_id": "<new_database_instance_id>"}'Replace the following information in the command above:
$SECRET_KEY
: Your Scaleway API secret key<region>
: The region of your Database Instance. It can either befr-par
,nl-ams
, orpl-waw
.<backup_id>
: The ID of your database backup, for example:7feaab67-0f22-4170-96cc-6fbb11c57256
.<my_database>
: The name of your PostgreSQL database, for example:customer_data
.<database_instance_id>
: The ID of your new Database Instance, for example:d401ff10-350d-4707-9571-c861677f0031
.
You will receive a JSON formatted output with information about the backup. The status should be
restoring
:{"created_at": "2021-06-24T12:07:17.368668Z","database_name": "dvdrental","download_url": "null","expires_at": "2021-06-26T13:00:00Z","id": "7feaab67-0f22-4170-96cc-6fbb11c57256","instance_id": "ad085d32-16e0-4ce6-862c-8e70c56b9ee7","instance_name": "rdb-nostalgic-dubinsky","name": "customer_data_backup","region": "fr-par","size": 661389,"status": "restoring","updated_at": "2021-06-24T12:07:21.775783Z"}If you want to restore several logical databases, repeat the command above for each of them.
NoteDepending on the size of your backup, restoring might take a moment.
-
The database appears in the Managed Databases tab of your new Database Instance, once the backup has been imported:
TipOptionally, you can verify the status of your database with a tool like pgAdmin 4.
-
Update your applications with the endpoint (IP address and port) of the new Database Instance.
-
Once the migration is complete, delete the old Database Instance from your account.
Automatic migration via a bash script using the Scaleway CLI v2
This solution enables you to migrate your database(s) automatically, with a bash script using the Scaleway CLI v2. The script will walk you through the migration process.
Before you start
To complete the actions presented below, you must have:
- A PostgreSQL database running on version 9.6
- Installed the Scaleway CLI v2 on your machine and the relevant PostgreSQL binaries on your machine
-
Copy and paste the following script into a text file, and save it as
db_mig_script.sh
.#!/bin/bash -lcolred='\033[0;31m' # Redcolgrn='\033[0;32m' # Greencolylw='\033[0;33m' # Yellowcolblu='\033[0;34m' # Bluecolrst='\033[0m' # Text Resetverbosity=5 # default to show infosilent_lvl=0crt_lvl=1err_lvl=2wrn_lvl=3inf_lvl=4dbg_lvl=5log_notify() { log $silent_lvl "NOTE: $1"; } # Always printslog_critical() { log $crt_lvl "${colred}CRITICAL:${colrst} --- $1"; }log_error() { log $err_lvl "${colred}ERROR${colrst} --- $1"; }log_warn() { log $wrn_lvl "${colylw}WARNING${colrst} - $1"; }log_info() { log $inf_lvl "${colgrn}INFO${colrst} ---- $1"; }log_debug() { log $dbg_lvl "${colblu}DEBUG${colrst} --- $1"; }log() {if [ $verbosity -ge "$1" ]; thendatestring=$(date +'%Y-%m-%d %H:%M:%S')echo -e "$datestring - $2"fi}# ----------------------------------------------------------------------------------------------------------------------------------------------# This script is designed to migrate a Scaleway RDB instance to a new PostgreSQL major version# It requires Scaleway CLI v2 as well as PostgreSQL binariesMIGRATION_USER_PASSWORD=$(openssl rand -base64 14)MIGRATION_USER="rdb_migration"REGION="fr-par"CLI_COMMAND="scw rdb"# Wrapper to call rdb cli with json output including errorrun_rdb_cli() {$CLI_COMMAND "$@" region=$REGION -o json 2>&1}# Get the instance major versioninstance_major_verion () {run_rdb_cli instance get "$1" | jq -r .engine | awk -F'-' '{print $2}' | awk -F'.' '{print $1}'}# Create a new migration usercreate_migration_user () {run_rdb_cli user create name="$MIGRATION_USER" password="$MIGRATION_USER_PASSWORD" instance-id="$1"}# Build and return PostgreSQL constringget_conn_string () {endpoint=$(run_rdb_cli instance get "$1" | jq -r '.endpoint')port=$(echo "$endpoint" | jq .port)ip=$(echo "$endpoint" | jq -r .ip)echo "-h $ip --port $port -U $MIGRATION_USER"}# Helper to validate or exit on user inputvalidate_action () {while true; doread -rp "Do you want to continue? [y/n] " yncase $yn in[Yy]* ) break;;[Nn]* ) exit;;* ) echo "Please answer yes or no.";;esacdone}# Helper to get the Scaleway region to useget_region () {echo ""while true; doread -rp "Specify your Scaleway Zone. Default fr-par [fr-par/nl-ams/pl-waw] " regionecho "$region"case $region in[fr-par|nl-ams|pl-waw]* ) REGION=$region; break;;'') break;;* ) echo "Please answer in [fr-par/nl-ams/pl-waw] ";;esacdone}# Check if instance_id exists and is reacheable with current cli configurationcheck_instance () {if [ "$1" = "" ]; thenlog_error "You must specify an instance_id"exit 1;fires=$(run_rdb_cli instance get "$1")if [ "$( jq 'has("message")' <<< "$res" )" == "true" ]; thenerr=$(jq -r .message <<< "$res")log_error "Error while getting database $1: $err"exit 1;fi}echo -e "Welcome to Scaleway RDB PostgreSQL Migration tool\nThis tool is designed to migrate your Databases from one major PostgreSQL version to another\n"echo -e "Please make sure, you got Scaleway V2 CLI (https://github.com/scaleway/scaleway-cli) as well as PostgreSQL binaries\n"scw version > /dev/nullif [ $? -ne 0 ]; thenlog_error "The Scaleway CLI must be installed see https://github.com/scaleway/scaleway-cli"exit 1fijq --version > /dev/nullif [ $? -ne 0 ]; thenlog_error "jq tool must be installed"exit 1figet_regionlog_info "Selected $REGION region"# List RDB Instances in the selected regionecho "$CLI_COMMAND instance list region=$REGION"$CLI_COMMAND instance list region="$REGION"read -rp "Which is the source database instance? " old_instancecheck_instance "$old_instance"read -rp "Which is the destination database instance? " new_instancecheck_instance "$new_instance"# Check instance versionsold_major_version=$(instance_major_verion "$old_instance")new_major_version=$(instance_major_verion "$new_instance")if [ "$old_major_version" -ge "$new_major_version" ]; thenlog_error "New instance major version must be superior to the old one";exit 1fibinaries_version=$(pg_restore --version | awk '{print $3}' | awk -F'.' '{print $1}')if [ $? -ne 0 ]; then"You need to install the PostgreSQL binaries"exit 1fiif [ "$binaries_version" != "$new_major_version" ]; thenlog_info "PostgreSQL binaries installed $binaries_version"log_error "You need to install $new_major_version PostgreSQL binaries version";exit 1fi# Get the source logical databasesinstance_dbs=$(run_rdb_cli database list instance-id="$old_instance")managed_dbs=$(jq -c '.[] | select( .managed == true )' <<< "$instance_dbs" | jq -r .name)not_managed_dbs=$(jq -c '.[] | select( .managed == false )' <<< "$instance_dbs" | jq -r .name)if [ -n "$managed_dbs" ]; thenlog_info "This script is able to migrate following databases:"for db in $managed_dbs; doecho "- $db"done;fiif [ -n "$not_managed_dbs" ]; thenlog_warn "The following databases are not managed by RDB and won't be migrated:"for db in $not_managed_dbs; doecho "- $db"done;validate_actionfi# Create a migration user to read and write data on both Instanceslog_info "Create migration users"for instance in $old_instance $new_instance; dolog_info "Create migration user on instance $instance";res=$(create_migration_user "$instance")# Error managementif [ "$( jq 'has("hint")' <<< "$res" )" == "true" ]; thenhint=$(jq -r .hint <<< "$res")if [ ! "$hint" = "User already exists" ]; thenlog_error "Unable to create user $MIGRATION_USER on $instance instance: $hint"exit 1;elselog_warn "User $USER_MIGRATION already exist on instance $instance. Update its password?"validate_actionrun_rdb_cli user update instance-id="$instance" name=$MIGRATION_USER password="$MIGRATION_USER_PASSWORD" is-admin=true > /dev/null;fi;fi;done# Dump and restore the databaseslog_info "Migrate databases"for db in $managed_dbs; doif [ "$db" != "rdb" ]; thenlog_info "Create $db on instance $new_instance"res=$(run_rdb_cli database create instance-id="$new_instance" name="$db")# Error managementif [ "$( jq 'has("hint")' <<< "$res" )" == "true" ]; thenhint=$(jq -r .hint <<< "$res")if [ ! "$hint" = "Database already exists" ]; thenlog_error "Unable to create database $db on new instance: $hint"exit 1;elselog_warn "Database $db already exists, it won't be recreated"fi;fi;fi;log_info "Set ALL privilege on $db to $MIGRATION_USER"run_rdb_cli privilege set instance-id="$new_instance" database-name="$db" user-name=$MIGRATION_USER permission=all > /dev/null# Dump and restore the database from old_instance to new instanceold_instance_connstring=$(get_conn_string "$old_instance")new_instance_connstring=$(get_conn_string "$new_instance")log_info "Dump and restore database $db"log_debug "pg_dump $old_instance_connstring -Fc --create $db | pg_restore $new_instance_connstring --dbname $db --no-privileges --no-owner"PGPASSWORD=$MIGRATION_USER_PASSWORD pg_dump $old_instance_connstring -Fc --create "$db" | PGPASSWORD=$MIGRATION_USER_PASSWORD pg_restore $new_instance_connstring --dbname "$db" --no-privileges --no-ownerlog_info "Database $db migrated to $new_instance"done; -
Make the script executable using the
chmod
command:chmod +x db_mig_script.sh -
Run the script from your terminal by typing
./db_mig_script.sh
The script will guide you through the migration process, asking you to complete the necessary information for your migration as required.
You now have successfully migrated your managed PostgreSQL database(s) to version 13 of the database engine. For more information about all new features and improvements in this version refer to the PostgreSQL release notes and the official documentation. To learn more about Scaleway’s powerful REST-API, refer to our developers’ documentation.