Jump toUpdate content
How to build your own Ceph distributed storage cluster on dedicated servers
Ceph is an open-source, software-defined storage solution designed to address object, block, and file storage needs. Ceph can handle up to several exabytes of data. It replicates data and makes it fault-tolerant, using standard hardware without expensive specific storage hardware. The system is designed to minimize administration time and costs. Therefore it is both self-healing and self-managing.
- You have an account and are logged into the Dedibox console
- You have three Dedibox servers running on Ubuntu Focal Fossa 20.04 LTS or later
- You have an additional admin machine available to install
ceph-deploy is a tool that allows you to deploy a Ceph cluster using a simple to use command-line interface and it will be installed on an independent admin machine.
Type the following command to connect to the admin machine using SSH:
Type the following command to add the Ceph release key to apt:
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
Type the following command to add the Ceph repository to the apt package manager:
echo deb https://eu.ceph.com/debian-octopus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
Type the following command to update the apt package manager to include Ceph’s repository:
sudo apt update
Type the following command to install
sudo apt install ceph-deploy
ceph-deploy needs access to a user that has passwordless sudo privileges to be able to install software and configuration files on the storage nodes.
Ceph deploy will be able to connect as any user, including the
root user, however this is not recommended. To simplify the software configuration, it is recommended to use the same username on all Ceph nodes in the cluster.
Do not use
ceph as login for your user, as it is reserved for Ceph-daemons. The installation of Ceph will fail if a
ceph user exists on a machine.
Type the following command to connect to your Ceph node using SSH:
Type the following command to create a user called
sudo useradd -d /home/ceph-deploy -m ceph-deployNote:
You can rename the user to your own preferences if required.
Type the following command to configure the password of the
sudo passwd ceph-deploy
Type the following command to add the user to the sudoers configuration so they have sudo rights:
echo "ceph-deploy ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph-deploy
sudo chmod 0440 /etc/sudoers.d/ceph-deploy
Type the following command to install an NTP client on your Ceph nodes:
sudo apt install ntpsecNote:
The installation of an NTP client allows you to avoid synchronization issues caused by time-drift.
Install Python to deploy the cluster:
sudo apt install python-minimal
Repeat all the previous steps 3 times to set up each of your 3 nodes.
In the following section, you will generate an SSH key and distribute the public key to each Ceph node to enable passwordless authentication on the nodes.
- Make sure not to use
- Make sure you are not logged as the
rootuser when generating the SSH keys.
Type the following command to generate a public or a private key-pair using the
ssh-keygenapplication on the admin node:
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ceph-admin/.ssh/id_rsa): <-- Press Enter to save the key in the default location
Created directory '/home/ceph-admin/.ssh'.
Enter passphrase (empty for no passphrase): <-- Leave the passphrase empty. Press Enter to confirm.
Enter same passphrase again:
Your identification has been saved in /home/ceph-admin/.ssh/id_rsa.
Your public key has been saved in /home/ceph-admin/.ssh/id_rsa.pub.
Type the following command to make sure that the hostnames of the ceph nodes are configured in your
sudo nano /etc/hosts
Your file should look like the following example:
Save the file and exit your text editor.
Transfer the public key of your admin node to each of your Ceph nodes:
In this section, you will deploy the Ceph cluster on your machines. To keep a clean directory structure, create a directory on your admin node for maintaining the configuration files and keys of your cluster, that
Type the following commands to create a new directory on the admin node and to enter into it:
Type the following command to create the cluster:
ceph-deploy --username ceph-deploy new ceph-node-aNote:
ceph-node-awith the FQDN of your node, for example:
If you have more than one network interface on the machine, type the following command to add the public network setting under the
[global]section of your Ceph configuration file:
public network = <ip-address>/<netmask>
public network = 220.127.116.11/24
Refer to the official documentation for more information on the network configuration.
Type the following command to install the ceph packages on the nodes:
ceph-deploy --username ceph-deploy install ceph-node-a ceph-node-b ceph-node-c
Type the following command to deploy the initial monitor(s) and gather the keys:
ceph-deploy --username ceph-deploy mon create-initial
Once completed, you should have the following files in your directory. Verify that they are generated using the
ceph.bootstrap-mds.keyring ceph.bootstrap-rgw.keyring ceph-deploy-ceph.log
ceph.bootstrap-mgr.keyring ceph.client.admin.keyring ceph.mon.keyring
Copy the configuration file and admin key to your Ceph Nodes using
ceph-deploy --username ceph-deploy admin ceph-node-a ceph-node-b ceph-node-cNote:
This step allows you to use the ceph CLI without having to specify the monitor address and
ceph.client.admin.keyringeach time you execute a command.
Type the following command to deploy the manager daemon on all Ceph nodes:
ceph-deploy --username ceph-deploy mgr create ceph-node-a ceph-node-b ceph-node-cNote:
These daemons operate in an active/standby pattern. Deploying multiple manager daemons ensures that if one of the daemons fails for any reason, another one can take over without interrupting service.
Type the following command to configure an Object Storage Device (OSD) on each of the ceph nodes:
ceph-deploy osd create --data /dev/sdb ceph-node-a
ceph-deploy osd create --data /dev/sdb ceph-node-b
ceph-deploy osd create --data /dev/sdb ceph-node-cImportant:
Ensure that the device is not in use and that it does not contain any important data. All data present on the device will be deleted!
Type the following command to check the status of your Ceph cluster from one of the Ceph nodes: If everything is up and running, your cluster should report
sudo ceph health
If everything is up and running, your cluster should report
Type the following command to see a complete status of your cluster
sudo ceph -s
The basic installation of your cluster is now complete. It is running and provides a distributed Ceph filesystem.
RGW (Ceph Object Gateway) is the S3/Swift gateway component of Ceph. It is required to access your files using any S3 compatible client or application. Deploy the component on one of your nodes from your admin machine using
Log into your admin machine and run the following command:
ceph-deploy --username ceph-deploy rgw create ceph-node-a
The following message displays once the installation is complete:
[ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host ceph-node-a and default port 7480Note:
By default, the RGW instance listens on port 7480.
Add the following lines to the file to change this port
/etc/ceph/ceph.confon the node running the RGW as follows:
sudo nano /etc/ceph/ceph.conf
rgw frontends = civetweb port=80
#To use an IPv6 address, add the following line:
rgw frontends = civetweb port=[::]:80
If you want to secure the connection using a TLS certificate, specify the HTTPS port and the bundled certificate file. You will need to either create a self-signed certificate or get a certificate from a certificate authority (CA), for example, Let’s Encrypt:
rgw frontends = civetweb port=443s ssl_certificate=/etc/ceph/private/bundle_keyandcert.pem
Open a web browser and point it to
http://ceph-node-a:7480to verify your installation. A response like the following displays:
Log into your gateway instance (in our example
ceph-node-a) and run the following command to create a new user:
sudo radosgw-admin user create --uid=johndoe --display-name="John Doe" --firstname.lastname@example.org
--uid: The unique user identification for the user (for example:
--display-name: The display name of the user (for example:
An output like the following displays. Write down the
user_key as you need these to configure your S3 client.
"display_name": "John Doe",
"op_mask": "read, write, delete",
You can now configure your S3 compatible tool, for example, aws-cli.
You can use AWS-CLI to manage objects in your Ceph storage cluster using the standardized S3 protocol. It allows you to create buckets easily and to manage your files using an efficient command-line tool. You can find a list of available S3 features in the Ceph documentation.
Type the following command to install
awscli-pluginon your machine using
pip, the Python package manager:
pip3 install awscli
pip3 install awscli-plugin-endpoint
Create a file
~/.aws/configand put the following contents in it. Edit the Endpoint URL to your RGW instance:
endpoint = awscli_plugin_endpoint
region = default
endpoint_url = http://ceph-node-a:7480
signature_version = s3v4
max_concurrent_requests = 100
max_queue_size = 1000
multipart_threshold = 50MB
multipart_chunksize = 10MB
endpoint_url = http://ceph-node-a:7480
Create a configuration file for your S3 credentials at
~/.aws/credentialsand edit it as follows:
<SECRET_KEY>with the credentials of the user account you created in a previous step.
Type the following command to create a bucket on your cluster:
aws s3 mb s3://MyBucket
Type the following command to create a test file:
echo "Hello World!" > testfile.txt
Type the following command to upload the file to the cluster:
aws s3 cp testfile.txt s3://MyBucket
upload: ./testfile.txt to s3://MyBucket/testfile.txt
Type the following command to check if the upload of the file was successful by listing the content of your bucket:
aws s3 ls s3://MyBucket
2020-08-20 14:40:38 13 testfile.txt
You have now configured an S3 compatible storage cluster using Ceph and three Dedibox dedicated servers. You can upload your data to the cluster and modify or share it using any S3 compatible tool. For more advanced configuration options, refer to the official Ceph documentation.