Jump toUpdate content

How to build your own Ceph distributed storage cluster on dedicated servers
- dedicated-servers
- dedibox
- storage-cluster
- Ceph
Ceph - Overview
Ceph is an open-source, software-defined storage solution designed to address object, block, and file storage needs. Ceph can handle up to several exabytes of data. It replicates data and makes it fault-tolerant, using standard hardware without expensive specific storage hardware. The system is designed to minimize administration time and costs. Therefore it is both self-healing and self-managing.
In this tutorial, you will learn how to deploy a three-node Ceph cluster using Dedibox dedicated servers running on Ubuntu Focal Fossa (20.04 LTS).
- You have an account and are logged into the Dedibox console
- You have three Dedibox servers running on Ubuntu Focal Fossa 20.04 LTS or later
- You have an additional admin machine available to install
ceph-deploy
Installing ceph-deploy on the admin machine
ceph-deploy
is a tool that allows you to deploy a Ceph cluster using a simple to use command-line interface and it will be installed on an independent admin machine.
Type the following command to connect to the admin machine using SSH:
ssh myuser@my.admin.server.ip
Type the following command to add the Ceph release key to apt:
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
Type the following command to add the Ceph repository to the apt package manager:
echo deb https://eu.ceph.com/debian-octopus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
Type the following command to update the apt package manager to include Ceph’s repository:
sudo apt update
Type the following command to install
ceph-deploy
:sudo apt install ceph-deploy
Creating a ceph-deploy user
ceph-deploy
needs access to a user that has passwordless sudo privileges to be able to install software and configuration files on the storage nodes.
Ceph deploy will be able to connect as any user, including the root
user, however this is not recommended. To simplify the software configuration, it is recommended to use the same username on all Ceph nodes in the cluster.
Do not use ceph
as login for your user, as it is reserved for Ceph-daemons. The installation of Ceph will fail if a ceph
user exists on a machine.
Type the following command to connect to your Ceph node using SSH:
ssh user@ceph-node
Type the following command to create a user called
ceph-deploy
:sudo useradd -d /home/ceph-deploy -m ceph-deploy
Note:You can rename the user to your own preferences if required.
Type the following command to configure the password of the
ceph-deploy
user:sudo passwd ceph-deploy
Type the following command to add the user to the sudoers configuration so they have sudo rights:
echo "ceph-deploy ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph-deploy
sudo chmod 0440 /etc/sudoers.d/ceph-deployType the following command to install an NTP client on your Ceph nodes:
sudo apt install ntpsec
Note:The installation of an NTP client allows you to avoid synchronization issues caused by time-drift.
Install Python to deploy the cluster:
sudo apt install python-minimal
Repeat all the previous steps 3 times to set up each of your 3 nodes.
Enabling passwordless SSH
In the following section, you will generate an SSH key and distribute the public key to each Ceph node to enable passwordless authentication on the nodes.
- Make sure not to use
sudo
. - Make sure you are not logged as the
root
user when generating the SSH keys.
Type the following command to generate a public or a private key-pair using the
ssh-keygen
application on the admin node:ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ceph-admin/.ssh/id_rsa): <-- Press Enter to save the key in the default location
Created directory '/home/ceph-admin/.ssh'.
Enter passphrase (empty for no passphrase): <-- Leave the passphrase empty. Press Enter to confirm.
Enter same passphrase again:
Your identification has been saved in /home/ceph-admin/.ssh/id_rsa.
Your public key has been saved in /home/ceph-admin/.ssh/id_rsa.pub.Type the following command to make sure that the hostnames of the ceph nodes are configured in your
/etc/hosts
file:sudo nano /etc/hosts
Your file should look like the following example:
127.0.0.1 localhost
127.0.1.1 ceph-admin
163.172.123.123 ceph-node-a
162.172.123.124 ceph-node-b
162.172.123.125 ceph-node-cSave the file and exit your text editor.
Transfer the public key of your admin node to each of your Ceph nodes:
ssh-copy-id ceph-deploy@ceph-node-a
ssh-copy-id ceph-deploy@ceph-node-b
ssh-copy-id ceph-deploy@ceph-node-c
Deploying a Ceph cluster
In this section, you will deploy the Ceph cluster on your machines. To keep a clean directory structure, create a directory on your admin node for maintaining the configuration files and keys of your cluster, that ceph-deploy
generates.
Type the following commands to create a new directory on the admin node and to enter into it:
mkdir my-ceph-cluster
cd my-ceph-clusterType the following command to create the cluster:
ceph-deploy --username ceph-deploy new ceph-node-a
Note:Replace
ceph-node-a
with the FQDN of your node, for example:ceph-node-a.example.com
.Tip:If you have more than one network interface on the machine, type the following command to add the public network setting under the
[global]
section of your Ceph configuration file:public network = <ip-address>/<netmask>
For example:
public network = 62.210.16.23/24
Refer to the official documentation for more information on the network configuration.
Type the following command to install the ceph packages on the nodes:
ceph-deploy --username ceph-deploy install ceph-node-a ceph-node-b ceph-node-c
Type the following command to deploy the initial monitor(s) and gather the keys:
ceph-deploy --username ceph-deploy mon create-initial
Once completed, you should have the following files in your directory. Verify that they are generated using the
ls
command:ceph.bootstrap-mds.keyring ceph.bootstrap-rgw.keyring ceph-deploy-ceph.log
ceph.bootstrap-mgr.keyring ceph.client.admin.keyring ceph.mon.keyring
ceph.bootstrap-osd.keyring ceph.confCopy the configuration file and admin key to your Ceph Nodes using
ceph-deploy
:ceph-deploy --username ceph-deploy admin ceph-node-a ceph-node-b ceph-node-c
Note:This step allows you to use the ceph CLI without having to specify the monitor address and
ceph.client.admin.keyring
each time you execute a command.Type the following command to deploy the manager daemon on all Ceph nodes:
ceph-deploy --username ceph-deploy mgr create ceph-node-a ceph-node-b ceph-node-c
Note:These daemons operate in an active/standby pattern. Deploying multiple manager daemons ensures that if one of the daemons fails for any reason, another one can take over without interrupting service.
Type the following command to configure an Object Storage Device (OSD) on each of the ceph nodes:
ceph-deploy osd create --data /dev/sdb ceph-node-a
ceph-deploy osd create --data /dev/sdb ceph-node-b
ceph-deploy osd create --data /dev/sdb ceph-node-cImportant:Ensure that the device is not in use and that it does not contain any important data. All data present on the device will be deleted!
Type the following command to check the status of your Ceph cluster from one of the Ceph nodes: If everything is up and running, your cluster should report
HEALTH_OK
sudo ceph health
If everything is up and running, your cluster should report HEALTH_OK
Type the following command to see a complete status of your cluster sudo ceph -s
The basic installation of your cluster is now complete. It is running and provides a distributed Ceph filesystem.
Deploying a Ceph Object Gateway (RGW)
RGW (Ceph Object Gateway) is the S3/Swift gateway component of Ceph. It is required to access your files using any S3 compatible client or application. Deploy the component on one of your nodes from your admin machine using ceph-deploy
.
Log into your admin machine and run the following command:
ceph-deploy --username ceph-deploy rgw create ceph-node-a
The following message displays once the installation is complete:
[ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host ceph-node-a and default port 7480
Note:By default, the RGW instance listens on port 7480.
Add the following lines to the file to change this port
/etc/ceph/ceph.conf
on the node running the RGW as follows:sudo nano /etc/ceph/ceph.conf
[client]
rgw frontends = civetweb port=80
#To use an IPv6 address, add the following line:
rgw frontends = civetweb port=[::]:80If you want to secure the connection using a TLS certificate, specify the HTTPS port and the bundled certificate file. You will need to either create a self-signed certificate or get a certificate from a certificate authority (CA), for example, Let’s Encrypt:
[client]
rgw frontends = civetweb port=443s ssl_certificate=/etc/ceph/private/bundle_keyandcert.pemOpen a web browser and point it to
http://ceph-node-a:7480
to verify your installation. A response like the following displays:
Creating S3 credentials
Log into your gateway instance (in our example ceph-node-a
) and run the following command to create a new user:
sudo radosgw-admin user create --uid=johndoe --display-name="John Doe" --email=john@example.com
--uid
: The unique user identification for the user (for example:johndoe
)--display-name
: The display name of the user (for example:John Doe
)--email
: The email address linked to the account (for example:john@example.com
)
An output like the following displays. Write down the access_key
and user_key
as you need these to configure your S3 client.
{
"user_id": "johndoe",
"display_name": "John Doe",
"email": "john@example.com",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "johndoe",
"access_key": "XCJNIP2QN5VCIMBUJNDK",
"secret_key": "YGmcv8vesn5gizeuVx79Oz2ycQY2MYZc8hY6OCAx"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
You can now configure your S3 compatible tool, for example, aws-cli.
Configuring AWS-CLI
You can use AWS-CLI to manage objects in your Ceph storage cluster using the standardized S3 protocol. It allows you to create buckets easily and to manage your files using an efficient command-line tool. You can find a list of available S3 features in the Ceph documentation.
Type the following command to install
aws-cli
andawscli-plugin
on your machine usingpip
, the Python package manager:pip3 install awscli
pip3 install awscli-plugin-endpointCreate a file
~/.aws/config
and put the following contents in it. Edit the Endpoint URL to your RGW instance:[plugins]
endpoint = awscli_plugin_endpoint
[default]
region = default
s3 =
endpoint_url = http://ceph-node-a:7480
signature_version = s3v4
max_concurrent_requests = 100
max_queue_size = 1000
multipart_threshold = 50MB
multipart_chunksize = 10MB
s3api =
endpoint_url = http://ceph-node-a:7480Create a configuration file for your S3 credentials at
~/.aws/credentials
and edit it as follows:[default]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>Note:Replace
<ACCESS_KEY>
and<SECRET_KEY>
with the credentials of the user account you created in a previous step.Type the following command to create a bucket on your cluster:
aws s3 mb s3://MyBucket
make_bucket: MyBucketType the following command to create a test file:
echo "Hello World!" > testfile.txt
Type the following command to upload the file to the cluster:
aws s3 cp testfile.txt s3://MyBucket
upload: ./testfile.txt to s3://MyBucket/testfile.txtType the following command to check if the upload of the file was successful by listing the content of your bucket:
aws s3 ls s3://MyBucket
2020-08-20 14:40:38 13 testfile.txt
Conclusion
You have now configured an S3 compatible storage cluster using Ceph and three Dedibox dedicated servers. You can upload your data to the cluster and modify or share it using any S3 compatible tool. For more advanced configuration options, refer to the official Ceph documentation.