How to build your own Ceph distributed storage cluster on dedicated servers

• dedicated-servers
• dedibox
• storage-cluster
• Ceph

Ceph - Overview

Ceph is an open-source, software-defined storage solution designed to address object, block, and file storage needs. Ceph can handle up to several exabytes of data. It replicates data and makes it fault-tolerant, using standard hardware without expensive specific storage hardware. The system is designed to minimize administration time and costs. Therefore it is both self-healing and self-managing.

In this tutorial, you will learn how to deploy a three-node Ceph cluster using Dedibox dedicated servers running on Ubuntu Focal Fossa (20.04 LTS).

Installing ceph-deploy on the admin machine

ceph-deploy is a tool that allows you to deploy a Ceph cluster using a simple to use command-line interface and it will be installed on an independent admin machine.

1. Type the following command to connect to the admin machine using SSH:

   ssh myuser@my.admin.server.ip
   Copy code

2. Type the following command to add the Ceph release key to apt:

   wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
   Copy code

3. Type the following command to add the Ceph repository to the apt package manager:

   echo deb https://eu.ceph.com/debian-octopus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
   Copy code

4. Type the following command to update the apt package manager to include Ceph’s repository:

   sudo apt update
   Copy code

5. Type the following command to install ceph-deploy:

   sudo apt install ceph-deploy
   Copy code

Creating a ceph-deploy user

ceph-deploy needs access to a user that has passwordless sudo privileges to be able to install software and configuration files on the storage nodes. Ceph deploy will be able to connect as any user, including the root user, however this is not recommended. To simplify the software configuration, it is recommended to use the same username on all Ceph nodes in the cluster.

1. Type the following command to connect to your Ceph node using SSH:

   ssh user@ceph-node
   Copy code

2. Type the following command to create a user called ceph-deploy:

   sudo useradd -d /home/ceph-deploy -m ceph-deploy
   Copy code

   Note:

   You can rename the user to your own preferences if required.

3. Type the following command to configure the password of the ceph-deploy user:

   sudo passwd ceph-deploy
   Copy code

4. Type the following command to add the user to the sudoers configuration so they have sudo rights:

   echo "ceph-deploy ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph-deploy
   sudo chmod 0440 /etc/sudoers.d/ceph-deploy
   Copy code

5. Type the following command to install an NTP client on your Ceph nodes:

   sudo apt install ntpsec
   Copy code

   Note:

   The installation of an NTP client allows you to avoid synchronization issues caused by time-drift.

6. Install Python to deploy the cluster:

   sudo apt install python-minimal
   Copy code

7. Repeat all the previous steps 3 times to set up each of your 3 nodes.

Enabling passwordless SSH

In the following section, you will generate an SSH key and distribute the public key to each Ceph node to enable passwordless authentication on the nodes.

1. Type the following command to generate a public or a private key-pair using the ssh-keygen application on the admin node:

   ssh-keygen
   Generating public/private rsa key pair.
   Enter file in which to save the key (/home/ceph-admin/.ssh/id_rsa): <-- Press Enter to save the key in the default location
   Created directory '/home/ceph-admin/.ssh'.
   Enter passphrase (empty for no passphrase):  <-- Leave the passphrase empty. Press Enter to confirm.
   Enter same passphrase again:
   Your identification has been saved in /home/ceph-admin/.ssh/id_rsa.
   Your public key has been saved in /home/ceph-admin/.ssh/id_rsa.pub.
   Copy code

2. Type the following command to make sure that the hostnames of the ceph nodes are configured in your /etc/hosts file:

   sudo nano /etc/hosts
   Copy code

   Your file should look like the following example:

   127.0.0.1   localhost
   127.0.1.1   ceph-admin
   163.172.123.123 ceph-node-a
   162.172.123.124 ceph-node-b
   162.172.123.125 ceph-node-c
   Copy code

   Save the file and exit your text editor.

3. Transfer the public key of your admin node to each of your Ceph nodes:

   ssh-copy-id ceph-deploy@ceph-node-a
   ssh-copy-id ceph-deploy@ceph-node-b
   ssh-copy-id ceph-deploy@ceph-node-c
   Copy code

Deploying a Ceph cluster

In this section, you will deploy the Ceph cluster on your machines. To keep a clean directory structure, create a directory on your admin node for maintaining the configuration files and keys of your cluster, that ceph-deploy generates.

1. Type the following commands to create a new directory on the admin node and to enter into it:

   mkdir my-ceph-cluster
   cd my-ceph-cluster
   Copy code

2. Type the following command to create the cluster:

   ceph-deploy --username ceph-deploy new ceph-node-a
   Copy code

   Note:

   Replace ceph-node-a with the FQDN of your node, for example: ceph-node-a.example.com.

   Tip:

   • If you have more than one network interface on the machine, type the following command to add the public network setting under the [global] section of your Ceph configuration file:

     public network = <ip-address>/<netmask>
     Copy code

     For example:

     public network = 62.210.16.23/24
     Copy code

   • Refer to the official documentation for more information on the network configuration.

3. Type the following command to install the ceph packages on the nodes:

   ceph-deploy --username ceph-deploy install ceph-node-a ceph-node-b ceph-node-c
   Copy code

4. Type the following command to deploy the initial monitor(s) and gather the keys:

   ceph-deploy --username ceph-deploy mon create-initial
   Copy code

   Once completed, you should have the following files in your directory. Verify that they are generated using the ls command:

   ceph.bootstrap-mds.keyring  ceph.bootstrap-rgw.keyring  ceph-deploy-ceph.log
   ceph.bootstrap-mgr.keyring  ceph.client.admin.keyring   ceph.mon.keyring
   ceph.bootstrap-osd.keyring  ceph.conf
   Copy code

5. Copy the configuration file and admin key to your Ceph Nodes using ceph-deploy:

   ceph-deploy --username ceph-deploy admin ceph-node-a ceph-node-b ceph-node-c
   Copy code

   Note:

   This step allows you to use the ceph CLI without having to specify the monitor address and ceph.client.admin.keyring each time you execute a command.

6. Type the following command to deploy the manager daemon on all Ceph nodes:

   ceph-deploy --username ceph-deploy mgr create ceph-node-a ceph-node-b ceph-node-c
   Copy code

   Note:

   These daemons operate in an active/standby pattern. Deploying multiple manager daemons ensures that if one of the daemons fails for any reason, another one can take over without interrupting service.

7. Type the following command to configure an Object Storage Device (OSD) on each of the ceph nodes:

   ceph-deploy osd create --data /dev/sdb ceph-node-a
   ceph-deploy osd create --data /dev/sdb ceph-node-b
   ceph-deploy osd create --data /dev/sdb ceph-node-c
   Copy code

   Important:

   Ensure that the device is not in use and that it does not contain any important data. All data present on the device will be deleted!

8. Type the following command to check the status of your Ceph cluster from one of the Ceph nodes: If everything is up and running, your cluster should report HEALTH_OK

   sudo ceph health
   Copy code

If everything is up and running, your cluster should report HEALTH_OK

The basic installation of your cluster is now complete. It is running and provides a distributed Ceph filesystem.

Deploying a Ceph Object Gateway (RGW)

RGW (Ceph Object Gateway) is the S3/Swift gateway component of Ceph. It is required to access your files using any S3 compatible client or application. Deploy the component on one of your nodes from your admin machine using ceph-deploy.

1. Log into your admin machine and run the following command:

   ceph-deploy --username ceph-deploy rgw create ceph-node-a
   Copy code

   The following message displays once the installation is complete:

   [ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-node-a and default port 7480
   Copy code

   Note:

   By default, the RGW instance listens on port 7480.

2. Add the following lines to the file to change this port /etc/ceph/ceph.conf on the node running the RGW as follows:

   sudo nano /etc/ceph/ceph.conf
   Copy code

   [client]
   rgw frontends = civetweb port=80
   
   #To use an IPv6 address, add the following line:
   rgw frontends = civetweb port=[::]:80
   Copy code

   If you want to secure the connection using a TLS certificate, specify the HTTPS port and the bundled certificate file. You will need to either create a self-signed certificate or get a certificate from a certificate authority (CA), for example, Let’s Encrypt:

   [client]
   rgw frontends = civetweb port=443s ssl_certificate=/etc/ceph/private/bundle_keyandcert.pem
   Copy code

3. Open a web browser and point it to http://ceph-node-a:7480 to verify your installation. A response like the following displays:

   

   

Creating S3 credentials

Log into your gateway instance (in our example ceph-node-a) and run the following command to create a new user:

sudo radosgw-admin user create --uid=johndoe --display-name="John Doe" --email=john@example.com
Copy code

An output like the following displays. Write down the access_key and user_key as you need these to configure your S3 client.

{
    "user_id": "johndoe",
    "display_name": "John Doe",
    "email": "john@example.com",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "johndoe",
            "access_key": "XCJNIP2QN5VCIMBUJNDK",
            "secret_key": "YGmcv8vesn5gizeuVx79Oz2ycQY2MYZc8hY6OCAx"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}
Copy code

You can now configure your S3 compatible tool, for example, aws-cli.

Configuring AWS-CLI

You can use AWS-CLI to manage objects in your Ceph storage cluster using the standardized S3 protocol. It allows you to create buckets easily and to manage your files using an efficient command-line tool. You can find a list of available S3 features in the Ceph documentation.

1. Type the following command to install aws-cli and awscli-plugin on your machine using pip, the Python package manager:

   pip3 install awscli
   pip3 install awscli-plugin-endpoint
   Copy code

2. Create a file ~/.aws/config and put the following contents in it. Edit the Endpoint URL to your RGW instance:

   [plugins]
   endpoint = awscli_plugin_endpoint
   
   [default]
   region = default
   s3 =
     endpoint_url = http://ceph-node-a:7480
     signature_version = s3v4
     max_concurrent_requests = 100
     max_queue_size = 1000
     multipart_threshold = 50MB
     multipart_chunksize = 10MB
   s3api =
     endpoint_url = http://ceph-node-a:7480
   Copy code

3. Create a configuration file for your S3 credentials at ~/.aws/credentials and edit it as follows:

   [default]
   aws_access_key_id=<ACCESS_KEY>
   aws_secret_access_key=<SECRET_KEY>
   Copy code

   Note:

   Replace <ACCESS_KEY> and <SECRET_KEY> with the credentials of the user account you created in a previous step.

4. Type the following command to create a bucket on your cluster:

   aws s3 mb s3://MyBucket
   make_bucket: MyBucket
   Copy code

5. Type the following command to create a test file:

   echo "Hello World!" > testfile.txt
   Copy code

6. Type the following command to upload the file to the cluster:

   aws s3 cp testfile.txt s3://MyBucket
   upload: ./testfile.txt to s3://MyBucket/testfile.txt
   Copy code

7. Type the following command to check if the upload of the file was successful by listing the content of your bucket:

   aws s3 ls s3://MyBucket
   2020-08-20 14:40:38         13 testfile.txt
   Copy code

Conclusion

You have now configured an S3 compatible storage cluster using Ceph and three Dedibox dedicated servers. You can upload your data to the cluster and modify or share it using any S3 compatible tool. For more advanced configuration options, refer to the official Ceph documentation.