Hometutorials
ceph cluster

Jump toUpdate content

How to build your own Ceph distributed storage cluster on dedicated servers

Reviewed on 19 April 2022Published on 29 June 2020
  • dedicated-servers
  • dedibox
  • storage-cluster
  • Ceph

Ceph - Overview

Ceph is an open-source, software-defined storage solution designed to address object, block, and file storage needs. Ceph can handle up to several exabytes of data. It replicates data and makes it fault-tolerant, using standard hardware without expensive specific storage hardware. The system is designed to minimize administration time and costs. Therefore it is both self-healing and self-managing.

In this tutorial, you will learn how to deploy a three-node Ceph cluster using Dedibox dedicated servers running on Ubuntu Focal Fossa (20.04 LTS).

Requirements:
  • You have an account and are logged into the Dedibox console
  • You have three Dedibox servers running on Ubuntu Focal Fossa 20.04 LTS or later
  • You have an additional admin machine available to install ceph-deploy

Installing ceph-deploy on the admin machine

ceph-deploy is a tool that allows you to deploy a Ceph cluster using a simple to use command-line interface and it will be installed on an independent admin machine.

  1. Type the following command to connect to the admin machine using SSH:

    ssh myuser@my.admin.server.ip
  2. Type the following command to add the Ceph release key to apt:

    wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
  3. Type the following command to add the Ceph repository to the apt package manager:

    echo deb https://eu.ceph.com/debian-octopus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
  4. Type the following command to update the apt package manager to include Ceph’s repository:

    sudo apt update
  5. Type the following command to install ceph-deploy:

    sudo apt install ceph-deploy

Creating a ceph-deploy user

ceph-deploy needs access to a user that has passwordless sudo privileges to be able to install software and configuration files on the storage nodes. Ceph deploy will be able to connect as any user, including the root user, however this is not recommended. To simplify the software configuration, it is recommended to use the same username on all Ceph nodes in the cluster.

Important:

Do not use ceph as login for your user, as it is reserved for Ceph-daemons. The installation of Ceph will fail if a ceph user exists on a machine.

  1. Type the following command to connect to your Ceph node using SSH:

    ssh user@ceph-node
  2. Type the following command to create a user called ceph-deploy:

    sudo useradd -d /home/ceph-deploy -m ceph-deploy
    Note:

    You can rename the user to your own preferences if required.

  3. Type the following command to configure the password of the ceph-deploy user:

    sudo passwd ceph-deploy
  4. Type the following command to add the user to the sudoers configuration so they have sudo rights:

    echo "ceph-deploy ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph-deploy
    sudo chmod 0440 /etc/sudoers.d/ceph-deploy
  5. Type the following command to install an NTP client on your Ceph nodes:

    sudo apt install ntpsec
    Note:

    The installation of an NTP client allows you to avoid synchronization issues caused by time-drift.

  6. Install Python to deploy the cluster:

    sudo apt install python-minimal
  7. Repeat all the previous steps 3 times to set up each of your 3 nodes.

Enabling passwordless SSH

In the following section, you will generate an SSH key and distribute the public key to each Ceph node to enable passwordless authentication on the nodes.

Note:
  • Make sure not to use sudo.
  • Make sure you are not logged as the root user when generating the SSH keys.
  1. Type the following command to generate a public or a private key-pair using the ssh-keygen application on the admin node:

    ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/ceph-admin/.ssh/id_rsa): <-- Press Enter to save the key in the default location
    Created directory '/home/ceph-admin/.ssh'.
    Enter passphrase (empty for no passphrase): <-- Leave the passphrase empty. Press Enter to confirm.
    Enter same passphrase again:
    Your identification has been saved in /home/ceph-admin/.ssh/id_rsa.
    Your public key has been saved in /home/ceph-admin/.ssh/id_rsa.pub.
  2. Type the following command to make sure that the hostnames of the ceph nodes are configured in your /etc/hosts file:

    sudo nano /etc/hosts

    Your file should look like the following example:

    127.0.0.1   localhost
    127.0.1.1 ceph-admin
    163.172.123.123 ceph-node-a
    162.172.123.124 ceph-node-b
    162.172.123.125 ceph-node-c

    Save the file and exit your text editor.

  3. Transfer the public key of your admin node to each of your Ceph nodes:

    ssh-copy-id ceph-deploy@ceph-node-a
    ssh-copy-id ceph-deploy@ceph-node-b
    ssh-copy-id ceph-deploy@ceph-node-c

Deploying a Ceph cluster

In this section, you will deploy the Ceph cluster on your machines. To keep a clean directory structure, create a directory on your admin node for maintaining the configuration files and keys of your cluster, that ceph-deploy generates.

  1. Type the following commands to create a new directory on the admin node and to enter into it:

    mkdir my-ceph-cluster
    cd my-ceph-cluster
  2. Type the following command to create the cluster:

    ceph-deploy --username ceph-deploy new ceph-node-a
    Note:

    Replace ceph-node-a with the FQDN of your node, for example: ceph-node-a.example.com.

    Tip:
    • If you have more than one network interface on the machine, type the following command to add the public network setting under the [global] section of your Ceph configuration file:

      public network = <ip-address>/<netmask>

      For example:

      public network = 62.210.16.23/24
    • Refer to the official documentation for more information on the network configuration.

  3. Type the following command to install the ceph packages on the nodes:

    ceph-deploy --username ceph-deploy install ceph-node-a ceph-node-b ceph-node-c
  4. Type the following command to deploy the initial monitor(s) and gather the keys:

    ceph-deploy --username ceph-deploy mon create-initial

    Once completed, you should have the following files in your directory. Verify that they are generated using the ls command:

    ceph.bootstrap-mds.keyring  ceph.bootstrap-rgw.keyring  ceph-deploy-ceph.log
    ceph.bootstrap-mgr.keyring ceph.client.admin.keyring ceph.mon.keyring
    ceph.bootstrap-osd.keyring ceph.conf
  5. Copy the configuration file and admin key to your Ceph Nodes using ceph-deploy:

    ceph-deploy --username ceph-deploy admin ceph-node-a ceph-node-b ceph-node-c
    Note:

    This step allows you to use the ceph CLI without having to specify the monitor address and ceph.client.admin.keyring each time you execute a command.

  6. Type the following command to deploy the manager daemon on all Ceph nodes:

    ceph-deploy --username ceph-deploy mgr create ceph-node-a ceph-node-b ceph-node-c
    Note:

    These daemons operate in an active/standby pattern. Deploying multiple manager daemons ensures that if one of the daemons fails for any reason, another one can take over without interrupting service.

  7. Type the following command to configure an Object Storage Device (OSD) on each of the ceph nodes:

    ceph-deploy osd create --data /dev/sdb ceph-node-a
    ceph-deploy osd create --data /dev/sdb ceph-node-b
    ceph-deploy osd create --data /dev/sdb ceph-node-c
    Important:

    Ensure that the device is not in use and that it does not contain any important data. All data present on the device will be deleted!

  8. Type the following command to check the status of your Ceph cluster from one of the Ceph nodes: If everything is up and running, your cluster should report HEALTH_OK

    sudo ceph health

If everything is up and running, your cluster should report HEALTH_OK

Tip:

Type the following command to see a complete status of your cluster sudo ceph -s

The basic installation of your cluster is now complete. It is running and provides a distributed Ceph filesystem.

Deploying a Ceph Object Gateway (RGW)

RGW (Ceph Object Gateway) is the S3/Swift gateway component of Ceph. It is required to access your files using any S3 compatible client or application. Deploy the component on one of your nodes from your admin machine using ceph-deploy.

  1. Log into your admin machine and run the following command:

    ceph-deploy --username ceph-deploy rgw create ceph-node-a

    The following message displays once the installation is complete:

    [ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-node-a and default port 7480
    Note:

    By default, the RGW instance listens on port 7480.

  2. Add the following lines to the file to change this port /etc/ceph/ceph.conf on the node running the RGW as follows:

    sudo nano /etc/ceph/ceph.conf
    [client]
    rgw frontends = civetweb port=80

    #To use an IPv6 address, add the following line:
    rgw frontends = civetweb port=[::]:80

    If you want to secure the connection using a TLS certificate, specify the HTTPS port and the bundled certificate file. You will need to either create a self-signed certificate or get a certificate from a certificate authority (CA), for example, Let’s Encrypt:

    [client]
    rgw frontends = civetweb port=443s ssl_certificate=/etc/ceph/private/bundle_keyandcert.pem
  3. Open a web browser and point it to http://ceph-node-a:7480 to verify your installation. A response like the following displays:

Creating S3 credentials

Log into your gateway instance (in our example ceph-node-a) and run the following command to create a new user:

sudo radosgw-admin user create --uid=johndoe --display-name="John Doe" --email=john@example.com
Note:
  • --uid: The unique user identification for the user (for example: johndoe)
  • --display-name: The display name of the user (for example: John Doe)
  • --email: The email address linked to the account (for example: john@example.com)

An output like the following displays. Write down the access_key and user_key as you need these to configure your S3 client.

{
"user_id": "johndoe",
"display_name": "John Doe",
"email": "john@example.com",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "johndoe",
"access_key": "XCJNIP2QN5VCIMBUJNDK",
"secret_key": "YGmcv8vesn5gizeuVx79Oz2ycQY2MYZc8hY6OCAx"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}

You can now configure your S3 compatible tool, for example, aws-cli.

Configuring AWS-CLI

You can use AWS-CLI to manage objects in your Ceph storage cluster using the standardized S3 protocol. It allows you to create buckets easily and to manage your files using an efficient command-line tool. You can find a list of available S3 features in the Ceph documentation.

  1. Type the following command to install aws-cli and awscli-plugin on your machine using pip, the Python package manager:

    pip3 install awscli
    pip3 install awscli-plugin-endpoint
  2. Create a file ~/.aws/config and put the following contents in it. Edit the Endpoint URL to your RGW instance:

    [plugins]
    endpoint = awscli_plugin_endpoint

    [default]
    region = default
    s3 =
    endpoint_url = http://ceph-node-a:7480
    signature_version = s3v4
    max_concurrent_requests = 100
    max_queue_size = 1000
    multipart_threshold = 50MB
    multipart_chunksize = 10MB
    s3api =
    endpoint_url = http://ceph-node-a:7480
  3. Create a configuration file for your S3 credentials at ~/.aws/credentials and edit it as follows:

    [default]
    aws_access_key_id=<ACCESS_KEY>
    aws_secret_access_key=<SECRET_KEY>
    Note:

    Replace <ACCESS_KEY> and <SECRET_KEY> with the credentials of the user account you created in a previous step.

  4. Type the following command to create a bucket on your cluster:

    aws s3 mb s3://MyBucket
    make_bucket: MyBucket
  5. Type the following command to create a test file:

    echo "Hello World!" > testfile.txt
  6. Type the following command to upload the file to the cluster:

    aws s3 cp testfile.txt s3://MyBucket
    upload: ./testfile.txt to s3://MyBucket/testfile.txt
  7. Type the following command to check if the upload of the file was successful by listing the content of your bucket:

    aws s3 ls s3://MyBucket
    2020-08-20 14:40:38 13 testfile.txt

Conclusion

You have now configured an S3 compatible storage cluster using Ceph and three Dedibox dedicated servers. You can upload your data to the cluster and modify or share it using any S3 compatible tool. For more advanced configuration options, refer to the official Ceph documentation.