Installing and Configuring Graphite on Ubuntu 20.04

• compute
• kubernetes
• instance
• elastic-metal
• Graphite
• Ubuntu-20.04
• -
• kubernetes
• -
• instances
• -
• elastic-metal

Graphite Overview

Graphite is a well known and widespread monitoring tool, that basically does two things:

• Storing of time-related data efficiently,
• Providing an interface for basic visualization of the stored data and the possibility to use mathematical functions to sum, group, or scale stored data in real-time.

Before you start

To complete the actions presented below, you must have:

• A Scaleway account logged into the console
• Owner status or IAM permissions allowing you to perform actions in the intended Organization
• An SSH key
• An Instance running Ubuntu 20.04 LTS
• A valid API key
• sudo privileges or access to the root user

Installing Graphite

1. Graphite and the required tools can be installed easily via apt. Update the apt cache and upgrade the packages already installed on the system:

   apt-get update && apt-get upgrade -y
   

2. Install Graphite:

   apt-get install graphite-web graphite-carbon
   

Configuring the Graphite web application

1. Open the configuration file in which you have to make the modifications explained afterword:

   nano /etc/graphite/local_settings.py
   

2. Set a SECRET_KEY that will be used as a salt when creating hashes. Uncomment the line and set it to secure key:

   SECRET_KEY = 'a_salty_string'
   

3. Uncomment the TIME_ZONE parameter and configure it to your local time-zone:

   TIME_ZONE = 'Europe/Paris'
   

4. Save the file and quit the editor.
5. Synchronize the database to create the required database layout:

   graphite-manage migrate
   

Configuring Carbon

As the database is ready now, continue with the configuration of Carbon, the storage backend of Graphite.

1. Open the service configuration file with a text editor:

   nano /etc/default/graphite-carbon
   

2. There is only one configuration parameter in the file, to make sure Carbon starts at boot, change the value to true.

   CARBON_CACHE_ENABLED=true
   

3. Save the file and quit the Editor once you have changed the value.
4. Open the Carbon configuration file:

   nano /etc/carbon/carbon.conf
   

5. Enable log rotation by setting the following value to true:

   ENABLE_LOGROTATION = True
   

6. Save the file and quit the Editor once you have changed the value.

Configuring Storage Schemas

1. Open the storage schemas file. It contains information about how long and how detailed values should be stored:

   nano /etc/carbon/storage-schemas.conf
   

2. The content of the file will look like this example:

   [carbon]
   pattern = ^carbon\.
   retentions = 60:90d
   [default_1min_for_1day]
   pattern = .*
   retentions = 60s:1d
   

   By default two sections are defined in the file:

   • The first one is to decide what to do with data coming from Carbon itself, as it is configured by default to store some of its own performance metrics.
   • The second one is a catch-all section that applies to any data that hasn’t been matched by any other section. It has to remain always the last section of the file.

   Each section is defined by section headers, namely the words in the brackets. Below each section header, a pattern definition and retention policy are defined.

   The pattern definition is a regular expression, used to match any information sent to Carbon. All information sent to Carbon include a metric name, which is checked by the pattern definition. In the first section, all metrics in question start with the string “carbon.”.

   The retention policy is defined by sets of numbers, consisting of a metric interval (defining how often a metric is recorded), followed by a colon and the storage duration of these values. It is possible to define multiple sets of retention policies, separated by commas.

3. Define a new retention policy that is triggered by a “test.” pattern and will be used later:

   [test]
   pattern = ^test\.
   retentions = 10s:10m,1m:1h,10m:1d
   

   Note

   Remember to place the policy before the catch-all block in the configuration file.

   This section will store the data it collects three times with different levels of details. The first collection 10s:10m will create a data point every ten seconds and data is stored for only 10 minutes.

   The second collection 1m:1h will create a data point every minute by gathering all the data from the past minute from the first collection. The information in the data point is aggregated by averaging the points (six points, as the first collection creates a point every 10 seconds). Data will be stored in this collection within one hour.

   The last collection 10m:1d will make a data point ever every 10 minutes by aggregating the information gathered from the second collection in the same way. Data will be stored in this collection for one day.

   Graphite returns the data from the most detailed collection that measures the requested time-frame when asking for information. This means:

   • If metrics for the past 5 minutes will be requested, Graphite will return information from the first collection.
   • If metrics for the past 50 minutes will be requested, Graphite will return information from the second collection.

4. Save and close the file when you have finished editing it.

Defining storage aggregation methods

To gather accurate metrics, it is essential to understand the way Carbon decides when it crunches detailed information into a generalized number. This happens every time when Graphite converts more detailed metrics into less detailed ones, like in the second and third collections in the test schema above.

The default behavior is to get the average value when aggregating. This means unlike the most detailed information in the first collection, information is less accurate in the second and third collection.

This may not always be useful though. For example to count up the total number of times an event occurred over different periods of time it is not useful to average them but to count each event.

It is possible to define the way Graphite aggregates metrics with a file called storage-aggregation.conf.

1. Copy the file from the examples’ directory to the actual configuration directory:

   cp /usr/share/doc/graphite-carbon/examples/storage-aggregation.conf.example /etc/carbon/storage-aggregation.conf
   

2. Open the file in a text editor:

   nano /etc/carbon/storage-aggregation.conf
   

3. You will see a file similar to this content:

   [min]
   pattern = \.min$
   xFilesFactor = 0.1
   aggregationMethod = min
   

   It is looking similar to the previous file, and the section name and pattern are exactly the same as in the storage-schemas file.

   The xFilesFactor is what we will take a closer look at. It allows specifying the minimum percentage of values that Carbon must have to create an aggregated data-point. For example, if the value is set to 0.5, it requires that 50% of the more detailed data points are available to create an aggregated point.

   This can be useful to avoid creating data-points misrepresenting the actual situation.

   aggregationMethod defines the way data is recorded. Possible values are average, sum, last, max and min. It is important to choose the correct value to avoid that your data will be recorded inaccurately. The correct selection depends on the kind of metrics that you are actually tracking.

4. Once you have edited the file towards your needs, save and close the file.

5. Start Carbon by typing:

   service carbon-cache start
   

Installing and configuring Apache

To use the web-interface of Graphite, a web server is required. The software comes with a pre-defined configuration for Apache, making configuration pretty easy.

1. Install the Apache web server and the required module:

   apt-get install apache2 libapache2-mod-wsgi-py3
   

2. Disable the default site of Apache, as it won’t be needed:

   a2dissite 000-default
   

3. Copy the Apache configuration file of Graphite into the available sites directory of Apache:

   cp /usr/share/graphite-web/apache2-graphite.conf /etc/apache2/sites-available
   

4. Enable the Graphite site:

   a2ensite apache2-graphite
   

5. Reload the configuration of Apache:

   service apache2 reload
   

6. Check if the web interface is working by pointing your browser to http://YOUR_SERVERS_IP. The following interface will be visible:
   

Discovering the web interface

1. Login by clicking on the Login-Button on the top of the page. Enter the username and password that you have set when you have synchronized the database. In case you do not remember these credentials or if you want to add another superuser, run the following command to create a new user:

   graphite-manage createsuperuser
   

2. You will notice a menu on the left of the Screen. Click carbon to see the metrics that are collected by Graphite. Currently, you will only see the data the application is gathering about its performance:
   
3. It is also possible to create dashboards from the data Graphite collects to have an overview over the different metrics:
   

Feeding data to graphite

Graphite is very flexible when it comes to the origin of data. There are three main methods for sending data to Graphite: Plaintext, Pickle, and AMQP.

1. Previously a test block has been created in the storage-schemas file. It will be used now to send some data to Graphite from a terminal.

2. Type the following command in a terminal. You can replace the value 42 with some different numbers to see what it does:

   echo "test.count 42 `date +%s`" | nc -q0 localhost 2003
   

   Metric messages need to contain a metric name, a value, and a timestamp.

   The example feeds the storage schema test with the metric count and the value 42. To get the timestamp, the date command is used.

3. Go back to the Graphite Web Interface and reload it. The new storage scheme test will appear in the menu.

4. Set the time range to a few minutes, by clicking on the clock-icon:

   

5. The values that you have sent will appear in the graph:

   

6. Wait for 15 minutes and refresh the graph by setting the time frame to the past 15 minutes. The graph will look different:

   

This is because our first collection does not store data for 15 minutes and Graphite will look into the second collection for rendering the graph.

As the data was sent with a count metric, Graphite adds up the values in the larger intervals instead of averaging them. It is therefore essential to choose the right metric for each use case.