Graphite is a well known and widespread monitoring tool, that basically does two things:
1 . Graphite and the required tools can be installed easily via apt. Update the apt cache and upgrade the packages already installed on the system:
apt-get update && apt-get upgrade -y
2 . Install Graphite:
apt-get install graphite-web graphite-carbon
1 . Open the configuration file in which you have to make the modifications explained afterwards:
2 . Set a
SECRET_KEY that will be used as a salt when creating hashes. Uncomment the line and set it to secure key:
SECRET_KEY = 'a_salty_string'
3 . Uncomment the
TIME_ZONE parameter and configure it to your local time-zone:
TIME_ZONE = 'Europe/Paris'
4 . Save the file and quit the editor.
5 . Synchronise the database to create the required database layout:
As the database is ready now, continue with the configuration of Carbon, the storage backend of Graphite.
1 . Open the service configuration file with a text editor:
2 . There is only one configuration parameter in the file, to make sure Carbon starts at boot, change the value to
3 . Save the file and quit the Editor once you have changed the value.
4 . Open the Carbon configuration file:
5 . Enable log rotation by setting the following value to true:
ENABLE_LOGROTATION = True
6 . Save the file and quit the Editor once you have changed the value.
1 . Open the storage schemas file. It contains information about how long and how detailed values should be stored:
2 . The content of the file will look like this example:
[carbon] pattern = ^carbon\. retentions = 60:90d [default_1min_for_1day] pattern = .* retentions = 60s:1d
By default two sections are defined in the file:
The first one is to decide what to do with data coming from Carbon itself, as it is configured by default to store some of its own performance metrics.
The second one is a catch-all section that applies to any data that hasn’t been matched by any other section. It has to remain always the last section of the file.
Each section is defined by section headers, namely the words in the brackets. Below each section header, a pattern definition and retention policy are defined.
The pattern definition is a regular expression, used to match any information sent to Carbon. All information sent to Carbon include a metric name, which is checked by the pattern definition. In the first section, all metrics in question start with the string “carbon.”.
The retention policy is defined by sets of numbers, consisting of a metric interval (defining how often a metric is recorded), followed by a colon and the storage duration of these values. It is possible to define multiple sets of retention policies, separated by commas.
3 . Define a new retention policy that is triggered by a “test.” pattern and will be used later:
[test] pattern = ^test\. retentions = 10s:10m,1m:1h,10m:1d
Note: Remember to place the policy before the catch-all block in the configuration file.
This section will store the data it collects three times with different levels of details. The first collection
10s:10m will create a data point every ten seconds and data is stored for only 10 minutes.
The second collection
1m:1h will create a data point every minute by gathering all of the data from the past minute from the first collection. The information in the data point is aggregated by averaging the points (six points, as the first collection creates a point every 10 seconds). Data will be stored in this collection within one hour.
The last collection
10m:1d will make a data point ever every 10 minutes by aggregating the information gathered from the second collection in the same way. Data will be stored in this collection for one day.
Graphite returns the data from the most detailed collection that measures the requested time-frame when asking for information. This means:
If metrics for the past 5 minutes will be requested, Graphite will return information from the first collection.
If metrics for the past 50 minutes will be requested, Graphite will return information from the second collection.
4 . Save and close the file when you have finished editing it.
To gather accurate metrics, it is essential to understand the way Carbon decides when it crunches detailed information into a generalized number. This happens every time when Graphite converts more detailed metrics into less detailed ones, like in the second and third collections in the test schema above.
The default behavior is to get the average value when aggregating. This means unlike the most detailed information in the first collection, information are less accurate in the second and third collection.
This may not always be useful though. For example to count up the total number of times an event occurred over different periods of time it is not useful to average them but to count each event.
It is possible to define the way Graphite aggregates metrics with a file called
1 . Copy the file from the the examples directory to the actual configuration directory:
cp /usr/share/doc/graphite-carbon/examples/storage-aggregation.conf.example /etc/carbon/storage-aggregation.conf
2 . Open the file in a text editor:
3 . You will see a file similar to this content:
[min] pattern = \.min$ xFilesFactor = 0.1 aggregationMethod = min
It is looking similar to the previous file, and the section name and pattern are exactly the same as in the storage-schemas file.
xFilesFactor is what we will have a closer look at. It allows specifying the minimum percentage of values that Carbon must have to create an aggregated data-point. For example, if the value is set to 0.5, it requires that 50% of the more detailed data points are available to create an aggregated point.
This can be useful to avoid creating data-points misrepresenting the actual situation.
aggregationMethod defines the way data is recorded. Possible values are
min. It is important to choose the correct value to avoid that your data will be recorded inaccurately. The correct selection depends on the kind of metrics that you are actually tracking.
4 . Once you have edited the file towards your needs, save and close the file.
5 . Start Carbon by typing:
service carbon-cache start
To use the web-interface of Graphite, a web server is required. The software comes with a pre-defined configuration for Apache, making configuration pretty easy.
1 . Install the Apache web server and the required module:
apt-get install apache2 libapache2-mod-wsgi
2 . Disable the default site of Apache, as it won’t be needed:
3 . Copy the Apache configuration file of Graphite into the available sites directory of Apache:
cp /usr/share/graphite-web/apache2-graphite.conf /etc/apache2/sites-available
4 . Enable the Graphite site:
5 . Reload the configuration of Apache:
service apache2 reload
6 . Check if the web interface is working by pointing your browser to
http://YOUR_SERVERS_IP. The following interface will be visible:
1 . Login by clicking on the Login-Button on the top of the page. Enter the username and password that you have set when you have synchronized the database. In case you don’t remember these credentials or if you want to add another superuser, run the following command to create a new user:
2 . You will notice a menu on the left of the Screen. Click on carbon to see the metrics that are collected by Graphite. Currently, you will only see the data the application is gathering about its performance:
3 . It is also possible to create dashboards from the data Graphite collects to have an overview over the different metrics:
1 . Previously a
test block has been created in the storage-schemas file. It will be used now to send some data to Graphite from a terminal.
2 . Type the following command in a terminal. You can replace the value
42 with some different numbers to see what it does:
echo "test.count 42 `date +%s`" | nc -q0 localhost 2003
Metric messages need to contain a metric name, a value, and a timestamp.
The example feeds the storage schema
test with the metric
count and the value
42. To get the timestamp, the
date command is used.
3 . Go back to the Graphite Web Interface and reload it. The new storage scheme test will appear in the menu.
4 . Set the time range to a few minutes, by clicking on the clock-icon:
5 . The values that you have sent will appear in the graph:
6 . Wait for 15 minutes and refresh the graph by setting the time frame to the past 15 minutes. The graph will look different:
This is because our first collection does not store data for 15 minutes and Graphite will look into the second collection for rendering the graph.
As the data was sent with a count metric, Graphite adds up the values in the larger intervals instead of averaging them. It is therefore essential to choose the right metric for each use case.
Note: Pushing Content from a terminal is not the usual way to send data to Graphite. Instead, you will use a tool to automatize the collection of data. A complete list of tools that work with Graphite is available in the official documentation.