Jump toUpdate content
Configuring a Prometheus monitoring server with a Grafana dashboard
This tutorial explains how to use a Prometheus Monitoring server with a Grafana dashboard.
Prometheus is a flexible monitoring solution that is in development since 2012. The software stores all its data in a time series database and offers a multi-dimensional data-model and a powerful query language to generate reports of the monitored resources.
Run the following command to create a user for Prometheus and Node Exporter. In this tutorial, we will create it without the login option.
To achieve this, we use the parameter
--no-create-homewhich skips the creation of a home directory and disable the shell with
sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus
sudo useradd --no-create-home --shell /bin/false node_exporter
Create the folders required to store the binaries of Prometheus and its configuration files.
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
Set the ownership of these directories to our
prometheususer, to make sure that Prometheus can access to these folders.
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
As your Prometheus is only capable of collecting metrics, we want to extend its capabilities by adding Node Exporter, a tool that collects information about the system including CPU, disk, and memory usage and exposes them for scraping.
Download the latest version of Node Exporter:
Unpack the downloaded archive. This will create a directory
node_exporter-1.3.1.linux-amd64, containing the executable, a readme and license file:
tar xvf node_exporter-1.3.1.linux-amd64.tar.gz
Copy the binary file into the directory
/usr/local/binand set the ownership to the user you have created in step previously:
sudo cp node_exporter-1.3.1.linux-amd64/node_exporter /usr/local/bin
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
Remove the leftover files of Node Exporter, as they are not needed any longer:
rm -rf node_exporter-1.3.1.linux-amd64.tar.gz node_exporter-1.3.1.linux-amd64
To run Node Exporter automatically on each boot, a Systemd service file is required. Create the following file by opening it in Nano:
sudo nano /etc/systemd/system/node_exporter.service
Copy the following information in the service file, save it and exit
Collectors are used to gather information about the system. By default, a set of collectors is activated. You can see the details about the set in the README file. If you want to use a specific set of collectors, you can define them in the
ExecStartsection of the service. Collectors are enabled by providing a
--collector.<name>flag. Collectors that are enabled by default can be disabled by providing a
Reload Systemd to use the newly defined service.
sudo systemctl daemon-reload
Run Node Exporter by typing the following command.
sudo systemctl start node_exporter.service
Verify that the software started successfully.
sudo systemctl status node_exporter.service
You will see an output like the following, showing you the status
active (running)as well as the main PID of the application:
● node_exporter.service - Node Exporter
Loaded: loaded (/etc/systemd/system/node_exporter.service; disabled; vendo>
Active: active (running) since Tue 2022-03-01 13:10:40 UTC; 11s ago
Main PID: 21976 (node_exporter)
Tasks: 4 (limit: 2335)
If everything is working, enable Node Exporter to start upon boot.
sudo systemctl enable node_exporter.service
Download and unpack Prometheus latest release of Prometheus. In this tutorial, we use version 2.34.0-rc.1.
sudo apt-get update && apt-get upgrade
tar xfz prometheus-*.tar.gz
The following two binaries are in the directory:
- prometheus - Prometheus’ main binary file
The following two folders (which contain the web interface, configuration files examples and the license) are in the directory:
Copy the binary files into the
sudo cp ./prometheus /usr/local/bin/
sudo cp ./promtool /usr/local/bin/
Set the ownership of these files to the
prometheususer previously created:
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo cp -r ./consoles /etc/prometheus
sudo cp -r ./console_libraries /etc/prometheus
Set the ownership of the two folders, as well as of all files that they contain, to our
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
In our home folder, remove the source files that are not needed anymore:
cd .. && rm -rf prometheus-\*
Prior to using Prometheus, it needs basic configuring. Thus, we need to create a configuration file named
The configuration file of Prometheus is written in YAML which strictly forbids to use tabs. If your file is incorrectly formatted, Prometheus will not start. Be careful when you edit it.
Open the file
prometheus.ymlin a text editor:
sudo nano /etc/prometheus/prometheus.yml
Prometheus’ configuration file is divided into three parts:
globalpart we can find the general configuration of Prometheus:
scrape_intervaldefines how often Prometheus scrapes targets,
evaluation_intervalcontrols how often the software will evaluate rules. Rules are used to create new time series and for the generation of alerts.
rule_filesblock contains information of the location of any rules we want the Prometheus server to load.
The last block of the configuration file is named
scape_configsand contains the information which resources Prometheus monitors.
Our file should look like this example:
# - "first.rules"
# - "second.rules"
- job_name: 'prometheus'
- targets: ['localhost:9090']
scrape_intervalis set to 15 seconds which is enough for most use cases.
We do not have any
rule_filesyet, so the lines are commented out and start with a
scrape_configspart we have defined our first exporter. It is Prometheus that monitors itself. As we want to have more precise information about the state of our Prometheus server we reduced the
scrape_intervalto 5 seconds for this job. The parameters
targetsdetermine where the exporters are running. In our case it is the same server, so we use
localhostand the port
As Prometheus scrapes only exporters that are defined in the
scrape_configspart of the configuration file, we have to add Node Exporter to the file, as we did for Prometheus itself.
We add the following part below the configuration for scrapping Prometheus:
- job_name: 'node_exporter'
- targets: ['localhost:9100']
Overwrite the global scrape interval again and set it to 5 seconds. As we are scarping the data from the same server as Prometheus is running on, we can use
localhostwith the default port of Node Exporter:
If you want to scrape data from a remote host, you have to replace
localhostwith the IP address of the remote server.Tip:
For all information about the configuration of Prometheus, you may check the configuration documentation.
Set the ownership of the file to our
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
Our Prometheus server is ready to run for the first time.
Start Prometheus directly from the command line with the following command, which executes the binary file as our
sudo -u prometheus /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries
The server starts displaying multiple status messages and the information that the server has started:
level=info ts=2022-03-01T13:16:52.918475753Z caller=main.go:222 msg="Starting Prometheus" version="(version=2.3.1, branch=HEAD, revision=188ca45bd85ce843071e768d855722a9d9dabe03)"
level=info ts=2022-03-01T13:16:52.918772613Z caller=main.go:223 build_context="(go=go1.10.3, user=root@82ef94f1b8f7, date=20180619-15:56:22)"
level=info ts=2022-03-01T13:16:52.918893461Z caller=main.go:224 host_details="(Linux 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 scw-elegant-greider (none))"
level=info ts=2022-03-01T13:16:52.919003688Z caller=main.go:225 fd_limits="(soft=1024, hard=1048576)"
level=info ts=2022-03-01T13:16:52.922302228Z caller=main.go:514 msg="Starting TSDB ..."
level=info ts=2022-03-01T13:16:52.922560093Z caller=web.go:415 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2022-03-01T13:16:52.949006143Z caller=main.go:524 msg="TSDB started"
level=info ts=2022-03-01T13:16:52.949114617Z caller=main.go:603 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2022-03-01T13:16:52.951124437Z caller=main.go:500 msg="Server is ready to receive web requests."
Open your browser and type
http://IP.OF.YOUR.SERVER:9090to access the Prometheus interface. If everything is working, we end the task by pressing on
CTRL + Con our keyboard.Note:
If you get an error message when you start the server, double check your configuration file for possible YAML syntax errors. The error message will tell you what to check.
The server is working now, but it cannot yet be launched automatically at boot. To achieve this, we have to create a new
systemdconfiguration file that will tell your OS which services should it launch automatically during the boot process.
sudo nano /etc/systemd/system/prometheus.service
The service file tells
systemdto run Prometheus as
prometheusand specifies the path of the configuration files.
Copy the following information in the file and save it, then exit the editor:
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
ExecReload=/bin/kill -HUP $MAINPID
To use the new service, reload
sudo systemctl daemon-reload
We enable the service so that it will be loaded automatically during boot:
sudo systemctl enable prometheus.service
sudo systemctl start prometheus.service
Your Prometheus server is ready to be used.
We have now installed Prometheus to monitor your instance.
Prometheus provides a basic web server running on
http://your.server.ip:9090 that provide access to the data collected by the software.
We can verify the status of our Prometheus server from the interface:
You can perform queries with the collected data.
The interface we set up is lightweight. The Prometheus team, therefore, recommends using a tool like Grafana if you want to do more than testing and debugging the installation.
- Install Grafana on our Instance which queries our Prometheus server.
sudo apt-get install -y adduser libfontconfig1
sudo dpkg -i grafana_8.4.3_amd64.deb
- Enable the automatic start of Grafana by
sudo systemctl daemon-reload && sudo systemctl enable grafana-server.service && sudo systemctl start grafana-server.service
Grafana is running now, and we can connect to it at
http://your.server.ip:3000. The default user and password is
Now you have to create a Prometheus data source:
- Click the cogwheel icon on the sidebar
- Click Data Sources
- Choose Add data source
- Select Prometheus as the data source
- Set the Prometheus server URL (in our case: http://localhost:9090/)
- Click Save & test to test the connection and to save the new data source
Your settings should look like this:
You are now ready to create your first dashboard from the information collected by Prometheus. You can also import some dashboards from a collection of shared dashboards
Below is an example of a dashboard that uses the CPU usage of our node and presents it in Grafana: