Pushing metrics and logs from a Scaleway Instance

Reviewed on 11 April 2024 • Published on 28 June 2023

cockpit
observability
metrics
logs
instance

This page shows you how to push your metrics and logs from a Scaleway Instance. You can also use this tutorial to push metrics and logs from external Instances (i.e. Instances outside of Scaleway).

Cockpit dashboard updates

Starting April 2024, a new version of Cockpit will be released.

In this version, the concept of regionalization will be introduced to offer you more flexibility and resilience for seamless monitoring. If you have created customized dashboards with data for your Scaleway resources before April 2024, you will need to update your queries in Grafana, with the new regionalized data sources.

Before you start

To complete the actions presented below, you must have:

A Scaleway account logged into the console
Owner status or IAM permissions allowing you to perform actions in the intended Organization

Important

Having the default configuration on your agents might lead to more of your resources’ metrics being sent, a high consumption, and a high bill at the end of the month. Find out how to avoid additional costs when using Cockpit.
Sending metrics and logs for Scaleway resources or personal data using an external path is a billable feature. In addition, any data that you push yourself is billed, even if you send data from Scaleway products. Refer to the product pricing for more information.

Configure your environment

Create an Instance from the Compute section of the Scaleway console.
Copy your Instance’s public IP as you will need it in the next steps.

Open a terminal and type the following command to connect to your Instance:

ssh root@<your Instance public IP>

Clone the GitHub repository

We have created a GitHub repository that contains all the code you need to configure the Grafana agent, to push your data.

Files in the directory

The tuto-cockpit-instance directory contains a workshop folder with several files inside of it

download.sh is a script responsible for downloading the necessary binaries and tools you need for the monitoring setup. It fetches the Grafana agent, for example.
agent.yaml is the Grafana agent configuration file responsible for collecting metrics and logs from your data sources, and pushing them to your Cockpit.
docker-compose.yaml is the Docker compose configuration file
setup.sh ensures that all dependencies required for monitoring are properly installed and configured
nginx.conf specifies server settings
run.sh executes the Grafana agent with the specified configuration file (agent.yaml).

The only file we will be editing in this tutorial is the agent.yaml file to add your Cockpit token.

Open another terminal and type the following command to clone our tuto-cockpit-instance directory on your local machine:

git clone git@github.com:scaleway/tuto-cockpit-instance.git

In the same terminal, run the following command to send the tuto-cockpit-instance directory and its content into your Instance:

scp -r tuto-cockpit-instance root@<your Instance public IP>:~/

Make sure the tuto-cockpit-instance directory is inside your Instance with the following command:

ls

Access the workshop folder inside tuto-cockpit-instance, using the following command:

cd tuto-cockpit-instance/workshop

Install Golang on your Instance using the following commands:
```
wget https://github.com/grafana/agent/releases/download/v0.34.3/grafana-agent-0.34.3-1.amd64.deb
sudo tar -C /usr/local/ -xzf go1.20.linux-amd64.tar.gz
```

Run the following command to set Golang in your environment:

export PATH=$PATH:/usr/local/go/bin

Run the following command to install docker. Docker allows you to run the docker-compose.yaml file.

apt install docker.io

Run the following command to unzip the files in the workshop folder:

sudo apt-get install zip unzip

Configure the Grafana agent file and push your data

Generate a token and select the push permissions for metrics and logs. Make sure you note down your token’s secret key as you will need it in the following steps.

In the same terminal where you are connected to your Instance, run the following command to download the Grafana agent:

bash ./download.sh

Run vim agent.yaml to access the Grafana agent configuration file.
Type i to edit the file.
Add your token’s secret key next to X-Token in the metrics section.
Add your token’s secret key next to tenant_id in the logs section.
Press escape, :wq and Enter to save and exit the file.

Run the following command to launch the Grafana agent:

bash ./run.sh

Visualize your metrics and logs

Log in to the Scaleway console and click Cockpit in the side menu.
Click Open dashboards at the top right corner of your screen.
Log in to Grafana using your credentials.
Click the downward pointing arrow at the top right corner of your screen and select Import dashboard.
Paste the following dashboard ID 12486 in the Import via grafana.com section and click Load. Alternatively, download the dashboard’s JSON template and upload it in the Import dasboard section.
Click Import.
Configure your dashboard as you please.
Select Scaleway metrics in the drop-down box and click Import.

Important
Select Metrics if you are using external data.

Your metrics should display in the dashboard.
Click Toggle menu and Explore.

Important
Select Logs if you are using external data.
Select the labels you want to apply in the Label filters section.
Click Run query at the top right corner of your screen. Your logs should display.