In this tutorial you will learn how to forward the applicative logs and the usage metrics of your Kubernetes Kapsule containers into the Observability Cockpit.
This process will be done using Fluent Bit, a lightweight logs and metrics processor which acts as a gateway between containers and the Cockpit endpoints, when configured in a Kubernetes cluster.
Cockpit dashboard updates
Starting April 2024, a new version of Cockpit will be released.
In this version, the concept of regionalization will be introduced to offer you more flexibility and resilience for seamless monitoring. If you have created customized dashboards with data for your Scaleway resources before April 2024, you will need to update your queries in Grafana, with the new regionalized data sources.
To complete the actions presented below, you must have:
helm, the Kubernetes package manager, on your local machine (version 3.2+)Fluent Bit will be installed as a Helm package configured to target your Kubernetes resources as inputs and your Observability cockpit as an output.
Add the Helm repository for Fluent Bit to your machine:
Create a values file for Helm named values.yaml that we will use to configure Fluent Bit.
Create a first section config.service in the values.yaml file to configure the Fluent Bit master process:
Flush 1: Collects logs every second.Log_level info: Displays informational logs in the Fluent Bit pods.Daemon off: Run Fluent Bit as the foreground process in its pods.Parsers_File custom_parsers.conf: Loads additional log parsers that we will define later on.HTTP_Server on: Enables Fluent Bit’s built-in HTTP server.HTTP_Listen 0.0.0.0: Listen on all interfaces exposed by your pod.HTTP_PORT 2020: Listen to port 2020.You need to enable Fluent Bit’s HTTP server in order for it to communicate with your Cockpit.
We will configure Fluent Bit to retrieve the metrics (e.g.: CPU, memory, disk usage) from your Kubernetes nodes and the applicative logs from your running pods.
Create a new section config.inputs in the values.yaml file:
The first subsection adds an input to Fluent Bit to retrieve the usage metrics from your containers:
Name node_exporter_metrics: This input plugin is used to collect various system-level metrics from your nodes.Tag node_metrics: The Tag parameter assigns a tag to the incoming data from the node_exporter_metrics plugin. In this case, the tag node_metrics is assigned to the collected metrics.Scrape_interval 60: The frequency at which metrics are retrieved. Metrics are collected every 60 seconds.Increasing the scrape interval allows you to push fewer metrics samples per minute to your Cockpit and thus, pay less. For instance, if your application exposes 100 metrics every 60 seconds, these 100 metrics are collected and pushed to the server. If you configure your scrape interval to 1 second, you will push 6000 samples per minute.
The second subsection adds an input to Fluent Bit to retrieve the logs from your containers:
Name tail: The tail input plugin is used to read logs from files.Path /var/log/containers/*.log: The tail plugin read logs from /var/log/containers/*.log which are the log dumps from your containers.Parser docker: The Parser parameter specifies the parser to be used for parsing log records. The docker parser is a custom parser that will be defined below.Tag logs.*: The Tag parameter assigns a tag to the incoming data from the tail plugin. The tag “logs.*” indicates that the collected logs will have a tag prefix of “logs” followed by any additional subtag.The inputs collected by Fluent Bit should be structured before sending them to the Cockpit to enable further filtering and better visualisation.
Create a config.customParsers section to define the docker parser which is referenced by the log parsing input:
This parser expects log records in JSON format. It assumes that the timestamp information is located under the key “time” in the JSON log record, and the timestamp format is in ISO 8601 date format.
Define a section named config.filters in order to filter incoming log files from the containers:
This sets up a filter plugin which will be applied to log records with tags starting with logs.. It enables log merging, extracts and parses Kubernetes log metadata, and allows log exclusion based on Kubernetes log metadata filters.
Define a section named config.extraFiles.'labelmap.json':
This defines a map for various Kubernetes labels and metadata to specific Fluent Bit field names in order to parse and structure the logs.
The last step in the Fluent Bit configuration is to define where the logs and metrics will be pushed.
Create a token and select push permissions for both logs and metrics.
Create a section named config.outputs in the values.yaml file:
Fill in the blanks as follows:
Host from the first subsection: paste your Metrics API URL defined in the API and Tokens tab section from the Cockpit. Remove the https:// protocol.Header: Next to Bearer, paste the token generated in the previous step.Host from the second subsection: paste your Logs API URL defined in the API and Tokens tab section from the Cockpit. Remove the https:// protocol.Http_passwd: paste the token generated in the previous step.In the first subsection, the prometheus_remote_write plugin is used to send metrics to the Prometheus server of your Cockpit using the remote write protocol.
In the second subsection, the loki plugin is used to send logs to the Loki server of your Cockpit, using the field mapping from labelmap.json defined above.
Run the following command in the same directory as your values.yaml file to install Fluent Bit:
You should see a DeamonSet named fluent-bit with running pods on all of your nodes.
You can find the logs and metrics from your Kubernetes cluster in your Cockpit’s dashboard in Grafana.
Grafana has a built-in dashboard for visualizing node metrics.
Kapsule.
Your Kapsule logs index can be queried in the Explore section of your Cockpit’s dashboard in Grafana. In the data source selector, pick the Logs index. The Kubernetes labels are already mapped and can be used as filters in queries.