Monitor your Kubernetes cluster with Grafana

Monitor your Kubernetes cluster with Grafana - Overview

When using a managed Kubernetes cluster, you may want to know what is going on inside it. With Grafana and its custom dashboards, you can monitor the CPU/Memory/Disk/Network usage of all your running pods and nodes, and dozens of other metrics!

We are using a monitoring tool called Prometheus, that gathers data about your cluster to achieve this.

Requirements:

Getting Started

1 . Ensure you are connected to your cluster with kubectl and helm.

2 . Set the RBAC authorization on your cluster and setup helm:

kubectl --namespace kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller --upgrade

3 . Start the prometheus-operator. More info about how it works is available in the official documentation.

helm repo update
helm install stable/prometheus-operator --name prometheus-operator --namespace monitoring

4 . Configure a port-forward for the prometheus GUI:

kubectl port-forward -n monitoring prometheus-prometheus-operator-prometheus-0 9090

Important: Keep this port-forward running in the background as Grafana will get its data from it.

5 . Open a web browser and access http://localhost:9090. You should see the Prometheus GUI. You can browse all metrics and see them in real-time.

Prometheus GUI

Installing Grafana

Refer to the Grafana docs for detailed information on how to install Grafana on your machine and OS.
The steps for Windows, MacOS X and Ubuntu/Debian Linux are listed below.

Steps for Windows

1 . Download the Grafana installer on the Grafana website.

2 . Important: Once you have downloaded the .zip file and before extracting it, make sure to open the properties (right-clicking on Properties) and select the unblock checkbox then Ok.

3 . The zip file contains a folder with the current Grafana version. Extract this folder to anywhere you want Grafana to run from. Go into the conf directory and copy sample.ini to custom.ini. You should edit custom.ini, never defaults.ini.

4 . The default Grafana port is 3000, this port requires extra permissions on windows. Edit custom.ini and uncomment the http_port configuration option (; is the comment character in ini files) and change it to something like 8080 or similar. That port should not require extra Windows privileges.

5 . Start Grafana by executing grafana-server.exe, located in the bin directory, preferably from the command line. If you want to run Grafana as windows service, download NSSM.

Steps for MacOS X

1 . Verify that Homebrew packet manager is installed on your local computer by typing the following command in a terminal window.

brew --version

If Homebrew is installed, the version information displays. If the packet manager is not installed, install it by running the following command:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2 . Update the packet sources and install Grafana using brew:

brew update
brew install grafana

3 . Grafana uses Homebrew Services. Before starting Grafana, make sure homebrew/services is installed.

brew tap homebrew/services

4 . Then start Grafana using the following command:

brew services start grafana

Steps for Ubuntu/Debian Linux

1 . Install the prerequisites via the apt packet manager:

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget

2 . Download the Grafana GPG key and add it to the system:

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

3 . Add the Grafana repository to the apt packet manager:

sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"

4 . Update apt and install Grafana:

sudo apt-get update
sudo apt-get install grafana

Accessing the Grafana GUI

In your web browser open the URL http://localhost:8080. You should see the Grafana GUI.

The default login is admin / admin. If you log in for the first time, Grafana may ask you to set a custom password. You can set up what you like and continue.

Setting up a Data Source

Grafana itself is just a display tool and without data it will only produce empty graphs. The data and metrics collecting is done by Prometheus; now we need to connect them together.

So let’s add Prometheus as the default data source of Grafana:

1 . Click on the Grafana logo to open the sidebar menu.
2 . Click on Data Sources in the sidebar.
> You can also go directly to http://localhost:8080/datasources

3 . Click on Add New.
4 . Select Prometheus as the type.
5 . Set the appropriate Prometheus server URL (in our case, http://localhost:9090/ from the port-forwarding) 6 . Adjust other data source settings as desired (for example, turning the proxy access off).
7 . Check the Default box
8 . Click Save & Test to save the new data source.

Adding a new Prometheus data source

Setting up the Dashboard

Now that the data source is configured, you can start to create dashboards to visualize it. You can configure your dashboard yourself, but there are a few already avaiable publically on grafana.com/dashboards.

You can find the one used in this tutorial here. If you want to use it, follow these steps:

1 . On the left menu, click on + then Import.
2 . In the field Grafana.com Dashboard, paste the id of the dashboard you found on grafana.com/dashboards.
3 . Click on Load.
4 . Give it a name, a uid and select the data source.
5 . Click on Import.
6 . You are now able to visualize your graphs.

Grafana GUI with custom dashboard

Exporting Data as CSV

If you want to do some analysis on your data, you can export each graph’s data into CSV format:

1 . On the graph to export, click the v button on the title.
2 . Then go to More… > Export CSV

Export CSV

Deleting Prometheus

Note: All data gathered since you installed Prometheus will be lost, be sure you want to do this!

If you want to stop gathering metrics, you can stop your local Grafana server, and remove Prometheus from your cluster by following these steps:

# Remove all prometheus
helm delete prometheus-operator --purge

# Remove all remaining resource definitions
kubectl delete --ignore-not-found customresourcedefinitions \
  prometheuses.monitoring.coreos.com \
  servicemonitors.monitoring.coreos.com \
  alertmanagers.monitoring.coreos.com \
  prometheusrules.monitoring.coreos.com

If you want to avoid the hassle of maintaining a Kubernetes cluster, Scaleway got you covered! Try out our managed Kubernetes service today and deploy your k8s cluster with a few clicks.

Discover a New Cloud Experience

Deploy SSD Cloud Servers in seconds.