--- title: Monitoring Kubernetes clusters description: Monitoring of your Kubernetes Kapsule cluster tags: kubernetes cluster pvc dates: validation: 2025-07-01 posted: 2020-10-14 --- import Requirements from '@macros/iam/requirements.mdx' import image from './assets/scaleway_prom_dashboard.webp' import image2 from './assets/scaleway_prom2.webp' import image3 from './assets/scaleway_prom3.webp' import image4 from './assets/scaleway_prom5.webp' import image5 from './assets/scaleway_grafana_login.webp' import image6 from './assets/scaleway_grafana_home.webp' import image7 from './assets/scaleway_grafana_datasource.webp' import image8 from './assets/scaleway_grafana_import.webp' import image9 from './assets/scaleway_grafana_dash_datasource.webp' import image10 from './assets/scaleway_grafana_dashboard.webp' import image11 from './assets/scaleway_monito3.webp' import image12 from './assets/scaleway_loki1.webp' import image13 from './assets/scaleway_loki2.webp' import image14 from './assets/scaleway_loki3.webp' Kubernetes Kapsule is fully integrated with Scaleway's [Observability Cockpit](/cockpit/quickstart/). You can [monitor your cluster](/kubernetes/how-to/monitor-cluster/) directly from the cluster's dashboard, eliminating the need to set up your own monitoring solution. The following content is provided for informational purposes only. - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization - [Created](/kubernetes/how-to/create-cluster/) a Kubernetes Kapsule cluster - Installed and configured `kubectl`, the [command line interface](https://kubernetes.io/docs/reference/kubectl/overview/) for Kubernetes, on your local machine - Installed `helm`, the Kubernetes [package manager](https://helm.sh/), on your local machine (version 3.2+) ## Deploying Prometheus Deploy the Prometheus stack in a dedicated Kubernetes [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) called `monitoring`. Set the retention time for 30 days and create a persistent volume (based on [Scaleway Block Storage](https://www.scaleway.com/en/block-storage/)) to store the Prometheus data. 1. Ensure you are connected to your cluster and `kubectl` and `helm` are installed on your local machine. 2. Add the Prometheus Community Helm repo and the stable Kubernetes repo and update them: ```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo add stable https://kubernetes-charts.storage.googleapis.com/ helm repo update ``` 3. Use the `helm` package manager to install the stable release of Prometheus. Set the following parameters to `helm` for both of these: * `server.persistentVolume`: 100Gi * `server.retention`: 30d ```bash helm install prometheus prometheus-community/prometheus --create-namespace --namespace monitoring --set server.persistentVolume.size=100Gi,server.retention=30d NAME: prometheus LAST DEPLOYED: Fri Oct 9 16:35:50 2020 NAMESPACE: monitoring STATUS: DEPLOYED [..] ``` 4. Verify that the created Pods are all running once the stack is deployed. You can also check whether the 100Gi block volume was created: ```bash kubectl get pods,pv,pvc -n monitoring NAME READY STATUS RESTARTS AGE pod/prometheus-alertmanager-6565668c85-5vdxc 2/2 Running 0 67s pod/prometheus-kube-state-metrics-6756bbbb8-6qs9r 1/1 Running 0 67s pod/prometheus-node-exporter-fbg6s 1/1 Running 0 67s pod/prometheus-pushgateway-6d75c59b7b-6knfd 1/1 Running 0 67s pod/prometheus-server-556dbfdfb5-rx6nl 1/2 Running 0 67s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-5a9def3b-22a1-4545-9adb-72823b899c36 100Gi RWO Delete Bound monitoring/prometheus-server scw-bssd 67s persistentvolume/pvc-c5e24d9b-3a69-46c1-9120-b16b7adf73e9 2Gi RWO Delete Bound monitoring/prometheus-alertmanager scw-bssd 67s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/prometheus-alertmanager Bound pvc-c5e24d9b-3a69-46c1-9120-b16b7adf73e9 2Gi RWO scw-bssd 68s persistentvolumeclaim/prometheus-server Bound pvc-5a9def3b-22a1-4545-9adb-72823b899c36 100Gi RWO scw-bssd 68s ``` 5. To access Prometheus use the Kubernetes [port forwarding](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) feature: ```bash export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace monitoring port-forward $POD_NAME 9090 ``` 6. Access the Prometheus dashboard using the following URL: [http://localhost:9090](http://localhost:9090) 7. Verify that both `node-exporter` and `kube-state-metrics` metrics are correctly scrapped by Prometheus: * The `node-exporter` metrics begin with “`node_`” * The `kube-state-metrics` begin with “`kube_`” 8. Prometheus is capable of generating graphs on its own. You can test the feature by choosing metrics to analyze directly in the application: ## Deploying Grafana Deploy `Grafana` to display the `Prometheus` metrics in some pre-defined dashboards. Deploy it in the `monitoring` namespace and enable the persistence: * `persistence.enable` : true * `persistence.type` : pvc * `persistence.size` : 10Gi 1. Run the following command to add the Helm repository and install Grafana using `helm`: ```bash helm repo add grafana https://grafana.github.io/helm-charts helm install grafana grafana/grafana \ --set persistence.enabled=true,persistence.type=pvc,persistence.size=10Gi \ --namespace=monitoring ``` 2. Retrieve the admin password once Grafana is installed: ```bash kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` 3. Configure the port forwarding to access the Grafana web interface at this address: `http://localhost:3000` ```bash kubectl port-forward --namespace monitoring service/grafana 3000:80 Forwarding from 127.0.0.1:3000 -> 3000 Forwarding from [::1]:3000 -> 3000 ``` 4. Open the Grafana Web Interface in a web browser at: `http://localhost:3000`. The login screen displays. Enter the user `admin` and the password recovered in step 2: The welcome screen displays and invites you to complete the configuration of Grafana. 5. Click **Add data source** to configure a new data source: 6. Choose **Prometheus** as data source from the list of available options. 7. Enter the details of the data source. You can leave the default settings and add the data source: `http://prometheus-server`. Click **Test & Save** to validate the connection to Prometheus and to save the settings: 8. Click **+** and then **Import** to import a ready-to-use dashboard from the Grafana website. To create a dashboard that uses *kube-state-metrics*, import the dashboard number [8588](https://grafana.com/grafana/dashboards/8588) and get information about your Deployment, StatefulSet and DaemonSet: 9. Choose `Prometheus` as data source and click **Import**. 10. Access the dashboard with metrics for Deployment, StatefulSet and DaemonSet: 11. You can also configure additional dashboards, for example, the node exporter full dashboard ([1860](https://grafana.com/grafana/dashboards/1860)) to display a dashboard with system metrics for each Kubernetes node: ## Deploying Loki Loki is a log aggregation system inspired by [Prometheus](https://prometheus.io/). The `loki` application is not included in the default Helm repositories. Since December 2020, Loki's Helm charts were moved from their initial location within the Loki repo to their new location at https://github.com/grafana/helm-charts which is hosted in https://grafana.github.io/helm-charts. 1. Add the Grafana repository to Helm and update it. ```bash helm repo add grafana https://grafana.github.io/helm-charts helm repo update ``` Which returns ``` "grafana" has been added to your repositories Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "loki" chart repository ...Successfully got an update from the "grafana" chart repository Update Complete. ⎈Happy Helming!⎈ ``` 2. Install the `loki-stack` with Helm. Install all the stack in a Kubernetes dedicated [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) named `loki-stack`. Deploy it to your cluster and enable persistence (allow Helm to create a Scaleway block device and attach it to the Loki Pod to store its data) using a Kubernetes [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) to survive a Pod re-schedule: ```bash helm install loki-stack grafana/loki-stack \ --create-namespace \ --namespace loki-stack \ --set promtail.enabled=true,loki.persistence.enabled=true,loki.persistence.size=100Gi ``` It will use Kapsule's default storage class, `scw-bsdd`, to create block volumes using Scaleway Block Storage. If you plan to use Loki on a production system, ensure that you set up a retention period to avoid filling the file systems. For instance, use these parameters if you want to enable a 30-day retention (logs older than 30 days will be deleted). Note that you have to choose the size of the persistent volume to fit the amount of volume your deployment will create. * `config.table_manager.retention_deletes_enabled` : true * `config.table_manager.retention_period`: 720h 3. Install Grafana in the loki-stack namespace with Helm. To allow Grafana to survive a re-schedule, enable persistence too: * `persistence.enabled`: true * `persistence.type`: pvc * `persistence.size`: 10Gi ```bash helm install loki-grafana grafana/grafana \ --set persistence.enabled=true,persistence.type=pvc,persistence.size=10Gi \ --namespace=loki-stack ``` 4. Check if the block devices were correctly created by Kubernetes: ```bash kubectl get pv,pvc -n loki-stack NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-88038939-24a5-4383-abe8-f3aab97b7ce7 10Gi RWO Delete Bound loki-stack/loki-grafana scw-bssd 18s persistentvolume/pvc-c6fce993-a73d-4423-9464-7c10ab009062 100Gi RWO Delete Bound loki-stack/storage-loki-stack-0 scw-bssd 4m30s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/loki-grafana Bound pvc-88038939-24a5-4383-abe8-f3aab97b7ce7 10Gi RWO scw-bssd 19s persistentvolumeclaim/storage-loki-stack-0 Bound pvc-c6fce993-a73d-4423-9464-7c10ab009062 100Gi RWO scw-bssd 5m3s ``` 5. Now that both Loki and Grafana are installed in the cluster, check if the Pods are correctly running: ```bash kubectl get pods -n loki-stack NAME READY STATUS RESTARTS AGE loki-grafana-67994589cc-7jq4t 0/1 Running 0 74s loki-stack-0 1/1 Running 0 5m58s loki-stack-promtail-dtf5v 1/1 Running 0 5m42s ``` 6. To be able to connect to Grafana get the admin password: ```bash kubectl get secret --namespace loki-stack loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` 7. Configure a `port-forward` to reach Grafana from your web browser: ```bash kubectl port-forward --namespace loki-stack service/loki-grafana 3000:80 Forwarding from 127.0.0.1:3000 -> 3000 Forwarding from [::1]:3000 -> 3000 ``` 8. Access [http://localhost:3000](http://localhost:3000) to reach the Grafana interface. Log in using the admin user and the password you got above. 9. Add the Loki source to Grafana (`http://loki-stack.loki-stack:3100`). 10. Check you can access your logs using the explore tab in Grafana: You now have a Loki stack up and running. All your Pods’ logs will be stored in Loki and you will be able to view and query your applications’ logs in Grafana. Refer to the [Loki documentation](https://grafana.com/docs/features/datasources/loki/), if you want to learn more about querying the Loki data source.