Setting Up Prometheus and Loki on Kubernetes with Longhorn Storage

Monitoring is a critical component of any Kubernetes environment. It's important not only to understand how your applications are performing but also to have the ability to troubleshoot issues as they arise. Two of the most popular monitoring tools in the Kubernetes ecosystem are Prometheus, which is used for metrics collection and alerting, and Loki, which is a log aggregation system. This blog post will guide you through the process of installing both Prometheus and Loki using Helm, with the added reliability of Longhorn persistent storage.

Before we begin, it's essential to have Longhorn installed on your Kubernetes cluster. Longhorn is a distributed block storage system for Kubernetes that is simple to deploy and upgrade, but highly available, with excellent performance. It also provides features like backup and restore. Make sure that Longhorn is ready to go because we will be using it to provision persistent storage for both Prometheus and Loki.

Installing Prometheus with Longhorn

Prometheus is an open-source monitoring system that offers a multi-dimensional data model with time series data identified by metric name and key/value pairs. Let's install Prometheus using the kube-prometheus-stack Helm chart and configure it to use Longhorn for storage.

Here's the command you'll need to run:

helm install prometheusstack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName="longhorn" \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.accessModes[0]="ReadWriteOnce" \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage="50Gi"

This command sets up Prometheus in the monitoring namespace, creates the namespace if it doesn't already exist, and configures the persistent storage to use Longhorn with 50Gi of space.

Installing Loki with Longhorn

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

To install Loki and configure it to collect logs even from the master nodes, use the following command:

helm upgrade --install loki grafana/loki-stack \
  --namespace monitoring \
  --set loki.persistence.enabled=true \
  --set loki.persistence.storageClassName=longhorn \
  --set loki.persistence.size=20Gi \
  --set 'promtail.tolerations[0].key=CriticalAddonsOnly' \
  --set 'promtail.tolerations[0].operator=Exists' \
  --set 'promtail.tolerations[0].effect=NoExecute' \
  --set 'promtail.tolerations[1].key=node-role.kubernetes.io/control-plane' \
  --set 'promtail.tolerations[1].operator=Exists' \
  --set 'promtail.tolerations[1].effect=NoSchedule'

This command ensures that Loki is set up with persistent storage backed by Longhorn, with 20Gi allocated for log storage. The tolerations are configured so that Promtail, the agent that ships logs to Loki, can run on master nodes despite their taints.

By following these steps and using the commands provided, you will have a robust monitoring setup on your Kubernetes cluster that utilizes the powerful features of Longhorn for storage. Happy monitoring!