Grafana Loki Log Persistance Configuration

This document help us in enabling persistent logs in Grafana. This setup is required when you want to store logs for longer duration for analysis & reference purpose.

The current Bahmni setup is using below tools for Monitoring & Observability:

1. Monitoring & Alerting: - Promethus, Grafana
2. Centralised Logging:- Loki Stack consisting of Promtail, Loki and Grafana

Grafana Loki needs to store two different types of data: chunks and indexes. Loki receives logs in separate streams, where each stream is uniquely identified by its tenant ID and its set of labels. As log entries from a stream arrive, they are compressed as “chunks” and saved in the chunks store. The index stores each stream’s label set and links them to the individual chunks.

https://grafana.com/docs/loki/latest/configuration/ is the detail example available on Grafana page for Loki configurations i.e. loki.yaml

Supported Stores:- Grafana supports wide variety of storage to setup log retention. A documented list is available on Grafana docs page as well.

Grafana Loki Storage Retention:- Retention in Grafana Loki is achieved either through the Table Manager or the Compactor.

Table of Contents

Log retention is needed whenever we want to do some detail analysis or debug an issue. Depends on the requirements log retention can be setup. We can setup for 1 day, 7 days, 15 days, 1 month or for a year depends how much budget is available for storage. With respect to Bahmni, we have implemented persistent logs configurations inside our helm-charts. All the implementation is automated & is executed using github workflows. Bahmni is using Compactor storage setup. In Bahmni, we are using below configurations to setup our log retention:

• Pre defined helm charts. https://artifacthub.io/packages/helm/grafana/loki-stack
• Storage: boltdb-shipper
• Storage size: 10 Gi
• Retention duration: 48 hrs

For detailed configuration refer this Grafana documentation. This explains all the supported contents and default values.

Below are the snippets of Bahmni configuration which is used for K8s & Docker-compose setup.

1. Change retention_period value under limits_config section

CHANGE THIS VALUE AS PER REQUIREMENT enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 168h max_entries_limit_per_query: 5000]]>Then restart loki service by running docker compose restart loki

limits_config:
      retention_period: 24h   --->CHANGE THIS VALUE AS PER REQUIREMENT
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_entries_limit_per_query: 5000

NOTE: Please contact Infra squad (Slack: #bahmni-infra) if you are facing any issues while accessing Grafana dashboards.