{"id":21660811,"url":"https://github.com/mohammadll/grafana-stack","last_synced_at":"2025-04-11T22:43:09.726Z","repository":{"id":215786145,"uuid":"736247446","full_name":"mohammadll/grafana-stack","owner":"mohammadll","description":"We're going to provide various examples to set up and work with the Grafana LGTM stack including Loki, Grafana, Tempo and Mimir on a kubernetes cluster","archived":false,"fork":false,"pushed_at":"2024-08-23T14:32:01.000Z","size":149,"stargazers_count":25,"open_issues_count":0,"forks_count":8,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-11T22:43:06.302Z","etag":null,"topics":["grafana","grafana-agent","grafana-alloy","grafana-loki","grafana-mimir","grafana-tempo"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mohammadll.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-27T11:35:26.000Z","updated_at":"2025-04-06T03:39:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"fbae1de5-b717-4fc9-a8af-8c7c5f8bc38b","html_url":"https://github.com/mohammadll/grafana-stack","commit_stats":null,"previous_names":["mohammadll/grafana-stack"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mohammadll%2Fgrafana-stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mohammadll%2Fgrafana-stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mohammadll%2Fgrafana-stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mohammadll%2Fgrafana-stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mohammadll","download_url":"https://codeload.github.com/mohammadll/grafana-stack/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248492990,"owners_count":21113159,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grafana","grafana-agent","grafana-alloy","grafana-loki","grafana-mimir","grafana-tempo"],"created_at":"2024-11-25T09:38:56.519Z","updated_at":"2025-04-11T22:43:09.704Z","avatar_url":"https://github.com/mohammadll.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src='grafana-lgtm.png' height=\"480\"\u003e\n\n## 💡 Description\nThe Grafana LGTM stack is a comprehensive set of open-source tools designed for monitoring, observability, and visualization. It includes several key components, each serving a specific purpose to provide a complete solution for monitoring applications and infrastructure. If you are interested in setting up a Grafana LGTM stack and working with it on a kubernetes clsuter, please follow the instructions provided in this repository.\n\n## :wrench: What tools do we want to use in this repository\n  - **Loki**: a log aggregation system designed to store and query logs from all your applications and infrastructure. https://grafana.com/oss/loki/\n  - **Grafana**: allows you to query, visualize, alert on and understand your metrics no matter where they are stored. https://grafana.com/oss/grafana/\n  - **Tempo**: an open source, easy-to-use, and high-scale distributed tracing backend. https://grafana.com/oss/tempo/\n  - **Mimir**: an open source, horizontally scalable, highly available, multi-tenant TSDB for long-term storage for Prometheus. https://grafana.com/oss/mimir/\n  - **Agent**: is a batteries-included, open source telemetry collector for collecting metrics, logs, and traces. https://grafana.com/oss/agent/\n\n## 🔎 What do we want to do\n  - **Traces with Tempo**:\n     - We want to write a simple Python app, collect its spans and traces using the `OpenTelemetry Kubernetes operator`, forward these traces to `Tempo`, and finally visualize the traces using `Grafana`.\n  - **Metrics with Mimir and Logs with Loki**:\n     - Using the `agent operator` to collect kubelet and cAdvisor metrics exposed by the kubelet service. Each node in your cluster exposes kubelet metrics at /metrics and cAdvisor metrics at /metrics/cadvisor. Finally, remotely writing these metrics to `Mimir` and visualizing them with `Grafana`\n     - Using the `agent operator` to collect logs from the cluster nodes, shipping them to a remote `Loki` endpoint and visualizing them with `Grafana`.\n\n\n## :one: Scenario-1: Traces with Tempo\n\nCreate a unique Kubernetes namespace for tempo and grafana:\n\n    kubectl create namespace tempo\n    kubectl create namespace grafana\n\nSet up a Helm repository using the following commands:\n\n    helm repo add grafana https://grafana.github.io/helm-charts\n    helm repo update\n\nUse the following command to install Tempo using the configuration options we’ve specified in the `helm-values/tempo-values.yml` file:\n\n    helm -n tempo install tempo grafana/tempo-distributed -f helm-values/tempo-values.yml\n\nUse the following command to install Grafana using the configuration options we’ve specified in the `helm-values/grafana-values-scenario-1.yml` file:\n\n    helm -n grafana install grafana grafana/grafana -f helm-values/grafana-values-scenario-1.yml\n\nTo install the Opentelemetry Operator in an existing cluster, make sure you have cert-manager installed and run:\n\n    kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml\n\nCreate an OpenTelemetry `Collector` and an `Instrumentation` resource with the configuration for the SDK and instrumentation.\n\n    kubectl apply -f open-telemetry-operator/colllector.yml\n    kubectl apply -f open-telemetry-operator/instrument.yml\n\nGo to `trace-python-app` directory and build the docker image using this command: `docker build -t myapp .` and then:\n\n    kubectl apply -f trace-python-app-manifests/deployment.yml\n    kubectl apply -f trace-python-app-manifests/service.yml\n\nUse `port-forward` to connect to `my-app`, like this: `kubectl port-forward POD_NAME 8080` and then Go to your browser and type:\n\n    http://localhost:8080/rolldice\n\nRefresh the page multiple times. if you want to see the traces, use `port-forward` to connect to `grafana` and Go to `Explore`. You'll see the traces of my-app\n\n\n## :two: Scenario-2: Metrics with Mimir and Logs with Loki, Collecting Metrics and Logs using Grafana Agent\n\nWe're going to use grafana/agent operator to collect metrics and logs from the kubernetes cluster and forward metrics to `Mimir` and pod logs to `loki` . Let's deploy an Nginx deployment with its related service and collect its logs using Grafana Agent. As you know, the log format of Nginx includes HTTP Method and Status Code, etc. Therefore, we will use pipelines in our Grafana Agent configuration to extract the HTTP Method and Status Code from Nginx access logs and add them as additional label to Loki, showcasing the power of Grafana Agent . Ready ? let's GO!\n\nFirst of all, you need to seutp MiniO as an external object storage to store loki index and chunks, check the `helm-values/loki-values.yml` for the `loki.storage` section to find out how we can integrate MiniO with Loki. We also need to stote `mimir tsdb blocks` inside minio. I highly recommend you to check both `helm-values/loki-values.yml` and `helm-values/mimir-values.yml` because you need to replace some existing data with your own appropriate data.\n\nInstall the charts of agent-operator, loki, mimir, grafana as well as the nginx manifests\n\n    helm install collector grafana/grafana-agent-operator\n    helm install -n loki loki grafana/loki -f helm-values/loki-values.yml\n    helm install -n mimir mimir grafana/mimir-distributed -f helm-values/mimir-values.yml\n    helm install -n grafana grafana grafana/grafana -f helm-values/grafana-values-scenario-2.yml\n    kubectl apply -f nginx/\n\nDeploy the GrafanaAgent resource, The root of the custom resource hierarchy is the `GrafanaAgent` resource—the primary resource Agent Operator looks for. `GrafanaAgent` is called the root because it discovers other sub-resources, `MetricsInstance` and `LogsInstance`\n\n    kubectl apply -f agent-operator/grafana-agent.yml\n\nDeploy a MetricsInstance resource, Defines where to ship collected metrics. This rolls out a Grafana Agent StatefulSet that will scrape and ship metrics to a remote_write endpoint.\n\n    kubectl apply -f agent-operator/metric-instance.yml\n\nCreate ServiceMonitors for kubelet and cAdvisor endpoints, Collects cAdvisor and kubelet metrics. This configures the MetricsInstance / Agent StatefulSet\n\n    kubectl apply -f agent-operator/kubelet-svc-monitor.yml\n    kubectl apply -f agent-operator/cadvisor-svc-monitor.yml\n\nDeploy LogsInstance resource, Defines where to ship collected logs. This rolls out a Grafana Agent DaemonSet that will tail log files on your cluster nodes.\n\n    kubectl apply -f agent-operator/log-instance.yml\n\nDeploy PodLogs resource, Collects container logs from Kubernetes Pods. This configures the LogsInstance / Agent DaemonSet.\n\n    kubectl apply -f agent-operator/pod-logs.yml\nThis example (`agent-operator/pod-logs.yml`) tails container logs for all Pods in the `nginx` namespace. You can restrict the set of matched Pods by using the `matchLabels` selector.\n\nIf you want to see the metrics and logs, use `port-forward` to connect to grafana and then Go to `Explore` and choose `mimir` as the datasource to see the metrics and choose `loki` to see the pods logs. you can see the pods logs of `nginx` namespace. if you want to add/remove any namespaces, Modify `matchLabels` selector in `agent-operator/pod-logs.yml`\n\nFor instance, you can use LogQL to extract log lines that include the status code `4.*`: `{namespace=\"nginx\", status_code=~\"4.*\"}`\n\n\n\n## :three: Scenario-3: Migrating from Grafana Agent to Grafana Alloy\n\nEnter Grafana Alloy, the spiritual successor to Grafana Agent. Alloy is the new open source distribution of the OpenTelemetry Collector that is 100% OTLP compatible and offers native pipelines for OpenTelemetry and Prometheus telemetry formats, supporting metrics, logs, traces, and profiles. It does everything you’d expect of the Grafana Agent project and other agents — plus so much more. If you are currently using Grafana Agent in your observability stack, we encourage you to migrate to Alloy. both Grafana Agent and Grafana Agent Operator will no longer receive any new feature updates.\n\nThe Monitor types (`PodMonitor`, `ServiceMonitor`, `Probe`, and `PodLogs`) are all supported natively by Alloy. The parts of Grafana Agent Operator that deploy Grafana Agent, `GrafanaAgent`, `MetricsInstance`, and `LogsInstance` CRDs, are deprecated.\n\nFirst of all, Let's start with `Metrics` (forwarding metrics to mimir) and then We'll move on to `Logs` (forwarding logs to loki)\n\n## Convert `MetricsInstance` to Alloy components\n\nUninstall `GrafanaAgent` and `MetricsInstance` resources inside `default` namespace and then deploy the below configmap as well as `cadvisor` and `kubelet` servicemonitors inside `alloy-metrics` namespace and finally install alloy-metrics using helm:\n\n    kubectl create ns alloy-metrics\n    kubectl apply -f migrate-from-agent-to-alloy/metrics/alloy-configMap.yml\n    migrate-from-agent-to-alloy/metrics/cadvisor-serviceMonitor.yml\n    migrate-from-agent-to-alloy/metrics/kubelet-serviceMonitor.yml\n    helm install alloy-metrics grafana/alloy -n alloy-metrics -f migrate-from-agent-to-alloy/metrics/alloy-helmValues.yml\n\n## Collecting logs\n\nUninstall `GrafanaAgent`, `LogsInstance` and `PodLogs` resources inside `default` namespace and then deploy the below configmap as well inside `alloy-logs` namespace and finally install alloy-logs using helm:\n\n    kubectl create ns alloy-logs\n    kubectl apply -f migrate-from-agent-to-alloy/logs/alloy-configMap.yml\n    helm install alloy-logs grafana/alloy -n alloy-logs -f migrate-from-agent-to-alloy/logs/alloy-helmValues.yml\nIf you pay attention to `loki.process` inside `migrate-from-agent-to-alloy/logs/alloy-configMap.yml`, you find out that i use an additional `stage` which adds `status_code` and `method` labels to the pods inside `nginx` namespace. if you take a look at `Explore` section of grafana, All logs within `nginx` namespace, have those added labels.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmohammadll%2Fgrafana-stack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmohammadll%2Fgrafana-stack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmohammadll%2Fgrafana-stack/lists"}