{"id":50683047,"url":"https://github.com/wlanboy/lgtm","last_synced_at":"2026-06-08T20:30:45.845Z","repository":{"id":352226415,"uuid":"940571143","full_name":"wlanboy/lgtm","owner":"wlanboy","description":"Docker compose for LGTM Stack (Loki, Grafana, Tempo, Mimir)","archived":false,"fork":false,"pushed_at":"2026-04-18T13:29:45.000Z","size":14,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-18T15:27:31.621Z","etag":null,"topics":["docker-compose","grafana","lgtm","loki","mimir","mirror","tempo"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wlanboy.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-02-28T12:12:04.000Z","updated_at":"2026-04-18T13:29:49.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/wlanboy/lgtm","commit_stats":null,"previous_names":["wlanboy/lgtm"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/wlanboy/lgtm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wlanboy%2Flgtm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wlanboy%2Flgtm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wlanboy%2Flgtm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wlanboy%2Flgtm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wlanboy","download_url":"https://codeload.github.com/wlanboy/lgtm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wlanboy%2Flgtm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34080025,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker-compose","grafana","lgtm","loki","mimir","mirror","tempo"],"created_at":"2026-06-08T20:30:42.624Z","updated_at":"2026-06-08T20:30:45.837Z","avatar_url":"https://github.com/wlanboy.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LGTM Observability Stack\n\nLokales Observability-Setup auf Basis des Grafana-Stacks. Kombiniert Metriken, Logs und Traces in einer einzigen Grafana-Oberfläche — vollständig mit Docker Compose betreibbar, ohne Cloud-Abhängigkeiten.\n\n---\n\n## Architektur\n\n```\nAnwendung / k6-tracing\n        │\n        ▼ OTLP (gRPC :4317 / HTTP :4318)\n    ┌───────┐       Traces      ┌───────┐\n    │ Alloy │ ─────────────────▶│ Tempo │──┐\n    └───────┘                   └───────┘  │ metrics_generator\n        │ Logs (/var/log)                  ▼\n        │                            ┌────────────┐\n        │                            │ Prometheus │\n        ▼                            └────────────┘\n    ┌──────┐                              │\n    │ Loki │                              │\n    └──────┘                              │\n        │                                 │\n        └──────────┬──────────────────────┘\n                   ▼\n              ┌─────────┐\n              │ Grafana │\n              └─────────┘\n                   │\n                   ▼\n            ┌─────────────┐\n            │ Alertmanager│\n            └─────────────┘\n```\n\n---\n\n## Komponenten\n\n| Service | Image | Version | Beschreibung |\n|---|---|---|---|\n| **Grafana** | `grafana/grafana` | 13.0.1 | Visualisierung, Dashboards, Explore |\n| **Loki** | `grafana/loki` | 3.7.1 | Log-Aggregation und -Abfrage |\n| **Tempo** | `grafana/tempo` | 2.10.4 | Distributed Tracing Backend |\n| **Prometheus** | `prom/prometheus` | v3.10.0 | Metriken-Scraping und -Speicherung |\n| **Alloy** | `grafana/alloy` | v1.15.1 | OTel Collector: OTLP→Tempo, Logs→Loki |\n| **Alertmanager** | `prom/alertmanager` | v0.32.0 | Alert-Routing und -Benachrichtigung |\n| **Memcached** | `memcached` | 1.6 | Cache für Tempo Frontend-Search |\n| **k6-tracing** *(optional)* | `xk6-client-tracing` | v0.0.9 | Synthetischer Trace-Generator |\n\n### k6-tracing (optional)\n\n`k6-tracing` sendet kontinuierlich synthetische Traces an Tempo und ist nur für Demo- und Testzwecke gedacht. Damit sind sofort Traces in Grafana sichtbar, ohne eine eigene Anwendung instrumentieren zu müssen.\n\n**Eigene Anwendungen senden Traces?** Dann `k6-tracing` aus `docker-compose.yaml` entfernen — sonst erzeugt es unnötigen Lärm in den Trace-Daten.\n\n### Alloy als zentraler Collector\n\nAlloy übernimmt mehrere Aufgaben und ersetzt Promtail:\n\n- **Traces**: Empfängt OTLP (gRPC + HTTP) und leitet an Tempo weiter\n- **Logs**: Liest `/var/log/*log` vom Host und sendet an Loki\n- **Host Metriken**: Erfasst Node-Metriken (entspricht node_exporter) und sendet an Prometheus\n\nKonfiguration: [shared/config.alloy](shared/config.alloy)\n\n### Tempo Metrics Generator\n\nTempo generiert automatisch Metriken aus Traces und schreibt sie per Remote Write in Prometheus:\n- **Service Graphs** — Abhängigkeiten und Latenz zwischen Services\n- **Span Metrics** — Rate, Fehlerrate, Dauer pro Operation\n- **Local Blocks** — Metriken-Abfragen direkt über TraceQL\n\n### Datenpersistenz\n\nAlle Daten werden in benannten Docker Volumes gespeichert und überleben `docker compose down`:\n\n| Volume | Service | Inhalt |\n|---|---|---|\n| `tempo-data` | Tempo | Trace-Daten |\n| `loki-data` | Loki | Log-Daten |\n| `prometheus-data` | Prometheus | Metriken-Daten |\n| `grafana-data` | Grafana | Dashboards, Einstellungen |\n| `alertmanager-data` | Alertmanager | Silences, Notification-Log |\n\n**Achtung:** `docker compose down -v` löscht alle Volumes unwiderruflich.\n\n---\n\n## URLs\n\n| Service | URL | Beschreibung |\n|---|---|---|\n| **Grafana** | http://localhost:3000 | Haupt-UI: Dashboards, Explore |\n| **Grafana Traces** | http://localhost:3000/a/grafana-traces-app | Trace Explorer |\n| **Prometheus** | http://localhost:9090 | PromQL, Targets, Alerts |\n| **Alertmanager** | http://localhost:9093 | Alert-Routing UI |\n| **Alloy UI** | http://localhost:12345 | Pipeline-Graph, Debug |\n| **Tempo API** | http://localhost:3200 | Tempo HTTP API |\n| **Loki API** | http://localhost:3100 | Loki HTTP API |\n| **Loki Ready** | http://localhost:3100/ready | Readiness Check |\n| **Loki Metrics** | http://localhost:3100/metrics | Loki interne Metriken |\n\n---\n\n## Start / Stop\n\n```bash\n# Stack starten\ndocker compose up -d\n\n# Status prüfen\ndocker compose ps\n\n# Logs eines Service anzeigen\ndocker compose logs -f alloy\n\n# Stack stoppen (Volumes behalten)\ndocker compose down\n\n# Stack stoppen und alle Volumes löschen\ndocker compose down -v\n```\n\n---\n\n## Daten senden\n\n### Traces via OTLP\n\nEigene Anwendungen können Traces direkt an Alloy senden:\n\n```\nOTLP gRPC:  localhost:4317\nOTLP HTTP:  localhost:4318\n```\n\nAlternativ direkt an Tempo (ohne Alloy-Pipeline):\n```\nJaeger HTTP:    localhost:14268\nZipkin:         localhost:9411\n```\n\n### Logs\n\nAlloy liest automatisch alle `*.log`-Dateien aus `/var/log` des Hosts und schickt sie an Loki. Für eigene Log-Pfade [shared/config.alloy](shared/config.alloy) anpassen:\n\n```alloy\nlocal.file_match \"my_app\" {\n  path_targets = [{\n    __path__ = \"/var/log/myapp/*.log\",\n    job      = \"myapp\",\n  }]\n}\n```\n\n---\n\n## Grafana verwenden\n\n### Dashboards\n\nBeim Start werden folgende Dashboards automatisch provisioniert (keine manuelle Installation nötig):\n\n| Dashboard | Inhalt |\n|---|---|\n| **Node Exporter Full** | CPU, RAM, Disk, Netzwerk des Hosts |\n| **Docker cAdvisor** | Container-Metriken (CPU, RAM, Restarts) |\n| **Loki Dashboard** | Log-Volumen, Fehlerrate, Log-Streams |\n| **Tempo / Traces** | Trace-Latenz, Service-Map, Error-Rate |\n| **Alertmanager** | Alert-Status, Silences, Gruppen |\n| **Kubernetes Cluster** | Pods, Deployments, Nodes (bei K8s-Anbindung) |\n| **Kubernetes Pods** | Pod-Ressourcen per Namespace (bei K8s-Anbindung) |\n| **Istio Mesh** | Service-Mesh Traffic, Fehlerrate (bei Istio) |\n\nDashboard-JSONs liegen unter [shared/dashboards/](shared/dashboards/).\n\n### Traces erkunden\n\n1. http://localhost:3000 öffnen (kein Login erforderlich)\n2. **Explore** → Datenquelle **Tempo** → Tab **Search** → **Run query**\n3. Einen Trace auswählen um das Trace-Diagramm zu sehen\n4. Tab **Service graph** für automatisch generierte Service-Maps (nach ~2 Minuten verfügbar)\n\n### Logs erkunden\n\n1. **Explore** → Datenquelle **Loki**\n2. Label-Filter: `{job=\"varlogs\"}` für System-Logs, `{container=\"alloy\"}` für Docker-Logs\n3. Mit LogQL filtern, z.B.: `{job=\"varlogs\"} |= \"error\"`\n\n### Metriken erkunden\n\n1. **Explore** → Datenquelle **Prometheus**\n2. Tempo-generierte Metriken z.B.: `rate(traces_spanmetrics_calls_total[5m])`\n3. Container-Metriken z.B.: `container_cpu_usage_seconds_total`\n\n### Metrik Targets überprüfen\n1. http://localhost:9090/targets öffnen\n\n### Traces mit Metriken verknüpfen (Exemplars)\n\nPrometheus ist mit Exemplar-Support konfiguriert. In Grafana unter **Explore → Prometheus** auf den Exemplar-Punkt in einem Graph klicken um direkt zum zugehörigen Trace zu springen.\n\n---\n\n## Alerting\n\nAlerts werden in Prometheus als Regeln definiert und über Alertmanager geroutet.\n\n### Alert-Regel anlegen\n\nNeue Datei `config/alerts.yaml` anlegen:\n\n```yaml\ngroups:\n  - name: example\n    rules:\n      - alert: HighErrorRate\n        expr: rate(traces_spanmetrics_calls_total{status_code=\"STATUS_CODE_ERROR\"}[5m]) \u003e 0.1\n        for: 2m\n        labels:\n          severity: warning\n        annotations:\n          summary: \"Hohe Fehlerrate bei {{ $labels.service }}\"\n          description: \"Fehlerrate: {{ $value | humanizePercentage }}\"\n\n      - alert: ContainerDown\n        expr: absent(container_last_seen{name!=\"\"})\n        for: 1m\n        labels:\n          severity: critical\n        annotations:\n          summary: \"Container nicht erreichbar: {{ $labels.name }}\"\n```\n\nDann in [shared/prometheus.yaml](shared/prometheus.yaml) einbinden:\n\n```yaml\nrule_files:\n  - /etc/prometheus/alerts.yaml\n```\n\nUnd in [docker-compose.yaml](docker-compose.yaml) das Volume mounten:\n\n```yaml\nprometheus:\n  volumes:\n    - ./config/alerts.yaml:/etc/prometheus/alerts.yaml\n```\n\n### Alertmanager konfigurieren\n\nEmpfänger und Routen in [config/alertmanager.yaml](config/alertmanager.yaml) definieren:\n\n```yaml\nglobal:\n  smtp_smarthost: 'localhost:25'\n\nroute:\n  receiver: default\n  group_by: [alertname, severity]\n  group_wait: 30s\n  group_interval: 5m\n  repeat_interval: 4h\n  routes:\n    - match:\n        severity: critical\n      receiver: oncall\n\nreceivers:\n  - name: default\n    email_configs:\n      - to: 'team@example.com'\n        from: 'alertmanager@example.com'\n\n  - name: oncall\n    webhook_configs:\n      - url: 'http://oncall-service/webhook'\n```\n\n### Alert-Status prüfen\n\n- Aktive Alerts: http://localhost:9090/alerts\n- Routing und Silences: http://localhost:9093\n\n---\n\n## Kubernetes\n\nFür die Anbindung eines Kubernetes-Clusters an diesen Stack siehe [kubernetes.md](kubernetes.md).\n\nKurzzusammenfassung: Grafana Alloy wird als DaemonSet im Cluster deployed und sendet Logs, Metriken und Traces an den laufenden Docker-Stack. Vier YAML-Dateien unter [kubernetes/](kubernetes/) sind deployfertig vorbereitet.\n\n---\n\n## Konfigurationsdateien\n\n```\n├── config/\n│   ├── alertmanager.yaml       # Alertmanager Routen und Receiver\n│   ├── loki-config.yaml        # Loki Storage, Schema, Pattern Ingester\n│   └── tempo.yaml              # Tempo Distributor, Ingester, Metrics Generator\n├── shared/\n│   ├── config.alloy            # Alloy Pipeline (Traces, Logs, Docker, Host-Metriken)\n│   ├── dashboards/             # Provisionierte Grafana Dashboard JSONs\n│   ├── grafana-dashboards.yaml # Grafana Dashboard-Provisioning\n│   ├── grafana-datasources.yaml# Grafana Datasource-Provisioning\n│   └── prometheus.yaml         # Prometheus Scrape-Config\n├── kubernetes/\n│   ├── namespace.yaml          # Namespace monitoring\n│   ├── rbac.yaml               # ServiceAccount, ClusterRole\n│   ├── configmap.yaml          # Alloy-Config für Kubernetes\n│   ├── configmap-with-istio.yaml # Alloy-Config inkl. Istio\n│   ├── daemonset.yaml          # Alloy DaemonSet\n│   └── istio.yaml              # Istio Telemetry + Service (optional)\n└── docker-compose.yaml\n```\n\n---\n\n## Troubleshooting\n\n### Grafana zeigt \"No data\"\n\n1. Alloy UI prüfen: http://localhost:12345 → Pipeline-Graph auf Fehler prüfen\n2. Prometheus Targets prüfen: http://localhost:9090/targets → alle Targets `UP`?\n3. Loki Readiness: `curl http://localhost:3100/ready` → muss `ready` zurückgeben\n4. Container-Logs prüfen: `docker compose logs -f \u003cservice\u003e`\n\n### Loki startet nicht / bleibt unhealthy\n\n```bash\ndocker compose logs loki\n# Häufige Ursache: Berechtigungsproblem auf loki-data Volume\ndocker compose down -v \u0026\u0026 docker compose up -d\n```\n\n### Tempo-Traces fehlen nach Neustart\n\nTraces liegen im `tempo-data` Volume — dieses überlebt `docker compose down`. Bei `docker compose down -v` gehen sie verloren. Volume-Status prüfen:\n\n```bash\ndocker volume ls | grep lgtm\ndocker volume inspect lgtm_tempo-data\n```\n\n### Alloy sammelt keine Docker-Logs\n\nDocker-Socket muss für Alloy erreichbar sein:\n\n```bash\n# Prüfen ob Socket gemountet ist\ndocker compose exec alloy ls -la /var/run/docker.sock\n\n# Alloy neu starten\ndocker compose restart alloy\n```\n\n### Alertmanager sendet keine Benachrichtigungen\n\n```bash\n# Alertmanager-Konfiguration validieren\ndocker compose exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yaml\n\n# Aktive Alerts anzeigen\ndocker compose exec alertmanager amtool alert query\n```\n\n### Prometheus Remote Write schlägt fehl (Kubernetes)\n\nPrometheus muss mit `--web.enable-remote-write-receiver` gestartet sein (bereits konfiguriert). Firewall-Regel prüfen: Cluster-Nodes müssen Port `9090` des LGTM-Hosts erreichen können.\n\n```bash\n# Vom Kubernetes-Node aus testen\ncurl -v http://\u003cLGTM-HOST\u003e:9090/-/healthy\n```\n\n---\n\n## Links\n\n- [Grafana Stack Übersicht](https://grafana.com/about/grafana-stack/)\n- [Alloy Dokumentation](https://grafana.com/docs/alloy/latest/)\n- [Loki Docker Setup](https://grafana.com/docs/loki/latest/setup/install/docker/)\n- [Tempo Getting Started](https://grafana.com/docs/tempo/latest/getting-started/docker-example/)\n- [Tempo Instrumentation](https://grafana.com/docs/tempo/latest/getting-started/instrumentation/)\n- [Prometheus Remote Write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write)\n- [OpenTelemetry Protokoll](https://opentelemetry.io/docs/specs/otlp/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwlanboy%2Flgtm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwlanboy%2Flgtm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwlanboy%2Flgtm/lists"}