{"id":30137320,"url":"https://github.com/deffz-finesse2nd/grafana-observability-stack","last_synced_at":"2026-04-17T05:03:03.118Z","repository":{"id":295353811,"uuid":"989885113","full_name":"Deffz-Finesse2nd/Grafana-Observability-Stack","owner":"Deffz-Finesse2nd","description":"Grafana Observability Stack — Production-ready observability platform with Prometheus, Loki, Tempo \u0026 Grafana. Collect metrics, logs, and traces in one stack. Easy to deploy with Docker Compose. Perfect for monitoring microservices, Kubernetes, and cloud-native apps.","archived":false,"fork":false,"pushed_at":"2025-05-28T01:04:23.000Z","size":36,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-08-11T00:20:33.957Z","etag":null,"topics":["devops","docker","docker-compose","grafana","logging","loki","metrics","microservices","monitoring","observability","prometheus","tempo","tracing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Deffz-Finesse2nd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-25T03:24:54.000Z","updated_at":"2025-05-28T06:24:17.000Z","dependencies_parsed_at":"2025-05-29T12:18:05.333Z","dependency_job_id":null,"html_url":"https://github.com/Deffz-Finesse2nd/Grafana-Observability-Stack","commit_stats":null,"previous_names":["deffz-finesse/grafana-observability-stack","deffz-finesse2nd/grafana-observability-stack"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Deffz-Finesse2nd/Grafana-Observability-Stack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deffz-Finesse2nd%2FGrafana-Observability-Stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deffz-Finesse2nd%2FGrafana-Observability-Stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deffz-Finesse2nd%2FGrafana-Observability-Stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deffz-Finesse2nd%2FGrafana-Observability-Stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Deffz-Finesse2nd","download_url":"https://codeload.github.com/Deffz-Finesse2nd/Grafana-Observability-Stack/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deffz-Finesse2nd%2FGrafana-Observability-Stack/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31915900,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T18:22:33.417Z","status":"online","status_checked_at":"2026-04-17T02:00:06.879Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["devops","docker","docker-compose","grafana","logging","loki","metrics","microservices","monitoring","observability","prometheus","tempo","tracing"],"created_at":"2025-08-11T00:13:51.820Z","updated_at":"2026-04-17T05:03:03.113Z","avatar_url":"https://github.com/Deffz-Finesse2nd.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Grafana Observability Stack\n\nA comprehensive, production-ready observability platform implementing the **three pillars of observability**: metrics, logs, and traces. Built with Docker Compose for easy deployment and management.\n\n## 🎯 Overview\n\nThis project offers a full-stack monitoring and observability solution using industry-standard tools from the Grafana ecosystem. It enables visibility into your applications and infrastructure via unified data collection, storage, and visualization.\n\n### Features at a Glance\n\n- **📊 Metrics Collection**: Scrapes and stores time-series metrics using Prometheus  \n- **📝 Log Aggregation**: Collects, processes, and stores logs via Loki and Promtail  \n- **🔍 Distributed Tracing**: Captures and analyzes request traces with Tempo  \n- **📈 Unified Visualization**: Correlated dashboards and alerts via Grafana  \n- **🔗 Data Correlation**: Links metrics, logs, and traces for end-to-end debugging  \n- **🕸️ Service Graphs**: Automatic service topology visualization from traces\n- **🌐 Node Graphs**: Interactive network topology views in Grafana\n- **📡 OpenTelemetry Integration**: Full OTLP support with Python instrumentation library\n- **📋 Structured Logging**: Production-ready logging with trace correlation and rotation\n\n### Key Benefits\n\n- ✅ **Production-Ready**: Persistent storage, automatic restarts, and robust networking  \n- ✅ **Zero-Config Setup**: Pre-configured data sources and dashboards  \n- ✅ **Correlation-Enabled**: Trace-to-log linking and cross-source queries  \n- ✅ **Scalable Architecture**: Container-based with volume persistence  \n- ✅ **Environment Flexible**: Ports and credentials configurable via `.env`  \n- ✅ **Service Discovery**: Automatic service topology mapping from trace data\n- ✅ **Metrics from Traces**: Span metrics and service graphs generated automatically\n\n## 🏗️ Architecture\n\n```mermaid\ngraph TB\n    subgraph \"Data Collection\"\n        A[Applications] --\u003e B[Promtail]\n        A --\u003e C[Prometheus]\n        A --\u003e D[Tempo]\n        A --\u003e E[OTLP Exporter]\n        E --\u003e D\n    end\n\n    subgraph \"Storage Layer\"\n        B --\u003e F[Loki\u003cbr/\u003eLog Storage]\n        C --\u003e G[Prometheus\u003cbr/\u003eMetrics Storage]\n        D --\u003e H[Tempo\u003cbr/\u003eTrace Storage]\n        D --\u003e I[Metrics Generator\u003cbr/\u003eService Graphs]\n        I --\u003e G\n    end\n\n    subgraph \"Visualization\"\n        F --\u003e J[Grafana]\n        G --\u003e J\n        H --\u003e J\n        J --\u003e K[Service Maps]\n        J --\u003e L[Node Graphs]\n    end\n\n    subgraph \"External Access\"\n        J --\u003e M[Dashboards \u0026 Alerts]\n        J --\u003e N[Query Interface]\n        K --\u003e O[Topology Views]\n        L --\u003e P[Network Analysis]\n    end\n```\n\n### Component Responsibilities\n\n| Component     | Purpose                         | Data Type                   | Port        |\n|---------------|----------------------------------|-----------------------------|-------------|\n| **Prometheus**| Metrics scraping and storage     | Time-series metrics         | 9090        |\n| **Grafana**   | Visualization and dashboards     | All data types              | 3000        |\n| **Loki**      | Log aggregation and storage      | Structured/unstructured logs| 3100        |\n| **Promtail**  | Log collection agent             | Log shipping                | 9080        |\n| **Tempo**     | Distributed tracing backend      | Request traces              | 3200 / 9095 |\n| **OTLP**      | OpenTelemetry Protocol receiver  | Traces via gRPC/HTTP        | 4317 / 4318 |\n\n**Note**: Loki, Tempo, and Promtail expose internal metrics to Prometheus for self-monitoring.\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- **Docker** (v20.10+)\n- **Docker Compose** (v2.0+)\n- **Python** (v3.8+) - for using the included instrumentation libraries\n- **4GB+ RAM** available\n- **Ports available**: 3000, 3100, 3200, 4317, 4318, 9090, 9095\n\n### 1. Clone and Setup\n\n```bash\ngit clone https://github.com/Deffz-Finesse/Grafana-Observability-Stack.git\ncd Grafana-Observability-Stack\n\n# Create required directories\nmkdir -p logs\n\n# Install Python dependencies (optional - for using lib modules)\npip install -r requirements.txt\n```\n\n### 2. Environment Variables\n\nEdit the included `.env` file:\n\n```bash\n# ─── Grafana\nGRAFANA_USER=admin\nGRAFANA_PASSWORD=secret\nGRAFANA_PORT=3000\n\n# ─── Prometheus\nPROMETHEUS_PORT=9090\n\n# ─── Loki\nLOKI_PORT=3100\n\n# ─── Tempo\nTEMPO_HTTP_PORT=3200\nTEMPO_GRPC_PORT=9095\n```\n\n### 3. Launch the Stack\n\n```bash\ndocker-compose up -d       # Start all services\ndocker-compose ps          # Verify containers\ndocker-compose logs -f     # View logs if needed\n```\n\n### 4. Access the Services\n\n| Service      | URL                                              | Credentials     |\n|--------------|---------------------------------------------------|-----------------|\n| **Grafana**  | [http://localhost:3000](http://localhost:3000)    | admin / secret  |\n| **Prometheus**| [http://localhost:9090](http://localhost:9090)   | None            |\n\n## 📊 Usage Guide\n\n### Accessing Grafana\n\n1. Visit [http://localhost:3000](http://localhost:3000)\n2. Login using configured credentials\n3. Use pre-configured data sources:\n   - Prometheus (metrics)\n   - Loki (logs)\n   - Tempo (traces)\n\n### Viewing Metrics\n\n```promql\n# Example Prometheus queries\nup                                  # Service health\nrate(http_requests_total[5m])       # Request rate\nprometheus_tsdb_head_series         # Metric count\n\n# Service Graph Metrics (generated by Tempo)\ntraces_service_graph_request_total                    # Service-to-service request count\ntraces_service_graph_request_failed_total             # Failed requests between services\ntraces_service_graph_request_server_seconds           # Server-side latency\ntraces_service_graph_request_client_seconds           # Client-side latency\n\n# Span Metrics (generated by Tempo)\ntraces_spanmetrics_latency                            # Span duration histogram\ntraces_spanmetrics_calls_total                        # Total span count\n```\n\n### Querying Logs\n\n```logql\n{container=\"app\"}                    # Logs from app container\n{level=\"error\"}                      # Error-level logs\n{service=\"api\"} |= \"timeout\"         # API logs containing \"timeout\"\n```\n\n### Exploring Traces\n\n1. Open **Explore** in Grafana  \n2. Select **Tempo** as data source  \n3. Search using:\n   - Trace ID\n   - Service name\n   - Operation\n   - Time range\n\n### Service Graphs \u0026 Node Graphs\n\n#### Service Maps\n1. Navigate to **Explore** in Grafana\n2. Select **Tempo** as data source\n3. Click on **Service Map** tab\n4. View automatic service topology generated from trace data\n5. Click on service nodes to drill down into traces\n\n#### Node Graphs\n1. In any Tempo trace view, look for the **Node Graph** button\n2. Interactive network topology shows:\n   - Service dependencies\n   - Request flow direction\n   - Error rates and latencies\n   - Service health indicators\n\n#### Metrics from Service Graphs\n- Service graph metrics are automatically scraped by Prometheus\n- Available in Grafana dashboards for alerting and monitoring\n- Provides RED metrics (Rate, Errors, Duration) per service pair\n\n### Adding Your Applications\n\n#### Metrics (Prometheus)\n\nAdd to `observability/prometheus/prometheus.yml`:\n\n```yaml\nscrape_configs:\n  - job_name: 'your-app'\n    static_configs:\n      - targets: ['your-app:8080']\n```\n\n#### Traces (Tempo)\n\nSend traces to:\n\n- **HTTP**: `http://localhost:3200`\n- **gRPC**: `http://localhost:9095`\n- **OTLP gRPC**: `http://localhost:4317`\n- **OTLP HTTP**: `http://localhost:4318`\n\n#### OpenTelemetry Integration\n\n**Installation:**\n\n```bash\n# Install Python dependencies\npip install -r requirements.txt\n```\n\n**Usage:**\n\nUse the provided Python libraries for automatic instrumentation and structured logging:\n\n```python\nfrom lib.otel import setup_otel, get_tracer\nfrom lib.logger import get_logger\n\n# Initialize OpenTelemetry (call once at startup)\nsetup_otel()\n\n# Get a tracer and logger for your service\ntracer = get_tracer(\"my-service\")\nlogger = get_logger(\"my-service\")\n\n# Create spans with correlated logging\nwith tracer.start_as_current_span(\"operation-name\") as span:\n    span.set_attribute(\"user.id\", \"12345\")\n    logger.info(\"Processing user request\", extra={\"user_id\": \"12345\"})\n    # Your application logic here\n```\n\n**Supported Instrumentations:**\n- HTTP requests (via `requests` library)\n- Database queries (SQLite, PostgreSQL)\n- Redis operations\n- Logging (automatic trace ID injection)\n\n**Environment Variables:**\n```bash\n# OpenTelemetry Configuration\nexport OTEL_SERVICE_NAME=\"my-service\"\nexport OTEL_EXPORTER_OTLP_ENDPOINT=\"localhost:4317\"\n\n# Logging Configuration\nexport LOG_LEVEL=\"INFO\"\nexport LOG_FILE=\"logs/app.log\"\nexport ENABLE_CONSOLE_LOG=\"true\"\nexport ENABLE_FILE_LOG=\"true\"\nexport ENV=\"development\"\n```\n\n## 📁 Project Structure\n\n```\n.\n├── docker-compose.yml                     # Service orchestration\n├── requirements.txt                       # Python dependencies for lib modules\n├── lib/\n│   ├── logger.py                          # Structured logging with OpenTelemetry integration\n│   └── otel.py                            # OpenTelemetry Python instrumentation\n├── observability/                         # Configuration files\n│   ├── grafana/\n│   │   └── provisioning/\n│   │       ├── datasources/\n│   │       │   └── datasource.yml         # Preloaded data sources (with node graphs enabled)\n│   │       └── dashboards/\n│   │           └── dashboard.yml          # Auto-loaded dashboards\n│   ├── prometheus/\n│   │   └── prometheus.yml                 # Scraping config (includes service graph metrics)\n│   ├── loki/\n│   │   └── loki-config.yml                # Log config\n│   ├── promtail/\n│   │   └── promtail-config.yml            # Log agent config\n│   └── tempo/\n│       └── tempo-config.yaml              # Tracing config (with metrics generator)\n└── logs/                                  # Log storage (create manually)\n```\n\n## ⚙️ Configuration\n\n### Tempo Service Graphs Configuration\n\nThe Tempo metrics generator is configured in `observability/tempo/tempo-config.yaml`:\n\n```yaml\nmetrics_generator:\n  storage:\n    path: /tmp/tempo/generator-wal\n    remote_write:\n      - url: http://prometheus:9090/api/v1/write\n  \n  traces_storage:\n    path: /tmp/tempo/generator-wal/traces\n  \n  registry:\n    collection_interval: 15s\n    external_labels:\n      source: tempo\n\noverrides:\n  defaults:\n    metrics_generator:\n      processors: [service-graphs, span-metrics, local-blocks]\n```\n\n### Grafana Node Graph Configuration\n\nNode graphs are enabled in `observability/grafana/provisioning/datasources/datasource.yml`:\n\n```yaml\n- name: Tempo\n  type: tempo\n  jsonData:\n    serviceMap:\n      datasourceUid: 'Prometheus'\n    nodeGraph:\n      enabled: true\n    tracesToLogsV2:\n      datasourceUid: 'Loki'\n```\n\n### Prometheus Service Graph Metrics\n\nService graph metrics are scraped via `observability/prometheus/prometheus.yml`:\n\n```yaml\n- job_name: 'tempo-metrics-generator'\n  static_configs:\n    - targets: ['tempo:3200']\n  metrics_path: /metrics\n```\n\n### Structured Logging Configuration\n\nThe logging library (`lib/logger.py`) supports flexible configuration via environment variables:\n\n```bash\n# Logging behavior\nLOG_LEVEL=INFO                    # DEBUG, INFO, WARNING, ERROR, CRITICAL\nLOG_FILE=logs/app.log            # Path to log file\nENABLE_CONSOLE_LOG=true          # Enable colored console output\nENABLE_FILE_LOG=true             # Enable file logging with rotation\nENV=development                  # Environment (affects log formatting)\n\n# File rotation settings (hardcoded in logger.py)\n# - Max file size: 5MB\n# - Backup count: 5 files\n# - Total log retention: ~25MB\n```\n\n**Usage Example:**\n\n```python\nfrom lib.logger import get_logger\n\n# Get a logger instance\nlogger = get_logger(\"my-component\")\n\n# Log with different levels\nlogger.debug(\"Detailed debugging information\")\nlogger.info(\"General information about program execution\")\nlogger.warning(\"Something unexpected happened\")\nlogger.error(\"A serious error occurred\", extra={\"error_code\": 500})\nlogger.critical(\"System is unusable\")\n\n# Structured logging with extra fields\nlogger.info(\"User login\", extra={\n    \"user_id\": \"12345\",\n    \"ip_address\": \"192.168.1.1\",\n    \"user_agent\": \"Mozilla/5.0...\"\n})\n```\n\n### Retention Settings\n\n**Prometheus** (`prometheus.yml`):\n\n```yaml\nglobal:\n  scrape_interval: 5s\n```\n\n**Loki** (`loki-config.yml`):\n\n```yaml\nlimits_config:\n  retention_period: 168h  # 7 days\n```\n\n**Tempo** (`tempo-config.yaml`):\n\n```yaml\ncompactor:\n  retention: 48h          # 2 days\n```\n\n## 🔧 Advanced Features\n\n### Service Graph Metrics\n\nTempo automatically generates service topology metrics from trace data:\n\n- **Request Rate**: `traces_service_graph_request_total`\n- **Error Rate**: `traces_service_graph_request_failed_total`  \n- **Duration**: `traces_service_graph_request_server_seconds`\n- **Client Latency**: `traces_service_graph_request_client_seconds`\n\nThese metrics enable:\n- Service dependency mapping\n- SLA monitoring and alerting\n- Performance regression detection\n- Capacity planning insights\n\n### Span Metrics\n\nIndividual span performance metrics:\n\n- **Latency Distribution**: `traces_spanmetrics_latency`\n- **Call Volume**: `traces_spanmetrics_calls_total`\n- **Error Tracking**: Per-operation error rates\n\n### OpenTelemetry Features\n\nThe included Python libraries provide:\n\n- **Automatic Instrumentation**: Popular libraries auto-traced\n- **Manual Instrumentation**: Custom span creation and attributes\n- **Context Propagation**: Distributed trace correlation\n- **Resource Detection**: Service name and version tagging\n- **Batch Export**: Efficient span batching to Tempo\n\n### Structured Logging Features\n\nThe included logging library (`lib/logger.py`) provides:\n\n- **Colored Console Output**: Environment-aware colored logging for development\n- **File Rotation**: Automatic log file rotation with configurable size limits\n- **Millisecond Precision**: High-precision timestamps for debugging\n- **OpenTelemetry Integration**: Automatic trace ID injection into log entries\n- **Environment Configuration**: Flexible logging behavior via environment variables\n- **Multiple Handlers**: Simultaneous console and file logging with different formats\n\n### Correlation Features\n\n- **Trace-to-Logs**: Click from spans to related log entries\n- **Metrics-to-Traces**: Drill down from service graphs to traces\n- **Service Maps**: Visual service topology from trace data\n- **Exemplars**: Link from metrics to example traces\n\n## 🚀 Production Considerations\n\n- Use **external storage** for persistence\n- Add **TLS and authentication**\n- Place behind **reverse proxy** (nginx, Traefik)\n- Set up **HA deployments** for redundancy\n- Configure **alerts and notifications** in Grafana\n- **Scale Tempo** with object storage (S3, GCS, Azure)\n- **Tune metrics generator** collection intervals for performance\n- **Monitor service graph cardinality** to prevent metric explosion\n\n## 📄 License\n\nLicensed under the MIT License. See the [LICENSE](LICENSE) file.\n\n## 🙏 Acknowledgments\n\n- [Grafana Labs](https://grafana.com/)\n- [Prometheus](https://prometheus.io/)\n- [OpenTelemetry](https://opentelemetry.io/)\n\n## 📚 Additional Resources\n\n- [Grafana Docs](https://grafana.com/docs/)\n- [Prometheus Querying](https://prometheus.io/docs/prometheus/latest/querying/examples/)\n- [Loki LogQL](https://grafana.com/docs/loki/latest/logql/)\n- [Tempo Tracing Guide](https://grafana.com/docs/tempo/latest/)\n- [Tempo Service Graphs](https://grafana.com/docs/tempo/latest/metrics-generator/service-graphs/)\n- [OpenTelemetry Python](https://opentelemetry.io/docs/instrumentation/python/)\n- [Grafana Node Graphs](https://grafana.com/docs/grafana/latest/panels/visualizations/node-graph/)\n\n---\n\n**⭐ Star this repository if it helped you build better observability!**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeffz-finesse2nd%2Fgrafana-observability-stack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeffz-finesse2nd%2Fgrafana-observability-stack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeffz-finesse2nd%2Fgrafana-observability-stack/lists"}