{"id":27521216,"url":"https://github.com/lakinduboteju/otel-python-logging","last_synced_at":"2026-04-15T14:34:58.051Z","repository":{"id":288147700,"uuid":"966983351","full_name":"lakinduboteju/otel-python-logging","owner":"lakinduboteju","description":"Integrating OpenTelemetry for Logging in Python","archived":false,"fork":false,"pushed_at":"2025-04-15T19:06:45.000Z","size":256,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-18T20:13:16.204Z","etag":null,"topics":["docker","grafana","logging","loki","monitoring","observability","opentelemetry","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lakinduboteju.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-15T18:41:01.000Z","updated_at":"2025-04-15T19:32:38.000Z","dependencies_parsed_at":"2025-04-18T07:59:58.095Z","dependency_job_id":null,"html_url":"https://github.com/lakinduboteju/otel-python-logging","commit_stats":null,"previous_names":["lakinduboteju/otel-python-logging"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lakinduboteju/otel-python-logging","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lakinduboteju%2Fotel-python-logging","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lakinduboteju%2Fotel-python-logging/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lakinduboteju%2Fotel-python-logging/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lakinduboteju%2Fotel-python-logging/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lakinduboteju","download_url":"https://codeload.github.com/lakinduboteju/otel-python-logging/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lakinduboteju%2Fotel-python-logging/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31846024,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T13:28:40.153Z","status":"ssl_error","status_checked_at":"2026-04-15T13:28:29.396Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","grafana","logging","loki","monitoring","observability","opentelemetry","python"],"created_at":"2025-04-18T07:59:54.710Z","updated_at":"2026-04-15T14:34:58.039Z","avatar_url":"https://github.com/lakinduboteju.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Python OpenTelemetry Logging Example\n\nThis project demonstrates how to integrate OpenTelemetry with Python's standard logging module to export logs to an OpenTelemetry Collector, which forwards them to Grafana Loki for visualization.\n\n## Architecture\n\nThis project implements a complete observability pipeline for logs using OpenTelemetry:\n\n```\n+-------------+       +------------------+       +------+       +---------+\n| Python App  | ----\u003e | OTel Collector   | ----\u003e | Loki | ----\u003e | Grafana |\n| (with OTel) |       | (Logs Processor) |       |      |       |         |\n+-------------+       +------------------+       +------+       +---------+\n```\n\nComponents:\n- **Python Application**: Uses standard Python logging with OpenTelemetry integration\n- **OpenTelemetry Collector**: Receives, processes, and exports logs\n- **Loki**: Log aggregation system designed for storing and querying logs\n- **Grafana**: Visualization platform for monitoring and observability\n\n## Project Structure\n\n```\n/\n├── compose.yaml         # Docker Compose configuration\n├── Dockerfile           # Docker configuration for the Python app\n├── loki.yaml            # Grafana Loki configuration\n├── main.py              # Example Python application that generates logs\n├── otel-collector.yaml  # OpenTelemetry Collector configuration\n├── otel_logging.py      # OpenTelemetry logging integration module\n└── pyproject.toml       # Python project dependencies\n```\n\n## Requirements\n\n- Python 3.11+\n- Docker and Docker Compose\n- Poetry (for dependency management)\n\n## Setup Instructions\n\n1. Clone this repository\n2. Build and start the Docker containers:\n\n```bash\ndocker-compose up -d\n```\n\nThis will start:\n- The Python application container\n- OpenTelemetry Collector\n- Loki log storage\n- Grafana dashboard\n\n3. Access Grafana at http://localhost:3000 (default credentials: admin/admin)\n4. Configure Loki as a data source in Grafana:\n   - Go to Configuration -\u003e Data Sources -\u003e Add data source\n   - Select Loki\n   - Set URL to http://loki:3100\n   - Click \"Save \u0026 Test\"\n\n## Code Explanation\n\n### OpenTelemetry Logging Setup\n\nThe `otel_logging.py` module contains the core functionality that integrates Python's standard logging with OpenTelemetry:\n\n```python\n# Key components:\n# 1. LoggerProvider - Manages the logging resources and processors\n# 2. OTLPLogExporter - Exports logs to the OTel Collector via gRPC\n# 3. BatchLogRecordProcessor - Batches logs for efficient export\n# 4. LoggingHandler - Connects standard Python logging to OTel\n```\n\nThe `setup_otel_logging` function configures:\n- A LoggerProvider with service name and instance ID\n- An OTLP exporter that sends logs to the collector\n- A batch processor for efficient log handling\n- A logging handler that's added to the root logger\n\n### Application Code\n\nThe `main.py` file demonstrates how to use the logging integration:\n\n1. Set up OpenTelemetry logging\n2. Create a logger instance\n3. Log messages at different levels\n4. Include structured attributes with log messages\n\n### OpenTelemetry Collector Configuration\n\nThe collector is configured in `otel-collector.yaml` to:\n1. Receive logs via OTLP/gRPC protocol\n2. Export logs to:\n   - Debug output (for troubleshooting)\n   - Loki storage backend\n\n## Running the Example\n\n1. Start the Docker environment:\n   ```bash\n   docker-compose up -d\n   ```\n\n2. Exec into the Python application container:\n   ```bash\n   docker exec -it otel-example-app bash\n   ```\n\n3. Run the example script:\n   ```bash\n   python main.py\n   ```\n\n4. View logs in Grafana:\n   - Open http://localhost:3000\n   - Navigate to Explore\n   - Select Loki data source\n   - Query for logs with labels like `{service_name=\"otel-example\"}`\n\n![Log visualization in Grafana](./screenshot.png)\n\n## Advanced Configuration\n\n### Customizing Log Attributes\n\nYou can add contextual information to logs using the `extra` parameter:\n\n```python\nlogger.info(\"Message with context\", extra={\n    \"user_id\": \"12345\",\n    \"request_id\": \"abcd-1234\"\n})\n```\n\n### Modifying OpenTelemetry Resource Attributes\n\nEdit the `otel_logging.py` file to change the service information:\n\n```python\nlogger_provider = LoggerProvider(\n    resource=Resource.create({\n        \"service.name\": \"your-service-name\",\n        \"service.instance.id\": \"instance-id\",\n        # Add additional resource attributes here\n        \"deployment.environment\": \"development\"\n    })\n)\n```\n\n## Troubleshooting\n\nCommon issues:\n- **Logs not appearing in Grafana**: Check OTel collector logs for connection issues\n- **Connection refused errors**: Verify that all services are running and properly networked\n- **Export failures**: Check that the OTel collector endpoint is correctly specified\n\n## References\n\n- [OpenTelemetry Python Documentation](https://opentelemetry.io/docs/instrumentation/python/)\n- [Grafana Loki Documentation](https://grafana.com/docs/loki/latest/)\n- [OpenTelemetry Collector Documentation](https://opentelemetry.io/docs/collector/)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flakinduboteju%2Fotel-python-logging","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flakinduboteju%2Fotel-python-logging","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flakinduboteju%2Fotel-python-logging/lists"}