{"id":19069117,"url":"https://github.com/cbos/observability-toolkit","last_synced_at":"2025-10-17T03:51:16.440Z","repository":{"id":218079495,"uuid":"745524384","full_name":"cbos/observability-toolkit","owner":"cbos","description":"Observability stack for local development and troubleshooting toolkit","archived":false,"fork":false,"pushed_at":"2025-03-10T08:58:57.000Z","size":2090,"stargazers_count":39,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-28T14:09:57.726Z","etag":null,"topics":["logs","metrics","observability","opentelemetry","opentelemetry-collector","traces"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cbos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-01-19T14:22:14.000Z","updated_at":"2025-04-18T09:36:38.000Z","dependencies_parsed_at":"2025-01-15T12:40:54.376Z","dependency_job_id":"9a4044ce-66b8-4e2e-9139-29e69e81ccee","html_url":"https://github.com/cbos/observability-toolkit","commit_stats":null,"previous_names":["cbos/observability-toolkit"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbos%2Fobservability-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbos%2Fobservability-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbos%2Fobservability-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbos%2Fobservability-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cbos","download_url":"https://codeload.github.com/cbos/observability-toolkit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251326849,"owners_count":21571636,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["logs","metrics","observability","opentelemetry","opentelemetry-collector","traces"],"created_at":"2024-11-09T01:13:23.063Z","updated_at":"2025-10-17T03:51:16.434Z","avatar_url":"https://github.com/cbos.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./docs/logo.png\" alt=\"Observability Toolkit\"/\u003e\u003cBR\u003e\n  \u003cimg src=\"https://goto.ceesbos.nl/badge/github/cbos/observability-toolkit\" alt=\"Badge\"/\u003e\n\u003c/div\u003e\n\n# Observability Toolkit for local usage\nA set of tools which can be used for local development which help with more insights in Observability.\n\n![](docs/setup.png)\n\n# Background\n\nTo get to know more about the background about this setup, please read: https://ceesbos.nl/posts/20240126-observability-toolkit/\n\n# Explanation of the components\n\n## OpenTelemetry Collector\n[OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) is the main entry point where all observability signals are collected.\nWith all kind of configuration you can choose what to do with it.\n\nOpenTelemetry Collector sends the data to:\n- [Loki](https://github.com/grafana/loki)   \n  Loki is a log storage created by Grafana\n- [Tempo](https://github.com/grafana/tempo)   \n  Tempo is a trace storage created by Grafana\n- [Prometheus](https://github.com/prometheus/prometheus)   \n  Prometheus is a metric storage\n\n## Grafana\n[Grafana](https://github.com/grafana/grafana) is the visualisation tool to get insights in all observability metrics collected and stored.\n\n# Power of correlation\nThe power of combining metrics, logs and traces together in 1 setup is that you can correlate these signals.    \n![](docs/correlation_between_signals.png)    \nThe advantage of OpenTelemetry is standardisation on naming, that makes it way easier to hop between the metrics, traces and logs in all directions.\nThat will help to find the problems.\n\nConfiguration in Grafana can help to make it easier to hop from metrics (with exemplars) to traces for example (see [prometheus.yaml](./config/grafana/provisioning/datasources/prometheus.yaml)).    \nThat is already configured in this setup. Same holds from logs to traces (see [loki.yaml](./config/grafana/provisioning/datasources/loki.yaml)).   \nAlso the setup of spanmetrics and servicegraph is already configured (see [tempo.yaml](./config/grafana/provisioning/datasources/tempo.yaml)).   \n\n# Start the setup\n\n```shell\ngit clone https://github.com/cbos/observability-toolkit\ncd observability-toolkit\ndocker compose up -d  \n\n# If you have just command runner installed you can run:\njust up\n```\nNow you can open http://localhost:3000 to open Grafana.\n\n# Run the demo app and generate load\nThere is a simple demo app, which helps to generate some traces, logs and metrics, but is not intended to show all capabilities.    \n\n\n```shell\n./run-observabilty-demo-app.sh \n```\n\nTo generate load:\n```shell\n./loadgen.sh \n```\n\n## OpenTelemetry Demo\nYou can use the [demo services of OpenTelemetry](https://opentelemetry.io/docs/demo/) to see what is all available for all languages.\nYou can change the file: `src/otelcollector/otelcol-config-extras.yml` and add this:\n\n```yaml\nexporters:\n  otlp/observabilitytoolkit:\n   # This assumes that the setup is running at the docker host on port 4317 (which is default of observability-toolkit)\n   endpoint: \"host.docker.internal:4317\"\n   tls:\n    insecure: true\n\nservice:\n  pipelines:\n   traces:\n    receivers: [otlp]\n    processors: [batch]\n    exporters: [otlp, otlp/observabilitytoolkit]\n   metrics:\n    receivers: [httpcheck/frontendproxy, redis, otlp]\n    processors: [batch]\n    exporters: [otlphttp/prometheus, otlp/observabilitytoolkit]\n   logs:\n    receivers: [otlp]\n    processors: [batch]\n    exporters: [otlp/observabilitytoolkit]\n```\n\n# Default dashboards\n- http://localhost:3000/d/observabilitystackOtelCollector/opentelemetry-collector\n![](docs/opentelemetry_collector_dashboard.png)\n\n# Settings \n\nAn number of settings can be tweaked by just setting environment variables.\n\n```shell\n# Specify the Grafana host port you want, other then the default 3000\nexport GRAFANA_HOST_PORT=3004\n\n# Start the stack (in the background with -d)\ndocker compose up -d \n\n# Now you can open Grafana at http://localhost:3004\n```\nSee [.env](.env) for actual values, below are the descriptions\n\n| Variable name                        | Description                                                                                                             |\n|--------------------------------------|-------------------------------------------------------------------------------------------------------------------------|\n| LOKI_IMAGE_NAME                      | Loki docker image                                                                                                       |\n| TEMPO_IMAGE_NAME                     | Tempo docker image                                                                                                      |\n| PROMETHEUS_IMAGE_NAME                | Prometheus docker image                                                                                                 |\n| GRAFANA_IMAGE_NAME                   | Grafana docker image                                                                                                    |\n| GRAFANA_HOST_PORT                    | Port on host on which Grafana will be available                                                                         |\n| OTEL_COLLECTOR_IMAGE_NAME            | OpenTelemetry Collector docker image                                                                                    |\n| OTEL_COLLECTOR_HOST_PORT_GRPC        | Port on host on which OpenTelemetry Collector will be available for OTLP format with GRPC                               |\n| OTEL_COLLECTOR_HOST_PORT_HTTP        | Port on host on which OpenTelemetry Collector will be available for OTLP format with HTTP                               |\n| OTEL_COLLECTOR_HOST_PORT_PROMETHEUS  | Port on host on which OpenTelemetry Collector will listen to expose prometheus data, like http://localhost:8889/metrics |\n| PROMTAIL_IMAGE_NAME                  | Promtail docker image                                                                                                   |\n| PYROSCOPE_IMAGE_NAME                 | Pyroscope docker image                                                                                                  |\n| PYROSCOPE_PORT                       | Port on which Pyroscope is available                                                                                    |\n\n\n# Inspired by \n\nThis setup is created based on what I already used locally.    \nThe idea of adding a simple application for showcasing the setup comes fromL [grafana/docker-otel-lgtm](https://github.com/grafana/docker-otel-lgtm)    \nThe script for generating random load with a simple curl command comes from the [Grafana Beyla project](https://github.com/grafana/beyla/blob/main/examples/greeting-apps/loadgen.sh)\n\nThe OpenTelemetry Collector Grafana Dashboard is coming from\nhttps://grafana.com/grafana/dashboards/15983-opentelemetry-collector/    \nBut it required some changes to get it working in this setup.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcbos%2Fobservability-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcbos%2Fobservability-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcbos%2Fobservability-toolkit/lists"}