{"id":15641601,"url":"https://github.com/blueswen/fastapi-jaeger","last_synced_at":"2025-10-17T13:52:09.870Z","repository":{"id":78525959,"uuid":"487194286","full_name":"blueswen/fastapi-jaeger","owner":"blueswen","description":"Trace FastAPI with Jaeger through OpenTelemetry Python API and SDK.","archived":false,"fork":false,"pushed_at":"2025-01-19T07:16:33.000Z","size":5082,"stargazers_count":99,"open_issues_count":0,"forks_count":25,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-30T19:57:27.749Z","etag":null,"topics":["fastapi","jaeger","opentelemetry"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blueswen.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"ko_fi":"blueswen"}},"created_at":"2022-04-30T05:51:32.000Z","updated_at":"2025-08-18T08:30:16.000Z","dependencies_parsed_at":"2023-03-25T22:03:15.255Z","dependency_job_id":"28bdc720-1168-4038-b37b-5a0a7ebca17f","html_url":"https://github.com/blueswen/fastapi-jaeger","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/blueswen/fastapi-jaeger","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blueswen%2Ffastapi-jaeger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blueswen%2Ffastapi-jaeger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blueswen%2Ffastapi-jaeger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blueswen%2Ffastapi-jaeger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blueswen","download_url":"https://codeload.github.com/blueswen/fastapi-jaeger/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blueswen%2Ffastapi-jaeger/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273728775,"owners_count":25157274,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-05T02:00:09.113Z","response_time":402,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastapi","jaeger","opentelemetry"],"created_at":"2024-10-03T11:43:19.323Z","updated_at":"2025-10-17T13:52:09.792Z","avatar_url":"https://github.com/blueswen.png","language":"Python","readme":"# FastAPI Tracing with Jaeger through OpenTelemetry\n\nTrace FastAPI with [Jaeger](https://www.jaegertracing.io/) through [OpenTelemetry Python API and SDK](https://github.com/open-telemetry/opentelemetry-python).\n\nThe span from the application could be collected with [Jaeger Collector](https://www.jaegertracing.io/docs/1.47/architecture/#collector)(jaeger-collector) or [OpenTelemetry Collector](https://www.jaegertracing.io/docs/1.47/architecture/#with-opentelemetry)(otel-collector):\n\n![Demo Project Architecture](./images/demo-arch.jpg)\n\nThere are four ways to push span:\n\n- A: Push span to OpenTelemetry Collector with gRPC (Port: 4317)\n- B: Push span to OpenTelemetry Collector over HTTP (Port: 4318)\n- C: Push span to Jaeger Collector with gRPC (Port: 4317)\n- D: Push span to Jaeger Collector with HTTP (Port: 4318)\n\nIn this architecture, OpenTelemetry Collector is an agent to collects, processes, and sends data to Jaeger Collector. Jaeger Collector is responsible for collecting span and writing span to DB, then Jaeger Query queries data from DB.\n\nJaeger Agent has been deprecated since version 1.43. Since version 1.43, OpenTelemetry SDK allows direct data transmission to Jaeger Collector or utilization of OpenTelemetry Collector as an Agent. If you still want to utilize Jaeger Agent for span collection, please refer to the previous version of this [project](https://github.com/blueswen/fastapi-jaeger/tree/jaeger-agent).\n\n## Table of contents\n\n- [FastAPI Tracing with Jaeger through OpenTelemetry](#fastapi-tracing-with-jaeger-through-opentelemetry)\n  - [Table of contents](#table-of-contents)\n  - [Quick Start](#quick-start)\n  - [Detail](#detail)\n    - [FastAPI Application](#fastapi-application)\n      - [Traces and Logs](#traces-and-logs)\n      - [Trace information](#trace-information)\n      - [Span Inject](#span-inject)\n    - [Jaeger](#jaeger)\n      - [Jaeger Collector](#jaeger-collector)\n      - [OpenTelemetry Collector](#opentelemetry-collector)\n      - [Storage](#storage)\n      - [Jaeger Query](#jaeger-query)\n  - [With Service Performance Monitoring](#with-service-performance-monitoring)\n    - [Quick Start](#quick-start-1)\n    - [Details](#details)\n      - [OpenTelemetry Collector](#opentelemetry-collector-1)\n      - [Prometheus](#prometheus)\n      - [Jaeger Query](#jaeger-query-1)\n  - [With Grafana and Loki](#with-grafana-and-loki)\n    - [Quick Start](#quick-start-2)\n    - [Explore with Grafana](#explore-with-grafana)\n      - [Traces to Logs](#traces-to-logs)\n      - [Logs to Traces](#logs-to-traces)\n    - [Detail](#detail-1)\n      - [Jaeger - Traces](#jaeger---traces)\n        - [Grafana Data Source](#grafana-data-source)\n      - [Loki - Logs](#loki---logs)\n        - [Loki Docker Driver](#loki-docker-driver)\n        - [Grafana Data Source](#grafana-data-source-1)\n      - [Grafana](#grafana)\n- [Reference](#reference)\n\n\n## Quick Start\n\n1. Start all services with docker-compose\n\n   ```bash\n   docker-compose up -d\n   ```\n\n   It may take some time for DB(Cassandra) to initialize. You can run `docker-compose ps` to check the `jaeger-query` status is running when DB is ready as below:\n\n   ![Container Status](./images/containers.png)\n\n2. Send requests with `curl` to the FastAPI application\n\n   ```bash\n   curl http://localhost:8000/chain\n   ```\n\n3. Check on Jaeger UI [http://localhost:16686/](http://localhost:16686/)\n\n   Jaeger UI screenshot:\n\n   ![Jaeger UI](./images/jaeger-ui.png)\n\n   ![Jaeger UI Trace](./images/jaeger-ui-trace.png)\n\n## Detail\n\n### FastAPI Application\n\nFor a more complex scenario, we use three FastAPI applications with the same code in this demo. There is a cross-service action in `/chain` endpoint, which provides a good example of how to use OpenTelemetry SDK process span and how Jaeger Query presents trace information.\n\n#### Traces and Logs\n\nUtilize [OpenTelemetry Python SDK](https://github.com/open-telemetry/opentelemetry-python) to send span to Jaeger. Each request span contains other child spans when using OpenTelemetry instrumentation. The reason is that instrumentation will catch each internal asgi interaction ([opentelemetry-python-contrib issue #831](https://github.com/open-telemetry/opentelemetry-python-contrib/issues/831#issuecomment-1005163018)). If you want to get rid of the internal spans, there is a [workaround](https://github.com/open-telemetry/opentelemetry-python-contrib/issues/831#issuecomment-1116225314) in the same issue #831 by using a new OpenTelemetry middleware with two overridden methods of span processing.\n\nUtilize [OpenTelemetry Logging Instrumentation](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/logging/logging.html) to apply logger format with trace id and span id.\n\n```py\n# fastapi_app/main.py\n\n# otlp-grpc, otlp-http\nMODE = os.environ.get(\"MODE\", \"otlp-grpc\")\n\nOTLP_GRPC_ENDPOINT = os.environ.get(\"OTLP_GRPC_ENDPOINT\", \"jaeger-collector:4317\")\nOTLP_HTTP_ENDPOINT = os.environ.get(\n    \"OTLP_HTTP_ENDPOINT\", \"http://jaeger-collector:4318/v1/traces\"\n)\n\ndef setting_jaeger(app: ASGIApp, log_correlation: bool = True) -\u003e None:\n    # set the tracer provider\n    tracer = TracerProvider()\n    trace.set_tracer_provider(tracer)\n\n    if MODE == \"otlp-grpc\":\n        tracer.add_span_processor(\n            BatchSpanProcessor(\n                OTLPSpanExporterGRPC(endpoint=OTLP_GRPC_ENDPOINT, insecure=True)\n            )\n        )\n    elif MODE == \"otlp-http\":\n        tracer.add_span_processor(\n            BatchSpanProcessor(OTLPSpanExporterHTTP(endpoint=OTLP_HTTP_ENDPOINT))\n        )\n    else:\n        # default otlp-grpc\n        tracer.add_span_processor(\n            BatchSpanProcessor(\n                OTLPSpanExporterGRPC(endpoint=OTLP_GRPC_ENDPOINT, insecure=True)\n            )\n        )\n\n    # override logger format which with trace id and span id\n    if log_correlation:\n        LoggingInstrumentor().instrument(set_logging_format=True)\n\n    FastAPIInstrumentor.instrument_app(app, tracer_provider=tracer)\n```\n\n#### Trace information\n\nThe instrumentation library will collect trace information automatically, e.g. HTTP status code, HTTP method, HTTP URL, etc. We can also add custom attributes to the span with SDK.\n\n```py\nfrom opentelemetry.sdk.resources import Resource\nfrom opentelemetry.sdk.trace import TracerProvider\n\nresource = Resource.create(attributes={\n  \"service.name\": \"fastapi-app\",\n  \"custom.data\": \"custom_data\",\n})\n\ntracer = TracerProvider(resource=resource)\n```\n\nOr we can use environment variables to set the attributes and service name(the service name displayed on Jaeger UI), which is used in this demo. Following is the example of setting environment variables in the compose file.\n\n```yaml\nservices:\n  app:\n    image: ghcr.io/blueswen/fastapi-jaeger/app:latest\n    environment:\n      OTEL_SERVICE_NAME: \"fastapi-app\"\n      OTEL_RESOURCE_ATTRIBUTES: \"custom.data=custom_data\"\n```\n\n#### Span Inject\n\nIf we want other services to use the same Trace ID as the source application, we have to pass the Trace ID to the header of the request. In this demo, we use `inject` function to add current span information to the header. Because OpenTelemetry FastAPI instrumentation only takes care of the asgi app's request and response, it does not affect any other modules or actions like sending HTTP requests to other servers or function calls.\n\n```py\n# fastapi_app/main.py\n\nfrom opentelemetry.propagate import inject\n\n@app.get(\"/chain\")\nasync def chain(response: Response):\n    headers = {}\n    inject(headers)  # inject trace info to header\n\n    async with httpx.AsyncClient() as client:\n        await client.get(f\"http://localhost:8000/\", headers=headers,)\n    async with httpx.AsyncClient() as client:\n        await client.get(f\"http://{TARGET_ONE_HOST}:8000/io_task\", headers=headers,)\n    async with httpx.AsyncClient() as client:\n        await client.get(f\"http://{TARGET_TWO_HOST}:8000/cpu_task\", headers=headers,)\n\n    return {\"path\": \"/chain\"}\n```\n\nOr we can just use the instrumentation library for [different libraries](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation#readme). Like [OpenTelemetry HTTPX Instrumentation](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-httpx) or [OpenTelemetry Requests Instrumentation](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-requests) according to the library we use. Following is the example of using OpenTelemetry HTTPX Instrumentation which will automatically inject trace info to the header.\n\n```py\nimport httpx\nfrom opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor\n\nHTTPXClientInstrumentor().instrument()\n\n@app.get(\"/chain\")\nasync def chain(response: Response):\n    async with httpx.AsyncClient() as client:\n        await client.get(f\"http://localhost:8000/\")\n    async with httpx.AsyncClient() as client:\n        await client.get(f\"http://{TARGET_ONE_HOST}:8000/io_task\")\n    async with httpx.AsyncClient() as client:\n        await client.get(f\"http://{TARGET_TWO_HOST}:8000/cpu_task\")\n\n    return {\"path\": \"/chain\"}\n```\n\n### Jaeger\n\nThere is an [all-in-one](https://www.jaegertracing.io/docs/1.33/getting-started/#all-in-one) Jaeger for quick testing, but in production running Jaeger backend components as a scalable distributed system is the most common method, as illustrated below.\n\n![Jaeger Architecture](./images/architecture-v1-2023.png)\n\nImage Source: [Jaeger](https://www.jaegertracing.io/docs/1.47/architecture/#direct-to-storage)\n\nOr with OpenTelemetry Collector:\n\n![Jaeger Architecture with OpenTelemetry Collector](./images/architecture-otel.png)\n\nImage Source: [Jaeger](https://www.jaegertracing.io/docs/1.47/architecture/#with-opentelemetry)\n\nWe create the Jaeger backend of this demo project based on the [docker compose example](https://github.com/jaegertracing/jaeger/blob/main/docker-compose/jaeger-docker-compose.yml) from Jaeger's official repository.\n\nCheck more details on [Jaeger docs about architecture](https://www.jaegertracing.io/docs/1.47/architecture/).\n\n#### Jaeger Collector\n\nThe Jaeger collector receives traces from OpenTelemetry SDKs or OpenTelemetry Agent and runs them through a processing pipeline.\n\n```yaml\n# docker-compose.yaml\nservices:\n  jaeger-collector:\n    image: jaegertracing/jaeger-collector:1.57.0\n    command: \n      - \"--cassandra.keyspace=jaeger_v1_dc1\"\n      - \"--cassandra.servers=cassandra\"\n      - \"--sampling.initial-sampling-probability=.5\"\n      - \"--sampling.target-samples-per-second=.01\"\n      - \"--collector.otlp.enabled=true\"\n    environment: \n      - SAMPLING_CONFIG_TYPE=adaptive\n    ports:\n      - \"4317\" # accept OpenTelemetry Protocol (OTLP) over gRPC\n      - \"4318\" # accept OpenTelemetry Protocol (OTLP) over HTTP\n    restart: on-failure\n    depends_on:\n      - cassandra-schema\n```\n\nCheck more details on Jaeger docs [Deployment about Collector](https://www.jaegertracing.io/docs/1.47/deployment/#collector), [CLI flags about Collector](https://www.jaegertracing.io/docs/1.47/cli/#jaeger-collector), and [Sampling](https://www.jaegertracing.io/docs/1.47/sampling/).\n\n#### OpenTelemetry Collector\n\nThe OpenTelemetry Collector receives traces from OpenTelemetry SDKs and processes them according to the configuration file `etc/otel-collector-config.yaml`.\n\n```yaml\n# docker-compose.yaml\nservices:\n  otel-collector:\n    image: otel/opentelemetry-collector-contrib:0.100.0\n    command:\n      - \"--config=/conf/config.yaml\"\n    volumes:\n      - ./etc/otel-collector-config.yaml:/conf/config.yaml\n    ports:\n      - \"4317\" # OTLP gRPC receiver\n      - \"4318\" # OTLP http receiver\n    restart: on-failure\n    depends_on:\n      - jaeger-collector\n```\n\n```yaml\n# etc/otel-collector-config.yaml\nreceivers:\n  otlp:\n    protocols:\n      grpc:\n      http: \n\nexporters:\n  otlp:\n    endpoint: jaeger-collector:4317\n    tls:\n      insecure: true\n  otlphttp:\n    endpoint: http://jaeger-collector:4318\n    tls:\n      insecure: true\n\nprocessors:\n  batch:\n\nservice:\n  pipelines:\n    traces:\n      receivers: [otlp]\n      processors: [batch]\n      exporters: [otlp] # export to jaeger-collector:4317\n      # exporters: [otlphttp] # export to jaeger-collector:4318\n```\n\nCheck more details on OpenTelemetry Collector docs [Configuration](https://opentelemetry.io/docs/collector/configuration/).\n\n#### Storage\n\nAll traces collected by Jaeger Collector will be validated, indexed, and then stored in storage. Jaeger supports multiple span storage backends:\n\n1. Cassandra 3.4+\n2. Elasticsearch 5.x, 6.x, 7.x\n3. Kafka\n4. memory storage\n5. Storage plugin\n\nIn this demo, we use Cassandra as the storage backend.\n\n```yaml\n# docker-compose.yaml\nservices:\n  # Cassandra instance container\n  cassandra:\n    image: cassandra:4.1.4\n\n  # initialize Cassandra\n  cassandra-schema:\n    image: jaegertracing/jaeger-cassandra-schema:1.57.0\n    depends_on:\n      - cassandra\n```\n\nIf you want to mitigate the pressure on Cassandra and prevent data loss, you can add Kafka and Jaeger Ingester to the architecture to process data asynchronously. The architecture will look like this:\n\n![Jaeger Architecture](./images/architecture-v2-2023.png)\n\nImage Source: [Jaeger](https://www.jaegertracing.io/docs/1.47/architecture/#via-kafka)\n\nThere is a [docker compose example](https://github.com/jaegertracing/jaeger/tree/main/docker-compose/kafka) on Jaeger's official repository.\n\nCheck more details on Jaeger docs [Deployment about Span Storage Backends](https://www.jaegertracing.io/docs/1.47/deployment/#span-storage-backends).\n\n#### Jaeger Query\n\nThe Jaeger Query is a service that retrieves traces from storage and hosts a UI to display them.\n\n```yaml\n# docker-compose.yaml\nservices:\n  jaeger-query:\n    image: jaegertracing/jaeger-query:1.57.0\n    command:\n      - \"--cassandra.keyspace=jaeger_v1_dc1\"\n      - \"--cassandra.servers=cassandra\"\n    ports:\n      - \"16686:16686\"\n      - \"16687:16687\"\n    restart: on-failure\n    depends_on:\n      - cassandra-schema\n```\n\nCheck more details on Jaeger docs [Deployment about Query Service \u0026 UI](https://www.jaegertracing.io/docs/1.47/deployment/#query-service--ui).\n\n## With Service Performance Monitoring\n\nJaeger Service Performance Monitoring become stable since Jaeger 1.43.0. It provides a new way to monitor the performance of services, which extracts the RED (Request, Error, Duration) metrics from span data. The data flow is shown below:\n\n![Demo Project Architecture with SPM](./images/demo-arch-spm.jpg)\n\n### Quick Start\n\n1. Start all services with docker-compose\n\n   ```bash\n   docker-compose -f docker-compose-spm.yaml up -d\n   ```\n\n   It may take some time for DB(Cassandra) to initialize. You can run `docker-compose ps` to check the `jaeger-query` status is running when DB is ready.\n\n2. Send requests with [siege](https://linux.die.net/man/1/siege) and curl to the FastAPI app\n\n   ```bash\n   bash request-script.sh\n   ```\n\n3. Check on Jaeger UI Monitoring tab [http://localhost:16686/monitor](http://localhost:16686/monitor)\n\n   Jaeger UI Monitoring tab screenshot:\n\n   ![Jaeger UI Monitoring Tab](./images/jaeger-ui-monitor.png)\n\n### Details\n\nTo enable Service Performance Monitoring, we need to:\n\n1. Add some configurations to OpenTelemetry Collector for extracting RED metrics from span data and exporting them to Prometheus.\n2. Create a Prometheus instance to store the metrics.\n3. Update Jaeger Query configurations to scrape metrics from Prometheus.\n\n#### OpenTelemetry Collector\n\nThe OpenTelemetry Collector receives traces from OpenTelemetry SDKs and processes them according to the configuration file `etc/otel-collector-config-spm.yaml`.\n\n```yaml\n# docker-compose-spm.yaml\nservice:\n  otel-collector:\n    image: otel/opentelemetry-collector-contrib:0.81.0\n    command:\n      - \"--config=/conf/config.yaml\"\n    volumes:\n      - ./etc/otel-collector-config-spm.yaml:/conf/config.yaml\n    ports:\n      - \"4317\" # OTLP gRPC receiver\n      - \"4318\" # OTLP http receiver\n      - \"8889\" # Prometheus metrics exporter\n    restart: on-failure\n    depends_on:\n      - jaeger-collector\n```\n\n```yaml\n# etc/otel-collector-config-spm.yaml\nreceivers:\n  otlp:\n    protocols:\n      grpc:\n      http: \n\nexporters:\n  otlp:\n    endpoint: jaeger-collector:4317\n    tls:\n      insecure: true\n  prometheus:\n    endpoint: \"0.0.0.0:8889\"\n\nconnectors:\n  spanmetrics:\n\nprocessors:\n  batch:\n\nservice:\n  pipelines:\n    traces:\n      receivers: [otlp]\n      processors: [batch]\n      exporters: [spanmetrics, otlp]\n    metrics/spanmetrics:\n      receivers: [spanmetrics]\n      exporters: [prometheus]\n```\n\nTo generate metrics from span data, we need to:\n\n1. Add `spanmetrics` to `connectors`: Enable and configure the spanmetrics connector\n   1. dimensions: Extract span attributes to Prometheus labels\n2. Add `spanmetrics` to traces pipeline `exporters`: Let the traces pipeline export traces to the spanmetrics connector\n3. Add `spanmetrics` to metrics pipeline `receivers`: Set the spanmetrics connector as the receiver of the metrics pipeline, and the data is from the traces pipeline exporter\n4. Add `prometheus` to metrics pipeline `exporters`: Expose metrics in Prometheus format on port 8889\n\nThe pipeline diagram and configuration file are as follows:\n\n![Span Metrics Pipeline](./images/spanmetrics-pipeline.png)\n\n#### Prometheus\n\nPrometheus collects metrics from OpenTelemetry Collector and stores them in its database. The metrics can be scraped by Jaeger Query.\n\n```yaml\n# docker-compose-spm.yaml\nservice:\n  prometheus:\n    image: prom/prometheus:v2.51.2\n    ports:\n      - \"9090:9090\"\n    volumes:\n      - ./etc/prometheus.yml:/workspace/prometheus.yml\n    command:\n      - --config.file=/workspace/prometheus.yml\n```\n\n```yaml\n# etc/prometheus.yml\nglobal:\n  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.\n  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.\n  # scrape_timeout is set to the global default (10s).\n\nscrape_configs:\n  - job_name: aggregated-trace-metrics\n    static_configs:\n    - targets: ['otel-collector:8889']\n```\n\n#### Jaeger Query\n\nJaeger Query scrapes metrics from Prometheus and displays them on the Monitoring tab.\n\n```yaml\n# docker-compose-spm.yaml\nservice:\n  jaeger-query:\n    image: jaegertracing/jaeger-query:1.57.0\n    environment:\n      - METRICS_STORAGE_TYPE=prometheus\n    command:\n      - \"--cassandra.keyspace=jaeger_v1_dc1\"\n      - \"--cassandra.servers=cassandra\"\n      - \"--prometheus.query.support-spanmetrics-connector=true\"\n      - \"--prometheus.server-url=http://prometheus:9090\"\n      - \"--prometheus.query.normalize-duration=true\"\n      - \"--prometheus.query.normalize-calls=true\"\n    ports:\n      - \"16686:16686\"\n      - \"16687:16687\"\n    restart: on-failure\n    depends_on:\n      - cassandra-schema\n```\n\n## With Grafana and Loki\n\nOnly viewing the trace information on Jaeger UI may not be good enough. How about also tracing with logs at the same time? [Grafana](https://github.com/grafana/grafana) started supporting [Jaeger data source](https://grafana.com/docs/grafana/latest/datasources/jaeger/) since Grafana 7.4+, and also provides a good log aggregation system [Loki](https://github.com/grafana/loki/). Grafana provides a great user experience tracing information across logs and traces.\n\n![Demo Project Architecture with Loki and Grafana](./images/demo-arch-grafana.jpg)\n\n### Quick Start\n\n1. Install [Loki Docker Driver](https://grafana.com/docs/loki/latest/clients/docker-driver/)\n\n   ```bash\n   docker plugin install grafana/loki-docker-driver:2.9.2 --alias loki --grant-all-permissions\n   ```\n\n2. Start all services with docker-compose\n\n   ```bash\n   docker-compose -f docker-compose-grafana.yaml up -d\n   ```\n\n   It may take some time for DB(Cassandra) to initialize. You can run `docker-compose ps` to check the `jaeger-query` status is running when DB is ready.\n\n   If got the error message `Error response from daemon: error looking up logging plugin loki: plugin loki found but disabled`, please run the following command to enable the plugin:\n\n   ```bash\n    docker plugin enable loki\n    ```\n\n3. Send requests with `curl` to the FastAPI application\n\n   ```bash\n   curl http://localhost:8000/chain\n   ```\n\n4. Explore the collected span on Grafana [http://localhost:3000/](http://localhost:3000/) with default user `admin` and password `admin`\n\n   Grafana Explore screenshot:\n\n   ![Explore Jaeger on Grafana](./images/explore-jaeger.png)\n\n### Explore with Grafana\n\n#### Traces to Logs\n\nGet Trace ID and tags defined in Jaeger data source from span, then query with Loki.\n\n![Traces to Logs](./images/traces-to-logs.png)\n\n#### Logs to Traces\n\nGet Trace ID pared from log (regex defined in Loki data source), then query in Jaeger.\n\n![Logs to Traces](./images/logs-to-traces.png)\n\n### Detail\n\n#### Jaeger - Traces\n\nReceives spans from applications.\n\n##### Grafana Data Source\n\n[Trace to logs](https://grafana.com/docs/grafana/latest/datasources/jaeger/#trace-to-logs) setting:\n\n1. Data source: target log source\n2. Tags: key of tags or process level attributes from the trace, which will be log query criteria if the key exists in the trace\n3. Map tag names: Convert existing key of tags or process level attributes from trace to another key, then used as log query criteria. Use this feature when the values of the trace tag and log label are identical but the keys are different.\n\nIn this demo, we use Loki Docker Driver to collect logs from applications. The Loki Docker Driver will add the following labels to the log: `container_name`, `compose_project`, `compose_service`, `source`, `filename`, and `host`. We have to map an attribute from the trace to a label of the log to query the log with the trace information. So we add a tag `compose_service` through environment variable `OTEL_RESOURCE_ATTRIBUTES` to the trace, then map the tag `compose_service` to the label `compose_service` of the log.\n\n```yaml\n# docker-compose-grafana.yaml\n\nservices:\n  app-a:\n    image: ghcr.io/blueswen/fastapi-jaeger/app:latest\n    depends_on:\n      - loki\n    ports:\n      - \"8000:8000\"\n    logging: *default-logging\n    environment:\n      MODE: \"otlp-grpc\"\n      OTEL_SERVICE_NAME: \"app-a\"\n      OTLP_GRPC_ENDPOINT: \"otel-collector:4317\"\n      OTEL_RESOURCE_ATTRIBUTES: \"compose_service=app-a\"\n```\n\nThe Jaeger data source setting in Grafana is as below:\n\n![Data Source of Jaeger: Trace to logs](./images/jaeger-trace-to-logs.png)\n\nWhich is produced by the following config file:\n\n```yaml\n# etc/grafana/datasource.yml\nname: Jaeger\ntype: jaeger\ntypeName: Jaeger\naccess: proxy\nurl: http://jaeger-query:16686\nuser: ''\ndatabase: ''\nbasicAuth: false\nisDefault: false\njsonData:\n  nodeGraph:\n    enabled: true\n  tracesToLogs:\n    datasourceUid: loki\n    filterBySpanID: false\n    filterByTraceID: true\n    mapTagNamesEnabled: true\n    mappedTags:\n      - key: compose_service\n```\n\n#### Loki - Logs\n\nCollects logs with Loki Docker Driver from applications.\n\n##### Loki Docker Driver\n\n1. Use [YAML anchor and alias](https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/) feature to set logging options for each service.\n2. Set [Loki Docker Driver options](https://grafana.com/docs/loki/latest/clients/docker-driver/configuration/)\n   1. loki-url: loki service endpoint\n   2. loki-pipeline-stages: processes multiline log from FastAPI application with multiline and regex stages ([reference](https://grafana.com/docs/loki/latest/clients/promtail/stages/multiline/))\n\n```yaml\n# docker-compose-grafana.yaml\nx-logging: \u0026default-logging # anchor(\u0026): 'default-logging' for defines a chunk of configuration\n  driver: loki\n  options:\n    loki-url: 'http://localhost:3100/api/prom/push'\n    loki-pipeline-stages: |\n      - multiline:\n          firstline: '^\\d{4}-\\d{2}-\\d{2} \\d{1,2}:\\d{2}:\\d{2}'\n          max_wait_time: 3s\n      - regex:\n          expression: '^(?P\u003ctime\u003e\\d{4}-\\d{2}-\\d{2} \\d{1,2}:\\d{2}:\\d{2},d{3}) (?P\u003cmessage\u003e(?s:.*))$$'\n# Use $$ (double-dollar sign) when your configuration needs a literal dollar sign.\n\nversion: \"3.4\"\n\nservices:\n  foo: # sample service\n    image: foo\n    logging: *default-logging # alias(*): refer to 'default-logging' chunk \n```\n\n##### Grafana Data Source\n\nAdd a TraceID derived field to extract the trace id and create a Jaeger link from the trace id.\n\nGrafana data source setting example:\n\n![Data Source of Loki: Derived fields](./images/loki-derive-filed.png)\n\nGrafana data source config example:\n\n```yaml\n# etc/grafana/datasource.yml\nname: Loki\ntype: loki\ntypeName: Loki\naccess: proxy\nurl: http://loki:3100\npassword: ''\nuser: ''\ndatabase: ''\nbasicAuth: false\nisDefault: false\njsonData:\nderivedFields:\n   - datasourceUid: jaeger\n      matcherRegex: (?:trace_id)=(\\w+)\n      name: TraceID\n      url: $${__value.raw}\n      # Use $$ (double-dollar sign) when your configuration needs a literal dollar sign.\nreadOnly: false\neditable: true\n```\n\n#### Grafana\n\n1. Add Jaeger, and Loki to the data source with config file ```etc/grafana/datasource.yml```.\n\n```yaml\n# grafana in docker-compose-grafana.yaml\nservice:\n  grafana:\n    image: grafana/grafana:10.4.2\n    ports:\n      - \"3000:3000\"\n    volumes:\n      - ./etc/grafana/:/etc/grafana/provisioning/datasources\n```\n\n# Reference\n\n1. [Jaeger](https://www.jaegertracing.io/)\n2. [Official Jaeger docker compose example](https://github.com/jaegertracing/jaeger/blob/main/docker-compose/jaeger-docker-compose.yml)\n3. [Official Jaeger with SPM docker compose example](https://github.com/jaegertracing/jaeger/tree/main/docker-compose/monitor)\n4. [Difference between OpenTelemetry Collector and OpenTelemetry Collector Contrib](https://uptrace.dev/opentelemetry/collector.html#otelcol-vs-otelcol-contrib)\n","funding_links":["https://ko-fi.com/blueswen"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblueswen%2Ffastapi-jaeger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblueswen%2Ffastapi-jaeger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblueswen%2Ffastapi-jaeger/lists"}