{"id":15137225,"url":"https://github.com/blueswen/spring-boot-observability","last_synced_at":"2025-05-16T13:02:47.557Z","repository":{"id":65286302,"uuid":"540778378","full_name":"blueswen/spring-boot-observability","owner":"blueswen","description":"Observe Spring Boot app with three pillars of observability: Traces (Tempo), Metrics (Prometheus), Logs (Loki) on Grafana through OpenTelemetry and OpenMetrics.","archived":false,"fork":false,"pushed_at":"2025-05-12T03:12:31.000Z","size":19384,"stargazers_count":301,"open_issues_count":13,"forks_count":105,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-12T19:18:52.629Z","etag":null,"topics":["grafana","loki","observability","openmetrics","opentelemetry","prometheus","spring-boot","tempo"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blueswen.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"ko_fi":"blueswen"}},"created_at":"2022-09-24T09:38:22.000Z","updated_at":"2025-05-09T17:36:52.000Z","dependencies_parsed_at":"2024-05-09T16:42:50.725Z","dependency_job_id":"4f625c42-90b7-49d9-8cfc-6966e749508b","html_url":"https://github.com/blueswen/spring-boot-observability","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blueswen%2Fspring-boot-observability","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blueswen%2Fspring-boot-observability/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blueswen%2Fspring-boot-observability/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blueswen%2Fspring-boot-observability/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blueswen","download_url":"https://codeload.github.com/blueswen/spring-boot-observability/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254535792,"owners_count":22087397,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grafana","loki","observability","openmetrics","opentelemetry","prometheus","spring-boot","tempo"],"created_at":"2024-09-26T07:00:30.477Z","updated_at":"2025-05-16T13:02:47.531Z","avatar_url":"https://github.com/blueswen.png","language":"Java","funding_links":["https://ko-fi.com/blueswen"],"categories":[],"sub_categories":[],"readme":"# Spring Boot with Observability\n\nObserve the Spring Boot application with three pillars of observability on [Grafana](https://github.com/grafana/grafana):\n\n1. Traces with [Tempo](https://github.com/grafana/tempo) and [OpenTelemetry Instrumentation for Java](https://github.com/open-telemetry/opentelemetry-java-instrumentation)\n2. Metrics with [Prometheus](https://prometheus.io/), [Spring Boot Actuator](https://docs.spring.io/spring-boot/docs/current/actuator-api/htmlsingle/), and [Micrometer](https://micrometer.io/)\n3. Logs with [Loki](https://github.com/grafana/loki) and [Logback](https://logback.qos.ch/)\n\n![Observability Architecture](./images/observability-arch.jpg)\n\nThis demo project is a Spring Boot version of [FastAPI with Observability](https://github.com/blueswen/fastapi-observability) and is also inspired by [Cloud Observability with Grafana and Spring Boot](https://github.com/qaware/cloud-observability-grafana-spring-boot).\n\n## Table of contents\n\n- [Spring Boot with Observability](#spring-boot-with-observability)\n  - [Table of contents](#table-of-contents)\n  - [Quick Start](#quick-start)\n  - [Explore with Grafana](#explore-with-grafana)\n    - [Metrics to Traces](#metrics-to-traces)\n    - [Traces to Logs](#traces-to-logs)\n    - [Logs to Traces](#logs-to-traces)\n  - [Detail](#detail)\n    - [Spring Boot Application](#spring-boot-application)\n      - [OpenTelemetry Instrumentation](#opentelemetry-instrumentation)\n      - [Logs](#logs)\n      - [Traces](#traces)\n      - [Metrics](#metrics)\n        - [Metrics with Exemplar](#metrics-with-exemplar)\n    - [Prometheus - Metrics](#prometheus---metrics)\n      - [Prometheus Config](#prometheus-config)\n      - [Grafana Data Source](#grafana-data-source)\n    - [Tempo - Traces](#tempo---traces)\n      - [Grafana Data Source](#grafana-data-source-1)\n    - [Loki - Logs](#loki---logs)\n      - [Loki Docker Driver](#loki-docker-driver)\n      - [Grafana Data Source](#grafana-data-source-2)\n    - [Grafana](#grafana)\n  - [Reference](#reference)\n\n## Quick Start\n\n1. Install [Loki Docker Driver](https://grafana.com/docs/loki/latest/clients/docker-driver/)\n\n   ```bash\n   docker plugin install grafana/loki-docker-driver:2.9.2 --alias loki --grant-all-permissions\n   ```\n\n2. Start all services with docker-compose\n\n   ```bash\n   docker-compose up -d\n   ```\n\n3. Send requests with [siege](https://linux.die.net/man/1/siege) and curl to the Spring Boot app\n\n   ```bash\n   bash request-script.sh\n   bash trace.sh\n   ```\n\n   Or you can send requests with [k6](https://k6.io/):\n\n   ```bash\n   k6 run --vus 3 --duration 300s k6-script.js\n   ```\n\n   Or send requests from applications' Swagger UI:\n\n    - app-a: [http://localhost:8080/swagger-ui/index.html](http://localhost:8080/swagger-ui/index.html)\n    - app-b: [http://localhost:8081/swagger-ui/index.html](http://localhost:8081/swagger-ui/index.html)\n    - app-c: [http://localhost:8082/swagger-ui/index.html](http://localhost:8082/swagger-ui/index.html)\n\n4. Check predefined dashboard ```Spring Boot Observability``` on Grafana [http://localhost:3000/](http://localhost:3000/) and login with default account ```admin``` and password ```admin```\n\n   Dashboard screenshot:\n\n   ![Spring Boot Monitoring Dashboard](./images/dashboard.png)\n\n   The dashboard is also available on [Grafana Dashboards](https://grafana.com/grafana/dashboards/17175).\n\n## Explore with Grafana\n\nGrafana provides a great solution, which could observe specific actions in service between traces, metrics, and logs through trace ID and exemplar.\n\n![Observability Correlations](./images/observability-correlations.jpeg)\n\nImage Source: [Grafana](https://grafana.com/blog/2021/03/31/intro-to-exemplars-which-enable-grafana-tempos-distributed-tracing-at-massive-scale/)\n\n### Metrics to Traces\n\nGet Trace ID from an exemplar in metrics, then query in Tempo.\n\nQuery with ```histogram_quantile(.99,sum(rate(http_server_requests_seconds_bucket{application=\"app-a\", uri!=\"/actuator/prometheus\"}[1m])) by(uri, le))``` and turn on Exemplars in options.\n\n![Metrics to Traces](./images/metrics-to-traces.png)\n\n### Traces to Logs\n\nGet Trace ID and tags (here is `compose.service`) defined in Tempo data source from span, then query with Loki.\n\n![Traces to Logs](./images/traces-to-logs.png)\n\n### Logs to Traces\n\nGet Trace ID from log (regex defined in Loki data source), then query in Tempo.\n\n![Logs to Traces](./images/logs-to-traces.png)\n\n## Detail\n\n### Spring Boot Application\n\nFor a more complex scenario, we use three Spring Boot applications with the same code in this demo. There is a cross-service action in `/chain` endpoint, which provides a good example of how OpenTelemetry Instrumentation work and how Grafana presents trace information.\n\n#### OpenTelemetry Instrumentation\n\n[OpenTelemetry Instrumentation for Java](https://github.com/open-telemetry/opentelemetry-java-instrumentation) provides a [zero-code way](https://opentelemetry.io/docs/zero-code/java/agent/) to instrument the application by the agent jar as follows:\n\n```bash\njava -javaagent:path/to/opentelemetry-javaagent.jar -jar myapp.jar\n```\n\nThe agent supports a lot of [libraries](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md), including Spring Web MVC. According to the document:\n\n\u003e It can be used to capture telemetry data at the “edges” of an app or service, such as inbound requests, outbound HTTP calls, database calls, and so on.\n\nSo we don't need to modify any line of code in our codebase. The agent will handle everything automatically. In this project, we have three kinds of actions that could be captured by the agent:\n\n1. HTTP requests: capture HTTP information like request method, status, and so on.\n   \n    ![Span data of HTTP requests](./images/span-data-http.png)\n   \n2. PostgreSQL actions(POST `/peanuts` and the first request of GET `/peanuts/{id}`): capture DB information like SQL statement, table name, and so on. \n\n    ![Span data of PostgreSQL actions](./images/span-data-postgresql.png)\n   \n3. Redis commands(From the second request of GET `/peanuts/{id}`): capture Redis information like commands, keys, and so on.\n\n    ![Span data of Redis commands](./images/span-data-redis.png)\n\nThe configurations, like the exporter setting, are listed on the [document](https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure), and are consumed by the agent from one or more of the following sources (ordered from highest to lowest priority):\n\n- system properties\n- environment variables\n- the configuration file\n- the [ConfigPropertySource](https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure#customizing-the-opentelemetry-sdk) SPI\n\nIn this demo project we use environment variables to set the agent configuration:\n\n```yaml\napp-a:\n  build: ./app/\n  environment:\n    - OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4317\n    - OTEL_SERVICE_NAME=app-a\n    - OTEL_RESOURCE_ATTRIBUTES=compose_service=app-a\n    - OTEL_METRICS_EXPORTER=none\n  ports:\n    - \"8080:8080\"\n```\n\nOr using a configuration file is another common way to set the agent:\n\n```properties\n# otel.properties\notel.exporter.otlp.endpoint=http://tempo:4317\notel.service.name=app-a\notel.resource.attributes=compose_service=app-a\notel.metrics.exporter=none\n```\n\n```bash\n# set otel.javaagent.configuration-file with system properties\njava -javaagent:path/to/opentelemetry-javaagent.jar \\\n     -Dotel.javaagent.configuration-file=path/to/otel.properties \\\n     -jar myapp.jar\n```\n\nMore configuration details can be found on the [official document](https://opentelemetry.io/docs/zero-code/java/agent/configuration/).\n\n#### Logs\n\nOpenTelemetry Agent will add information to each logging automatically.\n\n\u003e OpenTelemetry Agent injects the current span information into each logging event's MDC copy:\n\u003e - trace_id - the current trace id (same as Span.current().getSpanContext().getTraceId());\n\u003e - span_id - the current span id (same as Span.current().getSpanContext().getSpanId());\n\u003e - trace_flags - the current trace flags, formatted according to W3C traceflags format (same as Span.current().getSpanContext().getTraceFlags().asHex()).\n\n[Logger MDC auto-instrumentation](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/logger-mdc-instrumentation.md)\n\nIn this demo project, we override `logging.pattern.level` in `application.yaml` file:\n\n```yaml\nlogging:\n  pattern:\n    level: \"trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %p\"\n```\n\nThe log would be like this:\n\n```log\n2022-10-10 15:18:54.893 trace_id=605c7adf03bdb2f2917206de1eae8f72 span_id=c581b882e2d252c2 trace_flags=01 ERROR 1 --- [nio-8080-exec-6] com.example.app.AppApplication           : Hello World!!\n```\n\n#### Traces\n\nAs previously mentioned, OpenTelemetry Agent can capture telemetry data at the “edges” of an app or service, such as inbound requests, and outbound HTTP calls. We don't need to add anything to our code. To show this, we provide the `/chain` endpoint in the application as follows:\n\n```java\n@GetMapping(\"/chain\")\n    public String chain(@RequestParam(value = \"name\", defaultValue = \"World\") String name) throws InterruptedException, IOException {\n        String TARGET_ONE_HOST = System.getenv().getOrDefault(\"TARGET_ONE_HOST\", \"localhost\");\n        String TARGET_TWO_HOST = System.getenv().getOrDefault(\"TARGET_TWO_HOST\", \"localhost\");\n        logger.debug(\"chain is starting\");\n        Request.Get(\"http://localhost:8080/\")\n                .execute().returnContent();\n        Request.Get(String.format(\"http://%s:8080/io_task\", TARGET_ONE_HOST))\n                .execute().returnContent();\n        Request.Get(String.format(\"http://%s:8080/cpu_task\", TARGET_TWO_HOST))\n                .execute().returnContent();\n        logger.debug(\"chain is finished\");\n        return \"chain\";\n    }\n```\n\nWhen calling app-a `chain` endpoint(`curl localhost:8080/chain`), it will send requests to itself root(`/`) and the other two services' `io_task`, and `cpu_task` by order. In the whole process, we don't write any line of code about OpenTelemetry, trace, or span. But the log shows all inbound requests, and outbound HTTP calls were added span information as follows: \n\n```log\n# Start from app-a chain\n2022-10-10 15:57:12.828 trace_id=743ae05db90d00fd65998ff30cf7094d span_id=6cc84ac5ed4cf68c trace_flags=01 DEBUG 1 --- [nio-8080-exec-6] com.example.app.AppApplication           : chain is starting\n\n# In app-a root\n2022-10-10 15:57:13.106 trace_id=743ae05db90d00fd65998ff30cf7094d span_id=4745d1a1f588a949 trace_flags=01 ERROR 1 --- [nio-8080-exec-7] com.example.app.AppApplication           : [traceparent:\"00-743ae05db90d00fd65998ff30cf7094d-d72a1422522ce837-01\", host:\"localhost:8080\", connection:\"Keep-Alive\", user-agent:\"Apache-HttpClient/4.5.13 (Java/1.8.0_342)\", accept-encoding:\"gzip,deflate\"]\n2022-10-10 15:57:13.106 trace_id=743ae05db90d00fd65998ff30cf7094d span_id=4745d1a1f588a949 trace_flags=01 ERROR 1 --- [nio-8080-exec-7] com.example.app.AppApplication           : Hello World!!\n2022-10-10 15:57:13.106 trace_id=743ae05db90d00fd65998ff30cf7094d span_id=4745d1a1f588a949 trace_flags=01 DEBUG 1 --- [nio-8080-exec-7] com.example.app.AppApplication           : Debugging log\n2022-10-10 15:57:13.106 trace_id=743ae05db90d00fd65998ff30cf7094d span_id=4745d1a1f588a949 trace_flags=01 INFO 1 --- [nio-8080-exec-7] com.example.app.AppApplication           : Info log\n2022-10-10 15:57:13.106 trace_id=743ae05db90d00fd65998ff30cf7094d span_id=4745d1a1f588a949 trace_flags=01 WARN 1 --- [nio-8080-exec-7] com.example.app.AppApplication           : Hey, This is a warning!\n2022-10-10 15:57:13.106 trace_id=743ae05db90d00fd65998ff30cf7094d span_id=4745d1a1f588a949 trace_flags=01 ERROR 1 --- [nio-8080-exec-7] com.example.app.AppApplication           : Oops! We have an Error. OK\n\n# In app-b io_task\n2022-10-10 15:57:14.141 trace_id=743ae05db90d00fd65998ff30cf7094d span_id=b97df0b1834ab84a trace_flags=01 INFO 1 --- [nio-8080-exec-4] com.example.app.AppApplication           : io_task\n\n# In app-c cpu_task\n2022-10-10 15:57:14.191 trace_id=743ae05db90d00fd65998ff30cf7094d span_id=7fd693eefc0d3387 trace_flags=01 INFO 1 --- [nio-8080-exec-4] com.example.app.AppApplication           : cpu_task\n\n# Back to app-a chain\n2022-10-10 15:57:14.199 trace_id=743ae05db90d00fd65998ff30cf7094d span_id=6cc84ac5ed4cf68c trace_flags=01 DEBUG 1 --- [nio-8080-exec-6] com.example.app.AppApplication           : chain is finished\n```\n\nEach endpoint got the same trace_id `743ae05db90d00fd65998ff30cf7094d` start from app-a `chain`. The auto-injected `traceparent`(could saw in the first line in app-a root log) is how OpenTelemetry Agent passed through all these services.\n\n#### Metrics\n\nTo get Prometheus metrics from the Spring Boot application, we need two dependencies:\n\n1. [Spring Boot Actuator](https://github.com/spring-projects/spring-boot/tree/v2.7.4/spring-boot-project/spring-boot-actuator): Actuator provides a lot of monitoring features by HTTP or JMX endpoints for the Spring Boot applications.\n2. [Micrometer](https://github.com/micrometer-metrics/micrometer): Micrometer provides a general API to collect metrics and transform the format for different monitoring systems, including Prometheus.\n\nAdd these two dependencies to the `pom.xml` and config to the `application.yaml` as follows:\n\n```xml\n\u003cdependency\u003e\n   \u003cgroupId\u003eorg.springframework.boot\u003c/groupId\u003e\n   \u003cartifactId\u003espring-boot-starter-actuator\u003c/artifactId\u003e\n\u003c/dependency\u003e\n\u003cdependency\u003e\n   \u003cgroupId\u003eio.micrometer\u003c/groupId\u003e\n   \u003cartifactId\u003emicrometer-registry-prometheus\u003c/artifactId\u003e\n\u003c/dependency\u003e\n```\n\n```yaml\nmanagement:\n  endpoints:\n    web:\n      exposure:\n        include: prometheus # only exposure /actuator/prometheus endpoint\n  metrics:\n    tags:\n      application: app # add tag to each prometheus metric\n```\n\nThe Prometheus metrics look like this on `/actuator/prometheus`:\n\n```txt\n# HELP executor_active_threads The approximate number of threads that are actively executing tasks\n# TYPE executor_active_threads gauge\nexecutor_active_threads{application=\"app\",name=\"applicationTaskExecutor\",} 0.0\n...\n HELP http_server_requests_seconds Duration of HTTP server request handling\n# TYPE http_server_requests_seconds summary\nhttp_server_requests_seconds_count{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",} 1.0\nhttp_server_requests_seconds_sum{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",} 0.047062542\nhttp_server_requests_seconds_count{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/actuator/prometheus\",} 2.0\nhttp_server_requests_seconds_sum{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/actuator/prometheus\",} 0.053801375\n# HELP http_server_requests_seconds_max Duration of HTTP server request handling\n# TYPE http_server_requests_seconds_max gauge\nhttp_server_requests_seconds_max{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",} 0.047062542\nhttp_server_requests_seconds_max{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/actuator/prometheus\",} 0.045745625\n...\n```\n\n##### Metrics with Exemplar\n\nExemplar is a new datatype proposed in [OpenMetrics](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#exemplars). To enable the Exemplar feature there are some dependencies requirements:\n\n1. Spring Boot \u003e= 2.7.0: Spring Boot supported Prometheus Exemplars since [v2.7.0-RC1](https://github.com/spring-projects/spring-boot/releases/tag/v2.7.0-RC1).\n2. Micrometer \u003e= 1.10.0: Micrometer supported Exemplar for Prometheus Histogram and Prometheus Counter since [1.9.0](https://github.com/micrometer-metrics/micrometer/releases/tag/v1.9.0) and using io.prometheus.simpleclient_common 0.16.0 since [1.10.0](https://mvnrepository.com/artifact/io.micrometer/micrometer-registry-prometheus/1.10.0).\n\nAdditionally, we need to add an [Exemplar Sampler](app/src/main/java/com/example/app/PrometheusExemplarSamplerConfiguration.java) (Source from [qaware/cloud-observability-grafana-spring-boot](https://github.com/qaware/cloud-observability-grafana-spring-boot/blob/b331b87b1a7f0f5b5d57150e0356e6a26af967a2/spring-boot-app/src/main/java/de/qaware/demo/cloudobservability/PrometheusExemplarSamplerConfiguration.java)) as follows:\n\n```java\npackage com.example.app;\n\nimport io.prometheus.client.exemplars.tracer.otel_agent.OpenTelemetryAgentSpanContextSupplier;\nimport org.springframework.context.annotation.Bean;\nimport org.springframework.context.annotation.Configuration;\n\n@Configuration\npublic class PrometheusExemplarSamplerConfiguration {\n    @Bean\n    public OpenTelemetryAgentSpanContextSupplier openTelemetryAgentSpanContextSupplier() {\n        // OpenTelemetryAgentSpanContextSupplier is from the opentelemetry agent jar, without using the agent will cause class not found error when running.\n        return new OpenTelemetryAgentSpanContextSupplier();\n    }\n}\n```\n\nThe discussion about Exemplar Sampler is in [Exemplars support for Prometheus Histogram #2812](https://github.com/micrometer-metrics/micrometer/issues/2812#issuecomment-1086001766) on the Micrometer repository.\n\nWhen all dependencies are addressed. We can add a distribution metric to Prometheus metrics in the `application.yaml`.\n\n```yaml\nmanagement:\n  metrics:\n    distribution:\n      percentiles-histogram:\n        http:\n          server:\n            requests: 'true'\n```\n\nCheck more options for distribution metrics on the [Spring Boot document](https://docs.spring.io/spring-boot/docs/2.7.3/reference/html/actuator.html#actuator.metrics.customizing.per-meter-properties).\n\nAs previously mentioned, Exemplar is a new datatype proposed in OpenMetrics, and the default `/actuator/prometheus` provide metrics with Prometheus format. So we need to [add some headers](https://docs.spring.io/spring-boot/docs/current/actuator-api/htmlsingle/#prometheus.retrieving:~:text=The%20default%20response%20content%20type%20is%20text/plain%3Bversion%3D0.0.4.%20The%20endpoint%20can%20also%20produce%20application/openmetrics%2Dtext%3Bversion%3D1.0.0%20when%20called%20with%20an%20appropriate%20Accept%20header%2C%20as%20shown%20in%20the%20following%20curl%2Dbased%20example%3A) to get the metrics with OpenMetrics format as follows:\n\n```bash\ncurl 'http://localhost:8080/actuator/prometheus' -i -X GET \\\n   -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8'\n```\n\nThe histogram metrics with trace ID (starts with `#`) look like this:\n\n```txt\n# TYPE http_server_requests_seconds histogram\n# HELP http_server_requests_seconds Duration of HTTP server request handling\nhttp_server_requests_seconds_bucket{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",le=\"0.001\"} 0.0\nhttp_server_requests_seconds_bucket{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",le=\"0.001048576\"} 0.0\nhttp_server_requests_seconds_bucket{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",le=\"0.001398101\"} 0.0\nhttp_server_requests_seconds_bucket{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",le=\"0.001747626\"} 0.0\nhttp_server_requests_seconds_bucket{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",le=\"0.002097151\"} 0.0\nhttp_server_requests_seconds_bucket{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",le=\"0.002446676\"} 0.0\nhttp_server_requests_seconds_bucket{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",le=\"0.002796201\"} 1.0 # {span_id=\"55255da260e873d9\",trace_id=\"21933703cb442151b1cef583714eb42e\"} 0.002745959 1665676383.654\nhttp_server_requests_seconds_bucket{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",le=\"0.003145726\"} 1.0\nhttp_server_requests_seconds_bucket{application=\"app\",exception=\"None\",method=\"GET\",outcome=\"SUCCESS\",status=\"200\",uri=\"/\",le=\"0.003495251\"} 2.0 # {span_id=\"81222a08c4f050fe\",trace_id=\"eddcd9569d20b4aa48c06d3b905f32ea\"} 0.003224625 1665676382.620\n```\n\n### Prometheus - Metrics\n\nCollects metrics from applications.\n\n#### Prometheus Config\n\nDefine all Spring Boot applications metrics scrape jobs in `etc/prometheus/prometheus.yml`.\n\nPrometheus will scrape OpenMetrics format metrics automatically, there is no need to add specific headers config when scraping from `/actuator/prometheus`.\n\n```yaml\n...\nscrape_configs:\n  - job_name: 'app-a'\n    scrape_interval: 5s\n    metrics_path: \"/actuator/prometheus\"\n    static_configs:\n      - targets: ['app-a:8080']\n\n  - job_name: 'app-b'\n    scrape_interval: 5s\n    metrics_path: \"/actuator/prometheus\"\n    static_configs:\n      - targets: ['app-b:8080']\n\n  - job_name: 'app-c'\n    scrape_interval: 5s\n    metrics_path: \"/actuator/prometheus\"\n    static_configs:\n      - targets: ['app-c:8080']\n```\n\n#### Grafana Data Source\n\nAdd an Exemplars which uses the value of `TraceID` label to create a Tempo link.\n\nGrafana data source setting example:\n\n![Data Source of Prometheus: Exemplars](./images/prometheus-exemplars.png)\n\nGrafana data sources config example:\n\n```yaml\nname: Prometheus\ntype: prometheus\ntypeName: Prometheus\naccess: proxy\nurl: http://prometheus:9090\npassword: ''\nuser: ''\ndatabase: ''\nbasicAuth: false\nisDefault: true\njsonData:\nexemplarTraceIdDestinations:\n   - datasourceUid: tempo\n      name: TraceID\nhttpMethod: POST\nreadOnly: false\neditable: true\n```\n\n### Tempo - Traces\n\nReceives spans from applications.\n\n#### Grafana Data Source\n\n[Trace to logs](https://grafana.com/docs/grafana/latest/datasources/tempo/#trace-to-logs) setting:\n\n1. Data source: target log source\n2. Tags: key of tags or process level attributes from the trace, which will be log query criteria if the key exists in the trace\n3. Map tag names: Convert existing key of tags or process level attributes from trace to another key, then used as log query criteria. Use this feature when the values of the trace tag and log label are identical but the keys are different.\n\nGrafana data source setting example:\n\n![Data Source of Tempo: Trace to logs](./images/tempo-trace-to-logs.png)\n\nGrafana data sources config example:\n\n```yaml\nname: Tempo\ntype: tempo\ntypeName: Tempo\naccess: proxy\nurl: http://tempo\npassword: ''\nuser: ''\ndatabase: ''\nbasicAuth: false\nisDefault: false\njsonData:\nnodeGraph:\n   enabled: true\ntracesToLogs:\n   datasourceUid: loki\n   filterBySpanID: false\n   filterByTraceID: true\n   mapTagNamesEnabled: false\n   tags:\n      - compose_service\nreadOnly: false\neditable: true\n```\n\n### Loki - Logs\n\nCollects logs with Loki Docker Driver from all services.\n\n#### Loki Docker Driver\n\n1. Use [YAML anchor and alias](https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/) feature to set logging options for each service.\n2. Set [Loki Docker Driver options](https://grafana.com/docs/loki/latest/clients/docker-driver/configuration/)\n   1. loki-url: loki service endpoint\n   2. loki-pipeline-stages: processes multiline log from Spring Boot application with multiline and regex stages ([reference](https://grafana.com/docs/loki/latest/clients/promtail/stages/multiline/))\n\n```yaml\nx-logging: \u0026default-logging # anchor(\u0026): 'default-logging' for defines a chunk of configuration\n  driver: loki\n  options:\n    loki-url: 'http://localhost:3100/api/prom/push'\n    loki-pipeline-stages: |\n      - multiline:\n          firstline: '^\\d{4}-\\d{2}-\\d{2} \\d{1,2}:\\d{2}:\\d{2}'\n          max_wait_time: 3s\n      - regex:\n          expression: '^(?P\u003ctime\u003e\\d{4}-\\d{2}-\\d{2} \\d{1,2}:\\d{2}:\\d{2},d{3}) (?P\u003cmessage\u003e(?s:.*))$$'\n# Use $$ (double-dollar sign) when your configuration needs a literal dollar sign.\n\nversion: \"3.4\"\n\nservices:\n   foo:\n      image: foo\n      logging: *default-logging # alias(*): refer to 'default-logging' chunk \n```\n\n#### Grafana Data Source\n\nAdd a TraceID derived field to extract the trace id and create a Tempo link from the trace id.\n\nGrafana data source setting example:\n\n![Data Source of Loki: Derived fields](./images/loki-derive-filed.png)\n\nGrafana data source config example:\n\n```yaml\nname: Loki\ntype: loki\ntypeName: Loki\naccess: proxy\nurl: http://loki:3100\npassword: ''\nuser: ''\ndatabase: ''\nbasicAuth: false\nisDefault: false\njsonData:\nderivedFields:\n   - datasourceUid: tempo\n      matcherRegex: (?:trace_id)=(\\w+)\n      name: TraceID\n      url: $${__value.raw}\n      # Use $$ (double-dollar sign) when your configuration needs a literal dollar sign.\nreadOnly: false\neditable: true\n```\n\n### Grafana\n\n1. Add Prometheus, Tempo, and Loki to the data source with config file `etc/grafana/datasource.yml`.\n2. Load predefined dashboard with `etc/dashboards.yaml` and `etc/dashboards/spring-boot-observability.json`.\n\n```yaml\n# grafana in docker-compose.yaml\ngrafana:\n   image: grafana/grafana:10.4.2\n   volumes:\n      - ./etc/grafana/:/etc/grafana/provisioning/datasources # data sources\n      - ./etc/dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml # dashboard setting\n      - ./etc/dashboards:/etc/grafana/dashboards # dashboard json files directory\n```\n\n## Reference\n\n1. [Cloud Observability with Grafana and Spring Boot](https://github.com/qaware/cloud-observability-grafana-spring-boot)\n2. [Exemplars support for Prometheus Histogram](https://github.com/micrometer-metrics/micrometer/issues/2812)\n3. [OpenTelemetry SDK Autoconfigure](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md)\n4. [Java system properties and environment variables](https://stackoverflow.com/questions/7054972/java-system-properties-and-environment-variables)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblueswen%2Fspring-boot-observability","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblueswen%2Fspring-boot-observability","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblueswen%2Fspring-boot-observability/lists"}