{"id":23080111,"url":"https://github.com/opsdis/greetings_springboot_otel_demo","last_synced_at":"2025-04-03T13:42:11.787Z","repository":{"id":116137376,"uuid":"561258450","full_name":"opsdis/greetings_springboot_otel_demo","owner":"opsdis","description":null,"archived":false,"fork":false,"pushed_at":"2023-04-04T19:32:06.000Z","size":1572,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-09T02:34:22.446Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opsdis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-03T09:57:01.000Z","updated_at":"2022-11-21T06:57:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"1a1dfb83-5b89-49a2-a768-05508908b7c4","html_url":"https://github.com/opsdis/greetings_springboot_otel_demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdis%2Fgreetings_springboot_otel_demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdis%2Fgreetings_springboot_otel_demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdis%2Fgreetings_springboot_otel_demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opsdis%2Fgreetings_springboot_otel_demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opsdis","download_url":"https://codeload.github.com/opsdis/greetings_springboot_otel_demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247013335,"owners_count":20869251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-16T13:04:49.535Z","updated_at":"2025-04-03T13:42:11.769Z","avatar_url":"https://github.com/opsdis.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"A demo of Springboot with Opentelemetry\n------------------------\n\n# Overview\nThis is a small demo application based on Springboot. \nThe purpose with demo is to show the capabilities of using Springboot with Opentelemetry.\nThe application use:\n- Springboot micrometer for metrics\n- Springboot sleuth for tracing that include the opentelemetry sdk\n- Spingboot logback for logging configuration\n \n![Overview](./docs/springboot_demo.png)\n\nThe demo is about an enterprise greetings application that have 2 microservices, a frontend and \na backend. The frontend expose an api of the following url:\n```shell\nhttp://localhost:8080/greeting?name=XYZ\n```\nAnd just respond with:\n```json\n{\"id\":362,\"content\":\"Bonjure, XYZ!\"}\n```\nor\n```json\n{\"id\":362,\"content\":\"Hello, XYZ!\"}\n```\nThe id is just the sequence of calls, and used as our greetings id. In the logs and traces are\nthe business logic around greetings injected as:\n- greetingStatus\n- greetingsId\t\n- greetingsLanguage\n- greetingsName\n\nThe service is not very reliable, so you will find a high amount of problems in logs and traces.\nAnd there is metrics that counts the number of greetings for both service, both totally and\nthe errors.\n\nMy goals are to show:\n- Structured logs with json based layout output.\n- Minimal logging, instead using tracing in combination with logs, and just have log entry in the \"end\" of a \"session\" - either okay or catch exception and log failure\n- Metrics with focus on indicative problems - think your SLI\n- Enable traceid as a common key between traces, logs and metrics\n- Common labels that are injected on all different observability data\n- What auto instrumentation give you for free and how to interact with local child spans\n- Otel collector configuration - receivers. processors and exporters, especially what you can do with processors\n- Visualize it all in grafana, so you can navigate between metrics, logs and traces \n\n![Logs to traces](./docs/explore_logs_to_traces.png)\n\n# Todo\n- Metrics with exemplars not implemented\n- Native otel logging - currently to file and scraped by receiver `filelog`\n- Add more micro services\n\n# Design goals\nMove as much metadata information as possible to the otel collector and try to separate between \nwhat is application specific and what is infrastructure and environment specific.\n\nUsing the opentelemetry agent in the jvm traces and metrics are pushed according to the agent\nconfiguration.\n\n## Traces\nTraces are create by auto instrumentation. In the code we use opentelemetry annotations and\npackages to create additional ones. \n\n## Metrics\nLots of metrics are created by the auto instrumentation and custom metrics is created with \nmicrometer. \n\n## Logs\nWith logs the ambition is to show how to work with structured logs. This means:\n- using thread local MDC to save key/value attributes.\n- Static log messages, any dynamic content should be put in MDC's\n- Log in the end of the execution by success or by exception and not log every code line.\n- Output log data in a structured format  of json\n- Format exceptions\n\nWith the auto instrumentation the log lines will be injected with trace and span id. This is \nkey to be able to link logs to traces and the opposite.\n\n\u003e Some of this gluing is configuration in Grafana datasources but the information needs to be\n\u003e available \n\nThe tools in play is slf4j and logback, and the important configuration is done in logback. \nPlease see the logback.xml for details and explanation.\n\n# Run \n\nThe otel collector will send the observability data to the following:\n- Metrics -\u003e Mimir\n- Logs -\u003e Loki\n- Traces -\u003e Tempo\n\nIf you like to make it easy just set up a Grafana Cloud free account and start testing.\n\n```shell\ncp otel_conf_example.yml otel_conf.yml\n# Update otel/conf.yml with the Grafana Cloud credentials\nvi otel_conf.yml\n```\n\nNow it time to start the demo!\n\n```shell\n# Download the otel collector change version and os according to your needs\n# https://github.com/open-telemetry/opentelemetry-collector-releases/releases\n\nwget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.63.1/otelcol-contrib_0.63.1_linux_amd64.tar.gz\n\n# Download the opentelemetry java agent\nwget https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.19.2/opentelemetry-javaagent.jar\n\n# Build and package the backend\ncd backend\nexport JAVA_HOME=$HOME/.jdks/openjdk-19.0.1/\nexport PATH=$JAVA_HOME/bin:$PATH\n\n./mvnw clean package\n\n# Endpoint to the otel collector\nexport OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317\nexport OTEL_METRIC_EXPORT_INTERVAL=15000\n\n# Set the service name used on spans, for metrics this will be mapped to job\nexport OTEL_SERVICE_NAME=myBackendservice\n# These resources attributes will be added to all trace spans and to a timeseries named target_info\n# target_info can be used to promql join on the label job\nexport OTEL_RESOURCE_ATTRIBUTES=service.version=0.1.0, application=myBackendservice\n\njava -javaagent:../otel/opentelemetry-javaagent.jar -jar ./target/rest-backend-complete-0.0.1-SNAPSHOT.jar\n\n# In another window\n# Build and package the frontend\ncd frontend\nexport JAVA_HOME=$HOME/.jdks/openjdk-19.0.1/\nexport PATH=$JAVA_HOME/bin:$PATH\n./mvnw clean package\n\n# Endpoint to the otel collector\nexport OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317\nexport OTEL_METRIC_EXPORT_INTERVAL=15000\n\n# Set the service name used on spans, for metrics this will be mapped to job\nexport OTEL_SERVICE_NAME=myWebservice\n# These resources attributes will be added to all trace spans and to a timeseries named target_info\n# target_info can be used to promql join on the label job\nexport OTEL_RESOURCE_ATTRIBUTES=service.version=0.1.0, application=myWebservice\n\njava -javaagent:../otel/opentelemetry-javaagent.jar -jar ./target/rest-frontend-complete-0.0.1-SNAPSHOT.jar\n\n# In another window\n# local scrape\n./otelcol-contrib --config=conf_local.yml\n# remote \n~/programs//bin/otelcol-contrib --config=conf_micro.yml\n\n\n# In another window\nwhile [ true ] ; do curl -i http://localhost:8080/greeting?name=Foo; sleep 1; done\n\n# In another window\nwhile [ true ] ; do curl -i http://localhost:8080/greeting?name=Bar; sleep 2; done\n```\n\nPlease the [OpenTelemetry SDK Autoconfigure](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md) for\nmore configuration options.\n\n\n# Setup Grafana\n\nTo enable correlation between logs and traces and vice verse you need to create additional information\nto your data sources. \n\u003e In grafana cloud you can not change the existing so just create new ones but pointing to the \n\u003e endpoints you current datasources have. You must also create your own api keys.\n\nFor logs (Loki) create a datasource named `Loki-spring-to-traces` with the same endpoint and username\nas the one that exists. \n\n![Loki datasource setup](./docs/loki_datasource.png)\n\nIt's in the section `Derived fields` the magic happen of mapping the trace id field in the log to\nthe Tempo data source. \n\u003e Internal link should point to the tempo datasource.\n\n\nIn the directory `grafana/dashboards` there are two simple dashboards that can be imported to show some\napplication related dashboard and some jvm metrics.\n\n![Application](./docs/application_dashboard.png)\n\n# Links\notel contrib collector - https://github.com/open-telemetry/opentelemetry-collector-contrib/releases\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopsdis%2Fgreetings_springboot_otel_demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopsdis%2Fgreetings_springboot_otel_demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopsdis%2Fgreetings_springboot_otel_demo/lists"}