Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/openliberty/guide-microprofile-opentracing-jaeger
(Deprecated) A guide on how to enable and customize tracing of JAX-RS and non-JAX-RS methods by using MicroProfile OpenTracing and Jaeger.
https://github.com/openliberty/guide-microprofile-opentracing-jaeger
Last synced: 2 months ago
JSON representation
(Deprecated) A guide on how to enable and customize tracing of JAX-RS and non-JAX-RS methods by using MicroProfile OpenTracing and Jaeger.
- Host: GitHub
- URL: https://github.com/openliberty/guide-microprofile-opentracing-jaeger
- Owner: OpenLiberty
- License: other
- Created: 2020-07-06T13:00:53.000Z (over 4 years ago)
- Default Branch: prod
- Last Pushed: 2024-09-12T14:56:12.000Z (4 months ago)
- Last Synced: 2024-09-13T03:10:40.326Z (4 months ago)
- Language: Java
- Homepage: https://openliberty.io/guides/microprofile-opentracing-jaeger.html
- Size: 1.15 MB
- Stars: 3
- Watchers: 5
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.adoc
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
// Copyright (c) 2020, 2023 IBM Corporation and others.
// Licensed under Creative Commons Attribution-NoDerivatives
// 4.0 International (CC BY-ND 4.0)
// https://creativecommons.org/licenses/by-nd/4.0/
//
// Contributors:
// IBM Corporation
:projectid: microprofile-opentracing-jaeger
:page-layout: guide-multipane
:page-duration: 20 minutes
:page-guide-category: microprofile
:page-essential: false
:page-releasedate: 2020-08-20
:page-description: Explore how to enable and customize tracing of JAX-RS and non-JAX-RS endpoint methods by using Jaeger and MicroProfile OpenTracing.
:page-tags: ['MicroProfile']
:page-permalink: /guides/{projectid}
:imagesdir: /img/guide/{projectid}
:page-related-guides: ['cdi-intro', 'microprofile-opentracing']
:common-includes: https://raw.githubusercontent.com/OpenLiberty/guides-common/prod
:page-seo-title: Enabling distributed tracing in Java microservices using Eclipse MicroProfile OpenTracing and the Jaeger tracing system.
:page-seo-description: A getting started tutorial and an example on how to enable distributed tracing in Java microservices to easily trace request flows that span multiple resources by using MicroProfile OpenTracing and Jaeger tracing system.
:source-highlighter: prettify
:guide-author: Open Liberty
= Enabling distributed tracing in microservices with Jaeger
[.hidden]
NOTE: This repository contains the guide documentation source. To view the guide in published form, view it on the https://openliberty.io/guides/{projectid}.html[Open Liberty website].
Explore how to enable and customize tracing of JAX-RS and non-JAX-RS endpoint methods by using MicroProfile OpenTracing and Jaeger.
:inv-url: http://localhost:9081/inventory/systems
:inv-url-docker: http://localhost:9080/inventory/systems
:sys-url: http://localhost:9080/system/properties
:jaeger-url: http://localhost:16686
== What you'll learn
You will learn how to enable automatic tracing for JAX-RS endpoint methods and create custom tracers for non-JAX-RS endpoint methods by using MicroProfile OpenTracing.
OpenTracing is a standard API for instrumenting microservices for distributed tracing. Distributed tracing helps troubleshoot microservices by examining and logging requests as they propagate through a distributed system, allowing developers to tackle the otherwise difficult task of debugging these requests. Without a distributed tracing system in place, analyzing the workflows of operations becomes difficult, particularly in regard to pinpointing when and by whom a request is received or when a response is sent back.
`Tracer` and `Span` are two critical types in the OpenTracing specification. The `Span` type is the primary building block of a distributed trace, representing an individual unit of work done in a distributed system. The `Trace` type in OpenTracing can be thought of as a directed acyclic graph (DAG) of `Spans`, where the edges between `Spans` are called References. The `Tracer` interface creates `Spans` and `Traces` and understands how to serialize and deserialize their metadata across process boundaries.
MicroProfile OpenTracing enables distributed tracing in microservices. The MicroProfile OpenTracing specification doesn’t address the problem of defining, implementing, or configuring the underlying distributed tracing system. Rather, the specification makes it easier to instrument services with distributed tracing given an existing distributed tracing system.
https://www.jaegertracing.io/[Jaeger^] is an open source distributed tracing system that is compatible with the OpenTracing specification. Jaeger also provides an implementation of `Tracer` in the client package that is compatible with MicroProfile OpenTracing.
You’ll configure the provided `inventory` and `system` services to use Jaeger for distributed tracing with MicroProfile OpenTracing.
You’ll run these services in two separate JVMs made of two server instances to demonstrate tracing in a distributed environment.
If all the components were run on a single server, then any logging software would be sufficient.
== Additional prerequisites
Before you begin, deploy the Jaeger all-in-one executable file to start the Jaeger tracing system. The Jaeger all-in-one executable file is configured for quick local testing. You can find information about the Jaeger server and instructions for starting the all-in-one executable file in the https://www.jaegertracing.io/docs/1.22/getting-started/#all-in-one[Jaeger documentation^].
Before you proceed, make sure that your Jaeger server is up and running. Jaeger can be found at the
{jaeger-url}[{jaeger-url}^] URL.
// =================================================================================================
// Getting Started
// =================================================================================================
[role='command']
include::{common-includes}/gitclone.adoc[]
=== Try what you'll build
// cloud-hosted guide instructions:
ifdef::cloud-hosted[]
Run the following docker command to start Jaeger server:
```bash
docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 14250:14250 \
-p 9411:9411 \
jaegertracing/all-in-one:1.22
```
You can find information about the Jaeger server and instructions for starting the all-in-one executable file in the [Jaeger documentation](https://www.jaegertracing.io/docs/1.22/getting-started/#all-in-one).
Before you proceed, make sure that your Jaeger server is up and running. Click the following button to visit the Jaeger service:
::startApplication{port="16686" display="external" name="Visit Jaeger service" route="/"}
endif::[]
The `finish` directory in the root of this guide contains the finished application. Give it a try before you proceed.
ifndef::cloud-hosted[]
Open a command-line session and navigate to the `finish/inventory` directory. Run the following Maven goal to build the `inventory` service and deploy it to Open Liberty:
[role="command"]
----
mvn liberty:run
----
Open another command-line session and navigate to the `finish/system` directory. Run the following Maven goal to build the `system` service and deploy it to Open Liberty:
[role="command"]
----
mvn liberty:run
----
endif::[]
// cloud-hosted guide instructions:
ifdef::cloud-hosted[]
Navigate to the ***finish/inventory*** directory. Run the following Maven goal to build the ***inventory*** service and deploy it to Open Liberty:
```bash
cd /home/project/guide-microprofile-opentracing-jaeger/finish/inventory
mvn liberty:run
```
Open another command-line session and navigate to the ***finish/system*** directory. Run the following Maven goal to build the ***system*** service and deploy it to Open Liberty:
```bash
cd /home/project/guide-microprofile-opentracing-jaeger/finish/system
mvn liberty:run
```
endif::[]
After you see the following message in both command-line sessions, both of your services are ready:
[role="no_copy"]
----
The defaultServer server is ready to run a smarter planet.
----
// static guide instructions:
ifndef::cloud-hosted[]
Make sure that your Jaeger server is running and point your browser to the {inv-url}/localhost[{inv-url}/localhost^] URL. When you visit this endpoint, you make two GET HTTP requests, one to the `system` service and one to the `inventory` service. Both of these requests are configured to be traced, so a new trace is recorded in Jaeger.
To view the traces, go to the {jaeger-url}[{jaeger-url}^] URL. You can view the traces for the inventory or system services under the **Search** tab. Select the services in the **Select a service** menu and click the **Find Traces** button at the end of the section.
endif::[]
// cloud-hosted guide instructions:
ifdef::cloud-hosted[]
Open another command-line session and run the following curl command from the terminal:
```bash
curl -s http://localhost:9081/inventory/systems/localhost | jq
```
When you visit this endpoint, you make two GET HTTP requests, one to the ***system*** service and one to the **inventory** service. Both of these requests are configured to be traced, so a new trace is recorded in Jaeger.
To view the traces, click the following button to visit the Jaeger service:
::startApplication{port="16686" display="external" name="Visit Jaeger service" route="/"}
You can view the traces for the inventory or system services under the **Search** tab. Select the services in the **Select a service** menu and click the **Find Traces** button at the end of the section.
endif::[]
If you only see the **jaeger-query** option listed in the dropdown, you might need to wait a little longer and refresh the page to see the application services.
View the traces for `inventory`. You'll see the following trace:
image::tracelist.png[Trace result,align="center"]
{empty} +
The trace has four spans, three from inventory and one from system. Click the trace to view its details. Under **Service & Operation**, you see the spans in this trace. You can inspect each span by clicking it to reveal more detailed information, such as the time at which a request was received and the time at which a response was sent back.
Verify that there are three spans from `inventory` and one span from `system`:
image::trace01.png[Finished application's trace,align="center"]
{empty} +
After you’re finished reviewing the application, stop the Open Liberty servers by pressing `CTRL+C` in the command-line sessions where you ran the system and inventory services. Alternatively, you can run the following goals from the `finish` directory in another command-line session:
// static guide instructions:
ifndef::cloud-hosted[]
[role="command"]
----
mvn -pl system liberty:stop
mvn -pl inventory liberty:stop
----
endif::[]
// cloud-hosted guide instructions:
ifdef::cloud-hosted[]
```bash
cd /home/project/guide-microprofile-opentracing-jaeger/finish
mvn -pl system liberty:stop
mvn -pl inventory liberty:stop
```
endif::[]
// =================================================================================================
// Running the services
// =================================================================================================
== Building the application
You need to start the services to see basic traces appear in Jaeger.
When you run Open Liberty in https://openliberty.io/docs/latest/development-mode.html[dev mode^], the server listens for file changes and automatically recompiles and deploys your updates whenever you save a new change.
Open a command-line session and navigate to the `start/inventory` directory.
Run the following Maven goal to start the `inventory` service in dev mode:
// static guide instructions:
ifndef::cloud-hosted[]
[role="command"]
----
mvn liberty:dev
----
endif::[]
// cloud-hosted guide instructions:
ifdef::cloud-hosted[]
```bash
cd /home/project/guide-microprofile-opentracing-jaeger/start/inventory
mvn liberty:dev
```
endif::[]
Open a command-line session and navigate to the `start/system` directory.
Run the following Maven goal to start the `system` service in dev mode:
[role="command"]
// static guide instructions:
ifndef::cloud-hosted[]
[role="command"]
----
mvn liberty:dev
----
endif::[]
// cloud-hosted guide instructions:
ifdef::cloud-hosted[]
```bash
cd /home/project/guide-microprofile-opentracing-jaeger/start/system
mvn liberty:dev
```
endif::[]
After you see the following message, your application server in dev mode is ready:
[role="no_copy"]
----
**************************************************************
* Liberty is running in dev mode.
----
Dev mode holds your command-line session to listen for file changes. Open another command-line session to continue, or open the project in your editor.
// static guide instructions:
ifndef::cloud-hosted[]
When the servers start, you can find the `system` and `inventory` services at the following URLs:
* {sys-url}[{sys-url}^]
* {inv-url}[{inv-url}^]
endif::[]
// cloud-hosted guide instructions:
ifdef::cloud-hosted[]
When the servers start, you can find the ***system*** service by running the following curl command:
```bash
curl -s http://localhost:9080/system/properties | jq
```
and the ***inventory*** service by running the following curl command:
```bash
curl -s http://localhost:9081/inventory/systems | jq
```
endif::[]
// =================================================================================================
// Existing Tracer implementation
// =================================================================================================
== Enabling existing Tracer implementation
To collect traces across your systems, you need to implement the OpenTracing `Tracer` interface. Jaeger provides a `Tracer` implementation for the Jaeger server in the `jaeger-client` package.
This package is already added as a dependency for you in your [hotspot=jaeger]`pom.xml` file. It's downloaded and installed automatically into each service when you run a Maven build.
=== Configuring the Jaeger client
In a development environment, it is important that every trace is sampled. When every trace is sampled, all spans are available in the Jaeger UI.
The `JAEGER_SAMPLER_TYPE` and `JAEGER_SAMPLER_PARAM` environment variables are set as Open Liberty [hotspot=jaegerClientConfig]`configuration properties` to sample all traces.
The `const` value for `JAEGER_SAMPLER_TYPE` environment variable configures the Jaeger client sampler to make the same sampling decision for each trace, based on the sampler parameter. If the sampler parameter is 1, it samples all traces. If the sampler parameter is 0, it doesn't sample any traces.
The `1` value for [hotspot=samplerParamConfig]`JAEGER_SAMPLER_PARAM` variable configures the Jaeger sampler to sample all traces.
In a production environment, this configuration might cause a lot of overhead on the application and a lower sampling rate can be used. The different values for client sampling configuration can be found in the https://www.jaegertracing.io/docs/1.18/sampling/#client-sampling-configuration[sampling documentation^].
Similarly, in a production environment, Jaeger might not be running in the same host as the application. In this case, set the hostname of the Jaeger server to the `JAEGER_AGENT_HOST` environment variable and set the port that communicates with the Jaeger host to the `JAEGER_AGENT_PORT` environment variable.
You can view the configuration environment variables at the https://github.com/jaegertracing/jaeger-client-java/tree/master/jaeger-core#configuration-via-environment[Jaeger Java client documentation^].
pom.xml
[source, xml, linenums, role='code_column']
----
include::finish/inventory/pom.xml[]
----
// =================================================================================================
// Enabling distributed tracing
// =================================================================================================
== Enabling and disabling distributed tracing
The https://github.com/eclipse/microprofile-opentracing[MicroProfile OpenTracing feature^] enables tracing of all JAX-RS endpoint methods by default. To further control and customize these traces, use the `@Traced` annotation to enable and disable tracing of particular methods. You can also inject a custom `Tracer` object to create and customize spans.
This feature is already enabled in the `inventory` and `system` configuration files.
=== Enabling distributed tracing without code instrumentation
Because tracing of all JAX-RS endpoint methods is enabled by default, you only need to enable the [hotspot=mpOpenTracing file=0]`MicroProfile OpenTracing` feature in the `server.xml` file to see some basic traces in Jaeger.
The OpenTracing API is exposed as a third-party API in Open Liberty. To add the visibility of OpenTracing APIs to the application, add `third-party` to the types of API packages that this class loader supports. Instead of explicitly configuring a list of API packages that includes `third-party`, set the `+third-party` value to the [hotspot=thirdParty]`apiTypeVisibility` attribute in the [hotspot=thirdParty]`classLoader` configuration. This configuration adds `third-party` to the default list of API package types that are supported.
server.xml
[source, xml, linenums, role='code_column']
----
include::finish/inventory/src/main/liberty/config/server.xml[]
----
Make sure that your services are running. Then, point your browser to any of the services' endpoints and check your Jaeger server for traces.
=== Enabling explicit distributed tracing
Use the [hotspot=Traced file=0]`@Traced` annotation to define explicit span creation for specific classes and methods. If you place the annotation on a class, then the annotation is automatically applied to all methods within that class. If you place the annotation on a method, then the annotation overrides the class annotation if one exists.
The [hotspot=Traced file=0]`@Traced` annotation can be configured with the following two parameters:
* The `value=[true|false]` parameter indicates whether a particular class or method is traced. For example, while all JAX-RS endpoint methods are traced by default, you can disable their tracing by using the `@Traced(false)` annotation. This parameter is set to `true` by default.
* The `operationName=` parameter indicates the name of the span that is assigned to the method that is traced. If you omit this parameter, the span is named with the `..` format. If you use this parameter at a class level, then all methods within that class have the same span name unless they are explicitly overridden by another `@Traced` annotation.
[role="code_command hotspot file=0", subs="quotes"]
----
#Replace the `InventoryManager` class.#
`inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java`
----
InventoryManager.java
[source, Java, linenums, role='code_column tags=addToInvList hide_tags=copyright,Add']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java[]
----
[role="edit_command_text"]
Enable tracing of the [hotspot=list file=0]`list()` non-JAX-RS endpoint method by updating [hotspot=Traced file=0]`@Traced` as shown.
// static guide instructions:
ifndef::cloud-hosted[]
Go to the {inv-url}[{inv-url}^] URL and check your Jaeger server at the {jaeger-url}[{jaeger-url}^] URL. If you have the Jaeger UI open from a previous step, refresh the page. Select the `inventory` traces and click the **Find Traces** button.
endif::[]
// cloud-hosted guide instructions:
ifdef::cloud-hosted[]
Run the following curl command:
```bash
curl -s http://localhost:9081/inventory/systems | jq
```
Check your Jaeger server. If you have the Jaeger UI open from a previous step, refresh the page. Select the ***inventory*** traces and click the **Find Traces** button.
::startApplication{port="16686" display="external" name="Visit Jaeger service" route="/"}
endif::[]
You see a new trace record that is two spans long. One span is for the [hotspot=listContents file=1]`listContents()` JAX-RS endpoint method in the [hotspot file=1]`InventoryResource` class, and the other span is for the [hotspot=list file=0]`list()` method in the [hotspot file=0]`InventoryManager` class.
Verify that you see the following spans:
image::trace02.png[Explicit trace span,align="center"]
{empty} +
InventoryResource.java
[source, Java, linenums, role='code_column hide_tags=copyright,Traced-false']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[]
----
=== Disable automatic distributed tracing
You can use the `@Traced` annotation with a value of `false` to disable automatic distributed tracing of JAX-RS endpoint methods.
[role="code_command hotspot file=0", subs="quotes"]
----
#Replace the `InventoryResource` class.#
`inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java`
----
InventoryResource.java
[source, Java, linenums, role='code_column hide_tags=copyright']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[]
----
[role="edit_command_text"]
Disable tracing of the [hotspot=listContents file=0]`listContents()` JAX-RS endpoint method by setting [hotspot=Traced-false file=0]`@Traced(false)`.
// static guide instructions:
ifndef::cloud-hosted[]
Go to the {inv-url}[{inv-url}^] URL and check your Jaeger server at the {jaeger-url}[{jaeger-url}^] URL. If you have the Jaeger UI open from a previous step, refresh the page. Select the `inventory` traces and click the **Find Traces** button. You see a new trace record that is just one span long for the remaining [hotspot=list file=1]`list()` method in the `InventoryManager` class.
endif::[]
// cloud-hosted guide instructions:
ifdef::cloud-hosted[]
Run the following curl command:
```bash
curl -s http://localhost:9081/inventory/systems | jq
```
Check your Jaeger server. If you have the Jaeger UI open from a previous step, refresh the page. Select the **inventory** traces and click the **Find Traces** button. You see a new trace record that is just one span long for the remaining **list()** method in the **InventoryManager** class.
::startApplication{port="16686" display="external" name="Visit the Jaeger service" route="/"}
endif::[]
Verify that you see the following span:
image::trace03.png[Disable trace span,align="center"]
{empty} +
InventoryManager.java
[source, Java, linenums, role='code_column tags=addToInvList hide_tags=copyright,addSpan,endTry']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java[]
----
=== Injecting a custom Tracer object
The MicroProfile OpenTracing specification also makes the underlying OpenTracing `Tracer` instance available for use. You can access the configured `Tracer` by injecting it into a bean by using the [hotspot=customTracer file=0]`@Inject` annotation from the Contexts and Dependency Injections API.
Inject the `Tracer` object into the `inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java` file. Then, use it to define a new child scope in the [hotspot=addSpan file=0]`add()` call.
[role="code_command hotspot file=0", subs="quotes"]
----
#Replace the `InventoryManager` class.#
`inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java`
----
InventoryManager.java
[source, Java, linenums, role='code_column hide_tags=copyright']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java[]
----
This [hotspot=Try file=0]`try` block is called a `try-with-resources` statement, meaning that the `childScope` object is closed at the end of the statement. It's good practice to define custom spans inside such statements. Otherwise, any exceptions that are thrown before the span closes will leak the active span.
// static guide instructions:
ifndef::cloud-hosted[]
Go to the {inv-url}/localhost[{inv-url}/localhost^] URL and check your Jaeger server at the {jaeger-url}[{jaeger-url}^]
URL.
If you have the Jaeger UI open from a previous step, refresh the page.
Select the `inventory` traces and click the **Find Traces** button.
endif::[]
// cloud-hosted guide instructions:
ifdef::cloud-hosted[]
Run the following curl command:
```bash
curl -s http://localhost:9081/inventory/systems/localhost | jq
```
Check your Jaeger server. If you have the Jaeger UI open from a previous step, refresh the page. Select the **inventory** traces and click the **Find Traces** button.
::startApplication{port="16686" display="external" name="Visit the Jaeger service" route="/"}
endif::[]
Verify that there are three spans from `inventory` and one span from `system`:
image::trace01.png[Trace with custom span,align="center"]
{empty} +
This simple example shows what you can do with the injected `Tracer` object. More configuration options are available to you, including setting a timestamp for when a span was created and destroyed. However, these options require an implementation of their own, which doesn't come as a part of the Jaeger user feature that is provided. In a real-world scenario, implement all the OpenTracing interfaces that you deem necessary, which might include the `SpanBuilder` interface. You can use this interface for span creation and customization, including setting timestamps.
SystemResource.java
[source, Java, linenums, role='code_column hide_tags=copyright']
----
include::finish/system/src/main/java/io/openliberty/guides/system/SystemResource.java[]
----
InventoryResource.java
[source, Java, linenums, role='code_column hide_tags=copyright']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[]
----
// =================================================================================================
// Testing the services
// =================================================================================================
== Testing the services
No automated tests are provided to verify the correctness of the traces. Manually verify these traces by viewing them on the Jaeger server.
A few tests are included for you to test the basic functionality of the services. If a test failure occurs, then you might have introduced a bug into the code.
=== Running the tests
Since you started Open Liberty in dev mode, run the tests for the system and inventory services by pressing the `enter/return` key in the command-line sessions where you started the services.
When you are done checking out the services, exit dev mode by pressing `CTRL+C` in the shell sessions where you ran the `system` and `inventory` services, or by typing `q` and then pressing the `enter/return key`.
// static guide instructions:
ifndef::cloud-hosted[]
Finally, stop the `Jaeger` service that you started in the **Additional prerequisites** section.
endif::[]
// cloud-hosted guide instructions:
ifdef::cloud-hosted[]
Finally, stop the ***Jaeger*** service that you started in the previous step.
```bash
docker stop jaeger
docker rm jaeger
```
endif::[]
// =================================================================================================
// Great work! You're done!
// =================================================================================================
== Great work! You're done!
You just used MicroProfile OpenTracing in Open Liberty to customize how and which traces are delivered to Jaeger.
Try out one of the related MicroProfile guides. These guides demonstrate more technologies that you can learn to expand on what you built in this guide.
[role="command"]
include::{common-includes}/attribution.adoc[subs="attributes"]