Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ruanbekker/grafana-tempo-loki-tracing
Grafana Distributed Tracing Example with: Tempo, Prometheus, Loki, Grafana and Python Flask
https://github.com/ruanbekker/grafana-tempo-loki-tracing
distributed-tracing docker grafana loki prometheus tempo
Last synced: 5 days ago
JSON representation
Grafana Distributed Tracing Example with: Tempo, Prometheus, Loki, Grafana and Python Flask
- Host: GitHub
- URL: https://github.com/ruanbekker/grafana-tempo-loki-tracing
- Owner: ruanbekker
- Created: 2023-12-01T22:16:27.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2024-08-11T21:43:16.000Z (3 months ago)
- Last Synced: 2024-08-12T13:32:28.846Z (3 months ago)
- Topics: distributed-tracing, docker, grafana, loki, prometheus, tempo
- Language: Python
- Homepage: https://ruan.dev
- Size: 36.1 KB
- Stars: 19
- Watchers: 2
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# grafana-tempo-loki-tracing
Grafana Distributed Tracing Example with: Tempo, Prometheus, Loki, Grafana and Python Flask
## About
This project showcases microservices architecture with distributed tracing powered by OpenTelemetry, integrated with Grafana Tempo. It also leverages derived fields in Grafana Loki to seamlessly link trace IDs to Tempo, providing a unified and smooth experience for viewing traces directly from logs.
The request flow will look like the following:
## Getting Started
Boot the stack:
```bash
make up
```
The datasources will be auto-configured defined in `configs/grafana/datasources.yaml`.
## Example Request
Run the request in `./create_order.sh`:
```bash
./create_order.sh
```
The response will be similar to the following:
```json
{
"message": "Order 119 created and payment authorized",
"status": "success",
"trace_id": "25fb067b54465ea4ccecea93694c8824"
}
```
Logs using `make logs`:
```bash
api-gateway | [2024-08-11 20:28:42,387] DEBUG in app: api-gateway received post request
api-gateway | [2024-08-11 20:28:42,388] DEBUG in app: api-gateway makes a request to order-service trace_id=b6512704d84b6892c72f4b0706ba6d06
order-service | [2024-08-11 20:28:42,397] DEBUG in app: order-service received a post request
order-service | [2024-08-11 20:28:42,398] DEBUG in app: order-service makes a post request to inventory-service trace_id=b6512704d84b6892c72f4b0706ba6d06
inventory-service | [2024-08-11 20:28:42,408] DEBUG in app: inventory-service received a post request
inventory-service | [2024-08-11 20:28:42,409] DEBUG in app: inventory-service about to make a database query. trace_id=b6512704d84b6892c72f4b0706ba6d06
inventory-service | [2024-08-11 20:28:42,424] DEBUG in app: the availability in the inv db is 96
warehouse-service | [2024-08-11 20:28:42,434] DEBUG in app: warehouse-service received a post request
warehouse-service | [2024-08-11 20:28:42,435] INFO in app: warehouse-service about to make a database query. trace_id=b6512704d84b6892c72f4b0706ba6d06
warehouse-service | 192.168.128.8 - - [11/Aug/2024 20:28:42] "POST /warehouse/reserve HTTP/1.1" 200 -
inventory-service | 192.168.128.12 - - [11/Aug/2024 20:28:42] "POST /inventory/check HTTP/1.1" 200 -
payment-service | [2024-08-11 20:28:42,469] INFO in app: payment-service logged trace_id=b6512704d84b6892c72f4b0706ba6d06
fraud-service | [2024-08-11 20:28:42,479] INFO in app: fraud-service logged trace_id=b6512704d84b6892c72f4b0706ba6d06
fraud-service | [2024-08-11 20:28:42,480] INFO in app: Transaction passed fraud check: order_id=754, user_id=123
fraud-service | 192.168.128.9 - - [11/Aug/2024 20:28:42] "POST /fraud/check HTTP/1.1" 200 -
payment-service | 192.168.128.12 - - [11/Aug/2024 20:28:42] "POST /payment/authorize HTTP/1.1" 200 -
order-service | 192.168.128.13 - - [11/Aug/2024 20:28:42] "POST /order HTTP/1.1" 200 -
api-gateway | 192.168.128.1 - - [11/Aug/2024 20:28:42] "POST /api/order HTTP/1.1" 200 -
```
## Screenshots
Explore traces:
Query our cluster and Table view:
Node Graph:
When we use the `order-service` container logs in Loki:
From the response of our request, we get a `trace_id`, when we use TraceQL in Explore to search for our trace id and we can see all the spans from the given trace:
Since we are using `remote_write` in Tempo, we are writing metrics from Tempo to Prometheus, and we can access metrics such as http request rate from client to server:
Similarly, we can access the p90 percentile request latencies of requests using the request path `/api/order`:
## Extras
With `span.set_attribute` we can enrich some of the visuals:
```python
with tracer.start_as_current_span("query_inventory_database") as span:
inventory_item = Inventory.query.filter_by(id=item_id).first()
sql_query = f"SELECT * FROM inventory WHERE item_id = {item_id}"
span.set_attribute("db.query", sql_query)
span.set_attribute("inventory.item_id", item_id)
span.set_attribute("inventory.requested_quantity", quantity)
span.set_attribute("inventory.availability", inventory_item.availability)
```
Produces:
