{"id":20114046,"url":"https://github.com/davidb/sandbox_axum_observability","last_synced_at":"2025-08-23T02:07:23.494Z","repository":{"id":38978734,"uuid":"484842961","full_name":"davidB/sandbox_axum_observability","owner":"davidB","description":"Sandbox to experiment axum and observability","archived":false,"fork":false,"pushed_at":"2023-08-06T16:12:02.000Z","size":1427,"stargazers_count":31,"open_issues_count":6,"forks_count":0,"subscribers_count":4,"default_branch":"development","last_synced_at":"2025-04-09T12:22:14.647Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidB.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-23T19:51:58.000Z","updated_at":"2024-10-28T15:23:23.000Z","dependencies_parsed_at":"2024-11-13T18:33:02.439Z","dependency_job_id":"71a36842-aded-4c7a-b47e-d763b574b742","html_url":"https://github.com/davidB/sandbox_axum_observability","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidB%2Fsandbox_axum_observability","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidB%2Fsandbox_axum_observability/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidB%2Fsandbox_axum_observability/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidB%2Fsandbox_axum_observability/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidB","download_url":"https://codeload.github.com/davidB/sandbox_axum_observability/tar.gz/refs/heads/development","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252683422,"owners_count":21788036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T18:27:54.584Z","updated_at":"2025-05-06T12:30:54.662Z","avatar_url":"https://github.com/davidB.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sandbox_axum_observability \u003c!-- omit in toc --\u003e\n\n!!! WIP !!!\n\nSandbox I used to experiment [axum] and observability (for target platform), observability via infra (as most as possible). The stack and framework selected:\n\n- [App (Rust http service)](#app-rust-http-service)\n - [Main components for the app](#main-components-for-the-app)\n - [Usage on local shell](#usage-on-local-shell)\n - [direct to Jaeger](#direct-to-jaeger)\n- [Infra](#infra)\n - [Kubernetes](#kubernetes)\n - [Main components for the infra](#main-components-for-the-infra)\n - [Infra setup](#infra-setup)\n - [Links \u0026 inspiration](#links--inspiration)\n\n## App (Rust http service)\n\nThe setup of the app (microservice) defined under `/app`. The Goals of the app\n\n- [ ] Use axum, async api,...\n- [ ] Delegate collect of metrics, logs,... to the infra as much as possible (eg http status, rps, ...)\n- [ ] Try to be a cloud native app, follow 12 factor app recommendation via:\n - [x] Configuration dependent of the platform / stack override via Environment variable (use clap)\n - [x] Health-check via a `GET /health` endpoint\n - [x] Log printed on std output, in json format\n - [x] Log include trace_id to easily link response, log and trace\n - [x] on first log of the span, when incoming request has trace_id\n - [x] on following log of the span, when incoming request has trace_id\n - [x] on first log of the span, when incoming request has NO trace_id (imply start an new one)\n - [x] on following log of the span, when incoming request has trace_id\n- [x] To simulate a multi-level microservice architecture, the service can call `APP_REMOTE_URL` (to define as it-self in the infra)\n- [ ] Provide a endpoint `GET /depth/{:depth}` that wait a `duration` then call endpoint defined by `APP_REMOTE_URL` with the path parameter `depth` equals to current `depth - 1`\n - [x] `depth`: value between 0 and 10, if undefined a random value will be used.\n - [x] `duration_level_max`: duration in seconds, if undefined a random between 0.0 and 2.0\n - [x] the response of `APP_REMOTE_URL` is returned as wrapped response\n - [x] if `depth` is 0, then it returns the `{ \"trace_id\": \"....\"}`\n - [ ] if failure, then it returns the `{ \"err_trace_id\": \"....\"}`\n - [x] call `GET /` is like calling `GET /depth/{:depth}` with a random depth between 0 and 10\n- [ ] To simulate error\n - [ ] `GET /health` can failed randomly via configuration `APP_HEALTH_FAILURE_PROBABILITY` (value between `0.0` and `1.0`)\n - [ ] `GET /depth/{}` can failed randomly via query parameter `failure_probability` (value between `0.0` and `1.0`)\n- [ ] add test to validate and to demo feature above\n\n### Main components for the app\n\n- [ ] [tokio-rs/axum: Ergonomic and modular web framework built with Tokio, Tower, and Hyper](https://github.com/tokio-rs/axum) as rust web framework.\n- [ ] [tokio-rs/tracing: Application level tracing for Rust.](https://github.com/tokio-rs/tracing) (and also for log)\n- [ ] [OpenTelemetry](https://opentelemetry.io/)\n\n### Usage on local shell\n\nLaunch the server\n\n```sh\ncd app\ncargo run\n```\n\nSend http request from a curl client\n\n```sh\n# without client trace\n# FIXME the log on the server include an empty trace_id\n❯ curl -i \"http://localhost:8080/depth/0\"\nHTTP/1.1 200 OK\ncontent-type: application/json\ncontent-length: 67\naccess-control-allow-origin: *\nvary: origin\nvary: access-control-request-method\nvary: access-control-request-headers\ndate: Sat, 21 May 2022 15:35:32 GMT\n\n{\"simulation\":\"DONE\",\"trace_id\":\"522e44c536fec8020790c59f20560d1a\"}⏎\n\n# with client trace\n# for traceparent see [Trace Context](https://www.w3.org/TR/trace-context/#trace-context-http-headers-format)\n❯ curl -i \"http://localhost:8080/depth/2\" -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b9c7c989f97918e1-00'\nHTTP/1.1 200 OK\ncontent-type: application/json\ncontent-length: 113\naccess-control-allow-origin: *\nvary: origin\nvary: access-control-request-method\nvary: access-control-request-headers\ndate: Sat, 21 May 2022 15:33:54 GMT\n\n{\"depth\":2,\"response\":{\"depth\":1,\"response\":{\"simulation\":\"DONE\",\"trace_id\":\"0af7651916cd43dd8448eb211c80319c\"}}}⏎\n```\n\non jaeger web ui, service `example-opentelemetry` should be listed and trace should be like\n\n![trace in jaeger](doc/images/20220521164336.png)\n\n### direct to Jaeger\n\nLaunch a local jaeger (nased on [Jaeger \u003e Getting Started \u003e All in One](https://www.jaegertracing.io/docs/1.38/getting-started/#all-in-one))\n\n```nushell\n## docker cli can be used instead of nerdctl\n## to start jaeger (and auto remove on stop)\n(nerdctl run --name jaeger --rm\n -e COLLECTOR_ZIPKIN_HOST_PORT=:9411\n -e COLLECTOR_OTLP_ENABLED=true\n -p 6831:6831/udp\n -p 6832:6832/udp\n -p 5778:5778\n -p 16686:16686\n -p 4317:4317\n -p 4318:4318\n -p 14250:14250\n -p 14268:14268\n -p 14269:14269\n -p 9411:9411\n jaegertracing/all-in-one:1.38\n)\n\n## Ctrl-C to stop it\n```\n\nopen [Jaeger web UI](http://localhost:16686/)\n\nConfigure the exporter via environment variable [sdk-environment-variables](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md)\n\n```nushell\n# replace let-env by \"export\" for bash...\nlet-env OTEL_EXPORTER_OTLP_PROTOCOL = \"grpc\"\n\n# send trace via jaeger protocol to local jaeger (agent)\ncargo run -- --tracing-collector-kind jaeger\n```\n\n## Infra\n\n### Kubernetes\n\nThe setup of the infrastructure (cluster) defined under `/infra/kubernetes`.\n\n- Try to be more like a target / live environment, so it requires more resources on local than using \"local dev approach\":\n - use distributed solution (loki, tempo,...)\n - use S3 backend (minio).\n- no ingress or api gateway setup, access will be via port forward\n- use wrapper/adapter helm chart to install components, like if it is deployed by a gitops (pull mode) system\n- keep components separated to allow partial reuse and to identify integration point\n\n#### Main components for the infra\n\n- [x] S3 buckets to store logs, metrics, traces,...\n - [x] Use Minio to provide S3 buckets in on desktop cluster (use a deprecated version but easier to setup than the operator)\n- [x] [Grafana](https://grafana.com/oss/grafana/) for dashboard and integration of log, trace, metrics\n - artifacthub.io : [grafana 6.31.0 · grafana/grafana](https://artifacthub.io/packages/helm/grafana/grafana)\n - enable sidecars, to allow other components to register datasources, dashboards, notifiers, alerts\n- [x] [Grafana Tempo](https://grafana.com/oss/tempo/) to store trace\n - artifacthub.io: [tempo-distributed 0.20.2 · grafana/grafana](https://artifacthub.io/packages/helm/grafana/tempo-distributed)\n - Setup of tempo is based on [tempo/example/helm at main · grafana/tempo](https://github.com/grafana/tempo/tree/main/example/helm), in distributed mode (consume more resources, aka several pods)\n- [x] [Grafana Loki](https://grafana.com/oss/loki/) to store log\n- [x] [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) as log collector (also provide by Grafana)\n - artifacthub.io: [promtail 6.0.0 · grafana/grafana](https://artifacthub.io/packages/helm/grafana/promtail)\n- [x] [prometheus-operator/kube-prometheus: Use Prometheus to monitor Kubernetes and applications running on Kubernetes](https://github.com/prometheus-operator/kube-prometheus), a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.\n - artifacthub.io :[kube-prometheus-stack 36.2.0 · prometheus/prometheus-community](https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack))\n - provide(by default, see doc): grafana, prometheus-operator, prometheus, alertnamaner, node-exporter\n- [x] [Linkerd](https://linkerd.io/) a service-mesh but used for its observability feature\n- [x] [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) as collector for traces\n - artifacthub.io: [opentelemetry-collector 0.22.0 · opentelemetry/opentelemetry-helm](https://artifacthub.io/packages/helm/opentelemetry-helm/opentelemetry-collector)\n - alternatives: grafana agent, send directly to tempo, use the collector from linkerd-jaeger\n - [ ] TODO: use opentelemetry operator (currently some issue with port)\n- [x] [Rancher Desktop](https://rancherdesktop.io/) as kubernetes cluster for local test, but I hope the code to be easily portable for kind, minikube, k3d, k3s,...\n- [ ] Additional dashboards, alerts,... installed via grafana's sidecars\n- [ ] Use secrets for credentials\n\n#### Infra setup\n\nRequired:\n\n- `kubectl`, `helm` v3 : to manage the k8s cluster\n\nOptional:\n\n- `nushell`: to use `tools.nu` and to avoid too many manual commands\n- Lens / OpenLens / k9s / your favorite UI: to explore states of k8s cluster\n\n```nushell\n# launch nushell\nnu\n# after launch of your local (or remote) cluster, configure kubectl to access it as current context\ncd infra/kubernetes\nuse tools.nu\ntools install_all_charts\n# to uninstall stuff ;-)\ntools uninstall_all_charts\n# to have the list of subcommand\ntools \u003ctab\u003e\n```\n\n- manual creation of `loki` bucket\n\nsample list of components\n\n```sh\n❯ kubectl get service -A\nNAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ndefault kubernetes ClusterIP 10.43.0.1 \u003cnone\u003e 443/TCP 95m\nkube-system kube-dns ClusterIP 10.43.0.10 \u003cnone\u003e 53/UDP,53/TCP,9153/TCP 95m\nkube-system metrics-server ClusterIP 10.43.155.46 \u003cnone\u003e 443/TCP 95m\nkube-system traefik LoadBalancer 10.43.205.20 192.168.5.15 80:30073/TCP,443:31505/TCP 95m\ncert-manager cert-manager-webhook ClusterIP 10.43.208.146 \u003cnone\u003e 443/TCP 94m\ncert-manager cert-manager ClusterIP 10.43.60.191 \u003cnone\u003e 9402/TCP 94m\nminio minio ClusterIP 10.43.19.151 \u003cnone\u003e 9000/TCP 94m\ngrafana grafana ClusterIP 10.43.171.106 \u003cnone\u003e 80/TCP 93m\nkube-system kube-prometheus-stack-kube-scheduler ClusterIP None \u003cnone\u003e 10251/TCP 93m\nkube-system kube-prometheus-stack-coredns ClusterIP None \u003cnone\u003e 9153/TCP 93m\nkube-system kube-prometheus-stack-kube-proxy ClusterIP None \u003cnone\u003e 10249/TCP 93m\nkube-system kube-prometheus-stack-kube-controller-manager ClusterIP None \u003cnone\u003e 10257/TCP 93m\nkube-system kube-prometheus-stack-kube-etcd ClusterIP None \u003cnone\u003e 2379/TCP 93m\nkube-prometheus-stack kube-prometheus-stack-alertmanager ClusterIP 10.43.114.25 \u003cnone\u003e 9093/TCP 93m\nkube-prometheus-stack kube-prometheus-stack-operator ClusterIP 10.43.244.229 \u003cnone\u003e 443/TCP 93m\nkube-prometheus-stack kube-prometheus-stack-prometheus-node-exporter ClusterIP 10.43.173.60 \u003cnone\u003e 9100/TCP 93m\nkube-prometheus-stack kube-prometheus-stack-kube-state-metrics ClusterIP 10.43.147.90 \u003cnone\u003e 8080/TCP 93m\nkube-prometheus-stack kube-prometheus-stack-prometheus ClusterIP 10.43.139.178 \u003cnone\u003e 9090/TCP 93m\nkube-system kube-prometheus-stack-kubelet ClusterIP None \u003cnone\u003e 10250/TCP,10255/TCP,4194/TCP 93m\nloki-distributed loki-distributed-memberlist ClusterIP None \u003cnone\u003e 7946/TCP 93m\nloki-distributed loki-distributed-ingester-headless ClusterIP None \u003cnone\u003e 3100/TCP,9095/TCP 93m\nloki-distributed loki-distributed-query-frontend ClusterIP None \u003cnone\u003e 3100/TCP,9095/TCP,9096/TCP 93m\nloki-distributed loki-distributed-querier-headless ClusterIP None \u003cnone\u003e 3100/TCP,9095/TCP 93m\nloki-distributed loki-distributed-distributor ClusterIP 10.43.235.183 \u003cnone\u003e 3100/TCP,9095/TCP 93m\nloki-distributed loki-distributed-querier ClusterIP 10.43.35.214 \u003cnone\u003e 3100/TCP,9095/TCP 93m\nloki-distributed loki-distributed-gateway ClusterIP 10.43.245.76 \u003cnone\u003e 80/TCP 93m\nloki-distributed loki-distributed-ingester ClusterIP 10.43.168.198 \u003cnone\u003e 3100/TCP,9095/TCP 93m\ntempo-distributed tempo-distributed-gossip-ring ClusterIP None \u003cnone\u003e 7946/TCP 93m\ntempo-distributed tempo-distributed-query-frontend-discovery ClusterIP None \u003cnone\u003e 3100/TCP,9095/TCP,16686/TCP,16687/TCP 93m\ntempo-distributed tempo-distributed-query-frontend ClusterIP 10.43.85.84 \u003cnone\u003e 3100/TCP,9095/TCP,16686/TCP,16687/TCP 93m\ntempo-distributed tempo-distributed-ingester ClusterIP 10.43.242.5 \u003cnone\u003e 3100/TCP,9095/TCP 93m\ntempo-distributed tempo-distributed-querier ClusterIP 10.43.20.61 \u003cnone\u003e 3100/TCP,9095/TCP 93m\ntempo-distributed tempo-distributed-distributor ClusterIP 10.43.13.183 \u003cnone\u003e 3100/TCP,9095/TCP,4317/TCP,55680/TCP 93m\ntempo-distributed tempo-distributed-memcached ClusterIP 10.43.106.141 \u003cnone\u003e 11211/TCP,9150/TCP 93m\ntempo-distributed tempo-distributed-compactor ClusterIP 10.43.10.39 \u003cnone\u003e 3100/TCP 93m\ntempo-distributed tempo-distributed-metrics-generator ClusterIP 10.43.129.131 \u003cnone\u003e 9095/TCP,3100/TCP 93m\nopentelemetry-collector opentelemetry-collector ClusterIP 10.43.15.153 \u003cnone\u003e 6831/UDP,14250/TCP,14268/TCP,4317/TCP,4318/TCP,9411/TCP 93m\nlinkerd linkerd-dst ClusterIP 10.43.126.243 \u003cnone\u003e 8086/TCP 92m\nlinkerd linkerd-dst-headless ClusterIP None \u003cnone\u003e 8086/TCP 92m\nlinkerd linkerd-sp-validator ClusterIP 10.43.41.57 \u003cnone\u003e 443/TCP 92m\nlinkerd linkerd-policy ClusterIP None \u003cnone\u003e 8090/TCP 92m\nlinkerd linkerd-policy-validator ClusterIP 10.43.225.36 \u003cnone\u003e 443/TCP 92m\nlinkerd linkerd-identity ClusterIP 10.43.136.50 \u003cnone\u003e 8080/TCP 92m\nlinkerd linkerd-identity-headless ClusterIP None \u003cnone\u003e 8080/TCP 92m\nlinkerd linkerd-proxy-injector ClusterIP 10.43.51.211 \u003cnone\u003e 443/TCP 92m\napp app ClusterIP 10.43.108.47 \u003cnone\u003e 80/TCP 91m\nlinkerd-viz metrics-api ClusterIP 10.43.179.165 \u003cnone\u003e 8085/TCP 67m\nlinkerd-viz tap-injector ClusterIP 10.43.71.201 \u003cnone\u003e 443/TCP 67m\nlinkerd-viz tap ClusterIP 10.43.191.138 \u003cnone\u003e 8088/TCP,443/TCP 67m\nlinkerd-viz web ClusterIP 10.43.18.39 \u003cnone\u003e 8084/TCP,9994/TCP 67m\nlinkerd-jaeger jaeger-injector ClusterIP 10.43.72.101 \u003cnone\u003e 443/TCP 67m\n```\n\nUse port forward to access UI and service\n\n```sh\n# access grafana UI on http://127.0.0.1:8040\nkubectl port-forward -n grafana service/grafana 8040:80\n\n# access grafana UI on http://127.0.0.1:9009 (user/pass: minio/minio123)\nkubectl port-forward -n minio service/minio 9009:9000\n\n# access linerd-viz UI on http://127.0.0.1:8084\nkubectl port-forward -n linkerd-viz service/web 8084:8084\n\n# On rancher-desktop only\n# access traefik dashboard on http://127.0.0.1:9000/dashboard/#/\nbash -c 'kubectl port-forward -n kube-system $(kubectl -n kube-system get pods --selector \"app.kubernetes.io/name=traefik\" --output=name) 9000:9000'\n\n```\n\u003c!--\n# access grafana UI on https://127.0.0.1:9443\nkubectl port-forward -n minio-tenant-1 service/tenant-1-console 9443:9443\n\n# access minio Console of operator\n# - install kubectl krew (see [Installing · Krew](https://krew.sigs.k8s.io/docs/user-guide/setup/install/))\nkubectl krew install minio\nkubectl minio proxy -n minio-operator\n# console of tenant-1\nkubectl minio proxy -n minio-tenant-1\n--\u003e\n\nSetup the app and call it\n\n```sh\nkubectl port-forward -n app service/app 8080:80\ncurl -i \"http://localhost:8080/depth/2\"\n```\n\n![log+loki+grafana](assets/log_2022-07-24-212509_1737x1301_scrot.png)\n\n![trace+tempo+grafana](assets/trace_2022-07-24-212616_1622x1288_scrot.png)\n\n![node+tempo+grafana](assets/node_2022-07-24-212603_1639x1266_scrot.png)\n\nBut when using port-forward request doesn't go through linkerd proxy (so no monitoring of route,...) (see [port-forward traffic skips the proxy · Issue #2352 · linkerd/linkerd2](https://github.com/linkerd/linkerd2/issues/2352))\nSo If you don't have ingress setup,... you send request from inside the cluster:\n\n```sh\nkubectl run tmp-shell -n default --restart=Never --rm -i -tty --image curlimages/curl:7.84.0 -- curl -L -v http://app.app.svc.cluster.local/depth/3\n\n\n# Or via an interactive shell if you want\nkubectl run tmp-shell -n default --restart=Never --rm -i --tty --image curlimages/curl:7.84.0 -- sh\n\u003e curl -L -v http://app.app.svc.cluster.local/depth/3\n...\n\u003e exit\n\nkubectl delete pod tmp-shell -n default\n```\n\n![linkerd_route](assets/linkerd_route_2022-07-24-213720_1641x1289_scrot.png)\n\n![pod_usage](assets/pod_2022-07-24-214343_1662x1265_scrot.png)\n\nSample list of other dashboards\n![pod_usage](assets/dashboards_2022-07-24-214238_1388x1245_scrot.png)\n\n#### Links \u0026 inspiration\n\n- [tempo/integration/microservices at main · grafana/tempo](https://github.com/grafana/tempo/tree/main/integration/microservices)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidb%2Fsandbox_axum_observability","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidb%2Fsandbox_axum_observability","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidb%2Fsandbox_axum_observability/lists"}