{"id":15137139,"url":"https://github.com/rogerhmar/otel-ndc-2024","last_synced_at":"2025-10-23T12:30:22.526Z","repository":{"id":242394412,"uuid":"776026371","full_name":"rogerhmar/OTEL-NDC-2024","owner":"rogerhmar","description":"Setup used during workshop on NDC Oslo 2024","archived":false,"fork":false,"pushed_at":"2024-06-13T11:57:55.000Z","size":1870,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-30T18:47:49.396Z","etag":null,"topics":["dotnet8","grafana","loki","ndc","observability","opentelemetry","prometheus"],"latest_commit_sha":null,"homepage":"https://ndcoslo.com/agenda/opentelemetry-starter-pack-0ibp/03ljtvp5lbb","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rogerhmar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-22T14:29:27.000Z","updated_at":"2024-06-16T09:48:00.000Z","dependencies_parsed_at":"2024-09-21T11:02:07.943Z","dependency_job_id":"2f44f866-7546-47a3-8ee2-098c20b68ed8","html_url":"https://github.com/rogerhmar/OTEL-NDC-2024","commit_stats":null,"previous_names":["rogerhmar/otel-ndc-2024"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogerhmar%2FOTEL-NDC-2024","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogerhmar%2FOTEL-NDC-2024/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogerhmar%2FOTEL-NDC-2024/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogerhmar%2FOTEL-NDC-2024/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rogerhmar","download_url":"https://codeload.github.com/rogerhmar/OTEL-NDC-2024/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237821782,"owners_count":19371827,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dotnet8","grafana","loki","ndc","observability","opentelemetry","prometheus"],"created_at":"2024-09-26T07:00:22.791Z","updated_at":"2025-10-23T12:30:21.562Z","avatar_url":"https://github.com/rogerhmar.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenTelemetry Workshop\n[OpenTelemetry Starter Pack NDC {Oslo} 2024](https://ndcoslo.com/agenda/opentelemetry-starter-pack-0ibp/03ljtvp5lbb)\n\n## Run Infrastructure - Getting started\n\n### 1. Verify prerequisites\n* Docker is running? Run `docker ps`. You should not get any errors-\n  * `error during connect` means Rancher Desktop/Podman Desktop/Docker Desktop has not been starterd\n* Docker-Compose is installed? Run `Docker-Compose -v`\n* Dotnet is installed? `dotnet --version`. This should display the version.\n\n### 2. Run the infrastructure\n* Start by running `docker compose up` add `-d` to run detatched (just start it without displaying all logs)\n* To clean up any docker container run `docker-compose down` from this folder.\n\n### 3. Run the demo application - ExampleApi\nRun it with `dotnet run` or inside an IDE. This will give you more about URLs you can visit.\n\n### 4. Run a test to verify the setup\nGo to [localhost:8080/test](http://localhost:8080/test)\n\n## Setup\n\n### Prerequisites\n* Tool for building and running containers, e.g. Docker Desktop, Podman or Rancher Desktop - Including Compose.\n* Dotnet 8\n\n### Overall Architecture\n```mermaid\nflowchart LR\n    A[\"Application(example)\"] --\u003e B(\"OpenTelemetry Collector \\n :4317 (gRPC)\")\n    H[\"dependency1\"] --\u003e B\n    I[\"dependency2\"] --\u003e B\n    J[\"dependency3\"] --\u003e B\n    B --\u003e D(\"Loki \\n :3100\") --\u003e G(\"Grafana\")\n    B --\u003e E(\"Tempo \\n :3200\") --\u003e G\n    B --\u003e F(Prometheus \\n :9090)--\u003e G\n    K(k6)--\u003e|remote write|F\n\n    style A fill:red\n    style B fill:green\n    style H fill:blue\n    style I fill:blue\n    style J fill:blue\n    style D fill:yellow\n    style E fill:orange\n    style F fill:cyan\n    style G fill:purple\n```\n\n- Your application, called example, send data directly to the OTEl collector over gRPC\n- The \"legacy\" dependencies use autoinstrumentation and do the same\n- Prometheus scrapes data from the OTEL collector\n- Collector writes data to Loki and Tempo\n- Grafana uses the 3 sources to display data\n- k6 writes to prometheus using remote write\n\n### Versions\nVersions are defined in [.env](./.env)\n\nNote 09.06.2024: Tempo is currently running version 2.4.2. The latest version 2.5.0 have breaking changes related to the ownership of `Ownership of /var/tempo`. This was causing issues. Refer to https://github.com/grafana/tempo/releases/tag/v2.5.0 for more info.\n\n### Alternative to this setup\nGrafana has release a simplified setup with a single container, This is found here: https://github.com/grafana/docker-otel-lgtm/.\n\n## Instrumentation - What are our options?\nThere are 2 options for setting up OpenTelemetry in .NET applications.\n* Setup with code with the option of using both manual and automatic instrumentation\n* No-code automatic setup of automatic instrumentation. Manual instrumentation cannot be added\n\n### Manual setup\nThe example app uses this setup. Refer to [SetupOpentelemetry](source/example/SetupOpentelemetry.cs) to see how this may be done. For more infomation and examples refer to\n* https://opentelemetry.io/docs/languages/net/\n\nThe self-paced tasks focus on the manual setup.\n\n### Automatic instrumentation\nThis setup has include 3 containers (dependency1..3) with automatic instrumentation. The 2 main solutions for doing this is:\n* Download and run `otel-dotnet-auto-install.sh` or `OpenTelemetry.DotNet.Auto.psm1` or\n* Include Nuget `OpenTelemetry.AutoInstrumentation`\n\nWe have used the latter approch. Refer to dependency1 [docker-compose.yaml](./docker-compose.yaml) for an example. This example includes ENV variables for easier debugging.\n\nFor more information refer to:\n* https://opentelemetry.io/docs/zero-code/net/\n\n\n## Example App\n```\n    .\n    ├── exampleAPI.http        \u003c- To run HTTP command. An alternative to using Swagger or browser\n    ├── MapRoutesExtensions.cs \u003c- Sets up the routes\n    ├── SetupOpentelemetry.cs  \u003c- All OpenTelemetry setup for Logging, Tracing and Metrics\n    └── Program.cs...          \u003c- All the normal stuff\n```\n\nStartup app and go to `http://localhost:5000/`\n\n## Grafana and some fundamentals for viewing data\n\nGrafana shows the data using 3 data sources:\n* Tempo for tracing\n* Prometheus for metrics\n* Loki for Logging\n\nPS: Message `Failed to authenticate request` might appear. This should not have any impact, but is noisy. Go to `Sign in` in the top right corner and sign in with user and password, `admin` and `admin`\n\nBefore looking at data, you need to populate some data. You can do that by running some of the HTTP requests in:\n* [dependencyApi.http](./source/dependency/dependencyApi.http)\n* [exampleAPI.http](./source/example/exampleAPI.http) \n\nThen open Grafana on [localhost:3000](http://localhost:3000)\n\n### Tempo - Tracing - Connecting components\n```mermaid\nflowchart LR\n    A[\"Grafana Home\"] --\u003e B(\"Explore\") --\u003e C(\"Select source 'Tempo' - Default data source is 'Loki'\") --\u003e D(\"Select 'Search' - Default is 'TraceQL'\")\n\n    style A fill:blue\n    style B fill:Yellow\n    style C fill:Green\n    style D fill:Red\n```\nRead more about TraceQL here: https://grafana.com/docs/tempo/latest/traceql/\n\nShould look something like this:\n\n\u003cimg src=\"./images/Tempo.png\" width=\"80%\"\u003e\n\n### Prometheus - Metrics - Statistics\n```mermaid\nflowchart LR\n    A[\"Grafana Home\"] --\u003e B(\"Explore\") --\u003e C(\"Select source 'Prometheus' - Default data source is 'Loki'\") --\u003e D(\"Metrics browser'\")\n\n    style A fill:blue\n    style B fill:Yellow\n    style C fill:Green\n    style D fill:Red\n```\nPrometheus uses PromQL as a query language. Here are some examples: https://prometheus.io/docs/prometheus/latest/querying/examples/\n\nYou should be able to run `http_client_request_duration_seconds_count{}`. Send HTTP requests e.g. from [exampleAPI.http](./source/example/exampleAPI.http)\n\nShould look something like this:\n\n\u003cimg src=\"./images/Prometheus.png\" width=\"80%\"\u003e\n\n### Loki - Logging - Telling the story\n```mermaid\nflowchart LR\n    A[\"Grafana Home\"] --\u003e B(\"Explore\") --\u003e C(\"Default data source is 'Loki'\") --\u003e D(\"Add a LogQL\")\n\n    style A fill:blue\n    style B fill:Yellow\n    style C fill:Green\n    style D fill:Red\n```\nLoki uses LogQL. Refer to https://grafana.com/docs/loki/latest/query/.\nYou can start by adding LogQL:  `{exporter=\"OTLP\"}`. This will show all log records exported by the OpenTelemtry Collector\n\n\nShould look something like this:\n\n\u003cimg src=\"./images/Loki.png\" width=\"75%\"\u003e\n\n## Dashboards\n* [ASP .NET OTEL Metrics](http://localhost:3000/d/ASP_NET_OTLP_COL_SHARED/asp-net-otel-metrics-from-otel-collector?orgId=1\u0026var-job=exampleApiSetInEnv\u0026var-instance=\u0026var-http_client_peer_name=All\u0026from=now-30m\u0026to=now\u0026refresh=1m)\n  * Ref: https://grafana.com/grafana/dashboards/19896-asp-net-otel-metrics-from-otel-collector/\n* [k6 Prometheus](http://localhost:3000/d/a3b2aaa8-bb66-4008-a1d8-16c49afedbf0/k6-prometheus-native-histograms?orgId=1)\n  * Ref: https://grafana.com/grafana/dashboards/18030-k6-prometheus-native-histograms/\n* [OpenTelemetry Collector](http://localhost:3000/d/BKf2sowmj/opentelemetry-collector?orgId=1\u0026refresh=10s)\n  * Ref: https://grafana.com/grafana/dashboards/15983-opentelemetry-collector/\n* [Custom Counter](http://localhost:3000/d/c2b94126-3764-4cea-b30b-7207a0be87d9/counter-dashboard?orgId=1)\n\n# Self-paced Tasks\n\n## Verify Setup and getting started\n### Has everything started?\n* Start the infrastructure as in this [section](#run-infrastructure)\n* Start the `ExampleApi`\n* Verify that everything in up and running with [http://localhost:8080/test](http://localhost:8080/test)\n* Open the webpage in [http://localhost:5000/](http://localhost:5000/)\n\n### Is the .NET app connected correctly?\n* Go to [Loki last 5 min](http://localhost:3000/explore?schemaVersion=1\u0026panes=%7B%22eiw%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bexporter%3D%5C%22OTLP%5C%22%7D%20%7C%20json%20%7C%20line_format%20%5C%22%7B%7B.resources_service_name%7D%7D%20%7B%7B.resources_service_version%7D%7D%20%7B%7B.body%7D%7D%5C%22%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%22now-5m%22,%22to%22:%22now%22%7D%7D%7D\u0026orgId=1)\n* Confirm that Service name has not been set. It should be `unknown_service:dotnet` (at this stage)\n* Understand the basics for the setup. Refer to [SetupOpentelemetry.cs](./source/example/SetupOpentelemetry.cs)\n  * Add `\"OTEL_SERVICE_NAME\": \"exampleApiSetInEnv\"` to env e.g. here: [launchSettings.json](./source/example/Properties/launchSettings.json)\n  * Uncomment the code in for configure resources in [SetupOpentelemetry.cs](./source/example/SetupOpentelemetry.cs)\n  * Did it work as expected? How to fix it?\n* Verify that `ExampleApi` is also sending Metrics and Traces the OpenTelemetry Collector, by checking [Prometheus](http://localhost:3000/explore?schemaVersion=1\u0026panes=%7B%226oa%22:%7B%22datasource%22:%22prometheus%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22http_client_request_duration_seconds_sum%7B%7D%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22prometheus%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D\u0026orgId=1) and [Tempo](http://localhost:3000/goto/2uq52zUSg?orgId=1) in Grafana. Refer to `section Grafana and some fundamentals for viewing data`\n\n## Tracing\n### Task T1: Reducing noise\n* Locate the tracing RequestFilter in [SetupOpentelemetry.cs](./source/example/SetupOpentelemetry.cs)\n* Send request to the [http://localhost:5000/remove](http://localhost:5000/remove)\n* Update the request filter (input to SetupOpentelemetry) to remove this endpoint\n* Verify in Tempo that you succeeded.\n\n### Task T2: Use the API and understand the delay in the execution\n* send requests to the APIs\n  * [http://localhost:5000/parallel](http://localhost:5000/parallel)\n  * [http://localhost:5000/serial](http://localhost:5000/serial)\n* It is easy understand what is causing the delay? Check if you understand it in `Tempo`\n* Find out why tracing is missing in `ThisNeedsToBeTraced`. Note that you need to start the activity, not only create it.\n\n### Task T3: Use the API and get exceptions\n* Send requests to \n  * [http://localhost:5000/throwEx](http://localhost:5000/throwEx)\n  * [http://localhost:5000/error](http://localhost:5000/error)\n* Become familiar with the different ways of tracing errors. The mapping is done in [MapRoutesExtensions.cs](./source/example/MapRoutesExtensions.cs)\n* Look at the trace in `Tempo`\n\n## Metrics\n### Task M1: Understanding the data flow\n* I have added a custom metric to `SuperServiceWithMetrics` [SuperServiceWithMetrics.cs](./source/example/Service/SuperServiceWithMetrics.cs)\n* Use the endpoint [http://localhost:5000/metric/inc/10](http://localhost:5000/metric/inc/10) to increment the custom metric.\n* Are you seeing any data in [Grafana dashboard](http://localhost:3000/d/c2b94126-3764-4cea-b30b-7207a0be87d9/counter-dashboard?orgId=1)? No? Why?\n* PS:  the counter `super_service_counter` produces the metric `super_service_counter_total`\n* Verify that you can track the metric from the OpenTelemetry collector to Grafana-\n  * Find the metric in the output of the Open Telemetry Collector [http://localhost:8889/metrics](http://localhost:8889/metrics)\n  * Find the metric in prometheus:  [http://localhost:9090/graph](http://localhost:9090/graph)\n  * Find the metric in Grafana: http://localhost:3000/explore [http://localhost:3000/explore](http://localhost:3000/explore)\n* There is a bug in the custom metrics counter? Can you spot it? Fix it.\n\n### Task M2: What metrics are added?\n* Look into: AddAspNetCoreInstrumentation. It is called in [SetupOpentelemetry.cs](./source/example/SetupOpentelemetry.cs)\n  * You can also see it here: [AspNetCoreInstrumentationMeterProviderBuilderExtensions.cs](https://github.com/vishweshbankwar/opentelemetry-dotnet/blob/ca5b3aca0e1218a589556097b0537cb97f62da98/src/OpenTelemetry.Instrumentation.AspNetCore/AspNetCoreInstrumentationMeterProviderBuilderExtensions.cs)\n  * Read more about core metrics https://learn.microsoft.com/en-us/dotnet/core/diagnostics/built-in-metrics-aspnetcore\n* Generally, It is a bit hard to keep track which metrics are added. Use what you learned from task M1 to see what data is actually present.\n* The metrics have more than once broken the dashboards.\n\n### Task M3: Add a custom metric\n* Add another counter to the setup \n* Verify that this has been added to your metrics\n\n## Logging\n### Task L1: Add Logging and understand the log record\n* Open [/parallel](http://localhost:5000/parallel)\n* Open [Loki with LogQL {exporter=\"OTLP\"}](http://localhost:3000/explore?schemaVersion=1\u0026panes=%7B%22eiw%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bexporter%3D%5C%22OTLP%5C%22%7D%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%22now-30m%22,%22to%22:%22now%22%7D%7D%7D\u0026orgId=1)\n  * Get to know the log record. Some key fields are:\n    * body\n    * severety\n    * attributes\n    * resources\n    * instrumentation scope name\n* Open [/metric/inc/10](http://localhost:5000/metric/inc/10)\n  * How did instrumentation scope change?\n* Stop the application\n  * Look at the log record with body \"Application is shutting down...\". Does it have spanid and traceid? Why?\n* PS: LogQL `{exporter=\"OTLP\"} | json | line_format \"{{.body}}\"` gives a clean prinout of log record body. Try this. Observe how the log records are flattended\n\n### Task L2: Find Traces from log record\n* Send request to [http://localhost:5000/parallel](http://localhost:5000/parallel)\n* Go to log lines created during this request processing, and find the trace ID\n* Go to the trace in Tempo.\n\n### Task L3: Using scopes\n* Go to [SuperService.cs](./source/example/Service/SuperService.cs), and add `logger.BeginScope`\n* What do you see in Loki. What and where is the scope state added in the log record?\n* Advanced/optional:\n  * Try to duplicate the scope call\n  * Do you see the duplicated entry?\n  * What do you get if you mutate the value (and not the key) `logger.BeginScope`?\n  * Look at the log record in the custom processer in [SetupOpentelemetry.cs](./source/example/SetupOpentelemetry.cs)\n  * And look in he OtlpLogRecordTransformer. Can you find the duplicated attribute key?\n\n### Task L4: Include formatted message?\n* Go to [SetupOpentelemetry.cs](./source/example/SetupOpentelemetry.cs), and set `IncludeFormattedMessage = false` in the configuration of the logging.\n* Open [/parallel](http://localhost:5000/parallel)\n* Go to Loki and observe how the body changes.\n\n## Bonus\n### Task B1: Run k6 test and observe the system under load\n* Install k6. E.g. by running this https://dl.k6.io/msi/k6-latest-amd64.msi\n* Docs: https://k6.io/docs/\n* Run the test with `K6_PROMETHEUS_RW_SERVER_URL=http://localhost:9090/api/v1/write K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true k6 run -o experimental-prometheus-rw test/script.js`\n* View the result in the dashboard called `k6 Prometheus`\n\n### Task B2: Configure using Signal specific AddOtlpExpoerter methods\n* Update [SetupOpentelemetry.cs](./source/example/SetupOpentelemetry.cs) to use signal-specific (Logging, Trace, Metrics) setup of OTLP exporter.\n\n### Task B3: Debugging\n* Add the debug exporter to the pipeline in the [OTEL collector config](./config/otel-collector-config.yaml)\n  * traces\n  * Logging\n  * traces\n\n## Where can I go from here?\nYou can e.g. \n* Test with sampling https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/probabilisticsamplerprocessor/README.md\n* Check out frontend instrumentation using Faro https://github.com/grafana/faro-web-sdk\n* Replace the OTEL collector with Alloy https://grafana.com/docs/alloy/latest/\n* Check out the .NET Aspire dashboard.\n\n# Credit\nThis setup originally is based on: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/examples/demo\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frogerhmar%2Fotel-ndc-2024","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frogerhmar%2Fotel-ndc-2024","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frogerhmar%2Fotel-ndc-2024/lists"}