{"id":14971769,"url":"https://github.com/grafana/oats","last_synced_at":"2026-02-12T20:01:42.988Z","repository":{"id":195713425,"uuid":"674042654","full_name":"grafana/oats","owner":"grafana","description":null,"archived":false,"fork":false,"pushed_at":"2026-01-31T06:25:58.000Z","size":611,"stargazers_count":41,"open_issues_count":1,"forks_count":3,"subscribers_count":9,"default_branch":"main","last_synced_at":"2026-01-31T20:22:04.337Z","etag":null,"topics":["acceptance-testing","opentelemetry"],"latest_commit_sha":null,"homepage":"https://grafana.com/blog/2025/07/08/observability-in-under-5-seconds-reflecting-on-a-year-of-grafana/otel-lgtm/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grafana.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-08-03T02:38:44.000Z","updated_at":"2026-01-31T07:43:52.000Z","dependencies_parsed_at":"2025-11-29T18:03:09.145Z","dependency_job_id":null,"html_url":"https://github.com/grafana/oats","commit_stats":{"total_commits":61,"total_committers":7,"mean_commits":8.714285714285714,"dds":0.7049180327868853,"last_synced_commit":"9a79819efcde37f025613914708dd1ba721e5ddc"},"previous_names":["grafana/oats"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/grafana/oats","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grafana%2Foats","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grafana%2Foats/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grafana%2Foats/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grafana%2Foats/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grafana","download_url":"https://codeload.github.com/grafana/oats/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grafana%2Foats/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29379634,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T19:05:20.189Z","status":"ssl_error","status_checked_at":"2026-02-12T19:01:44.216Z","response_time":55,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acceptance-testing","opentelemetry"],"created_at":"2024-09-24T13:45:48.326Z","updated_at":"2026-02-12T20:01:42.981Z","avatar_url":"https://github.com/grafana.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenTelemetry Acceptance Tests (OATs)\n\nOpenTelemetry Acceptance Tests (OATs), or OATs for short, is a test framework for OpenTelemetry.\n\n- Declarative tests written in YAML\n- Supported signals: traces, logs, metrics\n- Full round-trip testing: from the application to the observability stack\n  - Data is stored in the LGTM stack ([Loki], [Grafana], [Tempo], [Prometheus], [OpenTelemetry Collector])\n  - Data is queried using LogQL, PromQL, and TraceQL\n  - All data is sent to the observability stack via OTLP - so OATs can also be used with other observability stacks\n- End-to-end testing\n  - Docker Compose with the [docker-otel-lgtm] image\n  - Kubernetes with the [docker-otel-lgtm] and [k3d]\n\n## Installation\n\n1. Install the `oats` binary:\n\n```sh\ngo install github.com/grafana/oats@latest\n```\n\n2. You can confirm it was installed with:\n\n```sh\n❯ ls $GOPATH/bin\noats\n```\n\n## Getting Started\n\n\u003e [!TIP]\n\u003e You can use the test cases in [prom_client_java](https://github.com/prometheus/client_java/tree/main/examples/example-exporter-opentelemetry/oats-tests) as a reference.\n\u003e The [GitHub action](https://github.com/prometheus/client_java/blob/main/.github/workflows/acceptance-tests.yml)\n\u003e uses a [script](https://github.com/prometheus/client_java/blob/main/scripts/run-acceptance-tests.sh) to run the tests.\n\n1. Create a folder `oats-tests` for the following files\n2. Create `Dockerfile` to build the application you want to test\n    ```Dockerfile\n    FROM eclipse-temurin:21-jre\n    COPY target/example-exporter-opentelemetry.jar ./app.jar\n    ENTRYPOINT [ \"java\", \"-jar\", \"./app.jar\" ]\n    ```\n3. Create `docker-compose.yaml` to start the application and any dependencies\n    ```yaml\n    services:\n      java:\n        build:\n          dockerfile: Dockerfile\n        environment:\n          OTEL_SERVICE_NAME: \"rolldice\"\n          OTEL_EXPORTER_OTLP_ENDPOINT: http://lgtm:4318\n          OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf\n    ```\n4. Create `oats.yaml` with the test cases\n    ```yaml\n    # OATs is an acceptance testing framework for OpenTelemetry - https://github.com/grafana/oats\n    oats-schema-version: 2\n    docker-compose:\n      files:\n        - ./docker-compose.yaml\n    expected:\n      metrics:\n        - promql: 'uptime_seconds_total{}'\n          value: '\u003e= 0'\n    ```\n5. Run the tests:\n```sh\noats /path/to/oats-tests/oats.yaml\n```\n\n## Running OATs Directly\n\nOATs can be run directly using the command-line interface:\n\n```sh\n# Run specific test files\noats /path/to/oats-tests/oats.yaml\n\n# Run multiple specific test files\noats /path/to/repo/test1.yaml /path/to/repo/test2.yaml\n\n# Run all tests in a directory (scans for .yaml/.yml files with oats-schema-version)\noats /path/to/oats-tests\n\n# With flags\noats --timeout=1m --lgtm-version=latest --manual-debug=false /path/to/oats-tests/oats.yaml\n```\n\n## Running multiple tests\n\nIt can run multiple tests:\n\n```sh\n# Scan directory for all test files\noats /path/to/repo\n\n# Or specify individual test files (better performance)\noats /path/to/repo/test1.yaml /path/to/repo/test2.yaml\n```\n\nWhen scanning a directory, OATs will search for all `.yaml` and `.yml` files that contain the `oats-schema-version` tag. Files marked with `oats-template: true` will be skipped as entry points but can still be included by other test files.\n\n## Flags \n\nThe following flags are available:\n\n- `-timeout`: Set the timeout for test cases (default: 30s)\n- `-absent-timeout`: Set the timeout for tests that assert absence (default: 10s)\n- `-lgtm-version`: Specify the version of [docker-otel-lgtm] to use (default: `\"latest\"`)\n- `-manual-debug`: Enable debug mode to keep containers running (default: `false`)\n- `-lgtm-log-all`: Enable logging for all containers (default: `false`)\n- `-lgtm-log-grafana`: Enable logging for Grafana (default: `false`)\n- `-lgtm-log-loki`: Enable logging for Loki (default: `false`)\n- `-lgtm-log-tempo`: Enable logging for Tempo (default: `false`)\n- `-lgtm-log-prometheus`: Enable logging for Prometheus (default: `false`)\n- `-lgtm-log-pyroscope`: Enable logging for Pyroscope (default: `false`)\n- `-lgtm-log-collector`: Enable logging for OpenTelemetry Collector (default: `false`)\n- `-host`: Override the host used to issue requests to applications and LGTM (default: `localhost`)\n- `-log-limit`: Maximum log output length per log entry\n\n## Run OATs in GitHub Actions\n\nHere's a [script](https://github.com/grafana/docker-otel-lgtm/blob/main/scripts/run-acceptance-tests.sh) that is used\nfrom GitHub Actions. It uses [mise](https://mise.jdx.dev/) to install OATs, but you also [install OATs directly](#installation).\n\n## Test Case Syntax\n\n\u003e [!TIP]\n\u003e All test files must include `oats-schema-version: 2` at the top level.\n\u003e Template files (used in `include` sections) must also include `oats-template: true` to prevent them from being run as entry points.\n\u003e You can use any file name with `.yaml` or `.yml` extension.\n\nThe syntax is a bit similar to [Tracetest](https://github.com/kubeshop/tracetest).\n\nHere is an example:\n\n```yaml\noats-schema-version: 2\ninclude:\n  - ../oats-template.yaml\ndocker-compose:\n  files:\n    - ../docker-compose.yaml\ninput:\n  - path: /stock\n    status: 200 # expected status code, 200 is the default\ninterval: 500ms # interval between requests to the input URL\nexpected:\n  traces:\n    - traceql: '{ name =~ \"SELECT .*product\"}'\n      regexp: 'SELECT .*'\n      attributes:\n        db.system: h2\n  logs:\n    - logql: '{exporter = \"OTLP\"}'\n      equals: 'hello LGTM'\n  metrics:\n    - promql: 'db_client_connections_max{pool_name=\"HikariPool-1\"}'\n      value: \"== 10\"\n```\n\n### Template Files\n\nTemplate files are used to share common configuration across multiple test files. They must include both `oats-schema-version` and `oats-template: true`:\n\n```yaml\n# oats-template.yaml\noats-schema-version: 2\noats-template: true\n\ndocker-compose:\n  files:\n    - ./docker-compose.yaml\n```\n\nHere is another example with a more specific input:\n\n```yaml\noats-schema-version: 2\ninclude:\n  - ../oats-template.yaml\ndocker-compose:\n  files:\n    - ../docker-compose.yaml\ninput:\n  - path: /users\n    method: POST\n    scheme: https\n    host: 127.0.0.1\n    status: 201\n    headers:\n      Authorization: Bearer my-access-token\n      Content-Type: application/json\n    body: |-\n      {\n        \"name\": \"Grot\"\n      }\ninterval: 500ms\nexpected:\n  traces:\n    - traceql: '{ name =~ \"SELECT .*product\"}'\n      regexp: 'SELECT .*'\n      attributes:\n        db.system: h2\n```\n\n### Query traces\n\nEach entry in the `traces` array is a test case for traces.\n\n```yaml\nexpected:\n  traces:\n    - traceql: '{ name =~ \"SELECT .*product\"}'\n      regexp: 'SELECT .*'\n      attributes:\n        db.system: h2\n      count:\n        min: 1 # allow multiple spans with the same attributes\n    - traceql: '{ span.kind = \"client\" }'\n      equals: 'HTTP GET'\n    - traceql: '{ name =~ \"dropped-span\" }'\n      count:\n        max: 0  # assert this span does NOT exist (e.g., filtered/dropped spans)\n```\n\n#### Trace assertion options\n\n- **`traceql`**: TraceQL query to find the trace (required)\n- **`equals`**: Exact string match for the span name (any span in the trace)\n- **`regexp`**: Regular expression pattern to match against the span name (any span in the trace)\n- **`attributes`**: Key-value pairs that must match exactly on the span (the span name matched by `equals` or `regexp`)\n- **`attribute-regexp`**: Key-value pairs where values are regex patterns to match against span attributes (the span name matched by `equals` or `regexp`)\n- **`no-extra-attributes`**: Set to `true` to fail if the span has attributes beyond those specified in `attributes` and `attribute-regexp`\n- **`count`**: Control expected number of matching spans, ignoring if they match other criteria\n  - **`min`**: Minimum number of spans expected (default: `1` if not specified)\n  - **`max`**: Maximum number of spans expected (`0` means no upper limit, or exactly `0` when `min` is also `0`)\n  - Examples:\n    - Not specified: at least 1 span expected\n    - `{ min: 2, max: 5 }`: between 2 and 5 spans (inclusive)\n    - `{ min: 3 }`: 3 or more spans\n    - `{ max: 0 }`: exactly 0 spans (assert absence)\n- **`matrix-condition`**: Regex to match against matrix test case names (only run this assertion for matching matrix cases)\n\n### Query logs\n\nEach entry in the `logs` array is a test case for logs.\n\n```yaml\nexpected:\n  logs:\n    - logql: '{service_name=\"rolldice\"} |~ `Anonymous player is rolling the dice.*`'\n      equals: 'Anonymous player is rolling the dice'\n      attributes:\n        service_name: rolldice\n      attribute-regexp:\n        container_id: \".*\"\n      no-extra-attributes: true # fail if there are extra attributes\n    - logql: '{service_name=\"rolldice\"} |~ `Anonymous player is rolling the dice.*`'\n      regexp: 'Anonymous player is .*'\n```\n\n#### Log assertion options\n\n- **`logql`**: LogQL query to find the log line (required)\n- **`equals`**: Exact string match for the log line\n- **`regexp`**: Regular expression pattern to match against the log line\n- **`attributes`**: Key-value pairs that must match exactly on the log labels\n- **`attribute-regexp`**: Key-value pairs where values are regex patterns to match against log labels\n- **`no-extra-attributes`**: Set to `true` to fail if the log has labels beyond those specified in `attributes` and `attribute-regexp`\n- **`count`**: Expected count range for returned log lines, ignoring if they match other criteria\n  - **`min`**: Minimum expected count (defaults to `0` if not specified)\n  - **`max`**: Maximum expected count. Set to `0` for no upper limit. To assert absence, set both `min: 0` and `max: 0`\n- **`matrix-condition`**: Regex to match against matrix test case names\n\nExample:\n```yaml\nexpected:\n  logs:\n    - logql: '{service_name=\"rolldice\"}'\n      equals: 'Rolling dice'\n      count:\n        min: 1\n        max: 5  # expect between 1-5 matching logs (inclusive)\n```\n\n### Query metrics\n\n```yaml\nexpected:\n  metrics:\n    - promql: 'db_client_connections_max{pool_name=\"HikariPool-1\"}'\n      value: \"== 10\"\n```\n\n#### Metric assertion options\n\n- **`promql`**: PromQL query to retrieve the metric (required)\n- **`value`**: Expected value with comparison operator. Supported operators: `==`, `!=`, `\u003e`, `\u003c`, `\u003e=`, `\u003c=` (e.g., `\"\u003e= 0\"`, `\"== 10\"`)\n- **`matrix-condition`**: Regex to match against matrix test case names\n\n### Query profiles\n\n```yaml\nexpected:\n  profiles:\n    - query: 'process_cpu:cpu:nanoseconds:cpu:nanoseconds{service_name=\"my-service\"}'\n      flamebearers:\n        equals: 'main'\n```\n\n#### Profile assertion options\n\n- **`query`**: Pyroscope query to retrieve the profile (required)\n- **`flamebearers`**: Assertions on the flamebearer response\n  - **`equals`**: String that must appear in the flamebearer names\n  - **`regexp`**: Regular expression pattern to match against the flamebearer names\n- **`matrix-condition`**: Regex to match against matrix test case names\n\n### Custom checks\n\nCustom checks allow you to run arbitrary scripts for advanced validation scenarios.\n\n```yaml\nexpected:\n  custom-checks:\n    - script: |\n        #!/bin/bash\n        # Your custom validation script here\n        exit 0\n```\n\n#### Custom check options\n\n- **`script`**: Script to execute (required)\n- **`matrix-condition`**: Regex to match against matrix test case names\n\n### Matrix of test cases\n\nMatrix tests are useful to test different configurations of the same application,\ne.g. with different settings of the otel collector or different flags in the application.\n\n```yaml\nmatrix:\n  - name: default\n    docker-compose:\n      files:\n        - ./docker-compose.oats.yml\n  - name: self-contained\n    docker-compose:\n      files:\n        - ./docker-compose.self-contained.oats.yml\n  - name: net8\n    docker-compose:\n      files:\n        - ./docker-compose.net8.oats.yml\n```\n\nYou can then make test cases depend on the matrix name:\n\n```yaml\nexpected:\n  metrics:\n    - promql: 'db_client_connections_max{pool_name=\"HikariPool-1\"}'\n      value: \"== 10\"\n      matrix-condition: default\n```\n\n`matrix-condition` is a regex that is applied to the matrix name. This field is available for all assertion types (traces, logs, metrics, profiles).\n\n## Docker Compose\n\nDescribes the docker-compose file(s) to use for the test.\nThe files typically define the instrumented application you want to test and optionally some dependencies,\ne.g. a database server to send requests to.\nYou don't need (and shouldn't have) to define the observability stack (e.g. Prometheus, Grafana, etc.),\nbecause this is provided by the test framework (and may test different versions of the observability stack,\ne.g. OTel Collector and Grafana Alloy).\n\nThis docker-compose file is relative to the `oats.yaml` file.\n\n## Kubernetes\n\nA local Kubernetes cluster can be used to test the application in a Kubernetes environment rather than in docker-compose.\nThis is useful to test the application in a more realistic environment - and when you want to test Kubernetes specific features.\n\nDescribes the Kubernetes manifest(s) to use for the test.\n\n```yaml\nkubernetes:\n  dir: k8s\n  app-service: dice\n  app-docker-file: Dockerfile\n  app-docker-context: ..\n  app-docker-tag: dice:1.1-SNAPSHOT\n  app-docker-port: 8080\n```\n\n[Tempo]: https://github.com/grafana/tempo\n[OpenTelemetry Collector]: https://opentelemetry.io/docs/collector/\n[Prometheus]: https://prometheus.io/\n[Grafana]: https://grafana.com/\n[Loki]: https://github.com/grafana/loki\n[docker-otel-lgtm]: https://github.com/grafana/docker-otel-lgtm/\n[k3d]: https://k3d.io/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrafana%2Foats","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrafana%2Foats","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrafana%2Foats/lists"}