{"id":50339668,"url":"https://github.com/shakhov-dmitrii/pytest-test-observer","last_synced_at":"2026-06-15T07:01:43.834Z","repository":{"id":358713302,"uuid":"1238008133","full_name":"shakhov-dmitrii/pytest-test-observer","owner":"shakhov-dmitrii","description":"A pytest plugin that ships per-test results to ClickHouse for trend analysis, flakiness tracking, and CI observability.","archived":false,"fork":false,"pushed_at":"2026-06-13T10:25:12.000Z","size":574,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-13T12:12:36.745Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shakhov-dmitrii.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-13T18:12:28.000Z","updated_at":"2026-06-13T10:12:28.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/shakhov-dmitrii/pytest-test-observer","commit_stats":null,"previous_names":["shakhov-dmitrii/pytest-test-observer"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/shakhov-dmitrii/pytest-test-observer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shakhov-dmitrii%2Fpytest-test-observer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shakhov-dmitrii%2Fpytest-test-observer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shakhov-dmitrii%2Fpytest-test-observer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shakhov-dmitrii%2Fpytest-test-observer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shakhov-dmitrii","download_url":"https://codeload.github.com/shakhov-dmitrii/pytest-test-observer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shakhov-dmitrii%2Fpytest-test-observer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34351451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-15T02:00:07.085Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-29T16:00:25.046Z","updated_at":"2026-06-15T07:01:43.828Z","avatar_url":"https://github.com/shakhov-dmitrii.png","language":"Python","funding_links":[],"categories":["Tools"],"sub_categories":[],"readme":"# pytest-test-observer\n\n[![PyPI - Version](https://img.shields.io/pypi/v/pytest-test-observer.svg?logo=pypi\u0026logoColor=white)](https://pypi.org/project/pytest-test-observer/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pytest-test-observer.svg?logo=python\u0026logoColor=white)](https://pypi.org/project/pytest-test-observer/)\n[![CI](https://github.com/shakhov-dmitrii/pytest-test-observer/actions/workflows/ci.yml/badge.svg)](https://github.com/shakhov-dmitrii/pytest-test-observer/actions/workflows/ci.yml)\n\nAn easy-to-use pytest plugin to take your test observability to the next level. Compatible with `allure-pytest`.\n\n![alt text](image.png)\n\n## Install\n\n```bash\npip install pytest-test-observer\n# or: uv add pytest-test-observer\n```\n\nFor Allure support, install the `allure` extra:\n\n```bash\npip install \"pytest-test-observer[allure]\"\n# or: uv add \"pytest-test-observer[allure]\"\n```\n\n### From source\n\n```bash\n# add the git version as a dependency\nuv add git+https://github.com/shakhov-dmitrii/pytest-test-observer\n\n# or develop inside a clone\ngit clone https://github.com/shakhov-dmitrii/pytest-test-observer\ncd pytest-test-observer \u0026\u0026 uv sync\n```\n\n## Quick start\n\n1. Start a local ClickHouse **and Grafana**:\n\n   ```bash\n   docker compose up -d\n   ```\n\n   This brings up:\n   - ClickHouse on \u003chttp://localhost:8123\u003e (HTTP) and `:9000` (native)\n   - Grafana on \u003chttp://localhost:3000\u003e (login: `admin` / `admin`) with the ClickHouse datasource auto-provisioned and a starter dashboard pre-loaded.\n\n2. Run your tests with the ClickHouse URL:\n\n   ```bash\n   pytest --ch-url=localhost:8123 --ch-table=pytest_results\n   ```\n\n3. Inspect the rows - either via SQL:\n\n   ```bash\n   docker exec -it pytest-test-observer-clickhouse clickhouse-client \\\n     -q \"SELECT nodeid, status, duration, ci_provider FROM default.pytest_results ORDER BY timestamp DESC LIMIT 20 FORMAT PrettyCompact\"\n   ```\n\n   or in the dashboard: \u003chttp://localhost:3000/d/pytest-test-observer-overview\u003e\n\nIf `--ch-url` is not provided, it does nothing and adds no overhead.\n\n## Configuration\n\nEvery connection setting can be supplied three ways, resolved in this order (highest priority first):\n\n1. **CLI flag** - `pytest --ch-url=...`\n2. **Environment variable** - `PYTEST_OBSERVER_CH_URL=...`\n3. **pyproject.toml** - `ch_url = \"...\"` under `[tool.pytest.ini_options]` (or any other pytest config file)\n4. Built-in default\n\n| CLI flag           | Env var                          | Ini key (`pyproject.toml`) | Default          |\n| ------------------ | -------------------------------- | -------------------------- | ---------------- |\n| `--ch-url`         | `PYTEST_OBSERVER_CH_URL`         | `ch_url`                   | none             |\n| `--ch-user`        | `PYTEST_OBSERVER_CH_USER`        | `ch_user`                  | `default`        |\n| `--ch-password`    | `PYTEST_OBSERVER_CH_PASSWORD`    | `ch_password`              | `\"\"`             |\n| `--ch-db`          | `PYTEST_OBSERVER_CH_DB`          | `ch_db`                    | `default`        |\n| `--ch-table`       | `PYTEST_OBSERVER_CH_TABLE`       | `ch_table`                 | `pytest_results` |\n| `--ch-send-from`   | `PYTEST_OBSERVER_CH_SEND_FROM`   | `ch_send_from`             | `any`            |\n| `--ch-auto-migrate`| `PYTEST_OBSERVER_CH_AUTO_MIGRATE`| `ch_auto_migrate`          | `true`           |\n| `--custom-events`  | `PYTEST_OBSERVER_CUSTOM_EVENTS`  | `custom_events`            | `false`          |\n\n### `--ch-send-from`: where rows come from\n\n- `any` (default) - send for both local and CI runs.\n- `ci` - only send when a provider-specific CI sentinel is set (`GITHUB_ACTIONS`, `GITLAB_CI`, `CIRCLECI`, or `JENKINS_URL`).\n\n### `--ch-auto-migrate`: schema migrations across plugin versions\n\nWhen a new plugin version adds columns, the plugin auto-applies `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` for each missing column. Existing rows get the column's default value. Old data doesn't change.\n\nIf your team's policy forbids the plugin running DDL, set `ch_auto_migrate = false` in `pyproject.toml` (or the env / CLI equivalent). The plugin will then refuse to migrate and surface the SQL you'd need to run yourself.\n\n### Example: defaults into `pyproject.toml`\n\n```toml\n[tool.pytest.ini_options]\nch_url   = \"clickhouse.internal:8123\"\nch_db    = \"ci_metrics\"\nch_table = \"pytest_results\"\n```\n\nThen a plain `pytest` picks them up - no flags, no env vars.\n\n### Example: CI secret via env var\n\n```yaml\n# GitHub Actions\n- run: pytest\n  env:\n    PYTEST_OBSERVER_CH_URL: ${{ secrets.CLICKHOUSE_URL }}\n    PYTEST_OBSERVER_CH_PASSWORD: ${{ secrets.CLICKHOUSE_PASSWORD }}\n```\n\n### Other environment variables\n\n| Variable                    | Effect                                                           |\n| --------------------------- | ---------------------------------------------------------------- |\n| `PYTEST_OBSERVER_RUN_ID`    | Override the auto-generated `run_id` (UUID) for the session      |\n| `XDG_CACHE_HOME`            | Base directory for the disk-buffer fallback                      |\n\n## ClickHouse schema\n\nThe table is auto-created on first flush:\n\n```sql\nCREATE TABLE IF NOT EXISTS pytest_results (\n    run_id             String,\n    timestamp          DateTime64(3),\n    started_at         UInt64,\n    finished_at        UInt64,\n    nodeid             String,\n    status             LowCardinality(String),\n    when_phase         LowCardinality(String),\n    duration           Float64,\n    markers            Array(String),\n    worker_id          LowCardinality(String),\n    ci_provider        LowCardinality(String),\n    ci_run_id          String,\n    git_commit         String,\n    git_branch         String,\n    allure_id          String,\n    allure_title       String,\n    allure_severity    LowCardinality(String),\n    allure_labels      Map(String, Array(String)),\n    allure_links       Array(Tuple(String, String, String))\n) ENGINE = MergeTree\nORDER BY (nodeid, timestamp)\nPARTITION BY toYYYYMM(timestamp);\n```\n\n## Allure compatibility\n\nWhen `allure-pytest` is installed and tests use the standard Allure decorators, the plugin captures:\n\n- Labels (`@allure.feature`, `@allure.story`, `@allure.tag`, `@allure.severity`, `@allure.id`, `@allure.epic`, `@allure.suite`, ...) -\u003e `allure_labels`\n- Links (`@allure.link`, `@allure.issue`, `@allure.testcase`) -\u003e `allure_links`\n- `@allure.title(...)` -\u003e `allure_title`\n- `@allure.severity(...)` -\u003e `allure_severity` (also stored in `allure_labels`)\n- `@allure.id(...)` -\u003e `allure_id`\n\n## Custom events\n\n**Custom events are off by default.** Enable them with `--custom-events=true` (or the env / ini equivalent):\n\n```bash\npytest --ch-url=localhost:8123 --custom-events=true\n```\n\n```python\ndef test_order(record_event):\n    record_event(\"inventory\", {\"sku\": \"WIDGET-A\", \"ok\": \"true\"})\n    record_event(\"payment\", {\"amount\": \"29.99\", \"ok\": \"false\"})\n    ...\n```\n\n- Payload values must be `str` (cast numbers/bools yourself, e.g. `str(latency_ms)`). A non-string value raises `TypeError`. Storing numeric metrics as strings lets Grafana chart them without schema changes.\n- Events are captured for both passing and failing tests, in call order.\n- The fixture is always defined, so tests using it stay runnable even when the plugin is inactive (no `--ch-url`) or custom events are disabled.\n\nThe events table is auto-created on first flush:\n\n```sql\nCREATE TABLE IF NOT EXISTS pytest_results_events (\n    run_id      String,\n    nodeid      String,\n    timestamp   DateTime64(3),\n    seq         UInt32,\n    event_name  LowCardinality(String),\n    payload     Map(String, String)\n) ENGINE = MergeTree\nORDER BY (nodeid, timestamp, seq)\nPARTITION BY toYYYYMM(timestamp);\n```\n\n## CI / git context detection\n\nDetected automatically.\n\n| Provider       | Env vars used                                                                        |\n| -------------- | -------------------------------------------------------------------------------------|\n| GitHub Actions | `GITHUB_ACTIONS`, `GITHUB_RUN_ID`, `GITHUB_SHA`, `GITHUB_HEAD_REF`/`GITHUB_REF_NAME` |\n| GitLab CI      | `GITLAB_CI`, `CI_PIPELINE_ID`, `CI_COMMIT_SHA`, `CI_COMMIT_REF_NAME`                 |\n| CircleCI       | `CIRCLECI`, `CIRCLE_BUILD_NUM`, `CIRCLE_SHA1`, `CIRCLE_BRANCH`                       |\n| Jenkins        | `JENKINS_URL`, `BUILD_ID`/`BUILD_NUMBER`, `GIT_COMMIT`, `GIT_BRANCH`                 |\n| Local          | `git rev-parse HEAD` and `git rev-parse --abbrev-ref HEAD` (`ci_provider=\"local\"`)   |\n\n## Disk buffer fallback\n\nWhen ClickHouse is unreachable, slow, or rejects the insert, the batch is written to:\n\n```\n$XDG_CACHE_HOME/pytest-test-observer/\u003crun_id\u003e.jsonl   # or ~/.cache/... if unset\n```\n\nOne JSON object per line, keys identical to the ClickHouse columns. The plugin emits a `warnings.warn` with the path. **The pytest exit code is unaffected.**\n\n### Replaying buffered files back into ClickHouse\n\nOnce ClickHouse is reachable again, run:\n\n```bash\npython -m pytest_test_observer.replay --ch-url=localhost:8123\n# or pick specific files:\npython -m pytest_test_observer.replay /path/to/run-abc.jsonl\n# or just see what would happen:\npython -m pytest_test_observer.replay --dry-run\n```\n\nAll the `PYTEST_OBSERVER_CH_*` env vars from the configuration table are honoured by the replay tool too.\n\n## Example queries\n\n```sql\n-- Top 10 flakiest tests in the last 30 days\nSELECT\n    nodeid,\n    countIf(status IN ('failed','broken')) AS non_passes,\n    count() AS total,\n    non_passes / total AS flakiness\nFROM pytest_results\nWHERE timestamp \u003e now() - INTERVAL 30 DAY\nGROUP BY nodeid\nHAVING total \u003e= 10 AND non_passes \u003e 0\nORDER BY flakiness DESC\nLIMIT 10;\n\n-- Slowest 10 tests (median duration)\nSELECT nodeid, quantileExact(0.5)(duration) AS p50_seconds, count() AS runs\nFROM pytest_results\nWHERE timestamp \u003e now() - INTERVAL 7 DAY AND status = 'passed'\nGROUP BY nodeid\nORDER BY p50_seconds DESC\nLIMIT 10;\n```\n\n## License\n\nThis project is licensed under the MIT License.\nSee the LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshakhov-dmitrii%2Fpytest-test-observer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshakhov-dmitrii%2Fpytest-test-observer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshakhov-dmitrii%2Fpytest-test-observer/lists"}