{"id":45155788,"url":"https://github.com/twsl/pytest-park","last_synced_at":"2026-04-01T18:58:59.981Z","repository":{"id":339505475,"uuid":"1157168948","full_name":"twsl/pytest-park","owner":"twsl","description":"Organise and analyse your pytest benchmarks","archived":false,"fork":false,"pushed_at":"2026-03-24T02:55:18.000Z","size":515,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-24T04:55:01.540Z","etag":null,"topics":["benchmark","compare","pytest","pytest-benchmark"],"latest_commit_sha":null,"homepage":"https://twsl.github.io/pytest-park/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/twsl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-13T14:10:10.000Z","updated_at":"2026-03-23T07:38:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/twsl/pytest-park","commit_stats":null,"previous_names":["twsl/pytest-park"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/twsl/pytest-park","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twsl%2Fpytest-park","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twsl%2Fpytest-park/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twsl%2Fpytest-park/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twsl%2Fpytest-park/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/twsl","download_url":"https://codeload.github.com/twsl/pytest-park/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twsl%2Fpytest-park/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31290992,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","compare","pytest","pytest-benchmark"],"created_at":"2026-02-20T04:05:07.572Z","updated_at":"2026-04-01T18:58:59.972Z","avatar_url":"https://github.com/twsl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pytest-park\n\n[![Build](https://github.com/twsl/pytest-park/actions/workflows/build.yaml/badge.svg)](https://github.com/twsl/pytest-park/actions/workflows/build.yaml)\n[![Documentation](https://github.com/twsl/pytest-park/actions/workflows/docs.yaml/badge.svg)](https://github.com/twsl/pytest-park/actions/workflows/docs.yaml)\n[![PyPI - Package Version](https://img.shields.io/pypi/v/pytest-park?logo=pypi\u0026style=flat\u0026color=orange)](https://pypi.org/project/pytest-park/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pytest-park?logo=pypi\u0026style=flat\u0026color=blue)](https://pypi.org/project/pytest-park/)\n[![Docs with MkDocs](https://img.shields.io/badge/MkDocs-docs?style=flat\u0026logo=materialformkdocs\u0026logoColor=white\u0026color=%23526CFE)](https://squidfunk.github.io/mkdocs-material/)\n[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)\n[![linting: ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n[![ty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json)](https://github.com/astral-sh/ty)\n[![prek](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/j178/prek/master/docs/assets/badge-v0.json)](https://github.com/j178/prek)\n[![security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg)](https://github.com/PyCQA/bandit)\n[![Semantic Versions](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--versions-e10079.svg)](https://github.com/twsl/pytest-park/releases)\n[![Copier](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-border.json)](https://github.com/copier-org/copier)\n[![License](https://img.shields.io/badge/license-MIT-blue)](LICENSE)\n\nOrganise and analyse your pytest benchmarks\n\n## Features\n\n- Inline benchmark comparison printed directly in `pytest` output — no extra commands needed.\n- Load pytest-benchmark JSON artifact folders and normalize runs, groups, marks, params, and custom grouping metadata.\n- Compare reference runs against candidate runs with per-case and per-group delta and speedup summaries.\n- Flexible grouping: custom keys, benchmark groups, marks, params, and postfix-based name normalization.\n- Associate optional profiler artifacts with benchmark runs for code-level context.\n- Serve an interactive local NiceGUI dashboard for historical exploration.\n\n## Installation\n\nWith `pip`:\n\n```bash\npython -m pip install pytest-park\n```\n\nWith [`uv`](https://docs.astral.sh/uv/):\n\n```bash\nuv add --group test pytest-park\n```\n\n## Usage\n\n### Step 1 — Run your tests\n\n```bash\npytest\n```\n\nAfter the normal pytest-benchmark tables, a `pytest-park` summary section is printed automatically. It compares the current run against the latest saved benchmark artifact found in pytest-benchmark storage. No extra arguments are needed.\n\n\u003e The plugin is registered automatically via the `pytest11` entry point when `pytest-park` is installed — no `conftest.py` changes are required.\n\n### Step 2 — Save runs to build a history (optional)\n\n```bash\n# Save and keep comparing against the latest saved run automatically\npytest --benchmark-autosave\n\n# Save with a meaningful name for a stable reference point\npytest --benchmark-save baseline\n\n# Compare against a specific saved run\npytest --benchmark-compare=0001\npytest --benchmark-compare=8d530304\n\n# Save a candidate and compare it against a specific baseline\npytest --benchmark-save candidate-v2 --benchmark-compare=0001\n```\n\n`pytest-park` reuses the baseline that `pytest-benchmark` resolves from its configured storage — it does not require a second format. `--benchmark-storage` is respected as usual.\n\n\u003e **VS Code Test Explorer**: if the run looks like a single-shot execution (benchmark timing disabled or reduced), `pytest-park` prints a warning so the output is not mistaken for a real comparison.\n\n### Name normalization and grouping (optional)\n\nIf your benchmark names encode variant postfixes (e.g. `test_func_orig`, `test_func_ref`, `test_func_np`, `test_func_pt`), add the `pytest_benchmark_group_stats` hook to group and label variants together:\n\n```python\n# tests/conftest.py\nfrom pytest_park.pytest_benchmark import default_pytest_benchmark_group_stats\n\n\ndef pytest_benchmark_group_stats(config, benchmarks, group_by):\n    return default_pytest_benchmark_group_stats(\n        config,\n        benchmarks,\n        group_by,\n        original_postfix=\"_orig\",      # or a list: [\"_np\", \"_numpy\"]\n        reference_postfix=\"_ref\",       # or a list: [\"_pt\", \"_torch\"]\n        group_values_by_postfix={\n            \"orig\": \"original\",         # leading underscores are stripped for matching\n            \"ref\": \"reference\",\n        },\n    )\n```\n\nThis stores parsed parts in `extra_info[\"pytest_park_name_parts\"]` (`base_name`, `parameters`, `postfix`) and groups paired variants under the same row in the comparison table.\n\nMultiple postfixes can be specified as a list or comma-separated string. Postfix matching is underscore-agnostic: `\"_original\"`, `\"original\"`, and `\"__original\"` all match the same postfix.\n\n### CLI postfix options\n\n`pytest-park` registers `--benchmark-original-postfix` and `--benchmark-reference-postfix` automatically. These accept comma-separated values and **override** any postfixes passed directly to `default_pytest_benchmark_group_stats`:\n\n```bash\n# Single postfix\npytest --benchmark-original-postfix=\"_original\" --benchmark-reference-postfix=\"_new\"\n\n# Multiple postfixes (comma-separated)\npytest --benchmark-original-postfix=\"_np,_numpy\" --benchmark-reference-postfix=\"_pt,_torch\"\n```\n\nWhen postfixes are configured, three output sections are produced:\n\n1. **Regression table** — flat per-method comparison of the current run vs the previous saved run (requires a reference benchmark file).\n2. **Postfix comparison table** — compares original-postfix methods vs reference-postfix methods within the current run (no saved reference needed).\n3. **Grouped comparison table** — the existing detailed comparison with grouping.\n\nDebug information (file names, postfixes, options) is always printed in the `pytest-park` section.\n\nPostfixes can also be set persistently in `pyproject.toml`, `pytest.ini`, or `setup.cfg` so you don't have to pass them on every run:\n\n```toml\n# pyproject.toml\n[tool.pytest.ini_options]\nbenchmark_original_postfix = \"_orig,_numpy\"\nbenchmark_reference_postfix = \"_ref,_torch\"\n```\n\nCLI flags always override ini-file values.\n\n### Custom grouping metadata (optional)\n\nStore arbitrary metadata on a benchmark for richer grouping:\n\n```python\ndef test_compute_optimized(benchmark):\n    benchmark.extra_info[\"custom_groups\"] = {\n        \"technique\": \"vectorization\",\n        \"scenario\": \"large-batch\",\n    }\n    benchmark(compute)\n```\n\nGroup by any key with `--group-by custom:technique` in the CLI.\n\n---\n\n## CLI — deeper analysis across saved artifacts\n\nUse the CLI when you want to compare specific saved runs, apply advanced grouping, or include profiler data.\n\n```bash\n# Compare latest run (candidate) against second-latest run (reference)\npytest-park analyze ./.benchmarks\n\n# Compare named runs\npytest-park analyze ./.benchmarks --reference baseline --candidate candidate-v2\n\n# When only --candidate is given, the preceding run is used as reference\npytest-park analyze ./.benchmarks --candidate candidate-v2\n\n# Group by benchmark group and a specific parameter\npytest-park analyze ./.benchmarks --group-by group --group-by param:device\n\n# Group by custom metadata key\npytest-park analyze ./.benchmarks --group-by custom:scenario\n\n# Exclude a parameter from comparison\npytest-park analyze ./.benchmarks --exclude-param device\n\n# Keep a parameter as a separate dimension\npytest-park analyze ./.benchmarks --group-by group --distinct-param device\n\n# Normalize method names by stripping postfixes\npytest-park analyze ./.benchmarks --original-postfix _orig --reference-postfix _ref\n\n# Include profiler artifacts\npytest-park analyze ./.benchmarks --profiler-folder ./.profiler --group-by group\n\n# Print installed version\npytest-park version\n```\n\n### Grouping reference\n\nDefault precedence (when no `--group-by` is given): `custom \u003e benchmark_group \u003e marks \u003e params`\n\n| Token          | Alias(es)         | Resolves to                            |\n| -------------- | ----------------- | -------------------------------------- |\n| `custom:\u003ckey\u003e` | —                 | `extra_info[\"custom_groups\"][\"\u003ckey\u003e\"]` |\n| `custom`       | `custom_group`    | All custom group keys combined         |\n| `group`        | `benchmark_group` | Benchmark group label                  |\n| `marks`        | `mark`            | Comma-joined pytest marks              |\n| `params`       | —                 | All parameter key=value pairs          |\n| `param:\u003cname\u003e` | —                 | Value of a specific parameter          |\n| `name`         | `method`          | Normalized method name                 |\n| `fullname`     | `nodeid`          | Full test node path                    |\n\nMultiple `--group-by` tokens can be combined; the resulting label is joined with `|`.\n\n### Artifact folder expectations\n\n- Input files are pytest-benchmark JSON files (`--benchmark-save` output) stored anywhere under the folder.\n- Default comparison: latest run as candidate, second-latest as reference.\n- When only `--candidate` is given, the run immediately preceding it is used as reference.\n- Run identity uses `metadata.run_id`, `metadata.tag`, or fallback datetime identifiers.\n\n---\n\n## Interactive dashboard\n\nFor exploratory, visual analysis across many saved runs:\n\n```bash\npytest-park serve ./.benchmarks --reference baseline --host 127.0.0.1 --port 8080\n\n# With profiler data\npytest-park serve ./.benchmarks --profiler-folder ./.profiler --port 8080\n```\n\nAccess the dashboard at `http://127.0.0.1:8080`. Features include run selection, history charts, delta distribution, and method-level drill-down.\n\nTo launch a guided interactive CLI session instead:\n\n```bash\npytest-park\n```\n\n---\n\n## Docs\n\n```bash\nuv run mkdocs build -f ./mkdocs.yml -d ./_build/\n```\n\n## Update template\n\n```bash\ncopier update --trust -A --vcs-ref=HEAD\n```\n\n## Credits\n\nThis project was generated with [![🚀 python project template.](https://img.shields.io/badge/python--project--template-%F0%9F%9A%80-brightgreen)](https://github.com/twsl/python-project-template)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwsl%2Fpytest-park","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftwsl%2Fpytest-park","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwsl%2Fpytest-park/lists"}