{"id":31776027,"url":"https://github.com/pytest-dev/pytest-snap","last_synced_at":"2025-10-10T05:22:17.161Z","repository":{"id":312937040,"uuid":"1049369266","full_name":"pytest-dev/pytest-snap","owner":"pytest-dev","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-07T00:26:42.000Z","size":513,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-08T00:03:24.497Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pytest-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-02T21:58:34.000Z","updated_at":"2025-09-11T02:05:32.000Z","dependencies_parsed_at":"2025-09-03T20:32:34.271Z","dependency_job_id":null,"html_url":"https://github.com/pytest-dev/pytest-snap","commit_stats":null,"previous_names":["cr5459/pytest-html-baseline","cr5459/pytest-snap","pytest-dev/pytest-snap"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pytest-dev/pytest-snap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytest-dev%2Fpytest-snap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytest-dev%2Fpytest-snap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytest-dev%2Fpytest-snap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytest-dev%2Fpytest-snap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pytest-dev","download_url":"https://codeload.github.com/pytest-dev/pytest-snap/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytest-dev%2Fpytest-snap/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002884,"owners_count":26083468,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-10T05:22:09.689Z","updated_at":"2025-10-10T05:22:17.147Z","avatar_url":"https://github.com/pytest-dev.png","language":"Python","readme":"# pytest-snap\n\nMinimal deterministic snapshot capture of a pytest run: per-test outcome +\nduration (ns) stored in a JSON file. Intended as a small foundation for\noptional future diff / perf / gating features.\n\nCurrent scope:\n* Pytest plugin auto‑loaded (entry point `snap`).\n* `--snap` flag enables capture.\n* `--snap-out PATH` chooses output file (default `.snap/current.json`).\n* CLI wrapper for repeated labeled runs (`pytest-snap run`, `pytest-snap all`).\n\nPyPi: https://pypi.org/project/pytest-snapcheck/#description\n\n## Installation\n\n```bash\npip install pytest-snapcheck\n```\n---\n\n## Quick Start\n\nInstall:\n\n```bash\npip install pytest-snapcheck\n```\n\nRun tests with snapshot capture:\n\n```bash\npytest --snap\n```\n\nResult written to `.snap/current.json` (create the directory if needed). Change\ndestination:\n\n```bash\npytest --snap --snap-out my_run.json\n```\n\nUse the helper CLI for labeled runs (writes `.artifacts/snap_\u003clabel\u003e.json`):\n\n```bash\npytest-snap run v1\npytest-snap run v2\n# or\npytest-snapcheck run v1\n```\n\nGenerate several labels in sequence:\n\n```bash\npytest-snap all               # default labels v1 v2 v3\n```\n\n### Using a custom tests folder\n\nBy default, the CLI runs your repo's `./tests` directory if it exists. To target a different folder, file, or a single test (node id), pass `--tests`:\n\n```bash\n# A specific directory\npytest-snap run v1 --tests ./path/to/tests\n\n# A subfolder of your test tree\npytest-snap run v1 --tests tests/integration\n\n# A single file or a single test node\npytest-snap run v1 --tests tests/test_api.py\npytest-snap run v1 --tests tests/test_api.py::test_happy_path\n\n# Add regular pytest filters (forwarded as-is)\npytest-snap run v1 --tests tests/integration -k \"smoke\" -m \"not flaky\"\n```\n\nPrefer using plain pytest? The plugin doesn't change discovery; just supply paths as usual and add the flags:\n\n```bash\npytest --snap --snap-out .artifacts/snap.json ./path/to/tests\n# If plugin autoload is disabled:\npytest -p pytest_snap.plugin --snap --snap-out .artifacts/snap.json ./path/to/tests\n```\n\n### Artifacts and outputs\n\nWhere results are written by default and how to change it:\n\n- Pure pytest (plugin)\n\t- Default file: `.snap/current.json`\n\t- Override with `--snap-out PATH`.\n\t- Example:\n\t\t```bash\n\t\tpytest --snap --snap-out .artifacts/snap_v1.json tests/\n\t\t```\n\n- CLI (pytest-snap)\n\t- Default directory: `.artifacts`\n\t- Files created per run:\n\t\t- `.artifacts/snap_\u003clabel\u003e.json` (always)\n\t\t- `.artifacts/run_\u003clabel\u003e.html` (only with `--html` and pytest-html installed)\n\t- Change directory with `--artifacts DIR`.\n\t- Examples:\n\t\t```bash\n\t\t# Default outputs\n\t\tpytest-snap run v1\n\n\t\t# Custom output directory\n\t\tpytest-snap run v1 --artifacts out/snapshots\n\n\t\t# Diff reads from the same directory\n\t\tpytest-snap diff v1 v2 --artifacts out/snapshots\n\t\t```\n\n- Housekeeping helpers\n\t```bash\n\tpytest-snap list                # list available snapshots\n\tpytest-snap show v1            # show summary for a snapshot\n\tpytest-snap clean              # remove the artifacts directory\n\t# (all accept --artifacts DIR)\n\t```\n\n---\n\n## Snapshot Schema (v0.1.0)\n\n```json\n{\n\t\"started_ns\": 1234567890,\n\t\"finished_ns\": 1234569999,\n\t\"env\": {\"pytest_version\": \"8.x\"},\n\t\"results\": [\n\t\t{\"nodeid\": \"tests/test_example.py::test_ok\", \"outcome\": \"passed\", \"dur_ns\": 10423}\n\t]\n}\n```\n\n## Future Roadmap (High Level)\nPlanned incremental additions (subject to change):\n1. Baseline diff \u0026 change bucket summarization.\n2. Slower test detection \u0026 perf thresholds.\n3. Budget YAML support and gating.\n4. Historical flake scoring.\n5. Rich diff / timeline CLI views.\n\nEarly adopters should pin minor versions if depending on emerging fields.\n\n### Code-level Diff (`--code`)\n\nIn addition to outcome \u0026 timing changes you can compare the test function source between two labeled versions.\n\nTypical layout:\n```\nproject/\n\tv1/tests/...\n\tv2/tests/...\n```\n\nRun a snapshot diff including code changes:\n```bash\npytest-snap diff v1 v2 --code\n```\n\nWhat happens:\n* Auto-detects version directories `\u003cA\u003e` and `\u003cB\u003e` under the current working directory (or under `--versions-base` if provided).\n* Lists added / removed / modified test functions (`def test_*`).\n* Shows a unified diff (syntax-colored) for modified tests with simple performance hints (range() growth, added sleep time).\n\nOptions:\n* `--code`        Combine snapshot diff + code diff.\n* `--code-only`   Suppress snapshot outcome section; only show code diff.\n* `--versions-base DIR`   Look for version subdirectories under `DIR` instead of `.`.\n\nExamples:\n```bash\n# Just code changes (no outcome buckets)\npytest-snap diff v1 v2 --code-only --code\n\n# Custom versions base path\npytest-snap diff release_old release_new --code --versions-base ./releases\n\n# Code + performance analysis together\npytest-snap diff v1 v2 --code --perf\n```\n\nLimitations:\n* Only inspects top-level `test_*.py` files; helper modules not diffed.\n* Function-level granularity (class-based tests appear as functions with node ids).\n* Large diffs are truncated after 20 modified tests (increase by editing source if needed).\n\n---\n\n### Performance Diff (`--perf`) in the CLI\n\nThe CLI snapshot diff (`pytest-snap diff A B`) ignores timing changes unless you opt in:\n\n```bash\npytest-snap diff v1 v2 --perf\n```\n\nThis adds a \"Slower Tests\" section listing tests whose elapsed time increased beyond BOTH thresholds:\n\n* ratio: new_duration / old_duration \u003e= `--perf-ratio` (default 1.30 ⇒ at least 30% slower)\n* absolute: new_duration - old_duration \u003e= `--perf-abs` (default 0.05s)\n\nOptional flags:\n\n| Flag | Meaning |\n|------|---------|\n| `--perf-ratio 1.5` | Require 50%+ slow-down (instead of 30%) |\n| `--perf-abs 0.02` | Require at least 20ms added latency |\n| `--perf-show-faster` | Also list significantly faster tests |\n\nTo see only timings + code changes (skip outcome buckets):\n```bash\npytest-snap diff v1 v2 --perf --code --code-only\n```\n\n### Performance Gating During Test Runs\n\nInside pytest runs (plugin), slower tests are tracked when you supply a baseline and choose a fail mode:\n\n```bash\npytest --snap-baseline .artifacts/snap_base.json \\\n\t--snap-fail-on slower \\\n\t--snap-slower-threshold-ratio 1.25 \\\n\t--snap-slower-threshold-abs 0.10\n```\n\nBehavior:\n\n* A test is considered slower if it exceeds both the ratio and absolute thresholds.\n* `--snap-fail-on slower` turns any slower test into a non‑zero exit (CI gating).\n* Adjust thresholds to tune sensitivity (raise ratio or abs to reduce noise).\n\nShortcut mental model: ratio filters relative regressions; absolute filters micro‑noise. Both must pass so a 2ms blip on a 1µs test won't alert even if ratio is large.\n\nIf you only care about functional changes, omit perf flags; if you want early perf regression visibility, add them.\n\n---\n\n### Timeline / Historical Progression (`timeline` subcommand)\n\nUse the timeline view to see how snapshots evolved over time and when failures first appeared.\n\nCreate snapshots (labels arbitrary):\n```bash\npytest-snap run v1\npytest-snap run v2\npytest-snap run v3\n```\n\nShow chronological summary:\n```bash\npytest-snap timeline\n```\nSample output:\n```\nTIMELINE (3 snapshots)\n2025-09-04T19:20:21Z v1 commit=8e05100 total=28 fail=0 new_fail=0 fixes=0 regressions=0\n2025-09-04T19:25:07Z v2 commit=8e05100 total=28 fail=1 new_fail=1 fixes=0 regressions=1\n2025-09-04T19:30:44Z v3 commit=8e05100 total=28 fail=1 new_fail=0 fixes=1 regressions=0\n```\n\nFlags:\n| Flag | Purpose |\n|------|---------|\n| `--since \u003ccommit\u003e` | Start listing from first snapshot whose `git_commit` matches (short hash) |\n| `--limit N` | Show only the last N snapshots after filtering |\n| `--json` | Emit machine-readable JSON array |\n| `--artifacts DIR` | Use alternate artifacts directory |\n\nComputed per row (vs previous snapshot):\n* `new_fail`: tests that newly failed.\n* `fixes`: previously failing tests that now pass.\n* `regressions`: passed → failed transitions.\n\nMetadata:\n* Each snapshot is enriched (best effort) with `git_commit` (short HEAD hash) after write.\n* If git metadata isn’t available (outside a repo), the commit shows as `unknown` or `None`.\n\nJSON example:\n```bash\npytest-snap timeline --json | jq .\n```\nProduces entries like:\n```json\n[\n\t{\"label\":\"v1\",\"git_commit\":\"8e05100\",\"total\":28,\"failed\":0,\"passed\":28,\"xfailed\":0,\"xpassed\":0,\"new_fail\":0,\"fixes\":0,\"regressions\":0},\n\t{\"label\":\"v2\",\"git_commit\":\"8e05100\",\"total\":28,\"failed\":1,\"passed\":27,\"xfailed\":0,\"xpassed\":0,\"new_fail\":1,\"fixes\":0,\"regressions\":1}\n]\n```\n\nUse cases:\n* Quickly pinpoint when a regression first appeared before diving into full diff.\n* Send the timeline JSON straight to a small dashboard (Prometheus push, simple web chart) without re-reading all snapshot files.\n* In Continuous Integration (CI) pipelines, fail the run (block the merge) if the timeline shows new failures or regressions. CI = automated test/build system that runs on every change.\n\n### Labels vs paths (what does `v1` mean?)\n\n- `pytest-snap run \u003clabel\u003e`\n\t- The label only names the output file: `.artifacts/snap_\u003clabel\u003e.json`.\n\t- It does not select a folder named `\u003clabel\u003e`; discovery defaults to `./tests` unless you pass `--tests`.\n\t- Examples:\n\t\t```bash\n\t\tpytest-snap run v1                      # runs ./tests, writes .artifacts/snap_v1.json\n\t\tpytest-snap run mylabel --tests tests/api\n\t\tpytest-snap run pr-123  --tests tests/test_api.py::test_happy_path\n\t\t```\n\n- `pytest-snap diff \u003cA\u003e \u003cB\u003e`\n\t- Labels refer to snapshot files in the artifacts directory (default `.artifacts`).\n\t- When you add `--code` (or `--code-only`), directories named `\u003cA\u003e` and `\u003cB\u003e` are looked up under `--versions-base` (default `.`).\n\t- You can control the base with `--versions-base PATH`.\n\n---\n\n## Flaky Detection\n\nWhen history logging is enabled (default in `pytest-snap run`), previous outcomes are tracked. A weighted score measures pass ↔ fail flips. Highly flaky tests can be excluded from \"new failures\" to reduce noise.\n\n---\n\n## Conceptual Model\n1. Enable capture (flag / CLI) → write snapshot.\n2. (Future) Compare snapshots → categorize changes.\n3. (Future) Apply gating policies.\n4. Refresh baseline as intent changes.\n\n---\n\n## FAQ\n**Do I need the CLI?** No; it's convenience sugar for labeled runs.\n\n**Why not a baseline diff yet?** Keeping 0.1.0 deliberately small; diffing lands next.\n\n**Will the schema change?** Potentially (still pre-1.0.0) but additions will prefer backward compatibility.\n\n---\n\n## Glossary\n| Term | Definition |\n|------|------------|\n| Snapshot | JSON record of one full test run |\n| Nodeid | Pytest's canonical test identifier |\n| Duration | Test call-phase elapsed time (ns stored) |\n\n---\n\n## Contributing\n\n1. Fork / clone.  \n2. (Optional) Create venv \u0026 install: `pip install -e .[dev]`.  \n3. Add or adjust tests for your changes.  \n4. Keep documentation clear and concise.  \n5. Open a PR.\n\n---\n\n## License\n\nMIT (see `LICENSE`).\n\n## Compatibility\nRuns on\n* Operating systems: macOS, Linux, and Windows (pure Python, no native extensions).\n* Python versions: 3.9+ (tox/CI test 3.9–3.12; builds also succeed on 3.13).\n* Requirements: pytest\u003e=8.0.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytest-dev%2Fpytest-snap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpytest-dev%2Fpytest-snap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytest-dev%2Fpytest-snap/lists"}