{"id":50916393,"url":"https://github.com/quinnjr/blackbox-bench","last_synced_at":"2026-06-16T15:32:14.953Z","repository":{"id":338263283,"uuid":"1157235537","full_name":"quinnjr/blackbox-bench","owner":"quinnjr","description":"A lightweight, zero-dependency Python benchmarking library with decorator and context manager APIs, CLI, and run comparison","archived":false,"fork":false,"pushed_at":"2026-05-20T16:44:57.000Z","size":242,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"develop","last_synced_at":"2026-05-20T21:55:27.020Z","etag":null,"topics":["benchmark","benchmarking","cli","developer-tools","optimization","performance","profiling","python","testing","timing"],"latest_commit_sha":null,"homepage":"https://quinnjr.github.io/blackbox-bench/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/quinnjr.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-13T15:35:35.000Z","updated_at":"2026-05-20T17:14:10.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/quinnjr/blackbox-bench","commit_stats":null,"previous_names":["quinnjr/pybench"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/quinnjr/blackbox-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quinnjr%2Fblackbox-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quinnjr%2Fblackbox-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quinnjr%2Fblackbox-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quinnjr%2Fblackbox-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/quinnjr","download_url":"https://codeload.github.com/quinnjr/blackbox-bench/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quinnjr%2Fblackbox-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34412788,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","benchmarking","cli","developer-tools","optimization","performance","profiling","python","testing","timing"],"created_at":"2026-06-16T15:32:14.766Z","updated_at":"2026-06-16T15:32:14.946Z","avatar_url":"https://github.com/quinnjr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# blackbox-bench\n\nA lightweight Python microbenchmarking library with a Rust core (PyO3 + maturin). Designed as a \"criterion for Python\" — minimal harness overhead, statistically rigorous output, and a small CLI that drops into CI.\n\n```python\nimport blackbox_bench\n\nbench = blackbox_bench.Bench()\n\n@bench.benchmark\ndef hash_1kb():\n    import hashlib\n    hashlib.sha256(b\"x\" * 1024).digest()\n\nbench.run()\nbench.report()                # human-readable table\nbench.report(format=\"json\", path=\"results.json\")\n```\n\n```\nblackbox-bench results\n─────────────────────────────────────────────────────────────────────────────────\nName        Mean   Median  StdDev    Min    Max  Ops/sec          CI 95%  Outliers\n─────────────────────────────────────────────────────────────────────────────────\nhash_1kb  1.4 µs   1.4 µs  12.0 ns 1.4 µs 1.5 µs  710,221  [1.4 µs, 1.4 µs]        3\n─────────────────────────────────────────────────────────────────────────────────\n```\n\n## Features\n\n- **Per-batch timing with auto-batch-sizing** — sub-microsecond functions are batched until each sample takes ≥5 µs, well above the timer resolution floor.\n- **Per-iteration overhead measurement and subtraction** — the harness probes itself at startup so reported nanoseconds attribute time to your code, not blackbox-bench.\n- **Bootstrap 95% confidence intervals** for the mean, plus Tukey or MAD outlier detection.\n- **`blackbox_bench.black_box(value)`** — opaque pass-through that the optimiser can't see through.\n- **`bench.iter_batched(setup, routine)`** — setup runs once per sample (untimed); routine runs the timed batch.\n- **`throughput=`** — `@bench.benchmark(throughput=1024)` reports MB/s alongside ops/sec.\n- **`params=[...]`** — parameterised benchmarks; one result per parameter.\n- **Opt-in HDR histograms** — `Bench(histogram=True)` attaches a percentile-queryable `HdrHistogram` to each result.\n- **`blackbox_bench.compare(baseline_json, current_json)`** — classifies each row as `unchanged` / `regressed` / `improved` / `new` / `removed` using CI overlap, not raw `change_pct`.\n- **Four reporters** — table, JSON, self-contained HTML (with inline SVG sparklines), JUnit-compatible XML (with a raw alternative).\n- **`blackbox-bench run --profile`** — wraps each benchmark in [`py-spy`](https://github.com/benfred/py-spy) and emits SVG flamegraphs alongside the results.\n\n## Install\n\n```bash\npip install blackbox-bench\n```\n\nPre-built abi3 wheels are published for cpython 3.10 / 3.11 / 3.12 / 3.13 on linux (x86_64 + aarch64), macOS (x86_64 + aarch64), and windows x86_64. Building from source requires a Rust toolchain.\n\nOptional extras:\n\n```bash\npip install blackbox-bench[profile]   # adds py-spy for `blackbox-bench run --profile`\n```\n\n## Usage\n\n### Decorator API\n\n```python\nimport blackbox_bench\n\nbench = blackbox_bench.Bench(\n    warmup=5,\n    target_time_ns=1_000_000_000,\n    outlier_method=\"tukey\",        # or \"mad\" / \"none\"\n    overhead_subtract=True,\n    histogram=False,\n)\n\n@bench.benchmark\ndef quick(): sum(range(100))\n\n@bench.benchmark(name=\"hash_kb\", throughput=1024, params=[10, 100, 1000])\ndef hashing(n):\n    import hashlib\n    hashlib.sha256(b\"x\" * n).digest()\n\nresults = bench.run()\nbench.report(format=\"html\", path=\"results.html\")\n```\n\n### `iter_batched` for setup-isolated timing\n\n```python\nimport random\n\n@bench.benchmark\ndef sort_random():\n    return bench.iter_batched(\n        setup=lambda: random.sample(range(1_000), 1_000),\n        routine=lambda xs: sorted(xs),\n    )\n```\n\n`setup` runs once per *sample* and isn't timed; `routine` is the timed call inside the batch.\n\n### Context manager\n\n```python\nwith bench.measure(\"payload_build\"):\n    payload = build_huge_payload()\n\nbench.run()  # appends measured contexts to results\n```\n\n### Module-level decorator + CLI\n\n```python\n# bench_hashing.py\nimport blackbox_bench\n\n@blackbox_bench.benchmark\ndef sha256_1kb():\n    import hashlib\n    hashlib.sha256(b\"x\" * 1024).digest()\n```\n\n```bash\nblackbox-bench run bench_hashing.py --warmup 5 --iterations 100\nblackbox-bench run benches/ --format html --output report.html\nblackbox-bench run benches/ --save baseline.json\nblackbox-bench compare baseline.json current.json   # CI-classified diff\n```\n\n### Comparing runs\n\n```python\nimport json\nreport = blackbox_bench.compare(\n    open(\"baseline.json\").read(),\n    open(\"current.json\").read(),\n)\nfor row in report.rows:\n    print(row.name, row.classification, row.change_pct)\n```\n\n`ComparisonReport.format(\"xml\")` emits JUnit with `\u003cfailure\u003e` on regressed rows — drop the file into Jenkins/GitHub Actions test reporters.\n\n## Why Rust under the hood\n\nThe harness has to stay small relative to the user's function:\n\n- **Tight sampling loop** — `Instant::now()` and the per-batch call dispatch are raw FFI (`PyObject_CallNoArgs` / `PyObject_CallObject`), skipping PyO3's higher-level wrappers inside the timed window.\n- **GIL released during stats** — bootstrap CI, Tukey/MAD, and the histogram run inside `py.detach(...)` so other Python threads aren't blocked while blackbox-bench crunches its samples.\n- **Reused scratch buffers** — `median`, `tukey`, `mad`, and `bootstrap_ci_mean` share two `Vec`s owned by the `Runner`; a 100-benchmark suite still allocates only once for the lot.\n\nThe criterion benchmarks at `benches/rust_internals.rs` measure these primitives directly. `benches/bench_dogfood.py` measures the assembled harness end-to-end (the empty-pass benchmark should report ~0–1 ns after overhead subtraction).\n\n## Migrating from 0.1.0\n\nSee [MIGRATION.md](MIGRATION.md). Most v0.1.0 code only needs an import swap; the one surface that changed without alias is `Bench.report(json_output=True)` → `Bench.report(format=\"json\")`. For exact v0.1.0 semantics:\n\n```python\nfrom blackbox_bench.legacy import Bench, benchmark, BenchmarkResult\n```\n\nThe legacy shim is removed in v1.1.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquinnjr%2Fblackbox-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquinnjr%2Fblackbox-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquinnjr%2Fblackbox-bench/lists"}