https://github.com/ryancswallace/benchmatrix
Build pytest-benchmark matrices and parse benchmark results with lightweight Python utilities.
https://github.com/ryancswallace/benchmatrix
benchmark benchmarking performance pytest pytest-benchmark python
Last synced: 25 days ago
JSON representation
Build pytest-benchmark matrices and parse benchmark results with lightweight Python utilities.
- Host: GitHub
- URL: https://github.com/ryancswallace/benchmatrix
- Owner: ryancswallace
- License: mit
- Created: 2026-06-19T11:53:57.000Z (30 days ago)
- Default Branch: main
- Last Pushed: 2026-06-22T21:38:27.000Z (26 days ago)
- Last Synced: 2026-06-22T22:06:44.664Z (26 days ago)
- Topics: benchmark, benchmarking, performance, pytest, pytest-benchmark, python
- Language: Python
- Homepage: https://ryancswallace.github.io/benchmatrix/
- Size: 217 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Citation: CITATION.cff
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
- Agents: AGENTS.md
Awesome Lists containing this project
README
# benchmatrix
[](https://github.com/ryancswallace/benchmatrix/actions/workflows/ci.yml)
[](https://github.com/ryancswallace/benchmatrix/actions/workflows/docs.yml)
[](https://github.com/ryancswallace/benchmatrix/actions/workflows/docker.yml)
[](https://github.com/ryancswallace/benchmatrix/actions/workflows/codeql.yml)
[](https://github.com/ryancswallace/benchmatrix/actions/workflows/scorecard.yml)
[](https://github.com/ryancswallace/benchmatrix/actions/workflows/workflow-lint.yml)
[](https://github.com/ryancswallace/benchmatrix/blob/main/pyproject.toml)
[](https://github.com/DetachHead/basedpyright)
[](https://docs.astral.sh/ruff/)
[](https://github.com/ryancswallace/benchmatrix/blob/main/pyproject.toml)
[](https://cyclonedx.org/)
**Benchmark matrices for Python projects that need performance data they can
trust, compare, and parse.**
benchmatrix sits on top of
[pytest-benchmark](https://pytest-benchmark.readthedocs.io/) and adds the layer
that benchmark suites usually grow by hand: implementation-by-case matrices,
strict JSON-safe metadata, metric-aware result parsing, and concise display of
saved benchmark runs.
| Build repeatable suites | Keep metrics honest | Parse saved runs |
| --- | --- | --- |
| Generate pytest benchmark tests across implementations, cases, and metric views. | Separate latency, throughput, and local distribution comparisons instead of mixing unlike numbers. | Load benchmatrix-tagged pytest-benchmark JSON rows into structured Python objects. |
## Quick Start
Create a benchmark matrix from ordinary synchronous callables:
```python
from benchmatrix import BenchmarkCase, make_benchmark_test
implementations = {
"builtin": sum,
"loop": lambda vs: sum(v for v in vs),
}
cases = [
BenchmarkCase.from_values(
"small",
list(range(100)),
work_units=100,
work_unit_name="items",
),
]
test_sum_matrix = make_benchmark_test(implementations, cases)
```
Run it with pytest-benchmark and keep the machine-readable output:
```bash
uv run pytest tests/test_sum_benchmark.py --benchmark-json benchmark.json
```
Read the tagged rows back later:
```python
from benchmatrix import display_benchmark_rows, load_benchmark_json
rows = load_benchmark_json("benchmark.json")
display_benchmark_rows(rows)
```
## Why It Exists
pytest-benchmark owns timing, calibration, statistics, terminal reporting, and
JSON export. benchmatrix owns the repeatable structure around those timings.
| Need | benchmatrix gives you |
| --- | --- |
| Compare multiple implementations | One generated pytest benchmark matrix instead of repeated parametrization code. |
| Track what each timing means | JSON-safe invocation metadata with implementation, case, and metric identity. |
| Report different metric views | Single-call latency, logical-work throughput, and local tail-latency summaries. |
| Reuse benchmark output | Parsers and display helpers for pytest-benchmark JSON produced by benchmatrix tests. |
benchmatrix is intentionally narrow: it benchmarks synchronous Python callables.
It is not a load-testing framework, production latency monitor, or replacement
for pytest-benchmark.
## Install
For local development from this repository:
```bash
make ready
```
To install the latest release from PyPI:
```bash
uv add benchmatrix
```
or
```bash
pip install benchmatrix
```
## Documentation
The documentation source lives under [`docs/`](docs/). The top-level Markdown
files are short project entry points; detailed guides, explanations, references,
and runbooks live in the MkDocs documentation.
| Start here | Use it for |
| --- | --- |
| [First benchmark](docs/tutorials/first-benchmark.md) | A complete first benchmark from test file to parsed JSON. |
| [Create a benchmark matrix](docs/how-to/create-benchmark-matrix.md) | Cases, work units, fresh inputs, and synchronous target wrappers. |
| [Parse benchmark results](docs/how-to/parse-results.md) | Loading and displaying benchmatrix-tagged pytest-benchmark JSON. |
| [Performance model](docs/explanation/performance.md) | What the metrics mean and what they do not prove. |
| [Development](docs/development.md) | Local setup, test commands, and repository layout. |
| [Compatibility](docs/reference/compatibility.md) | Supported Python versions, API stability, and support policy. |
| [Publishing](docs/explanation/publishing.md) | Release artifacts, draft releases, PyPI publishing, and verification. |
| [Configuration and automation](docs/reference/configuration.md) | Make targets, CI workflows, Docker checks, docs, and SBOM generation. |
The MkDocs site builds in strict mode and generates API reference pages from the
package docstrings.
## Project Links
* [Examples](examples/)
* [Contributing](CONTRIBUTING.md)
* [Changelog](CHANGELOG.md)
* [Security policy](SECURITY.md)
* [Release policy](RELEASING.md)
* [Code of conduct](CODE_OF_CONDUCT.md)
* [Citation metadata](CITATION.cff)
## License
benchmatrix is distributed under the [MIT License](LICENSE).