{"id":50926300,"url":"https://github.com/hypercoil/nitrix-perf-bench","last_synced_at":"2026-06-16T23:30:57.716Z","repository":{"id":359905229,"uuid":"1239991077","full_name":"hypercoil/nitrix-perf-bench","owner":"hypercoil","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-06T08:17:00.000Z","size":931,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-06T10:09:38.163Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hypercoil.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-15T16:49:15.000Z","updated_at":"2026-06-06T08:17:12.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hypercoil/nitrix-perf-bench","commit_stats":null,"previous_names":["hypercoil/nitrix-perf-bench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hypercoil/nitrix-perf-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hypercoil%2Fnitrix-perf-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hypercoil%2Fnitrix-perf-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hypercoil%2Fnitrix-perf-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hypercoil%2Fnitrix-perf-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hypercoil","download_url":"https://codeload.github.com/hypercoil/nitrix-perf-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hypercoil%2Fnitrix-perf-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34428196,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-16T23:30:53.821Z","updated_at":"2026-06-16T23:30:57.711Z","avatar_url":"https://github.com/hypercoil.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nitrix-perf-bench\n\nPerformance benchmark suite for [`nitrix`](../nitrix) — rigorous, fair,\nmulti-outcome, multi-platform, with structured results as the source of truth.\n\n- **Architecture:** [`DESIGN.md`](DESIGN.md)\n- **Row schema + worker lifecycle (implementation contract):**\n  [`SCHEMA_AND_LIFECYCLE.md`](SCHEMA_AND_LIFECYCLE.md)\n- **Optimising a nitrix op against the suite (feedback-loop playbook for the\n  nitrix agent):** [`OPTIMIZATION_LOOP.md`](OPTIMIZATION_LOOP.md)\n\n## Status: P3 done (gate · bundles · HTML site)\n\nThe L4 result schema is **frozen at `schema_version = 1`** (additive-only;\n`SCHEMA_AND_LIFECYCLE.md`). The first **real** nitrix case ships —\n`semiring_matmul` — comparing the JAX reference, the Pallas/Triton kernel, and\na naive materialise-then-reduce baseline against an fp64 oracle, across the\n`real` / `log` / `tropical` / `euclidean` algebras. Its rendered report\nsupersedes the hand-built `nitrix/bench/PERF_SEMIRING_MATMUL.md`. (The\nthrowaway `dense_matmul` case from P0a remains, for core smoke tests.)\n\n**P1 (in progress):** the runner spawns **one subprocess per attempt** via a\npluggable interpreter — making per-attempt `peak_hbm` and cold `compile_time`\nhonest. A **resource-aware scheduler** (`schedule.py`) serialises GPU attempts\nunder a per-device **lock** (clean timings + clock stability) while **parallel\nCPU** attempts run on **disjoint pinned cores** (`--cpu-slots N`, honest because\nslots don't contend). **Multi-platform** works end to end: `--platforms a,b`\nfans attempts across platforms in one run (CPU + a GPU overlap), and\n`--render-from f1 f2 …` combines separate runs/devices into one `platform`-column\nreport with within-platform ratios. **Registries** are in: a metric registry\n(units / direction / kind + the fidelity gate threshold, the single source of\ntruth the driver stamps and cases are validated against) and a **provider**\nregistry (the cross-case framework + env-isolation a baseline runs under — the\n\"baseline registry\", realised on providers because baseline *names* are\ncase-local and would collide). **Multi-GPU fan-out** is in: `--gpus N` (default\nauto-probed) gives each device its own lock, so attempts fan across GPUs (one\nper device at a time, N concurrent), each pinned via `CUDA_VISIBLE_DEVICES`. A\n**durable store** (`--store`; `results/store/\u003ccase\u003e/\u003crun_id\u003e.jsonl`) accumulates\neach run as a file, `--render-from \u003cdir\u003e --latest` combines runs/devices into\nthe current-state report, and `--prune-keep N` caps history. **P1 is complete**\nmodulo the cross-machine store transport policy (DESIGN §8).\n\n**P2 (in progress) — cross-framework refs.** A `torch-dense` baseline (the same\nmaterialise-then-reduce a torch practitioner writes for a non-real semiring\nmatmul) now runs as a `torch` **provider**: a separate, uv-isolated interpreter\n(torch CPU wheels are on the PyTorch index, so it is its own *interpreter*, not\na second package manager). Build it reproducibly with\n`tools/setup_refs_env.sh`, then point the runner at it with\n`NPERF_PYTHON_TORCH`; the worker interpreter is now resolved per **(framework,\nplatform)**, so a torch attempt picks its own env even on a jax platform. With\nno refs env configured, `torch-dense` records a clean `env_failed` row and the\njax baselines still run. The committed report combines the A10G GPU run with a\nCPU cross-framework run.\n\nA **PyG** baseline lands on a second case, `ell_edge_aggregate`, where it is the\n*natural* reference: nitrix's `semiring_ell_edge_aggregate` is message passing\n(gather ELL neighbours → per-edge `edge_fn` → semiring reduce) — exactly PyG's\n`message`/`aggregate`. A torch `MessagePassing` baseline (GCN-style linear\n`edge_fn`, so JAX / torch / the fp64 oracle compute identical math) competes\nagainst `nitrix-jax` for sum- and max-aggregation (combined CPU+A10G report\n`reports/PERF_ELL_EDGE_AGGREGATE.md`). The finding shows why measuring on the\ntarget matters: PyG is ~2–5× faster on **CPU**, but on the **A10G** the gap\nvanishes (nitrix within ~5% for sum, ~15% faster for max) — XLA fuses the\ngather+vmap+reduce well on Ampere. Modern PyG message-passes on torch-native\n`scatter_reduce`, so it installs pure-Python via uv (the same refs env) —\n**not** the pixi escape hatch; pixi stays reserved for genuinely conda-only\ncompiled extensions. The GPU run uses a CUDA refs env\n(`NPERF_REFS_VARIANT=cuda tools/setup_refs_env.sh`; jax[cuda12] + cuda torch\ncoexist, and torch's HBM is read from torch's own allocator).\n\nThe **op_matrix feed** (`tools/op_matrix_feed.py`) reads the accumulated rows\nand emits the `perf_{cpu,gpu}_{baseline,ratio}` fields nitrix's\n`docs/op_matrix.json` wants (ratio = nitrix.min / reference.min at the\nrepresentative point; `\u003c1` = nitrix faster) — never mutating nitrix (`--apply`\nwrites a merged copy for review).\n\nPublished reports live in [`reports/`](reports/) (the rendered markdown **and**\nthe L4 rows it was generated from, so the report is reproducible from committed\ndata). Scratch runs go to `results/` (git-ignored).\n\n## Run\n\nThe default runner spawns **one subprocess per attempt** (P1) so per-attempt\nmemory and cold-compile are honest. `--in-process` keeps the faster P0 driver\n(memory metrics become process high-water marks — the report says so).\n\n```bash\n# CPU smoke (subprocess workers reuse this uv interpreter). The Pallas baseline\n# records a `skipped` row off-GPU; nitrix-jax and naive-dense run; torch-dense\n# records `env_failed` until the refs env below exists.\nJAX_PLATFORMS=cpu uv run nperf --quick\n\n# Cross-framework (P2): build the torch refs env once (off the root overlay --\n# torch is ~1 GB), then point the runner at it. torch-dense now runs.\ntools/setup_refs_env.sh                                  # -\u003e $NPERF_REFS_ENV_DIR\nNPERF_PYTHON_TORCH=\"${NPERF_REFS_ENV_DIR:-/output/nperf-refs-env}/bin/python\" \\\n  JAX_PLATFORMS=cpu uv run nperf --quick\n\n# Full sweep targeting a CUDA host. The orchestrator coordinates on CPU and\n# spawns GPU workers via a pluggable interpreter; point it at a jax[cuda] env\n# that can import nitrix:\nNPERF_PYTHON_JAX_CUDA12=/path/to/cuda-env/bin/python \\\n  uv run nperf --platforms jax-cuda12 \\\n  --out reports/semiring_matmul.jsonl --report reports/PERF_SEMIRING_MATMUL.md\n\n# Mixed run: CPU + GPU in one invocation (distinct resources run in parallel).\nNPERF_PYTHON_JAX_CUDA12=/path/to/cuda-env/bin/python \\\n  uv run nperf --platforms jax-cpu,jax-cuda12\n\n# Accumulate runs durably (one file per run), then render current state across\n# every accumulated run/device:\nNPERF_PYTHON_JAX_CUDA12=/path/to/cuda-env/bin/python \\\n  uv run nperf --platforms jax-cuda12 --store      # ingest A10G run\nuv run nperf --platforms jax-cpu --store           # ingest a CPU run\nuv run nperf --render-from results/store/semiring_matmul --latest \\\n  --report reports/combined.md                      # newest per (plat,param,baseline)\n\n# Regression gate (P3): diff a current run against a stored baseline on\n# steady_time min (tight) + p95 (loose); exits nonzero if either trips -\u003e CI.\nuv run nperf --gate-baseline reports/semiring_matmul.jsonl \\\n  --gate-current results/store/semiring_matmul \\\n  --gate-out results/gate.json --report results/gate.md\n\n# Decision-input bundle (P3): structured evidence for a human call, no verdict.\npython tools/decision_bundle.py --case semiring_matmul \\\n  --from results/store/semiring_matmul\n\n# HTML site (P3): one self-contained page (tables + inline-SVG plots) from the\n# whole store (or --render-from \u003crows\u003e); /site is git-ignored.\nuv run nperf --site site --render-from results/store\n```\n\n`--platforms` is a comma-list of worker env-groups (`jax-cpu` / `jax-cuda12`);\nattempts fan out across them and distinct resources run in parallel. Worker\ninterpreter resolves as `NPERF_PYTHON_\u003cPLATFORM\u003e` → `NPERF_WORKER_PYTHON` → this\ninterpreter. `--cpu-slots N` runs N CPU attempts in parallel on disjoint pinned\ncores (timings reflect the slot's core budget; `1` = full machine); `--gpus N`\nfans GPU attempts across N devices (default: auto-probed), one lock each;\n`--gpu-settle S` holds a device's lock S seconds between its attempts.\n`--store [DIR]` ingests the run durably (default `results/store`); `--prune-keep\nN` caps history. `--out`/`--report` default to `results/\u003ccase\u003e.{jsonl,md}`;\n`--quick` runs the representative point, `--point '\u003cjson\u003e'` a single explicit\none, `--baselines A,B` / `--skip-baselines X,Y` select (allowlist) / drop\n(denylist) baselines — a dropped baseline is **recorded as a `skipped` row**\n(omission is data, DESIGN §1), so you can dodge a pathological cold compile\n(e.g. `--skip-baselines naive-dense`) without a silent gap. `--skip-slow` drops\nthe case's *declared* `slow_baselines` for fast dev cycles and stamps the run\n`coverage_mode=fast`, so the op_matrix feed + regression gate **refuse to treat\nit as authoritative** — run the full sweep (omit `--skip-slow`) at sprint end\n(`COVERAGE_MANDATE.md` §7). `--in-process` uses the P0 driver, `--render-from\n\u003cfiles/dirs\u003e [--latest]` re-renders (and combines) saved rows. `--gate-baseline \u003cfiles/dirs\u003e` runs the\nregression gate (`--gate-current`, default the store; `--gate-min`/`--gate-p95`\nthresholds; `--gate-out` artifact) and exits nonzero on a regression; `--site\n[DIR]` renders the self-contained HTML site. The op_matrix feed\n(`tools/op_matrix_feed.py`), decision-input bundles\n(`tools/decision_bundle.py`), and the **coverage-\u0026-deficit report**\n(`tools/coverage_report.py` — joins the op catalogue with the store and ranks\nunder-covered + on-target-lagging ops for the nitrix agent) are sibling L5\nartifacts over the same rows. The **drift check** (`tools/drift_check.py`)\nfingerprints each op's signature + canonical-output digest into\n`reports/op_drift_manifest.json` and flags when a nitrix change has silently\nmoved a case's assumptions (a change detector, not a correctness gate; the fast\nsignature half also runs in the suite). Tests:\n`JAX_PLATFORMS=cpu uv run pytest` (CPU-only; schema, fidelity, case build,\nworker round-trip, scheduler invariants, multi-platform, registries, store,\ngate, bundle, html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhypercoil%2Fnitrix-perf-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhypercoil%2Fnitrix-perf-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhypercoil%2Fnitrix-perf-bench/lists"}