https://github.com/Hanno-Labs/decision-bench
Open benchmark runtime for document-grounded decision models
https://github.com/Hanno-Labs/decision-bench
benchmark decision-model jev llm-evaluation model-evaluation typesafe-ai
Last synced: about 2 months ago
JSON representation
Open benchmark runtime for document-grounded decision models
- Host: GitHub
- URL: https://github.com/Hanno-Labs/decision-bench
- Owner: Hanno-Labs
- License: apache-2.0
- Created: 2026-09-20T23:17:33.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2026-09-24T04:29:53.000Z (about 2 months ago)
- Last Synced: 2026-09-24T07:35:33.911Z (about 2 months ago)
- Topics: benchmark, decision-model, jev, llm-evaluation, model-evaluation, typesafe-ai
- Language: Python
- Size: 219 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Citation: CITATION.cff
Awesome Lists containing this project
- awesome-jev-tools - DecisionBench - Model evaluation: scores Jev Noul, Choice, and Score answers on pinned document-grounded tasks, counting malformed probability distributions as misses so model comparisons remain reproducible. (Full list / Evaluation & Benchmarking)
- awesome-typesafe-jev - DecisionBench - grounded decision models, with pinned tasks and a native OpenRouter adapter for Jev. (Community projects / Evaluations and independent research)
- awesome-jev-by-typesafe - DecisionBench - model evaluation | Open runner with pinned document-grounded tasks and a native Jev adapter; inspect the model version, task mix, and scoring protocol before comparing results. | (Use-case map / Benchmarks, calibration, and open reproductions)
- awesome-jev - DecisionBench - Labs/decision-bench?style=flat-square&label=%E2%98%85) - Model evaluation: scores Jev Noul, Choice, and Score answers on pinned document-grounded tasks, counting malformed probability distributions as misses so model comparisons remain reproducible. (Full list / Evaluation & Benchmarking)
README
DecisionBench
---
The evaluation ecosystem for decision models
Installation ·
Documentation ·
Leaderboard ·
Results ·
Issues ·
Citing
## Installation
```bash
pip install git+https://github.com/Hanno-Labs/decision-bench.git
```
```bash
uv add git+https://github.com/Hanno-Labs/decision-bench.git
```
## Example Usage
Smoke-test a supported decision model on the pinned benchmark. This example uses
[Bosun v3.1 0.6B](https://huggingface.co/Hanno-Labs/bosun-v3.1-0.6b), whose
native decision-token readout is supported directly by `run-hf`.
```bash
hf download Hanno-Labs/bosun-v3.1-0.6b \
--revision aaa9dd06d4d6501b33df61942472fed9284bc5e6 \
--local-dir models/bosun-v3.1-0.6b
decision-bench run-hf task_specs/decisionbench-dev.toml \
models/bosun-v3.1-0.6b results/bosun-v3.1-0.6b --smoke
```
Before running anything else, see the complete
[supported adapters and models](https://hanno-labs.github.io/decision-bench/overview/models/).
If your model is listed, use its runner; only add an adapter when its native
decision readout is not already supported.
## Overview
| | |
|---|---|
| 📈 [Leaderboard] | Compare reviewed results and filter by task, family, domain, or primitive |
| 🏃 [Get Started] | Install DecisionBench and run the frozen suite |
| 📋 [Tasks and Views] | Understand the 23,900 rows, nine families, three primitives, and reasoning track |
| 🤖 [Models] | See supported adapters and models, or add a new native readout contract |
| 📊 [Results] | Load, inspect, and submit reproducible results |
| 🧪 [Evaluation] | Learn the metrics, artifacts, and comparability rules |
| 🤝 [Contributing] | Add models, tasks, benchmarks, and result records |
[Leaderboard]: https://huggingface.co/spaces/Hanno-Labs/decision-bench-leaderboard
[Get Started]: https://hanno-labs.github.io/decision-bench/
[Tasks and Views]: https://hanno-labs.github.io/decision-bench/overview/tasks/
[Models]: https://hanno-labs.github.io/decision-bench/overview/models/
[Results]: https://hanno-labs.github.io/decision-bench/get_started/usage/loading_results/
[Evaluation]: https://hanno-labs.github.io/decision-bench/get_started/usage/running_the_evaluation/
[Contributing]: https://hanno-labs.github.io/decision-bench/contributing/
## Contribute
Report bugs and request features for any DecisionBench component in the
[central issue tracker](https://github.com/Hanno-Labs/decision-bench/issues).
Send code changes to the repository that owns that component.
Choose the path that matches what you want to bring to DecisionBench:
🤖 Add a Model →
Add a compatibility adapter so DecisionBench can evaluate a new decision model.
📊 Submit Results →
Run a supported model and submit its reviewed, reproducible scores.
🧩 Add a Task →
Contribute one dataset-backed decision problem with labels, provenance, and tests.
🗂️ Add a Benchmark →
Curate existing tasks into a named evaluation for a domain or purpose.
## Citing
DecisionBench is under active development. Until the benchmark paper is published, cite the
repository and the individual datasets listed in the
[task catalog](https://hanno-labs.github.io/decision-bench/overview/tasks/). Machine-readable
citation metadata lives in [`CITATION.cff`](CITATION.cff).