An open API service indexing awesome lists of open source software.

https://github.com/Hanno-Labs/decision-bench

Open benchmark runtime for document-grounded decision models
https://github.com/Hanno-Labs/decision-bench

benchmark decision-model jev llm-evaluation model-evaluation typesafe-ai

Last synced: about 2 months ago
JSON representation

Open benchmark runtime for document-grounded decision models

Awesome Lists containing this project

README

          

DecisionBench

---

The evaluation ecosystem for decision models


License
CI


Installation ·
Documentation ·
Leaderboard ·
Results ·
Issues ·
Citing

## Installation

```bash
pip install git+https://github.com/Hanno-Labs/decision-bench.git
```

```bash
uv add git+https://github.com/Hanno-Labs/decision-bench.git
```

## Example Usage

Smoke-test a supported decision model on the pinned benchmark. This example uses
[Bosun v3.1 0.6B](https://huggingface.co/Hanno-Labs/bosun-v3.1-0.6b), whose
native decision-token readout is supported directly by `run-hf`.

```bash
hf download Hanno-Labs/bosun-v3.1-0.6b \
--revision aaa9dd06d4d6501b33df61942472fed9284bc5e6 \
--local-dir models/bosun-v3.1-0.6b

decision-bench run-hf task_specs/decisionbench-dev.toml \
models/bosun-v3.1-0.6b results/bosun-v3.1-0.6b --smoke
```

Before running anything else, see the complete
[supported adapters and models](https://hanno-labs.github.io/decision-bench/overview/models/).
If your model is listed, use its runner; only add an adapter when its native
decision readout is not already supported.

## Overview

| | |
|---|---|
| 📈 [Leaderboard] | Compare reviewed results and filter by task, family, domain, or primitive |
| 🏃 [Get Started] | Install DecisionBench and run the frozen suite |
| 📋 [Tasks and Views] | Understand the 23,900 rows, nine families, three primitives, and reasoning track |
| 🤖 [Models] | See supported adapters and models, or add a new native readout contract |
| 📊 [Results] | Load, inspect, and submit reproducible results |
| 🧪 [Evaluation] | Learn the metrics, artifacts, and comparability rules |
| 🤝 [Contributing] | Add models, tasks, benchmarks, and result records |

[Leaderboard]: https://huggingface.co/spaces/Hanno-Labs/decision-bench-leaderboard
[Get Started]: https://hanno-labs.github.io/decision-bench/
[Tasks and Views]: https://hanno-labs.github.io/decision-bench/overview/tasks/
[Models]: https://hanno-labs.github.io/decision-bench/overview/models/
[Results]: https://hanno-labs.github.io/decision-bench/get_started/usage/loading_results/
[Evaluation]: https://hanno-labs.github.io/decision-bench/get_started/usage/running_the_evaluation/
[Contributing]: https://hanno-labs.github.io/decision-bench/contributing/

## Contribute

Report bugs and request features for any DecisionBench component in the
[central issue tracker](https://github.com/Hanno-Labs/decision-bench/issues).
Send code changes to the repository that owns that component.

Choose the path that matches what you want to bring to DecisionBench:



🤖 Add a Model →


Add a compatibility adapter so DecisionBench can evaluate a new decision model.




📊 Submit Results →


Run a supported model and submit its reviewed, reproducible scores.






🧩 Add a Task →


Contribute one dataset-backed decision problem with labels, provenance, and tests.




🗂️ Add a Benchmark →


Curate existing tasks into a named evaluation for a domain or purpose.



## Citing

DecisionBench is under active development. Until the benchmark paper is published, cite the
repository and the individual datasets listed in the
[task catalog](https://hanno-labs.github.io/decision-bench/overview/tasks/). Machine-readable
citation metadata lives in [`CITATION.cff`](CITATION.cff).