https://github.com/janteichertkluge/synth-bench
synthbench is a Python library for generating reproducible, metadata-rich synthetic datasets for benchmarking.
https://github.com/janteichertkluge/synth-bench
benchmark benchmarking classification data datasets dgp evaluation regression synthetic-data
Last synced: about 24 hours ago
JSON representation
synthbench is a Python library for generating reproducible, metadata-rich synthetic datasets for benchmarking.
- Host: GitHub
- URL: https://github.com/janteichertkluge/synth-bench
- Owner: JanTeichertKluge
- License: mit
- Created: 2026-04-24T13:48:29.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-05-06T08:58:09.000Z (2 months ago)
- Last Synced: 2026-05-09T05:16:39.466Z (2 months ago)
- Topics: benchmark, benchmarking, classification, data, datasets, dgp, evaluation, regression, synthetic-data
- Language: Python
- Homepage: https://janteichertkluge.github.io/synth-bench/
- Size: 4.54 MB
- Stars: 2
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
---
synthbench is a small Python library for generating synthetic datasets that are actually useful for benchmarking. You control the signal complexity, add noise or missing data on top, and get back a dataset with full provenance so you know exactly what you generated and why. Every result is reproducible from a single integer seed.
It covers eight DGP families, five corruptors, metadata enrichment (Bayes error, effective rank), Parquet/CSV serialization, and sweep helpers for running ablation grids.
## Installation
```bash
pip install synthbench
```
For Parquet support:
```bash
pip install "synthbench[io]"
```
For `RandomNeuralDGP` (needs PyTorch):
```bash
pip install "synthbench[neural]"
```
## Basic usage
```python
from synthbench import BenchPipeline, LinearDGP, MissingDataCorruptor
pipeline = BenchPipeline(
LinearDGP(complexity="medium", task_type="classification"),
corruptors=[MissingDataCorruptor(proportion=0.1, mechanism="mar")],
)
result = pipeline.run(n_samples=500, n_features=10, random_state=42)
print(result.X.shape) # (500, 10)
print(result.metadata["bayes_error"]) # empirical difficulty estimate
print(result.metadata["effective_rank"]) # feature space dimensionality
```
## What it does
**Data-generating processes** — Linear, Polynomial, Tree, Friedman (variants 1/2/3), Additive, Sparse, Geometric, and RandomNeural. Each takes a `complexity` parameter and records ground-truth feature importances alongside the data.
**Corruptors** — MeasurementNoise, Outlier, MissingData, Collinearity, and Categorical corruptors for the feature matrix, plus `LabelNoiseCorruptor` for flipping labels or injecting regression noise. They chain together in a canonical order and track how much signal they degrade.
**Metadata** — every result carries `bayes_error`, `effective_rank`, corruptor parameters, and version provenance. Enough to reconstruct the generating pipeline from scratch.
**Sweeps** — `severity_sweep` and `difficulty_sweep` for single-axis ablations, and `experiment_grid` for full factorial runs across sample size, complexity, and severity. Seeds are derived hierarchically so cells are independent but deterministic.
**Named suites** — `BenchSuite("easy-classification").run()` returns a labelled dict of results for a curated collection. Good for quick sanity checks or as a shared benchmark baseline.
**Serialization** — `to_parquet` / `from_parquet` and `to_csv` / `from_csv` round-trip everything including metadata. `BenchPipeline.from_metadata` reconstructs and re-runs the pipeline for bit-identical replay.
## Ablation example
```python
from synthbench import LinearDGP, OutlierCorruptor, experiment_grid
grid = experiment_grid(
LinearDGP,
OutlierCorruptor,
n_samples_list=[200, 500, 1000],
complexities=["low", "medium", "high"],
severities=["low", "medium", "high"],
n_features=10,
random_state=0,
task_type="classification",
)
result = grid[(500, "high", "medium")]
print(result.metadata["bayes_error"])
```
## Docs
Full reference at [JanTeichertKluge.github.io/synth-bench](https://JanTeichertKluge.github.io/synth-bench).