https://github.com/decoderesearch/synth-sae-bench-experiments
code for the paper: SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data
https://github.com/decoderesearch/synth-sae-bench-experiments
ai interpretability machine-learning sparse-autoencoders
Last synced: 3 months ago
JSON representation
code for the paper: SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data
- Host: GitHub
- URL: https://github.com/decoderesearch/synth-sae-bench-experiments
- Owner: decoderesearch
- License: mit
- Created: 2026-02-12T14:11:37.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2026-02-17T12:28:24.000Z (4 months ago)
- Last Synced: 2026-02-17T17:53:38.326Z (4 months ago)
- Topics: ai, interpretability, machine-learning, sparse-autoencoders
- Language: Python
- Homepage: https://arxiv.org/abs/2602.14687
- Size: 198 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Citation: CITATION.cff
Awesome Lists containing this project
README
# SynthSAEBench
This repo contains code for the paper: [SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data](https://arxiv.org/abs/2602.14687).
Trained SAEs and results are online at [https://huggingface.co/decoderesearch/synth-sae-bench-16k-v1-saes](https://huggingface.co/decoderesearch/synth-sae-bench-16k-v1-saes).
## Structure
Experiments for the paper are in the `experiments` directory. The extended SAE classes used in the paper (`XStandardTrainingSAE` and `XJumpReLUTrainingSAE` containing L0 autotuning) are in the `saes` directory.
## Setup
This project uses uv for package management. To install the dependencies, run:
```bash
uv sync
```
## Running Experiments
To run the experiments, use the `uv run` command. For example, to run the superposition experiment, run:
```bash
uv run experiments/sweeps/sweep_superposition.py
```
## Loading the benchmark model
The main benchmark model is on Huggingface at [decoderesearch/synth-sae-bench-16k-v1](https://huggingface.co/decoderesearch/synth-sae-bench-16k-v1). To load the model, run:
```bash
from sae_lens.synthetic.synthetic_model import SyntheticModel
model = SyntheticModel.from_pretrained("decoderesearch/synth-sae-bench-16k-v1")
```
## Development
### Linting and Formatting
This project uses ruff for linting and formatting. To run the linting and formatting, run:
```bash
uv run ruff check .
uv run ruff format .
```
### Testing
This project uses pytest for testing. To run the tests, run:
```bash
uv run pytest
```
### Type Checking
This project uses pyright for type checking. To run the type checking, run:
```bash
uv run pyright
```