https://github.com/abundant-ai/oddish
Run Harbor tasks in the cloud
https://github.com/abundant-ai/oddish
eval llm rl
Last synced: 4 days ago
JSON representation
Run Harbor tasks in the cloud
- Host: GitHub
- URL: https://github.com/abundant-ai/oddish
- Owner: abundant-ai
- License: apache-2.0
- Created: 2026-02-02T04:31:29.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-03-26T20:42:17.000Z (about 1 month ago)
- Last Synced: 2026-03-26T20:58:17.074Z (about 1 month ago)
- Topics: eval, llm, rl
- Language: Python
- Homepage: https://oddish.app/
- Size: 3.58 MB
- Stars: 5
- Watchers: 0
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Oddish
> Run evals on [Harbor](https://github.com/laude-institute/harbor) tasks in the cloud.
Oddish extends Harbor with:
- Provider-aware queuing and automatic retries for LLM providers
- Real-time monitoring via dashboard or CLI
- Postgres-backed state and S3 storage for logs
Just replace `harbor run` with `oddish run`.
## Quick Start
### 1. Install
```bash
uv pip install oddish
```
### 2. Generate an API key [here](https://oddish.app/)
- API key generation is restricted during the beta. To request access, contact the [maintainer](https://github.com/RishiDesai).
```bash
export ODDISH_API_KEY="ok_..."
```
### 3. Submit a job
```bash
# Run a single agent
oddish run -d terminal-bench@2.0 -a codex -m gpt-5.4 --n-trials 3
```
```bash
# Or sweep multiple agents
oddish run -d terminal-bench@2.0 -c job.yaml
```
Example [job.yaml](assets/light-run.yaml)
### 4. Monitor Progress
```bash
oddish status
```
## Documentation
- [CLI docs](DOCS.md)
- [Core library](oddish/README.md)
- [Web dashboard](frontend/README.md)
- [Cloud backend](backend/README.md)
- [Self-hosting](SELF_HOSTING.md)
- [Agents](AGENTS.md)
## License
[Apache License 2.0](LICENSE)