https://github.com/automl/promptolution
A unified, modular Framework for Prompt Optimization
https://github.com/automl/promptolution
automated-prompt-engineering automated-prompt-optimization llm prompt prompt-engineering prompt-optimization prompt-tuning
Last synced: 3 months ago
JSON representation
A unified, modular Framework for Prompt Optimization
- Host: GitHub
- URL: https://github.com/automl/promptolution
- Owner: automl
- License: apache-2.0
- Created: 2024-07-21T13:29:31.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2026-02-15T19:39:22.000Z (4 months ago)
- Last Synced: 2026-02-16T00:14:50.249Z (4 months ago)
- Topics: automated-prompt-engineering, automated-prompt-optimization, llm, prompt, prompt-engineering, prompt-optimization, prompt-tuning
- Language: Python
- Homepage: https://automl.github.io/promptolution/
- Size: 8.86 MB
- Stars: 109
- Watchers: 2
- Forks: 8
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
[](https://github.com/automl/promptolution/actions/workflows/ci.yml)
[](https://github.com/automl/promptolution/actions/workflows/docs.yml)
[](https://colab.research.google.com/github/automl/promptolution/blob/main/tutorials/getting_started.ipynb)
## 🚀 What is Promptolution?
**Promptolution** is a unified, modular framework for prompt optimization built for researchers and advanced practitioners who want full control over their experimental setup. Unlike end-to-end application frameworks with high abstraction, promptolution focuses exclusively on the optimization stage, providing a clean, transparent, and extensible API. It allows for simple prompt optimization for one task up to large-scale reproducible benchmark experiments.
### Key Features
* Implementation of many current prompt optimizers out of the box.
* Unified LLM backend supporting API-based models, Local LLMs, and vLLM clusters.
* Built-in response caching to save costs and parallelized inference for speed.
* Detailed logging and token usage tracking for granular post-hoc analysis.
Have a look at our [Release Notes](https://automl.github.io/promptolution/release-notes/) for the latest updates to promptolution.
## 📦 Installation
```
pip install promptolution[api]
```
Local inference via vLLM or transformers:
```
pip install promptolution[vllm,transformers]
```
From source:
```
git clone https://github.com/automl/promptolution.git
cd promptolution
poetry install
```
## 🔧 Quickstart
Start with the **Getting Started tutorial**:
[https://github.com/automl/promptolution/blob/main/tutorials/getting_started.ipynb](https://github.com/automl/promptolution/blob/main/tutorials/getting_started.ipynb)
Full docs:
[https://automl.github.io/promptolution/](https://automl.github.io/promptolution/)
## 🧠 Featured Optimizers
| **Name** | **Paper** | **Init prompts** | **Exploration** | **Costs** | **Parallelizable** | **Few-shot** |
| ---- | ---- | ---- |---- |---- | ----|---- |
| `CAPO` | [Zehle et al., 2025](https://openreview.net/forum?id=UweaRrg9D0) | required | 👍 | 💲 | ✅ | ✅ |
| `EvoPromptDE` | [Guo et al., 2023](https://openreview.net/forum?id=ZG3RaNIsO8) | required | 👍 | 💲💲 | ✅ | ❌ |
| `EvoPromptGA` | [Guo et al., 2023](https://openreview.net/forum?id=ZG3RaNIsO8) | required | 👍 | 💲💲 | ✅ | ❌ |
| `OPRO` | [Yang et al., 2023](https://openreview.net/forum?id=Bb4VGOWELI) | optional | 👎 | 💲💲 | ❌ | ❌ |
## 🏗 Components
* **`Task`** – Manages the dataset, evaluation metrics, and subsampling.
* **`Predictor`** – Defines how to extract the answer from the model's response.
* **`LLM`** – A unified interface handling inference, token counting, and concurrency.
* **`Optimizer`** – The core component that implements the algorithms that refine prompts.
* **`ExperimentConfig`** – A configuration abstraction to streamline and parametrize large-scale scientific experiments.
## 🤝 Contributing
Open an issue → create a branch → PR → CI → review → merge.
Branch naming: `feature/...`, `fix/...`, `chore/...`, `refactor/...`.
Please ensure to use pre-commit, which assists with keeping the code quality high:
```
pre-commit install
pre-commit run --all-files
```
We encourage every contributor to also write tests, that automatically check if the implementation works as expected:
```
poetry run python -m coverage run -m pytest
poetry run python -m coverage report -i
```
Developed by **Timo Heiß**, **Moritz Schlager**, and **Tom Zehle** (LMU Munich, MCML, ELLIS, TUM, Uni Freiburg).