https://github.com/plastic-labs/dspy-opentom
Exploration using DSPy to optimize modules to maximize performance on the OpenToM dataset
https://github.com/plastic-labs/dspy-opentom
Last synced: 17 days ago
JSON representation
Exploration using DSPy to optimize modules to maximize performance on the OpenToM dataset
- Host: GitHub
- URL: https://github.com/plastic-labs/dspy-opentom
- Owner: plastic-labs
- Created: 2024-02-24T21:57:46.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-03-06T00:13:01.000Z (about 1 year ago)
- Last Synced: 2025-05-06T03:01:57.179Z (25 days ago)
- Language: Python
- Homepage:
- Size: 18.4 MB
- Stars: 16
- Watchers: 2
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-dspy - DSPy OpenTOM - Exploration using DSPy to optimize modules to maximize performance on the OpenToM dataset. ([DSPy](https://github.com/stanfordnlp/dspy) - A library for compiling declarative language model calls into self-improving pipelines. / Projects)
README
# DSPy OpenTOM
This repo contains scripts for optimizing DSPy modules for the OpenTOM Benchmark. We support Chain of Thought and a method we thought might work where we generate a "thought" about the context to aid in answering the question (spoiler -- it didn't work better than just `BootstrapFewShotWithRandomSearch`).
CLI Usage:
```
usage: main.py [-h] [--student STUDENT] [--teacher TEACHER] [--train_size TRAIN_SIZE] [--download_dataset DOWNLOAD_DATASET]
[--question_types [QUESTION_TYPES ...]]
experiment_title dspy_method dspy_optimizer
Run DSPY method.
positional arguments:
experiment_title Title of new experiment
dspy_method The DSPY method to run
dspy_optimizer The DSPY optimizer to use
options:
-h, --help show this help message and exit
--student STUDENT The LLM to optimize prompts for
--teacher TEACHER Teacher LLM for optimizing prompts. Defaults to Student LLM
--train_size TRAIN_SIZE
Number of training examples to use for optimization
--download_dataset DOWNLOAD_DATASET
Download dataset
--question_types [QUESTION_TYPES ...]
Question types. Defaults to all
```
Come chat with us in our [discord](https://discorg.gg/plasticlabs) or in the [DSPy thread](https://discord.com/channels/1161519468141355160/1214629969318252574)