https://github.com/shreyashankar/spade-experiments

Experiments to assess SPADE on different LLM pipelines.
https://github.com/shreyashankar/spade-experiments

Last synced: about 1 month ago
JSON representation

Experiments to assess SPADE on different LLM pipelines.

• Host: GitHub
• URL: https://github.com/shreyashankar/spade-experiments
• Owner: shreyashankar
• Created: 2023-12-29T21:45:47.000Z (12 months ago)
• Default Branch: main
• Last Pushed: 2024-04-07T03:24:50.000Z (9 months ago)
• Last Synced: 2024-11-05T13:40:44.551Z (about 2 months ago)
• Language: Python
• Size: 4.41 MB
• Stars: 16
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# SPADE (System for Prompt Analysis and Delta-Based Evaluation)

This is the code for the paper SPADE: Auto-Generated Assertions for Large Language Model

Pipelines.

The SPADE pipeline has the following steps:

1. Decomposing a list of prompts and an example into a set of prompt deltas, assertion concepts, and candidate assertions. This is in `candidate_gen.py`.

2. Generating a set of candidate assertions for each prompt delta. This is also in `candidate_gen.py` (uses a helper function in `assertion_gen.py`).

3. Generates synthetic data and evaluates the assertions (`execute_assertions.py`). The user should label the synthetic data responses as correct or incorrect.

4. Evaluates subsumption of the assertions (`check_subsumes.py`).

5. Runs the assertion selection algorithm (`optimizer.py``).

All the experiments live in paper_experiments/ and have a notebook for each experiment. To run any of the experiments (leveraging cached data so you don't need to run any LLMs), run the `pipeline.ipynb` notebook in the root directory, using the correct experiment name.