https://github.com/dpguthrie/braintrust-playground
https://github.com/dpguthrie/braintrust-playground
Last synced: 9 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/dpguthrie/braintrust-playground
- Owner: dpguthrie
- Created: 2025-05-20T21:03:48.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-06-02T21:17:09.000Z (about 1 year ago)
- Last Synced: 2025-06-03T11:47:31.079Z (about 1 year ago)
- Language: Python
- Size: 55.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Braintrust Playground
A collection of cookbooks demonstrating Braintrust's evaluation and dataset management capabilities.
## Setup
1. Install dependencies:
```bash
uv sync
```
2. Set up environment variables:
```bash
cp .env.example .env
# Edit .env with your API keys
```
## Development
This project uses:
- Python 3.12+
- uv
- Braintrust for experiment tracking
- Pydantic for data validation
## Makefile Commands
The project includes a Makefile to simplify common tasks:
```bash
# Show available commands and their descriptions
make help
# Run agent evaluation for a specific cookbook
make run-evals code_conversion
# Create example dataset in Braintrust
make push-datasets
```
### Command Details
- `make help`: Displays all available commands with their descriptions
- `make run-evals `: Runs agent evaluation for the specified cookbook
- Example: `make run-evals code_conversion`
- Evaluates model outputs against ground truth
- Generates metrics and scores
- Creates evaluation reports
- `make push-datasets`: Creates example dataset in Braintrust using `code_conversion/push_datasets.py`
- Pushes dataset versions to Braintrust's platform
- Enables tracking and versioning of your datasets
- Makes datasets available for experiments
## License
MIT