https://github.com/langwatch/cookbooks
example projects that use langwatch's features.
https://github.com/langwatch/cookbooks
Last synced: 10 months ago
JSON representation
example projects that use langwatch's features.
- Host: GitHub
- URL: https://github.com/langwatch/cookbooks
- Owner: langwatch
- License: mit
- Created: 2025-04-13T21:27:51.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-05-26T12:57:52.000Z (about 1 year ago)
- Last Synced: 2025-06-04T01:13:43.827Z (about 1 year ago)
- Language: Jupyter Notebook
- Size: 13.6 MB
- Stars: 6
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# AI Evaluations Cookbooks
A collection of practical notebooks demonstrating systematic approaches to building, evaluating, and improving AI applications. These cookbooks provide hands-on guidance for developing more effective AI systems through data-driven methodologies.
## Overview
Building effective AI applications requires more than just connecting to the latest LLM API. This repository provides structured approaches to developing systems that are reliable, efficient, and continuously improving. Each notebook in this collection focuses on a specific technique and walks through a methodical process for:
1. **Establishing evaluation frameworks** - Creating robust metrics to measure performance
2. **Systematic improvement** - Using data-driven approaches to enhance capabilities
3. **Performance visualization** - Tracking improvements and identifying bottlenecks
## Notebooks
### [Simple RAG Application](/notebooks/simple-rag-app.ipynb)
A step-by-step guide to building and improving a Retrieval-Augmented Generation (RAG) application. This notebook covers:
- Implementing effective retrieval strategies
- Evaluating RAG performance with meaningful metrics
- Systematically improving retrieval and generation quality
### [Fine-Tuning Embedding Models](/notebooks/finetune-embedding-models.ipynb)
Learn how to fine-tune embedding models to significantly improve retrieval performance. This notebook covers:
- Fine-tuning open-source embedding models using triplet loss
- Evaluating and visualizing performance improvements
- Applying techniques from industry case studies (like Ramp's transaction categorization)
### [Metadata Filtering](/notebooks/metadata-filtering.ipynb)
Explore how to enhance retrieval performance by implementing metadata filtering in RAG applications. This notebook covers:
- Implementing both semantic search and metadata-filtered search approaches
- Evaluating and comparing approaches using industry-standard metrics
- Drawing data-driven insights to optimize your own retrieval systems
### [Evaluating Tool Selection in AI Assistants](/notebooks/tool-calling.ipynb)
Learn how to measure and improve tool calling capabilities in AI assistants using precision and recall metrics. This notebook covers:
- Creating a framework for evaluating tool selection decisions
- Analyzing per-tool performance to identify specific improvement areas
- Systematically enhancing multi-tool coordination for complex tasks
## Getting Started
1. Clone this repository
2. Install the required dependencies: `pip install -r requirements.txt`
3. Open the notebooks in Jupyter or your preferred notebook environment
4. Follow along with the step-by-step instructions
## Contributing
Contributions are welcome! If you have ideas for new notebooks or improvements to existing ones, please open an issue or submit a pull request.
## License
This project is licensed under the MIT License - see the LICENSE file for details.