https://github.com/firstbatchxyz/dria-cookbook
All Dria related examples and cookbooks
https://github.com/firstbatchxyz/dria-cookbook
Last synced: over 1 year ago
JSON representation
All Dria related examples and cookbooks
- Host: GitHub
- URL: https://github.com/firstbatchxyz/dria-cookbook
- Owner: firstbatchxyz
- Created: 2024-12-25T16:46:38.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2025-02-20T13:16:54.000Z (over 1 year ago)
- Last Synced: 2025-03-30T03:24:04.499Z (over 1 year ago)
- Language: Jupyter Notebook
- Size: 9.06 MB
- Stars: 5
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Dria RAG Evaluation Cookbook
This repository contains a comprehensive guide and set of tools for evaluating AI agents using Dria SDK. The notebook provided in this repository demonstrates how to generate an evaluation set for your AI agents and assess their performance using various tools and datasets.
## Table of Contents
- [Introduction](#introduction)
- [Installation](#installation)
- [Usage](#usage)
- [Step 1: Initialization](#step-1-initialization)
- [Step 2: Setting Environmental Variables](#step-2-setting-environmental-variables)
- [Step 3: Scraping Content](#step-3-scraping-content)
- [Step 4: Combining Data](#step-4-combining-data)
- [Step 5: Evaluation](#step-5-evaluation)
- [Dependencies](#dependencies)
- [Contributing](#contributing)
- [License](#license)
## Importance of Evaluation
Evaluating Retrieval-Augmented Generation (RAG) agents is crucial for ensuring their effectiveness and reliability across diverse datasets and scenarios. By testing these agents with detailed questions and varied personas, you can better understand their strengths and weaknesses. This process helps in refining the agents to perform optimally in real-world applications.
Moreover, evaluating different models with various RAG methodologies allows for a comprehensive comparison of their capabilities. It highlights the nuances in performance and adaptability, guiding the selection of the most suitable model for specific tasks. This evaluation is essential for advancing AI technologies and ensuring they meet the desired standards of accuracy and efficiency.
## Introduction
This notebook shows how to generate an evaluation set for your AI agents by using [Dria](https://docs.dria.co/). In the end, you can evaluate these agents with [promptfoo](https://www.promptfoo.dev/) and see the evaluation and assessment results.
## Installation
To get started, clone this repository and install the necessary dependencies. We recommend using a Python virtual environment to manage dependencies.
## Usage
### Step 1: Initialization
Begin by installing the necessary dependencies. This can be done by running the provided code block in the notebook. It is recommended to use your local machine instead of Google Colab due to potential incompatibilities.
### Step 2: Setting Environmental Variables
To run and use external applications in this notebook, you need to have API keys from various providers such as Firecrawl, Jina Reader, Upstash, Cohere, and OpenAI. Create an `.env` file with the required API keys.
### Step 3: Scraping Content
Utilize the command-line interface provided in the notebook to scrape content from web domains. You can choose to scrape an entire domain or a single URL.
### Step 4: Combining Data
The notebook demonstrates how to combine scraped content with personas to create a comprehensive dataset for evaluation.
### Step 5: Evaluation
Finally, use the combined data to evaluate your AI agents. By leveraging Dria, it produces synthetic QA pairs for each context-persona combination. These pairs simulate real-world scenarios, offering insights into the performance of different RAG configurations. The notebook also provides guidance on how to perform this evaluation with promptfoo.
## Dependencies
The project requires several Python packages, including but not limited to:
- requests
- openai
- pandas
- nltk
- matplotlib
- firecrawl
- upstash_vector
- cohere
- python-dotenv
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## License
This project is licensed under the MIT License. See the LICENSE file for more details.