https://github.com/couchbase-examples/evaluate-rag
Evaluation of RAG systems using Ragas
https://github.com/couchbase-examples/evaluate-rag
Last synced: about 1 month ago
JSON representation
Evaluation of RAG systems using Ragas
- Host: GitHub
- URL: https://github.com/couchbase-examples/evaluate-rag
- Owner: couchbase-examples
- License: mit
- Created: 2024-09-25T07:56:04.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-05-12T08:01:25.000Z (11 months ago)
- Last Synced: 2025-05-12T08:56:33.786Z (11 months ago)
- Language: Python
- Size: 15.6 KB
- Stars: 1
- Watchers: 5
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Evaluate RAG using Ragas
The code examples to evaluate RAG responses using [Ragas](https://ragas.io/) using the LLM (OpenAI) as the judge approach. The RAG pipelines is based off the RAG application in this [repo](https://github.com/couchbase-examples/qa-bot-demo).
> Note that you need Couchbase Server 7.6 or higher for Vector Search.
A detailed tutorial for this demo can be found on the [Developer Portal](https://developer.couchbase.com/tutorial-evaluate-rag-responses-using-ragas)
## How to Run
- Install the dependencies
`pip install -r requirements.txt`
- Set the environment variables
Copy the .env.example file and rename it to .env and replace the placeholders with the actual values for your environment.
```
OPENAI_API_KEY=
DB_CONN_STR=
DB_USERNAME=
DB_PASSWORD=
DB_BUCKET=
DB_SCOPE=
DB_COLLECTION=
INDEX_NAME=
EMBEDDING_MODEL=text-embedding-3-small
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
LANGCHAIN_API_KEY=
```
- Run the evaluation script
`python rag_evaluation.py`
Note: For the RAG pipeline to work, you would need to ingest some documents to answer the questions related to Couchbase. You could do that based on the ingestion script in this [repo](https://github.com/couchbase-examples/qa-bot-demo).
## Adding more Test Data
The test data can be updated by adding more question & answer pairs to the list in the `create_dataset.py` file.