https://github.com/bklieger-groq/rag-verification
Technique for verifying RAG citations
https://github.com/bklieger-groq/rag-verification
Last synced: 5 months ago
JSON representation
Technique for verifying RAG citations
- Host: GitHub
- URL: https://github.com/bklieger-groq/rag-verification
- Owner: bklieger-groq
- Created: 2024-10-23T18:31:28.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2024-10-23T20:36:00.000Z (12 months ago)
- Last Synced: 2025-04-03T15:52:37.846Z (7 months ago)
- Language: Python
- Size: 53.7 KB
- Stars: 11
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# RAG-verification
Technique for verifying RAG citations
### Quickstart
~~~
python3 -m venv venv
~~~
~~~
source venv/bin/activate
~~~
~~~
pip3 install -r requirements.txt
~~~
~~~
export GROQ_API_KEY=gsk...
~~~
~~~
python3 app.py
~~~
```
Question: When did Paul Graham start working on Bel?
Response:
{
"citations": [
{
"number": 1,
"quote": "In late 2015 I spent 3 months writing essays, and when I went back to working on Bel I could barely understand the code."
}
],
"response": "Paul Graham started working on Bel in late 2015 [1]."
}
Verification:
Citation 1 verified with 100.00 confidence
```
The evaluation shows citation helps in certain cases!
```
Results Comparison:
Citation-based accuracy: 100.00%
Direct answer accuracy: 94.74%
CITATION
Question: What was the key insight Paul Graham had about making online stores look legitimate?
Correct Answer: C. Getting page layouts, fonts, and colors right
LLM Answer: C. Getting page layouts, fonts, and colors right
Correct: True
Citations:
1: It helped to have studied art, because the main goal of an online store builder is to make users look legit, and the key to looking legit is high production values.
2: If you get page layouts and fonts and colors right, you can make a guy running a store out of his bedroom look more legit than a big company.
Explanation: Based on the citations, the key insight Paul Graham had about making online stores look legitimate is having high production values, which includes getting page layouts, fonts, and colors right [1]. This is because high production values can make a small online store look more legitimate than a big company [2].
DIRECT
Question: What was the key insight Paul Graham had about making online stores look legitimate?
Correct Answer: C. Getting page layouts, fonts, and colors right
LLM Answer: B. Having high production values
Correct: False
```
### Credits
Groq implementation developed by Benjamin Klieger and Rick Lamers with inspiration from Trelis Research [(Original Video)](https://www.youtube.com/watch?v=-wGzSnhQKPM).