https://github.com/dettinjo/llm-fact-auditor
A post-processing pipeline to fact-check, entity-link, and verify answers from Large Language Models (LLMs). Developed for the Web Data Processing Systems course at Vrije Universiteit Amsterdam.
https://github.com/dettinjo/llm-fact-auditor
data-processing docker entity-linking fact-checking llm nlp portfolio python pytorch question-answering spacy stanza transformers wikidata
Last synced: 2 months ago
JSON representation
A post-processing pipeline to fact-check, entity-link, and verify answers from Large Language Models (LLMs). Developed for the Web Data Processing Systems course at Vrije Universiteit Amsterdam.
- Host: GitHub
- URL: https://github.com/dettinjo/llm-fact-auditor
- Owner: dettinjo
- License: mit
- Created: 2025-10-14T20:04:49.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2026-06-03T09:48:17.000Z (2 months ago)
- Last Synced: 2026-06-03T11:22:29.380Z (2 months ago)
- Topics: data-processing, docker, entity-linking, fact-checking, llm, nlp, portfolio, python, pytorch, question-answering, spacy, stanza, transformers, wikidata
- Language: Python
- Homepage:
- Size: 926 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
LLM Fact Auditor
A post-processing pipeline to fact-check, entity-link, and verify answers from Large Language Models.
About the Project
·
Getting Started
·
Usage
Table of Contents
## About The Project
Large Language Models (LLMs) are powerful, but they can produce factually incorrect or unverifiable information—a phenomenon often called "hallucination." This project, **LLM Fact Auditor**, serves as a robust post-processing pipeline designed to address this challenge. It takes a question and a raw LLM-generated answer, then enriches and verifies it through a multi-stage process.
Here's what it does:
* **Entity Linking**: It identifies named entities (like people, places, and organizations) in the text and links them to their corresponding Wikipedia pages, grounding the response in factual data.
* **Answer Extraction**: It distills the often verbose LLM response into a concise, direct answer, such as a "yes/no" or a specific entity.
* **Fact-Checking**: It verifies the extracted answer's correctness by cross-referencing it with structured knowledge from Wikidata and the content of the linked Wikipedia pages.
This system was developed as a university project to create a practical tool for improving the reliability of AI-generated content.
### Built With
This project leverages a powerful stack of modern NLP tools and libraries.
## Getting Started
Follow these steps to set up and run the project locally within the provided Docker environment.
### Prerequisites
* **Docker**: You must have Docker installed and running.
* **WDPS Docker Image**: The project is designed to run inside the `karmaresearch/wdps2` Docker container. Ensure you have this container running.
```sh
docker ps
```
### Installation
1. **Clone the Repository**:
```sh
git clone https://github.com/dettinjo/LLM-Fact-Auditor.git
cd LLM-Fact-Auditor
```
2. **Copy Project Files to Docker**: From your host machine's terminal, copy the entire project directory into your running Docker container.
```sh
docker cp ./ :/home/user/submission
```
3. **Access the Container and Set Up Environment**:
```sh
# Enter the container's shell
docker exec -it bash
# Navigate to the project directory
cd /home/user/submission
# Switch to root user to install dependencies
sudo su
# Create and activate a virtual environment
python3 -m venv virtual_env
source virtual_env/bin/activate
```
4. **Install Dependencies**: Install all required Python packages and download the necessary NLP models. This step may take some time.
```sh
# Install Python packages
pip install -r requirements.txt
# Run the setup script to download all models
python src/setup.py
```
## Usage
The main script is designed to read questions from standard input and write the processed output to standard output.
### Running with Llama 2 (Default)
This command reads questions from `test_data/input.txt` and saves the structured output to `test_data/output.txt`.
```sh
python3 main.py < ./test_data/input.txt > ./test_data/output.txt
```
### Running with Llama 3
For higher quality answers and faster performance, you can use the Llama 3 model by adding the `--llama_ver=3` flag.
```sh
python3 main.py --llama_ver=3 < ./test_data/input.txt > ./test_data/output.txt
```
### Example Input & Output
**Input Question in `input.txt`:**
```
question-001 Is Managua the capital of Nicaragua?
```
**Corresponding Output in `output.txt`:**
```
question-001 R"Yes, Managua is the capital and largest city of Nicaragua."
question-001 A"yes"
question-001 C"correct"
question-001 E"Managua" "https://en.wikipedia.org/wiki/Managua"
question-001 E"Nicaragua" "https://en.wikipedia.org/wiki/Nicaragua"
```
The output format includes the raw **R**esponse, extracted **A**nswer, **C**orrectness check, and linked **E**ntities.
## Roadmap
- [ ] Implement a more robust relation extraction module.
- [ ] Add support for additional knowledge bases beyond Wikidata.
- [ ] Develop a simple web interface for interactive demonstrations.
- [ ] Expand fact-checking capabilities to handle more complex and nuanced claims.
See the [open issues](https://github.com/dettinjo/LLM-Fact-Auditor/issues) for a full list of proposed features (and known issues).
## License
Distributed under the MIT License. See `LICENSE` for more information.
## Contact
This project was created for the Web Data Processing Systems course (XM_40020) at Vrije Universiteit Amsterdam.
**Group Members:**
* Joel Dettinger - j.dettinger@student.vu.nl
* Ruida Zhou - r.zhou4@student.vu.nl
* Hongqian Xia - h.xia@student.vu.nl
* Angelo De Nadai - a.denadai@student.vu.nl
Project Link: [https://github.com/dettinjo/LLM-Fact-Auditor](https://github.com/dettinjo/LLM-Fact-Auditor)
## Acknowledgments
* Vrije Universiteit Amsterdam
* Hugging Face for the incredible `transformers` library and model hosting.
* The developers of spaCy, Stanza, and the Wikidata platform.
* [Othneil Drew's Best-README-Template](https://github.com/othneildrew/Best-README-Template)