https://github.com/hasaniqbal777/openfactcheck
An Open-source Factuality Evaluation Demo for LLMs
https://github.com/hasaniqbal777/openfactcheck
artificial-intelligence fact-checking llm natural-language-processing
Last synced: 10 months ago
JSON representation
An Open-source Factuality Evaluation Demo for LLMs
- Host: GitHub
- URL: https://github.com/hasaniqbal777/openfactcheck
- Owner: hasaniqbal777
- License: gpl-3.0
- Created: 2024-07-16T09:53:08.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2025-06-11T14:39:21.000Z (about 1 year ago)
- Last Synced: 2025-07-12T03:50:26.102Z (12 months ago)
- Topics: artificial-intelligence, fact-checking, llm, natural-language-processing
- Language: Python
- Homepage: https://openfactcheck.com
- Size: 10.1 MB
- Stars: 27
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
An Open-source Factuality Evaluation Demo for LLMs
---
---
Overview •
Installation •
Usage •
HuggingFace Demo •
Documentation
## Overview
OpenFactCheck is an open-source repository designed to facilitate the evaluation and enhancement of factuality in responses generated by large language models (LLMs). This project aims to integrate various fact-checking tools into a unified framework and provide comprehensive evaluation pipelines.
## Installation
You can install the package from PyPI using pip:
```bash
pip install openfactcheck
```
## Usage
First, you need to initialize the OpenFactCheckConfig object and then the OpenFactCheck object.
```python
from openfactcheck import OpenFactCheck, OpenFactCheckConfig
# Initialize the OpenFactCheck object
config = OpenFactCheckConfig()
ofc = OpenFactCheck(config)
```
### Response Evaluation
You can evaluate a response using the `ResponseEvaluator` class.
```python
# Evaluate a response
result = ofc.ResponseEvaluator.evaluate(response: str)
```
### LLM Evaluation
We provide [FactQA](https://raw.githubusercontent.com/hasaniqbal777/OpenFactCheck/main/src/openfactcheck/templates/llm/questions.csv), a dataset of 6480 questions for evaluating LLMs. Onc you have the responses from the LLM, you can evaluate them using the `LLMEvaluator` class.
```python
# Evaluate an LLM
result = ofc.LLMEvaluator.evaluate(model_name: str,
input_path: str)
```
### Checker Evaluation
We provide [FactBench](https://raw.githubusercontent.com/hasaniqbal777/OpenFactCheck/main/src/openfactcheck/templates/factchecker/claims.jsonl), a dataset of 4507 claims for evaluating fact-checkers. Once you have the responses from the fact-checker, you can evaluate them using the `CheckerEvaluator` class.
```python
# Evaluate a fact-checker
result = ofc.CheckerEvaluator.evaluate(checker_name: str,
input_path: str)
```
## Cite
If you use OpenFactCheck in your research, please cite the following:
```bibtex
@article{wang2024openfactcheck,
title = {OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs},
author = {Wang, Yuxia and Wang, Minghan and Iqbal, Hasan and Georgiev, Georgi and Geng, Jiahui and Nakov, Preslav},
journal = {arXiv preprint arXiv:2405.05583},
year = {2024}
}
@article{iqbal2024openfactcheck,
title = {OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs},
author = {Iqbal, Hasan and Wang, Yuxia and Wang, Minghan and Georgiev, Georgi and Geng, Jiahui and Gurevych, Iryna and Nakov, Preslav},
journal = {arXiv preprint arXiv:2408.11832},
year = {2024}
}
@software{hasan_iqbal_2024_13358665,
author = {Hasan Iqbal},
title = {hasaniqbal777/OpenFactCheck: v0.3.0},
month = {aug},
year = {2024},
publisher = {Zenodo},
version = {v0.3.0},
doi = {10.5281/zenodo.13358665},
url = {https://doi.org/10.5281/zenodo.13358665}
}
```