https://github.com/imoonlab/hyper-rag
"Hyper-RAG: Combating LLM Hallucinations using Hypergraph-Driven Retrieval-Augmented Generation" by Yifan Feng, Hao Hu, Xingliang Hou, Shiquan Liu, Shihui Ying, Shaoyi Du, Han Hu, and Yue Gao.
https://github.com/imoonlab/hyper-rag
graph-rag llms rag
Last synced: 7 days ago
JSON representation
"Hyper-RAG: Combating LLM Hallucinations using Hypergraph-Driven Retrieval-Augmented Generation" by Yifan Feng, Hao Hu, Xingliang Hou, Shiquan Liu, Shihui Ying, Shaoyi Du, Han Hu, and Yue Gao.
- Host: GitHub
- URL: https://github.com/imoonlab/hyper-rag
- Owner: iMoonLab
- License: apache-2.0
- Created: 2025-04-03T09:10:33.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2025-04-27T06:25:29.000Z (24 days ago)
- Last Synced: 2025-05-08T05:02:39.362Z (13 days ago)
- Topics: graph-rag, llms, rag
- Language: Python
- Homepage: https://arxiv.org/abs/2504.08758
- Size: 4.63 MB
- Stars: 112
- Watchers: 5
- Forks: 11
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Hyper-RAG
About |
Features |
Installation |
Quick Start |
Evaluation |
License |
Author
We show that Hyper-RAG is a powerful RAG that can enhance the performance of various LLMs and outperform other SOTA RAG methods in the NeurologyCorp dataset. **Our paper is available at here**.
## :dart: About
Abstract
Large language models (LLMs) have transformed various sectors, including education, finance, and medicine, by enhancing content generation and decision-making processes. However, their integration into the medical field is cautious due to hallucinations, instances where generated content deviates from factual accuracy, potentially leading to adverse outcomes. To address this, we introduce Hyper-RAG, a hypergraph-driven Retrieval-Augmented Generation method that comprehensively captures both pairwise and beyond-pairwise correlations in domain-specific knowledge, thereby mitigating hallucinations. Experiments on the NeurologyCrop dataset with six prominent LLMs demonstrated that Hyper-RAG improves accuracy by an average of 12.3% over direct LLM use and outperforms Graph RAG and Light RAG by 6.3% and 6.0%, respectively. Additionally, Hyper-RAG maintained stable performance with increasing query complexity, unlike existing methods which declined. Further validation across nine diverse datasets showed a 35.5% performance improvement over Light RAG using a selection-based assessment. The lightweight variant, Hyper-RAG-Lite, achieved twice the retrieval speed and a 3.3\% performance boost compared with Light RAG. These results confirm Hyper-RAG's effectiveness in enhancing LLM reliability and reducing hallucinations, making it a robust solution for high-stakes applications like medical diagnostics.
Schematic diagram of the proposed Hyper-RAG architecture. a, The patient poses a question. b, A knowledge base is constructed from relevant domainspecific corpora. c, Responses are generated directly using LLMs. d, Hyper-RAG generates responses by first retrieving relevant prior knowledge from the knowledge base and then inputting this knowledge, along with the patient’s question, into the LLMs to formulate the reply.
More details about hypergraph modeling
Example of hypergraph modeling for entity space. Hypergraph can model the beyond-pairwise relationship among entities, which is more powerful than the pairwise relationship in traditional graph modeling. With hypergraphs, we can avoid the information loss caused by the pairwise relationship.
Illustration of Entity and Correlation Extraction from Raw Corpus: Dark brown boxes represent entities, blue arrows denote low-order correlations between entities, and red arrows indicate high-order correlations. Yellow boxes contain the original descriptions of the respective entities or their correlations.
## :sparkles: Why Hyper-RAG is More Powerful
:heavy_check_mark: **Comprehensive Relationship Modeling with Hypergraphs**: Utilizes hypergraphs to thoroughly model the associations within the raw corpus data, providing more complex relationships compared to traditional graph-based data organization.;\
:heavy_check_mark: **Native Hypergraph-DB Integration**: Employs the native hypergraph database, Hypergraph-DB, as the foundation, supporting rapid retrieval of higher-order associations.;\
:heavy_check_mark: **Superior Performance**: Hyper-RAG outperforms Graph RAG and Light RAG by 6.3% and 6.0% respectively.;\
:heavy_check_mark: **Broad Validation**: Across nine diverse datasets, Hyper-RAG shows a 35.5% performance improvement over Light RAG based on a selection-based assessment.;\
:heavy_check_mark: **Efficiency**: The lightweight variant, Hyper-RAG-Lite, achieves twice the retrieval speed and a 3.3% performance boost compared to Light RAG.;
## :rocket: Installation
```bash
# Clone this project
git clone https://github.com/iMoonLab/Hyper-RAG.git
# Access
cd Hyper-RAG
# Install dependencies
pip install -r requirements.txt
```
## :white_check_mark: Quick Start
### Configure your LLM API
Copy the `config_temp.py` file to `my_config.py` in the root folder and set your LLM `URL` and `KEY`.
```python
LLM_BASE_URL = "Yours xxx"
LLM_API_KEY = "Yours xxx"
LLM_MODEL = "gpt-4o-mini"
EMB_BASE_URL = "Yours xxx"
EMB_API_KEY = "Yours xxx"
EMB_MODEL = "text-embedding-3-small"
EMB_DIM = 1536
```
### Run the toy example
```bash
python examples/hyperrag_demo.py
```
### Or Run by Steps
1. Prepare the data. You can download the dataset from here. Put the dataset in the root direction. Then run the following command to preprocess the data.
```bash
python reproduce/Step_0.py
```
2. Build the knowledge hypergraphs, and entity and relation vector database with following command.
```bash
python reproduce/Step_1.py
```
3. Extract questions from the orignial datasets with following command.
```bash
python reproduce/Step_2_extract_question.py
```
Those questions are saved in the `cache/{{data_name}}/questions` folder.
4. Run the Hyper-RAG to response those questions with following command.
```bash
python reproduce/Step_3_response_question.py
```
Those response are saved in the `cache/{{data_name}}/response` folder.
You can also change the `mode` parameter to `hyper` or `hyper-lite` to run the Hyper-RAG or Hyper-RAG-Lite.
### Hypergraph Visualization
We provide a web-based visualization tool for hypergraphs and lightweight Hyper-RAG QA system. For more information, please refer to [Hyper-RAG Web-UI](./web-ui/README.md).
*Note: The web UI is still under development and may not be fully functional. We welcome any contributions to improve it.*
## :checkered_flag: Evaluation
In this work, we propose two evaluation strategys: the **selection-based** and **scoring-based** evaluation.
### Scoring-based evaluation
Scoring-Based Assessment is designed to facilitate the comparative evaluation of multiple model outputs by quantifying their performance across various dimensions. This approach allows for a nuanced assessment of model capabilities by providing scores on several key metrics. However, a notable limitation is its reliance on reference answers. In our preprocessing steps, we leverage the source chunks from which each question is derived as reference answers.
You can use the following command to use this evaluation method.
```bash
python evaluate/evaluate_by_scoring.py
```
The results of this evaluation are shown in the following figure.
### Selection-based evaluation
Selection-Based Assessment is tailored for scenarios where preliminary candidate models are available, enabling a comparative evaluation through a binary choice mechanism. This method does not require reference answers, making it suitable for diverse and open-ended questions. However, its limitation lies in its comparative nature, as it only allows for the evaluation of two models at a time.
You can use the following command to use this evaluation method.
```bash
python evaluate/evaluate_by_selection.py
```
The results of this evaluation are shown in the following figure.
### Efficiency Analysis
We conducted an efficiency analysis of our Hyper-RAG method using GPT-4o mini on the NeurologyCrop dataset, comparing it with standard RAG, Graph RAG, and Light RAG. To ensure fairness by excluding network latency, we measured only the local retrieval time for relevant knowledge and the construction of the prior knowledge prompt. While standard RAG focuses on the direct retrieval of chunk embeddings, Graph RAG, Light RAG, and Hyper-RAG also include retrieval from node and correlation vector databases and the time for one layer of graph or hypergraph information diffusion. We averaged the response times over 50 questions from the dataset for each method. The results are shown in the following figure.
## :memo: License
This project is under license from Apache 2.0. For more details, see the [LICENSE](LICENSE.md) file.
Hyper-RAG is maintained by [iMoon-Lab](http://moon-lab.tech/), Tsinghua University.
Made with :heart: by Yifan Feng, Hao Hu, Xingliang Hou, Shiquan Liu, Yifan Zhang, Xizhe Yu.
If you have any questions, please feel free to contact us via email: [Yifan Feng](mailto:[email protected]).
This repo benefits from [LightRAG](https://github.com/HKUDS/LightRAG) and [Hypergraph-DB](https://github.com/iMoonLab/Hypergraph-DB). Thanks for their wonderful works.
## 🌟Citation
```
@misc{feng2025hyperrag,
title={Hyper-RAG: Combating LLM Hallucinations using Hypergraph-Driven Retrieval-Augmented Generation},
author={Yifan Feng and Hao Hu and Xingliang Hou and Shiquan Liu and Shihui Ying and Shaoyi Du and Han Hu and Yue Gao},
year={2025},
eprint={2504.08758},
archivePrefix={arXiv},
primaryClass={cs.IR},
url={https://arxiv.org/abs/2504.08758},
}
```