https://github.com/yixuantt/MultiHop-RAG

Repository for "MultiHop-RAG: A Dataset for Evaluating Retrieval-Augmented Generation Across Documents" (COLM 2024)
https://github.com/yixuantt/MultiHop-RAG

Last synced: 15 days ago
JSON representation

Repository for "MultiHop-RAG: A Dataset for Evaluating Retrieval-Augmented Generation Across Documents" (COLM 2024)

• Host: GitHub
• URL: https://github.com/yixuantt/MultiHop-RAG
• Owner: yixuantt
• Created: 2024-01-23T07:00:54.000Z (12 months ago)
• Default Branch: main
• Last Pushed: 2024-11-19T09:24:38.000Z (about 2 months ago)
• Last Synced: 2024-11-19T10:27:14.804Z (about 2 months ago)
• Language: Python
• Homepage:
• Size: 7.82 MB
• Stars: 205
• Watchers: 3
• Forks: 14
• Open Issues: 6
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• Awesome-LLMs-Datasets - https://github.com/yixuantt/MultiHop-RAG/
• StarryDivineSky - yixuantt/MultiHop-RAG - RAG：用于评估跨文档检索增强生成的数据集”（COLM 2024） 的存储库。MultiHop-RAG：一个 QA 数据集，用于评估 RAG 管道中元数据跨文档的检索和推理。它包含 2556 个查询，每个查询的证据分布在 2 到 4 个文档中。查询还涉及文档元数据，反映了实际 RAG 应用程序中常见的复杂场景。 (A01_文本生成_文本对话 / 大语言对话模型及数据)

README

        
# 💡 MultiHop-RAG

A Dataset for **Evaluating Retrieval-Augmented Generation Across Documents**  

   

## 🚀 Overview

**MultiHop-RAG**: a QA dataset to evaluate retrieval and reasoning across documents with metadata in the RAG pipelines. It contains 2556 queries, with evidence for each query distributed across 2 to 4 documents. The queries also involve document metadata, reflecting complex scenarios commonly found in real-world RAG applications.  

📄 Paper Link **(Accepted by COLM 2024)**: [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://arxiv.org/pdf/2401.15391.pdf)  

🤗 [Hugging Face dataloader](https://huggingface.co/datasets/yixuantt/MultiHopRAG)

![rag.png](resource/rag.png)

## Simple Use Case

**1. For Retrieval**

Please try '**simple_retrieval.py**,' a sample use case demonstrating retrieval using this dataset. 

```

pip install llama-index==0.9.40

```

```shell

# test simple retrieval and save results

python simple_retrieval.py --retriever BAAI/llm-embedder

# test simple retrieval with rerank and save results

python simple_retrieval.py --retriever BAAI/llm-embedder --rerank

```

**2. For QA**

Please try '**qa_llama.py**,' a sample use case demonstrating query and answer with llama using this dataset. 

```

python qa_llama.py

```

## Evaluation

**1. For Retrieval**: 'retrieval_evaluate.py' 

**2. For QA**: 'qa_evaluate.py' 

```

python retrieval_evaluate.py --file {saved_file_path}

```

## Construction Pipeline

For research purposes, we open-sourced part of the code to construct the dataset. However, the current structure of the code is not very tidy. We will organize it in the future.

💡 Just For Reference: pipeline/

## Citation

```

@misc{tang2024multihoprag,

      title={MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries}, 

      author={Yixuan Tang and Yi Yang},

      year={2024},

      eprint={2401.15391},

      archivePrefix={arXiv},

      primaryClass={cs.CL}

}

```

## License

MultiHop-RAG is licensed under [ODC-BY](https://opendatacommons.org/licenses/by/1-0/)