{"id":20298105,"url":"https://github.com/QingFei1/LongRAG","last_synced_at":"2025-05-07T20:34:16.667Z","repository":{"id":261633085,"uuid":"867269098","full_name":"QingFei1/LongRAG","owner":"QingFei1","description":"[EMNLP 2024] LongRAG: A Dual-perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering","archived":false,"fork":false,"pushed_at":"2025-01-29T05:22:18.000Z","size":2200,"stargazers_count":90,"open_issues_count":1,"forks_count":12,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-01-29T06:23:33.635Z","etag":null,"topics":["llm","long-context","rag"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2410.18050","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/QingFei1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-03T18:39:29.000Z","updated_at":"2025-01-29T05:22:21.000Z","dependencies_parsed_at":"2024-11-07T16:24:50.115Z","dependency_job_id":"61b42d38-60eb-422c-8b8e-c49fbc3e2737","html_url":"https://github.com/QingFei1/LongRAG","commit_stats":null,"previous_names":["qingfei1/longrag"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QingFei1%2FLongRAG","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QingFei1%2FLongRAG/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QingFei1%2FLongRAG/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QingFei1%2FLongRAG/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/QingFei1","download_url":"https://codeload.github.com/QingFei1/LongRAG/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252953717,"owners_count":21830890,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","long-context","rag"],"created_at":"2024-11-14T16:02:09.920Z","updated_at":"2025-05-07T20:34:16.654Z","avatar_url":"https://github.com/QingFei1.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话"],"sub_categories":["大语言对话模型及数据"],"readme":"# [EMNLP2024] LongRAG: A Dual-perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering\n\n**LongRAG** is a general, dual-perspective, and robust LLM-based RAG system paradigm for LCQA to en-hance RAG’s understanding of complex long-context knowledge (i.e., global information and factual details)\n\n![LongRAG](assets/overall.png)\n\n\n## ⚙️ Environmental Setup\nInstall the requirements with pip: `pip install -r requirements.txt`. We recommend using FlashAttention 2 for optimization and saving GPU memory. The relevant dependencies can be installed according to the code base of [FlashAttention](https://github.com/Dao-AILab/flash-attention).\n\n\n## ⚙️ Data Preparation\n\nOur raw training data comes from [HotpotQA,2WikiMultihopQA,MuSiQue](https://github.com/StonyBrookNLP/ircot) and [Qasper](https://allenai.org/data/qasper). The evaluation data and the corresponding retrieval corpus raw data are sourced from [LongBench](https://github.com/THUDM/LongBench).\n\nWe have standardized the data format for the aforementioned datasets. You can download our standardized raw datasets by running the following command:\n\n```bash\nbash download/raw_data.sh\n```\n\nThe data will be downloaded in the `data/`.\n\n\n\n### Data Processing\n\nBuild the LRGinstruction dataset for SFT:\n\n```bash\ncd src\npython gen_instruction.py --per_task_num 200 --min_res_tokens 20 --long_ratio 0.2\n```\n\nSave the processed data in `data/train/processed`.\n\nBuild an index for retrieval and save the mapping relationship between chunks and the original text:\n\n```bash\ncd src\npython gen_index.py --dataset hotpotqa --chunk_size 200 --min_sentence 2 --overlap 2\n```\n\nSave the processed data in `data/corpus/processed`.\n\n## 🖥️ LongRAG Training\n\nFirst, you need to download [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory/tree/v0.6.3) to our project. Then put our constructed instruction data into `LLaMA-Factory/data` and add the following entry to `dataset_info.json`:\n\n```json\n\"LRGinstruction\": {\n  \"file_name\": \"LRGinstruction.json\",\n  \"columns\": {\n    \"prompt\": \"instruction\",\n    \"query\": \"input\",\n    \"response\": \"output\"\n  }\n}\n```\n\nThen run the following script to start fine-tuning:\n\n```bash\ncd scripts\nbash sft.sh $model_name_or_path $template $cutoff_len\n```\n\n`model_name_or_path` should correspond to the [template](https://github.com/hiyouga/LLaMA-Factory/tree/v0.6.3), and `cutoff_len` is the truncation length.\n\n## 📊 Evaluation\n\nHere are some example scripts for performing inference and evaluation on HotpotQA. To get started, first navigate to the `src` directory.\n\n### Using different methods\n\nWe provide examples of inference using the ChatGLM3-6B-32k model.\n\n**LongRAG-ChatGLM3-6B-32k (without SFT)**:\n```bash\nCUDA_VISIBLE_DEVICES=0 python main.py --dataset hotpotqa --model chatGLM3-6b-32k --rb --rl --ext --fil --ext_fil \n```\n\n**LongRAG-ChatGLM3-6B-32k (with SFT)**:\n```bash\nCUDA_VISIBLE_DEVICES=0 python main.py --dataset hotpotqa --model LongRAG-chatglm3-32k --rb --rl --ext --fil --ext_fil \n```\n\n### Component Transferability\n\nUsing only the Extractor, with the generator using GPT-3.5-turbo and the Extractor using LongRAG-chatglm3-32k:\n```bash\nCUDA_VISIBLE_DEVICES=0,1 python main.py --dataset hotpotqa --model gpt-3.5-turbo --lrag_model LongRAG-chatglm3-32k --ext \n```\n\nUsing only the Filter, with the generator using GPT-3.5-turbo and the Filter using LongRAG-chatglm3-32k:\n```bash\nCUDA_VISIBLE_DEVICES=0,1 python main.py --dataset hotpotqa --model gpt-3.5-turbo --lrag_model LongRAG-chatglm3-32k --fil \n```\n\nUsing both Extractor \u0026 Filter, with the generator using GPT-3.5-turbo and both the Extractor \u0026 Filter using LongRAG-chatglm3-32k:\n```bash\nCUDA_VISIBLE_DEVICES=0,1 python main.py --dataset hotpotqa --model gpt-3.5-turbo --lrag_model LongRAG-chatglm3-32k --ext_fil \n```\n\nNote: The parameters `--rb`, `--rl`, `--ext`, `--fil`, and `--ext_fil` represent running RAG-Base, RAG-Long, Extractor, Filter, and Extractor \u0026 Filter, respectively. These parameters can be combined arbitrarily.\n\nEvaluation results will be saved in the `log` directory.\n\n### Evaluation Result on Each Dataset\nBelow are partial experimental results, showcasing the F1 scores on three multi-hop datasets from [LongBench](https://github.com/THUDM/LongBench), using the LongRAG paradigm.\n\u003e Note: Following the LongBench settings, for text that exceeds the model's processing length, we truncate it from the middle of the text and retain the beginning and end information.\n\n|                   | HotpotQA | 2WikiMultiHopQA  | MusiQue |Avg |\n| ----------------- | :-----------: | :----------: | :-----------: | :-----: |\n| LongRAG-Qwen-1.5-7B-32k w/ SFT| 52.91 | 46.65 | 31.85 | 43.80 |\n| LongRAG-Llama3-8B-8k w/ SFT| 52.39 | 49.67 | 31.70 | 44.59 |\n| LongRAG-Vicuna-v1.5-7B-16k w/ SFT| 55.55 | 50.13 | 28.29 | 44.66 |\n| LongRAG-ChatGLM3-6B-32k w/ SFT| 55.93 | 54.85 | 33.00 | 47.93 |\n| LongRAG-GPT-3.5-Turbo w/o SFT| 56.17 | 51.37 | 32.83 | 46.79 |\n| LongRAG-GPT-3.5-Turbo-16k w/o SFT| 59.11 | 51.25 | 30.37 | 46.91 |\n| LongRAG-GLM-4 w/o SFT| 62.11 | 57.16 | 38.40 | 52.56 |\n\n## 📝 Citation\n\nIf you find our work useful, please consider citing LongRAG:\n\n```\n@article{zhao2024longrag,\n  title={LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering},\n  author={Qingfei Zhao and Ruobing Wang and Yukuo Cen and Daren Zha and Shicheng Tan and Yuxiao Dong and Jie Tang},\n  journal={arXiv preprint arXiv:2410.18050},\n  year={2024}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FQingFei1%2FLongRAG","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FQingFei1%2FLongRAG","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FQingFei1%2FLongRAG/lists"}