{"id":31034663,"url":"https://github.com/freedomintelligence/rag-instruct","last_synced_at":"2025-09-14T02:46:51.579Z","repository":{"id":271102828,"uuid":"910404012","full_name":"FreedomIntelligence/RAG-Instruct","owner":"FreedomIntelligence","description":"RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions","archived":false,"fork":false,"pushed_at":"2025-04-13T02:57:29.000Z","size":12992,"stargazers_count":137,"open_issues_count":0,"forks_count":4,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-13T03:40:35.885Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-31T07:19:25.000Z","updated_at":"2025-04-13T02:57:33.000Z","dependencies_parsed_at":"2025-04-13T03:32:33.711Z","dependency_job_id":"5bcb8d61-d6c9-415e-a7e1-38c290669b52","html_url":"https://github.com/FreedomIntelligence/RAG-Instruct","commit_stats":null,"previous_names":["freedomintelligence/rag-instruct"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FreedomIntelligence/RAG-Instruct","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FRAG-Instruct","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FRAG-Instruct/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FRAG-Instruct/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FRAG-Instruct/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/RAG-Instruct/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FRAG-Instruct/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275054971,"owners_count":25397576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-14T02:00:10.474Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-14T02:46:45.172Z","updated_at":"2025-09-14T02:46:51.562Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions\n\u003cdiv align=\"center\"\u003e\n\u003ch3\u003e\n  RAG-Instruct\n\u003c/h3\u003e\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n📃 \u003ca href=\"assets/paper.pdf\" target=\"_blank\"\u003ePaper\u003c/a\u003e ｜🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/RAG-Instruct-Llama3-3B\" target=\"_blank\"\u003eRAG-Instruct-Llama3-3B\u003c/a\u003e ｜🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/RAG-Instruct-Llama3-8B\" target=\"_blank\"\u003eRAG-Instruct-Llama3-8B\u003c/a\u003e ｜  📚 \u003ca href=\"https://huggingface.co/datasets/FreedomIntelligence/RAG-Instruct\" target=\"_blank\"\u003eRAG-Instruct Dataset\u003c/a\u003e\n\u003c/p\u003e\n\n\n## ⚡ Introduction\nHello! Welcome to the repository for [RAG-Instruct](https://arxiv.org/abs/2501.00353)!\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"assets/RAG-Instruct.png\"  width = \"90%\" alt=\"RAG-Instruct\" align=center/\u003e\n\u003c/div\u003e\n\n\n**RAG-Instruct** is a method for generating diverse and high-quality RAG instruction data. It synthesizes instruction datasets based on any source corpus, leveraging the following approaches:\n\n- **Five RAG paradigms**, which represent diverse query-document relationships to enhance model generalization across tasks.\n- **Instruction simulation**, which enriches instruction diversity and quality by utilizing the strengths of existing instruction datasets.\n\nUsing this approach, we constructed a 40K instruction dataset from Wikipedia, covering a wide range of RAG scenarios and tasks. \nOur RAG-Instruct significantly enhances the RAG ability of LLMs, demonstrating remarkable improvements in RAG performance across various tasks.\n\n| Model                          | WQA (acc) | PQA (acc) | TQA (acc) | OBQA (EM) | Pub (EM) | ARC (EM) | 2WIKI (acc) | HotP (acc) | MSQ (acc) | CFQA (EM) | PubMed (EM) |\n|--------------------------------|-----------|-----------|-----------|-----------|----------|----------|-------------|------------|-----------|-----------|-------------|\n| Llama3.2-3B                    | 58.7 | 61.8 | 69.7 |  77.0 | 55.0 | 66.8 | 55.6 | 40.2 | 13.2 | 46.8 | 70.3 |\n| Llama3.1-8B                    | 59.5                      | 60.8                | 73.4               |  82.0                           | 56.7                    | 77.1                    | 65.6                 | 45.6           | 18.7            | 56.5                     | 73.9                    |\n| Llama3.2-3B + **RAG-Instruct**     | 65.3                      | 64.0                | 77.0               |  81.2                           | 66.4                    | 73.0                    | 72.9                 | 52.7           | 25.0            | 50.3                     | 72.6                    |\n| Llama3.1-8B + **RAG-Instruct**     | 69.7                      | 68.4                | 79.3               |  84.8                           | 77.2                    | 79.9                    | 79.3                 | 56.4           | 30.3            | 57.8                     | 77.0                    |\n\n\nWe open-sourced our models, data, and code here.\n\n\u003c!-- ## 💭 Environment\nYou can create a conda environment by running the command below.\n```\npip install -r requirements.txt\n``` --\u003e\n\n## 💻 Model\n- **Model Access**\n\n|          Model Name                  | Base LLMs       | Link                                                                         |\n| -------------------------- | ------------ | ---------------------------------------------------------------------------- |\n| **RAG-Instruct-Llama3-3B** | LLaMA-3.2-3B | [HF Link](https://huggingface.co/FreedomIntelligence/RAG-Instruct-Llama3-3B) |\n| **RAG-Instruct-Llama3-8B** | LLaMA-3.1-8B | [HF Link](https://huggingface.co/FreedomIntelligence/RAG-Instruct-Llama3-8B) |\n\n\n- **Deploy**\n\nRAG-Instruct models can be used just like `Llama-3.1-8B-Instruct`. You can deploy it with tools like [vllm](https://github.com/vllm-project/vllm) or [Sglang](https://github.com/sgl-project/sglang),  or perform direct inference:\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load the model and tokenizer\nmodel = AutoModelForCausalLM.from_pretrained(\"FreedomIntelligence/RAG-Instruct-Llama3-8B\",torch_dtype=\"auto\",device_map=\"auto\")\ntokenizer = AutoTokenizer.from_pretrained(\"FreedomIntelligence/RAG-Instruct-Llama3-8B\")\n\n# Example input\ninput_text = \"\"\"### Paragraph:\n[1] structure is at risk from new development...\n[2] as Customs and Excise stores...\n[3] Powis Street is partly underway...\n...\n\n### Instruction:\nWhich organization is currently using a building in Woolwich that holds historical importance?\n\"\"\"\n\n# Tokenize and prepare input\nmessages = [{\"role\": \"user\", \"content\": input_text}]\ninputs = tokenizer(tokenizer.apply_chat_template(messages, tokenize=False,add_generation_prompt=True), return_tensors=\"pt\").to(model.device)\n\n# Generate output\noutputs = model.generate(**inputs, max_new_tokens=2048)\nprint(tokenizer.decode(outputs[0], skip_special_tokens=True))\n```\n\n\n## 📚 Data\nWe’ve open-sourced a 40K instruction dataset for RAG. Download it here:\n\n| Data                  | Description                                                                                   | Link                                                                                           |\n| -------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |\n| RAG-Instruct (Wikipedia) | Diverse RAG instruction data based on Wikipedia | [Link](https://huggingface.co/datasets/FreedomIntelligence/RAG-Instruct)  |\n\n\n## 🛠️ Data Construction\n\nWe provide scripts to **synthesize a diverse RAG instruction dataset**.\n\n**1. Download Source Documents.**  \nWe use preprocessed passage data from DPR and embeddings generated with [Contriever-MSMARCO](https://github.com/facebookresearch/contriever) :\n\n- Download the preprocessed passage data:\n  ```bash\n  cd retrieval_lm\n  wget https://dl.fbaipublicfiles.com/dpr/wikipedia_split/psgs_w100.tsv.gz\n  ```\n  \n- Download the generated embeddings:\n  ```bash\n  wget https://dl.fbaipublicfiles.com/contriever/embeddings/contriever-msmarco/wikipedia_embeddings.tar\n  ```\n\n**2. Prepare Exemplar Datasets.**  \n\nWe utilize several high-quality datasets as exemplars, including [ShareGPT](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca), [WizardLM-70K](https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_V70K), [Lmsys-chat-1M](https://huggingface.co/datasets/lmsys/lmsys-chat-1m), and [SlimOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca).\n\nTo ensure high-quality data, we filtered and sampled these datasets using GPT-4o to extract **knowledge-intensive data** (Q). Using the exemplar data (Q), we retrieve source documents to construct (D*). Specifically, we match the exemplar instructions or questions with source documents by ranking their relevance. For convenience, we provide a processed dataset containing source documents and exemplar data across five RAG scenarios [here](data_gen/examplar_data/data.json).\n\n**3. Synthesize Data with Prompts.**  \nUsing the retrieved documents (D*) and exemplar data (Q), we synthesize new data points with tailored prompts to create diverse and high-quality instruction-following datasets.\n\n```bash\ncd data_gen\npython generate_data.py \\\n    --data_path examplar_data/data.json \\\n    --max_workers 16 \\\n    --save_dir ./output_data/RAG-Instruct.json\n```\n\n**4. Run Retriever**  \nBefore training, we need to perform retrieval on the synthesized RAG-Instruct dataset. For each data entry, we ensure that the retrieval documents includes all source documents (D*) and supplement them with enough unrelated documents (D-) to total 10 documents.\nWe use preprocessed passage data from DPR and embeddings generated with [Contriever](https://github.com/facebookresearch/contriever). To retrieve noisy documents (D-), use the following command:\n\n```bash\ncd retrieval_lm\npython passage_retrieval.py \\\n    --model_name_or_path facebook/contriever-msmarco \\\n    --passages psgs_w100.tsv \\\n    --passages_embeddings \"wikipedia_embeddings/*\" \\\n    --input_name RAG_INSTRCT_DATA_PATH \\\n    --output_dir YOUR_OUTPUT_FILE \\\n    --n_docs 250\n```\n\n`RAG_INSTRUCT_DATA_PATH` is the final location of the synthesized `RAG-Instruct.json` file. The input file must be in `json` or `jsonl` format. Each instance should include either a `question` or `instruction` field, which will be used as the query during retrieval. \n\nNext, we sample documents ranked beyond the top 200 as (D-) and get the final training data. \n\n## 🚀 Training\n\n**Fine-tuning with RAG-Instruct**\n\nYou can fine-tune your large model using the `RAG-Instruct` dataset to significantly boost RAG capabilities. Use the following code:\n\n```bash\naccelerate launch --config_file ./configs/sft.yaml \\\n    --num_processes 8  \\\n    --num_machines 1 \\\n    --machine_rank 0 \\\n    --deepspeed_multinode_launcher standard train_rag_sft.py \\\n    --experiment_name RAG-Instruct-training \\\n    --model_path meta-llama/Llama-3.1-8B-Instruct \\\n    --data_path FreedomIntelligence/RAG-Instruct \\\n    --max_seq_len 4096 \\\n    --learning_rate 5e-6 \\\n    --train_bsz_per_gpu 2 \\\n    --gradient_accumulation_steps 16 \\\n    --output_dir ./ckpts \\\n    --log_dir ./train_logs \\\n    --n_epochs 3 \\\n    --gradient_checkpointing\n```\n\n\n## 🧐 Evaluation\n1. You first need to install [Sglang](https://github.com/sgl-project/sglang). After installation, deploy the model you want to test using Sglang with the following command:\n```bash\nlog_num=0\nmodel_name=\"FreedomIntelligence/RAG-Instruct-Llama3-3B\" # Path to the model you are deploying\nport=21${log_num}35\nCUDA_VISIBLE_DEVICES=0  python -m sglang.launch_server --model-path $model_name --port $port --mem-fraction-static 0.8 --dp 1 --tp 1  \u003e sglang${log_num}.log 2\u003e\u00261 \u0026\n```\n2. Wait for the model to be deployed. After deployment, you can run the following code for evaluation. \n```bash\nmodel_name=\"FreedomIntelligence/RAG-Instruct-Llama3-3B\" # Path to the model you are deploying\npython eval/eval_sglang.py --model_name $model_name --input_file eval/data/eval_data.json --port $port --max_new_tokens 500  \n```\nHere, we provide the evaluation example using the PopQA dataset in the file `eval/data/eval_data.json`. For other evaluation datasets, please first use the retriever to retrieve (You can refer to the retriever code in the training section), and then use the above script for evaluation.\n\n3. After completing the evaluation, run the following code to stop the Sglang service and release GPU memory.\n```bash\nbash evaluation/kill_sglang_server.sh\n```\nThe evaluation code above can be used to test most models supported by Sglang.\n\n\n## 📖 Citation\n```\n@misc{liu2024raginstructboostingllmsdiverse,\n      title={RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions}, \n      author={Wanlong Liu and Junying Chen and Ke Ji and Li Zhou and Wenyu Chen and Benyou Wang},\n      year={2024},\n      eprint={2501.00353},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2501.00353}, \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Frag-instruct","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Frag-instruct","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Frag-instruct/lists"}