{"id":19984479,"url":"https://github.com/intellabs/rag-fit","last_synced_at":"2025-05-15T08:10:50.552Z","repository":{"id":251810531,"uuid":"832661327","full_name":"IntelLabs/RAG-FiT","owner":"IntelLabs","description":"Framework for enhancing LLMs for RAG tasks using fine-tuning.","archived":false,"fork":false,"pushed_at":"2025-02-20T11:01:03.000Z","size":947,"stargazers_count":737,"open_issues_count":0,"forks_count":56,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-14T14:59:38.890Z","etag":null,"topics":["evaluation","fine-tuning","information-retrieval","llm","nlp","question-answering","rag","semantic-search"],"latest_commit_sha":null,"homepage":"https://intellabs.github.io/RAG-FiT/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IntelLabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-23T13:20:35.000Z","updated_at":"2025-04-12T16:18:42.000Z","dependencies_parsed_at":"2024-10-08T00:02:55.950Z","dependency_job_id":"473f0967-2499-45a0-a48c-3532e814fab6","html_url":"https://github.com/IntelLabs/RAG-FiT","commit_stats":{"total_commits":30,"total_committers":4,"mean_commits":7.5,"dds":"0.16666666666666663","last_synced_commit":"c367036ea36b29c0fb9fbdff7aec20eda92c31cf"},"previous_names":["intellabs/ragfoundry","intellabs/rag-fit"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2FRAG-FiT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2FRAG-FiT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2FRAG-FiT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2FRAG-FiT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IntelLabs","download_url":"https://codeload.github.com/IntelLabs/RAG-FiT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254301432,"owners_count":22047904,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation","fine-tuning","information-retrieval","llm","nlp","question-answering","rag","semantic-search"],"created_at":"2024-11-13T04:19:08.699Z","updated_at":"2025-05-15T08:10:50.530Z","avatar_url":"https://github.com/IntelLabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"assets/rag_fit.png\" width=\"400\"/\u003e\n\u003c/div\u003e\n\n----------\n\n**RAG-FiT** is a library designed to improve LLMs ability to use external information by fine-tuning models on\nspecially created RAG-augmented datasets. The library helps create the data for training, given a RAG technique, helps\neasily train models using parameter-efficient finetuning (PEFT), and finally can help users measure the improved\nperformance using various, RAG-specific metrics. The library is modular, workflows are customizable using configuration\nfiles. Formerly called *RAG Foundry*.\n\nComments, suggestions, issues and pull-requests are welcomed! ❤️\n\n### Installation\nClone and run:\n\n```sh\npip install -e .\n```\n\nOptional packages can be installed:\n```sh\npip install -e .[haystack]\npip install -e .[deepeval]\n```\n\n### Quick Start\n\nFor a simple, end-to-end example, see the [PubmedQA Tutorial](./docs/pubmed.md).\n\n## Overview\n\nThe RAG-FiT framework facilitates fast prototyping and experimentation with various RAG settings and configurations,\nincluding data selection and filtering, processing, retrieval, ranking, query manipulation, prompt generation, training,\ninference, output processing and evaluation. The library is comprised of 4 modules: dataset creation, training,\ninference and evaluation.\n\n* **Dataset Creation**: The processing module creates datasets, persisting RAG interactions, to be used for RAG training\nand inference. RAG interactions include dataset loading, columns normalization, data aggregation (fewshot creation),\ninformation retrieval using external tools and frameworks, API integration, template-based prompt creation and any other\nform of pre-processing. The data is saved in a consistent, model-independent, input-output format, along with all other\nfields and metadata. See [Processing.md](docs/processing.md).\n\n* **Training**: using PEFT for efficient training and TRL (e.g. supervised FT) users can train any model on the augmented\ndatasets. Training is done on the completions. Models can be pushed to HF Hub. See [Training.md](docs/training.md).\n\n* **Inference**: generating predictions using the augmented datasets with trained or untrained LLMs. See [Inference.md](docs/inference.md).\n\n* **Evaluation**: running evaluation on the generated output from the inference module. Users can provide a list of\nmetrics to run; custom metrics can be implemented easily. Current metrics include EM, F1, ROUGE, BERTScore, Deepeval,\nRAGAS, HF `evaluate` and classification. Metrics can be *local*—run on each example, or *global*—run on the entire\ndataset, e.g. recall. Metrics can utilize any feature in the dataset, like retrieval results, reasoning,\ncitations and attributions, not just the input and output texts. See [Evaluation.md](docs/evaluation.md).\n\n\n## Running\nThe 4 modules are represented as scripts: `processing.py`, `training.py`, `inference.py` and `evaluation.py` at the top\nlevel. Every call has the form `python SCRIPT options...`.\n\nThe library utilizes the [Hydra](https://hydra.cc/docs/intro/) configuration tool; it enables the use of hierarchical\nconfigurations, easily overridden of values in the CLI and the ability to run multiple jobs remotely (e.g. integrations with\nSLURM and Ray). It represents a *configuration-as-code* approach, as it can instantiate python classes according to\nconfiguration (the `_target_` keyword indicates the python class to use in a given context).\n\nThere are default configurations for each module in the [configs](./configs/) folder. A configuration file can be\noverridden like so:\n\n```sh\npython processing -cp configs/paper -cn processing-asqa-retrieval\n```\n\nIndividual keywords can be overridden as well:\n```sh\npython processing -cp configs/paper -cn processing-asqa-retrieval   \\\n       output_path=/store/data/here                                 \\\n       cache=true\n```\n\nFor a complete set of configurations, **reproducing the experimentation in the paper with the ASQA dataset**, see the\nconfigurations in the [Paper](./configs/paper) folder.\n\n## Citation\n\nPlease cite our paper if it helps your research: [RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation](https://arxiv.org/abs/2408.02545).\n\n```BibTex\n@article{fleischerRAGFoundryFramework2024,\n  title =        {{RAG} {Foundry}: {A} {Framework} for {Enhancing} {LLMs} for {Retrieval} {Augmented} {Generation}},\n  author =       {Fleischer, Daniel and Berchansky, Moshe and Wasserblat, Moshe and Izsak, Peter},\n  year =         2024,\n  note =         {arXiv:2408.02545 [cs]},\n  annote =       {Comment: 10 pages},\n  url =          {http://arxiv.org/abs/2408.02545},\n  publisher =    {arXiv},\n}\n```\n\n## License\n\nThe code is licensed under the [Apache 2.0 License](LICENSE).\n\n## Disclaimer\n\nThis is not an official Intel product.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintellabs%2Frag-fit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintellabs%2Frag-fit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintellabs%2Frag-fit/lists"}