{"id":18832975,"url":"https://github.com/declare-lab/robustmift","last_synced_at":"2025-04-14T04:31:53.455Z","repository":{"id":223853343,"uuid":"761721691","full_name":"declare-lab/RobustMIFT","owner":"declare-lab","description":"[Arxiv 2024] Official Implementation of the paper: \"Towards Robust Instruction Tuning on Multimodal Large Language Models\"","archived":false,"fork":false,"pushed_at":"2024-06-22T08:46:38.000Z","size":70828,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-27T18:21:39.824Z","etag":null,"topics":["instruction-following","large-language-models","llm","llm-robustness","multimodal-instruction-tuning","multimodal-large-language-models"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/declare-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-22T11:14:17.000Z","updated_at":"2024-12-15T04:12:07.000Z","dependencies_parsed_at":"2024-05-15T21:28:12.397Z","dependency_job_id":null,"html_url":"https://github.com/declare-lab/RobustMIFT","commit_stats":null,"previous_names":["declare-lab/instr-auger","declare-lab/instraug","declare-lab/robustmift"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FRobustMIFT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FRobustMIFT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FRobustMIFT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2FRobustMIFT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/declare-lab","download_url":"https://codeload.github.com/declare-lab/RobustMIFT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248821749,"owners_count":21166950,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["instruction-following","large-language-models","llm","llm-robustness","multimodal-instruction-tuning","multimodal-large-language-models"],"created_at":"2024-11-08T01:59:44.425Z","updated_at":"2025-04-14T04:31:48.446Z","avatar_url":"https://github.com/declare-lab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Robust Instruction Tuning on MLLMs\nOfficial Implementation of the paper: [InstrAug: Automatic Instruction Augmentation for Multimodal Instruction Fine-tuning](https://arxiv.org/pdf/2402.14492.pdf)\n\n## Introduction\nInstrAug is a framework for instruction augmentation. It can expand extant small instruction set to __up to 30x__ larger.\nThe whole pipeline of InstrAug includes (as illustrated in the figure below):\n1. Meta-prompt Generation\n2. Augmented Instruction Generation and Rule-based Filtering\n    * Multi-temp sampling  ($\\rm MIns+_{\\rm MT}$)\n    * Iterative rephrasing ($\\rm MIns+_{\\rm Iter}$)\n3. Instruction-following Dataset Construction\n\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"assets/Framework.png\"/\u003e\n    \u003cbr\u003e\n\u003cp\u003e\n\nWe apply InstrAug to Multimodal Instruction Fine-tuning (MIFT) benchmarks and test on 12 downstream tasks from __MultiInstruct__ and __InstrutBLIP-Bench__ and the whole __MMMU benchmark__.\nThe results show that the model's capability on instruction-augmented dataset (59K) is competitive to or even exceeds non-augmented but larger datasets (564K).\n\n## Repo Hierarchy\nThe file structure in this repository is as below, we only show important folders/files\n\n    .\n    ├── IBLIP                   # Implementation code on Instruct-BLIP\n    ├── OFA                     # Implementation code on OFA\n    ├── MultiInstruct           # Code to create MINS+\n        ├──llama                # Code to generate augmented instructions using LLaMA\n        ├──mminstr_dataset      # folder to store MINS and MINS+ dataset \n        └──instruction_data     # folder to store original and generated instruction set \n    ├── LICENSE\n    └── README.md\n\n## Usage\nPlease refer to the README.md under individual folder for more details.\n\n## Results\n### 1. Results on MultiInstruct\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"assets/res_mins.png\"/\u003e\n    \u003cbr\u003e\n\u003cp\u003e\n\n### 2. Results on IBLIP-Bench\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"assets/res_iblip.png\"/\u003e\n    \u003cbr\u003e\n\u003cp\u003e\n\n### 3. Results on MMMU\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"assets/res_mmmu.png\"/\u003e\n    \u003cbr\u003e\n\u003cp\u003e\n\n## Citation\nPlease cite our paper if you find this work useful for your research and applications\n\n```bibtex\n@misc{han2024robust,\n      title={Towards Robust Instruction Tuning on Multimodal Large Language Models}, \n      author={Wei Han and Hui Chen and Soujanya Poria},\n      year={2024},\n      eprint={2402.14492},\n      archivePrefix={arXiv},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Frobustmift","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeclare-lab%2Frobustmift","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Frobustmift/lists"}