{"id":23587683,"url":"https://github.com/NVlabs/DoRA","last_synced_at":"2025-08-30T04:31:20.137Z","repository":{"id":234133546,"uuid":"785300720","full_name":"NVlabs/DoRA","owner":"NVlabs","description":"[ICML2024 (Oral)] Official PyTorch implementation of DoRA: Weight-Decomposed Low-Rank Adaptation","archived":false,"fork":false,"pushed_at":"2024-10-01T10:14:11.000Z","size":3211,"stargazers_count":797,"open_issues_count":17,"forks_count":57,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-06-11T23:06:58.312Z","etag":null,"topics":["commonsense-reasoning","deep-learning","deep-neural-networks","instruction-tuning","large-language-models","large-vision-language-models","lora","parameter-efficient-fine-tuning","parameter-efficient-tuning","vision-and-language"],"latest_commit_sha":null,"homepage":"https://nbasyl.github.io/DoRA-project-page/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-11T15:55:29.000Z","updated_at":"2025-06-09T23:57:29.000Z","dependencies_parsed_at":"2024-06-07T19:44:36.373Z","dependency_job_id":"d2609f7c-91b8-47ca-8579-cc7938ee893b","html_url":"https://github.com/NVlabs/DoRA","commit_stats":null,"previous_names":["nvlabs/dora"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/NVlabs/DoRA","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2FDoRA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2FDoRA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2FDoRA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2FDoRA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVlabs","download_url":"https://codeload.github.com/NVlabs/DoRA/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2FDoRA/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272805294,"owners_count":24995909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["commonsense-reasoning","deep-learning","deep-neural-networks","instruction-tuning","large-language-models","large-vision-language-models","lora","parameter-efficient-fine-tuning","parameter-efficient-tuning","vision-and-language"],"created_at":"2024-12-27T05:01:38.641Z","updated_at":"2025-08-30T04:31:19.145Z","avatar_url":"https://github.com/NVlabs.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话"],"sub_categories":["大语言对话模型及数据"],"readme":"\u003ch1 align=\"center\"\u003e\n    \u003cp\u003e DoRA: Weight-Decomposed Low-Rank Adaptation \u003cbr\u003e [ICML2024 (Oral)]\u003c/p\u003e\n\u003c/h1\u003e\n\n\u003ch1 align=\"center\"\u003e \n    \u003cimg src=\"./imgs/dora.png\" width=\"600\"\u003e\n\u003c/h1\u003e\n\nThe Official PyTorch implementation of [**DoRA: Weight-Decomposed Low-Rank Adaptation**](https://arxiv.org/abs/2402.09353) [ICML2024 (Oral, acceptance rate: ***1.5%***)].\n\n[![Star on GitHub](https://img.shields.io/github/stars/NVlabs/DoRA.svg?style=social)](https://github.com/NVlabs/DoRA/stargazers)\n\n[Shih-Yang Liu*](https://nbasyl.github.io/), [Chien-Yi Wang](https://chienyiwang.github.io/), [Hongxu Yin](https://hongxu-yin.github.io/), [Pavlo Molchanov](https://www.pmolchanov.com/), [Yu-Chiang Frank Wang](http://vllab.ee.ntu.edu.tw/ycwang.html), [Kwang-Ting Cheng](https://seng.hkust.edu.hk/about/people/faculty/tim-kwang-ting-cheng), [Min-Hung Chen](https://minhungchen.netlify.app/) \u003cbr\u003e\n(*Work done during the internship at NVIDIA Research)\n\n[[`Paper`](https://arxiv.org/abs/2402.09353)] [[`Website`](https://nbasyl.github.io/DoRA-project-page/)] [[`NV Blog`](https://developer.nvidia.com/blog/introducing-dora-a-high-performing-alternative-to-lora-for-fine-tuning/)] [[`BibTeX`](#citation)]\n\nDoRA decomposes the pre-trained weight into two components, *magnitude* and *direction*, for fine-tuning, specifically employing LoRA for directional updates to minimize the number of trainable parameters efficiently. By employing DoRA, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. DoRA consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding.\n\nFor business inquiries, please visit our website and submit the form: [NVIDIA Research Licensing](https://www.nvidia.com/en-us/research/inquiries/).\n\n## 💥 News 💥\n- **[07.02.2024]** 🔥🔥 The official NVIDIA Tech Blog of DoRA is released [HERE](https://developer.nvidia.com/blog/introducing-dora-a-high-performing-alternative-to-lora-for-fine-tuning/)!!\n- **[06.02.2024]** 🔥🔥 DoRA is selected as an ***Oral*** paper in [ICML 2024](https://icml.cc/Conferences/2024)!!\n- **[05.24.2024]** 🔥🔥 Add step-by-step instructions on how to reproduce the QDoRA/FSDP results, see [/QDoRA](https://github.com/NVlabs/DoRA/tree/main/QDoRA)\n- **[05.02.2024]** 🔥🔥 DoRA is accepted to [**ICML 2024**](https://icml.cc/Conferences/2024)!! See you in Vienna!!\n- **[04.27.2024]** 🔥🔥 We have added the source code and the DoRA weight for finetuning LLaMA2-7B and LLaMA3-8B on commonsense reasoning tasks!\n- **[04.22.2024]** 🔥🔥 Check out an awesome blog post [FSDP/QDoRA](https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html) from Answer.ai which shows that QDoRA significantly outperforms QLoRA and even edges out full finetuning!\n- **[04.18.2024]** 🔥🔥 We have released the source code and the DoRA weight for reproducing the results in our paper!\n- **[03.20.2024]** 🔥🔥 DoRA is now fully supported by the HuggingFace PEFT package and can now support Linear, Conv1d, and Conv2d layers, as well as linear layers quantized with bitsandbytes! \n\n## Useful Links\n\n- An amazing tutorial about implementing DoRA from scratch by Sebastian Raschka, see https://magazine.sebastianraschka.com/p/lora-and-dora-from-scratch \n- An amazing blog post from Answer.AI about QDoRA/FSDP which allows finetuning LLMs on consumer-level GPUs, see https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html\n\n## Quick Start and some tricks regarding finetuning with DoRA\n### HuggingFace PEFT\nDoRA is now supported by the [Huggingface PEFT package](https://github.com/huggingface/peft/releases/tag/v0.10.0). You can install the PEFT package using\n```bash\npip install git+https://github.com/huggingface/peft.git -q\n```\n\nAfter PEFT is installed, you can simply set the `use_dora` argument of `LoraConfig()` to `True` for applying DoRA.\n\nAn example could be as follows:\n```bash\nfrom peft import LoraConfig\n\n# Initialize DoRA configuration\nconfig = (\n    use_dora=True, ...\n)\n```\nPlease refer to the official [documentation](https://huggingface.co/docs/peft/en/developer_guides/lora#weight-decomposed-low-rank-adaptation-dora) for more details.\n\n### HuggingFace Diffusers\nYou can also toy with DoRA on finetuning diffusion models. See [huggingface/diffusers](https://github.com/huggingface/diffusers/tree/main/examples/advanced_diffusion_training#dora-training). Another good tutorial would be this [Colab notebook](https://colab.research.google.com/drive/134mt7bCMKtCYyYzETfEGKXT1J6J50ydT?usp=sharing#scrollTo=23d6bb49-3469-4e23-baf5-25b2344b599d) from [Linoy Tsaban](https://twitter.com/linoy_tsaban).\n\nIn general, DoRA finetuning on diffusion models is still _experimental_ and is likely to require different hyperparameter values to perform best compared to LoRA.\n\u003e Specifically, people have noticed 2 differences to take into account in your training: \n\u003e 1. **LoRA seem to converge faster than DoRA** (so a set of parameters that may lead to overfitting when training a LoRA may be working well for a DoRA)\n\u003e 2. **DoRA quality superior to LoRA especially in lower ranks**: The difference in quality of DoRA of rank 8 and LoRA of rank 8 appears to be more significant than when training ranks of 32 or 64 for example.  \n\n#### Some DoRA vs. LoRA diffusion finetuning results\n- Example From [Linoy Tsaban](https://twitter.com/linoy_tsaban)(Images generated by DoRA are on the left and LoRA on the right):\n\u003ch1 align=\"center\"\u003e \n    \u003cimg src=\"./imgs/dora_lora_yoda_emoji.jpg\" width=\"500\"\u003e\n\u003c/h1\u003e\n\n- Example From [merve](https://twitter.com/mervenoyann):\n\u003ch1 align=\"center\"\u003e \n    \u003cimg src=\"./imgs/dora_lora_lego.jpeg\" width=\"500\"\u003e\n\u003c/h1\u003e\n\n\n### DoRA hyperparameters settings\n\u003e [!NOTE]\n\u003e 💡 While fine-tuning with DoRA, utilizing the configuration of LoRA can already achieve better results most of the time, achieving optimal performance compared to LoRA still requires adjustments to the hyperparameters. \n\n\u003e We suggest starting with a slightly lower learning rate than that of LoRA, and users may also experiment with varying LoRA dropout ratios.\n\n\u003e User may also start with half of the rank of the LoRA configuration which oftentimes can already result in comparable or even superior accuracy compared to that of LoRA.\n\n## Reproducing the results in the paper\n\nThis repo contains four directories:\n\n`./commonsense_reasoning` contains the code to finetune LLaMA-7B/13B using DoRA on the commonsense reasoning tasks. This directory is modified based on [LLM-Adapter](https://github.com/AGI-Edgerunners/LLM-Adapters).\n\n`./instruction_tuning_dvora` contains the code to finetune LLaMA-7B and LLaMA2-7B using DoRA and DVoRA (DoRA+VeRA) with the cleaned Alpaca instruction tuning dataset. This directory is modified based on [VeRA](https://openreview.net/attachment?id=NjNfLdxr3A\u0026name=supplementary_material).\n\n`./image_video_text_understanding` contains the code to finetune VL-BART using DoRA for the image/video-text understanding tasks. This directory is modified based on [VL-Adapter](https://github.com/ylsung/VL_adapter).\n\n`./visual_instruction_tuning` contains the code to finetune LLaVA-1.5-7B on the visual instruction tuning tasks with DoRA. This directory is modified based on [LLaVA](https://github.com/haotian-liu/LLaVA).\n\n\n## DoRA vs LoRA on the commonsense reasoning tasks \n| Model                 | r |    BoolQ  |  PIQA  |  SIQA  |  HellaS  |  WinoG  |  ARC-e  |  ARC-c  |  OBQA  |  Average  |\n|-----------------------|-------|---------|--------|--------|-------------|--------------|---------|---------|--------|-----------|\n| LLaMA-7B-LoRA\t\t  |   32  |    67.5  |  80.8  |  78.2  |  83.4  |  80.4   |  78.0   |  62.6   |  79.1  |  76.3     |\n| LLaMA-7B-DoRA(ours)\t  |  [16](https://huggingface.co/sliuau/DoRA-weights/tree/main/llama_dora_commonsense_checkpoints/LLama-7B/dora_r16)   |    70.0 | 82.6 | 79.7 | 83.2 | 80.6 | 80.6 | 65.4 | 77.6 | **77.5**   |\n| LLaMA-7B-DoRA(ours) \t  |  [32](https://huggingface.co/sliuau/DoRA-weights/tree/main/llama_dora_commonsense_checkpoints/LLama-7B/dora_r32)   |   69.7 | 83.4 | 78.6 | 87.2 | 81.0 | 81.9 | 66.2 | 79.2 | **78.4**   |\n| LLaMA2-7B-LoRA\t\t  |   32  |   69.8 | 79.9| 79.5| 83.6| 82.6| 79.8|64.7| 81.0| 77.6    |\n| LLaMA2-7B-DoRA(ours)\t\t  |  [16](https://huggingface.co/sliuau/DoRA-weights/tree/main/llama_dora_commonsense_checkpoints/LLama2-7B/dora_r16)   |   72.0 |83.1 |79.9| 89.1 |83.0| 84.5| 71.0 |81.2 |**80.5**  |\n| LLaMA2-7B-DoRA(ours) \t  |  [32](https://huggingface.co/sliuau/DoRA-weights/tree/main/llama_dora_commonsense_checkpoints/LLama2-7B/dora_r32)   |  71.8 |83.7 |76.0 |89.1 |82.6 |83.7 |68.2| 82.4 |**79.7**   |\n| LLaMA3-8B-LoRA\t\t  |   32  |   70.8 |85.2| 79.9| 91.7 |84.3 |84.2| 71.2| 79.0| 80.8    |\n| LLaMA3-8B-DoRA(ours)\t\t  |  [16](https://huggingface.co/sliuau/DoRA-weights/tree/main/llama_dora_commonsense_checkpoints/LLama3-8B/dora_r16)   |  74.5 |88.8 |80.3| 95.5| 84.7| 90.1| 79.1| 87.2| **85.0**   |\n| LLaMA3-8B-DoRA(ours) \t  |  [32](https://huggingface.co/sliuau/DoRA-weights/tree/main/llama_dora_commonsense_checkpoints/LLama3-8B/dora_r32)   |   74.6| 89.3| 79.9 |95.5| 85.6| 90.5| 80.4 |85.8 |**85.2**  |\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=NVlabs/DoRA\u0026type=Date)](https://star-history.com/#NVlabs/DoRA\u0026Date)\n\n\n## Contact\nShih-Yang Liu: [shihyangl@nvidia.com](shihyangl@nvidia.com) or [sliuau@connect.ust.hk](sliuau@connect.ust.hk)\n\n## Citation\nIf you find DoRA useful, please consider giving a star and citation:\n```bibtex\n@article{liu2024dora,\n  title={DoRA: Weight-Decomposed Low-Rank Adaptation},\n  author={Liu, Shih-Yang and Wang, Chien-Yi and Yin, Hongxu and Molchanov, Pavlo and Wang, Yu-Chiang Frank and Cheng, Kwang-Ting and Chen, Min-Hung},\n  journal={arXiv preprint arXiv:2402.09353},\n  year={2024}\n}\n```\n\n## Licenses\nCopyright © 2024, NVIDIA Corporation. All rights reserved.\n\nThis work is made available under the NVIDIA Source Code License-NC. Click [here](https://github.com/nbasyl/DoRA/LICENSE) to view a copy of this license.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNVlabs%2FDoRA","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNVlabs%2FDoRA","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNVlabs%2FDoRA/lists"}