{"id":14964513,"url":"https://github.com/chongjie-si/subspace-tuning","last_synced_at":"2025-04-05T14:04:52.821Z","repository":{"id":247509678,"uuid":"824955070","full_name":"Chongjie-Si/Subspace-Tuning","owner":"Chongjie-Si","description":"A generalized framework for subspace tuning methods in parameter efficient fine-tuning.","archived":false,"fork":false,"pushed_at":"2025-02-07T06:51:49.000Z","size":55869,"stargazers_count":133,"open_issues_count":1,"forks_count":7,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-05T14:04:33.931Z","etag":null,"topics":["adapter","commonsense-reasoning","glue","llama","llama2-7b","llama3-8b","lora","lora-dash","low-rank-adaptation","natural-language-generation","natural-language-processing","natural-language-understanding","parameter-efficient-fine-tuning","pretrained-models","soft-prompt-tuning","subject-driven-generation","subspace-tuning"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2407.05417","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Chongjie-Si.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-06T11:44:15.000Z","updated_at":"2025-04-02T12:59:21.000Z","dependencies_parsed_at":"2025-02-17T22:56:57.180Z","dependency_job_id":null,"html_url":"https://github.com/Chongjie-Si/Subspace-Tuning","commit_stats":{"total_commits":15,"total_committers":4,"mean_commits":3.75,"dds":0.2666666666666667,"last_synced_commit":"7a9c80518248b302cb72cc2a4ef5bbebd6f89251"},"previous_names":["chongjie-si/subspace-tuning"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Chongjie-Si%2FSubspace-Tuning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Chongjie-Si%2FSubspace-Tuning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Chongjie-Si%2FSubspace-Tuning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Chongjie-Si%2FSubspace-Tuning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Chongjie-Si","download_url":"https://codeload.github.com/Chongjie-Si/Subspace-Tuning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247345850,"owners_count":20924102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adapter","commonsense-reasoning","glue","llama","llama2-7b","llama3-8b","lora","lora-dash","low-rank-adaptation","natural-language-generation","natural-language-processing","natural-language-understanding","parameter-efficient-fine-tuning","pretrained-models","soft-prompt-tuning","subject-driven-generation","subspace-tuning"],"created_at":"2024-09-24T13:33:17.463Z","updated_at":"2025-04-05T14:04:52.796Z","avatar_url":"https://github.com/Chongjie-Si.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://github.com/Chongjie-Si/Subspace-Tuning/\"\u003e\n    \u003cimg src=\"resources/logo.png\" width=\"600\" alt=\"Logo\"/\u003e\n\u003c/a\u003e\n  \u003cdiv\u003e\u0026nbsp;\u003c/div\u003e\n  \u003cdiv align=\"center\"\u003e\n    \u003cb\u003e\u003cfont size=\"5\"\u003eA Generalized Framework of Subspace Tuning for PEFT\u003c/font\u003e\u003c/b\u003e\n  \u003c/div\u003e\n  \u003cdiv\u003e\u0026nbsp;\u003c/div\u003e\n\n  \u003cdiv style=\"display: inline-block;\"\u003e\n    \u003ca href=\"https://github.com/Chongjie-Si/Subspace-Tuning/blob/main/LICENSE\" title=\"License\"\u003e\n      \u003cimg src=\"https://img.shields.io/github/license/Chongjie-Si/Subspace-Tuning\" alt=\"License\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/Chongjie-Si/Subspace-Tuning/issues\" title=\"Issues or Pull Requests\"\u003e\n      \u003cimg src=\"https://img.shields.io/github/issues/Chongjie-Si/Subspace-Tuning\" alt=\"GitHub Issues or Pull Requests\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/Chongjie-Si/Subspace-Tuning/stargazers\" title=\"GitHub Repo stars\"\u003e\n      \u003cimg src=\"https://img.shields.io/github/stars/Chongjie-Si/Subspace-Tuning?style=flat\" alt=\"GitHub Repo stars\"\u003e\n    \u003c/a\u003e\n  \u003c/div\u003e\n\n  \u003cdiv\u003e\n    \u003ca href=\"#introduction\"\u003e📘 Introduction\u003c/a\u003e |\n    \u003ca href=\"#news\"\u003e💥 News\u003c/a\u003e |\n    \u003ca href=\"#usage\"\u003e🛠️ Usage\u003c/a\u003e |\n    \u003ca href=\"#tasks\"\u003e 🎯 Tasks \u003c/a\u003e|\n    \u003ca href=\"#algorithms\"\u003e🔍 Algorithms\u003c/a\u003e |\n    \u003ca href=\"https://github.com/Chongjie-Si/Subspace-Tuning/issues/new/choose\"\u003e🤔 Reporting Issues\u003c/a\u003e |\n    \u003ca href=\"mailto:chongjiesi@sjtu.edu.cn?subject=Contact%20Us\"\u003e📧 Contact Us\u003c/a\u003e\n  \u003c/div\u003e\n\u003c/div\u003e\n\n## \u003ca id=\"introduction\"\u003e📘 Introduction\u003c/a\u003e\n\nWelcome to our repository, which contains a diverse collection of Subspace Tuning methods for Parameter-Efficient Fine-Tuning (PEFT). Subspace Tuning are essential for adapting large pre-trained models to specific tasks with minimal changes to the original parameters. It endeavors to identify the maximal projection of the optimal weight $\\mathbf{W}^{*}$ onto the subspace spanned by the bases of $\\phi(\\mathbf{W})$, where $\\phi(\\mathbf{W})$ denotes the subspace transformation of the original frozen weight $\\mathbf{W}$. For more details, please refer to [the original paper](https://arxiv.org/abs/2407.05417).\n\n![Framework](./resources/framework.png)\n\nWe aim to provide a comprehensive resource for researchers and practitioners in this field, and facilitate easy integration into your projects. Whether you are here to find resources for your projects or to contribute, we hope this repository will be a valuable and inspiring part of your research journey.\n\n### Information Box\n\nThis repository also contains some of the other projects we have worked on, which might have led you here.\n\n- [**LoRA-Dash**](https://chongjiesi.site/full-publications/2024-arxiv-lora-dash/): Unleashing the Power of Task-Specific Directions in Parameter Efficient Fine-tuning.\n- [**FLoRA**](https://chongjiesi.site/full-publications/2024-arxiv-flora/): Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning.\n\n## \u003ca id=\"news\"\u003e💥 News\u003c/a\u003e\n\n- **[2024.09.04]** 🔥🔥 Add ***Method*** LoRA-Dash and ***Task*** Subject-driven Generation to Our Repo!\n- **[2024.08.18]** 🔥🔥 Add ***Task*** Math Reasoning to Our Repo!\n- **[2024.07.22]** 🔥🔥 Add ***Methods*** PISSA, MiLoRA and Spectral Adapter to Our Repo!\n- **[2024.07.09]** 🔥🔥 Repository Constructed!\n\n## 📝 Todo List\n\n- Nothing to do yet.\n\n## \u003ca id=\"usage\"\u003e🛠️ Usage\u003c/a\u003e\n\nTo use the algorithms in this repository, clone the repository and install the necessary dependencies.\n\n1. Clone this Repository:\n\n    ```bash\n    git clone https://github.com/Chongjie-Si/Subspace-Tuning.git\n    cd Subspace-Tuning\n    ```\n\n2. Follow the Instructions in Each Folder.\n\n## \u003ca id=\"tasks\"\u003e🎯 Tasks\u003c/a\u003e\n\nWe support several tasks including:\n\n- Natural Language Understanding ([NLU](./NLU/))\n- Natural Language Generation ([NLG](./NLG_QA/))\n- Question Answering ([QA](./NLG_QA/))\n- Commonsense Reasoning ([CR](./CR_MR/))\n- Math Reasoning ([MR](./CR_MR/))\n- Subject-driven Generation ([SdG](./SdG/))\n- ...\n\n## \u003ca id=\"algorithms\"\u003e🔍 Algorithms\u003c/a\u003e\n\nBased on subspace tuning theory, PEFT methods are classified into three categories: reconstruction-based, extension-based and combination-based.\n\n![Method](./resources/method.png)\n\nWe implement different methods mainly in [loralib/](./loralib/loralib/).\n\u003cdiv style=\"display: flex;\"\u003e\n\u003ctable cellspacing=\"0\" cellpadding=\"5\" style=\"border-collapse: collapse; width: 100%; max-width: 800px;\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"background-color: #f2f2f2;\"\u003e\n      \u003cth style=\"border: 1px solid #ddd; padding: 8px;\"\u003eCategory\u003c/th\u003e\n      \u003cth style=\"border: 1px solid #ddd; padding: 8px;\"\u003eAlgorithm\u003c/th\u003e\n      \u003cth style=\"border: 1px solid #ddd; padding: 8px;\"\u003eCode\u003c/th\u003e\n      \u003cth style=\"border: 1px solid #ddd; padding: 8px;\"\u003ePaper\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003c!-- Reconstruction-based--\u003e\n    \u003c!-- Reconstruction-based--\u003e\n    \u003c!-- Reconstruction-based--\u003e\n    \u003ctr\u003e\n      \u003ctd rowspan=\"10\" style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eReconstruction\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eSAM-PARSER\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_SAMPARSER.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2308.14604\"\u003e2024 AAAI\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eIA3\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_IA3.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://proceedings.neurips.cc/paper_files/paper/2022/hash/0cde695b83bd186c1fd456302888454c-Abstract-Conference.html\"\u003e2022 NeurIPS\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eSSB\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_SSB.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2407.05417\"\u003e2024 Arxiv\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eSSL\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_SSL.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2407.05417\"\u003e2024 Arxiv\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eBitFit\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003eN/A\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2106.10199\"\u003e2022 ACL\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003ePrefix-tuning\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./CR/peft/src/peft/tuners/prefix_tuning.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://aclanthology.org/2021.acl-long.353.pdf\"\u003e2021 ACL\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003ePrompt-tuning\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./CR/peft/src/peft/tuners/prompt_tuning.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/pdf/2104.08691\"\u003e2021 EMNLP\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eP-tuning\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./CR/peft/src/peft/tuners/p_tuning.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://aclanthology.org/2022.acl-short.8.pdf\"\u003e2022 ACL\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n     \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003ePISSA\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_PISSA.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2404.02948\"\u003e2024 NIPS\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eMiLoRA\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_MiLoRA.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2406.09044\"\u003e2024 Arxiv\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003c!-- Extension-based--\u003e\n    \u003c!-- Extension-based--\u003e\n    \u003c!-- Extension-based--\u003e\n    \u003ctr\u003e\n      \u003ctd rowspan=\"8\" style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eExtension\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eLoRA\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_LoRA.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2106.09685\"\u003e2022 ICLR\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eAdaLoRA\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/adalora.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://par.nsf.gov/servlets/purl/10471451\"\u003e2023 ICLR\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eFLoRA\u003c/strong\u003e\u003c/td\u003e\n    \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_FLoRA.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n    \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://openreview.net/forum?id=OALIb8oNfl\u0026referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DICLR.cc%2F2025%2FConference%2FAuthors%23your-submissions)\"\u003e2025 ICLR\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eMoSLoRA\u003c/strong\u003e\u003c/td\u003e\n    \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_MosLoRA.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n    \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/pdf/2406.11909\"\u003e2024 EMNLP\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eTriLoRA\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_TriLoRA.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2405.11236\"\u003e2024 Arxiv\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eAdapter (Houlsby)\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003eN/A\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"http://proceedings.mlr.press/v97/houlsby19a.html\"\u003e2019 ICML\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eAdapter (Pfeiffer)\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003eN/A\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2005.00247\"\u003e2021 ACL\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eParallel Adapter\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_PA.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2110.04366\"\u003e2022 ICLR\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003c!-- Combination-based--\u003e\n    \u003c!-- Combination-based--\u003e\n    \u003ctr\u003e\n      \u003ctd rowspan=\"5\" style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eCombination\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eDoRA\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_DoRA.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2402.09353\"\u003e2024 ICML\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n     \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003eSVDiff\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_SVDiff.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://openaccess.thecvf.com/content/ICCV2023/html/Han_SVDiff_Compact_Parameter_Space_for_Diffusion_Fine-Tuning_ICCV_2023_paper.html\"\u003e2023 ICCV\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003e Spectral Adapter\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_Spectral_Adapter.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://arxiv.org/abs/2405.13952\"\u003e2024 NIPS\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cstrong\u003e LoRA-Dash\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"./loralib/loralib/layers_LoRA-Dash.py\"\u003eCode\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003ca href=\"https://openreview.net/forum?id=RYrJqz44p4\u0026referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DICLR.cc%2F2025%2FConference%2FAuthors%23your-submissions)\"\u003e2025 ICLR\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003e\u003cem\u003eMore algorithms and updates are continually added...\u003c/em\u003e\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003eN/A\u003c/td\u003e\n      \u003ctd style=\"border: 1px solid #ddd; padding: 8px;\"\u003eN/A\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\nWe have also tested the performance of some algorithms on NLU and [CR](./Fair_Comparison/) tasks.\n\n![result](./resources/result.png)\n\n## 🎁 Contribution\n\nWe welcome contributions to this repository! Whether you’re fixing bugs, adding new features, or improving documentation, your help is appreciated. Please follow the [guidelines](./resources/Contributions.md) to ensure a smooth contribution process.\n\n## 💡 Further Information\n\nThank you for your interest in our PEFT code repository. We strive to make this a valuable resource for your projects and research endeavors.\n\nOur goal is to foster a collaborative environment where both you and our researchers can exchange ideas and cooperate. Beyond discussing code-related issues, we encourage you to share your perspectives on any PEFT methodology and address any potential challenges you encounter. We welcome discussions that may spark new insights and innovations.\n\nBesides, this code repository is of a more private nature, containing tasks and algorithms that I use during my experiments.\nIf you have any algorithms you’d like to implement or wish to add more task scenarios, please feel free to send [email](mailto:chongjiesi@sjtu.edu.cn) to me. You can also visit my [personal homepage](https://chongjiesi.github.io) for more details.\n\n## 📧 Contact\n\nIf you have any questions, suggestions, or feedback, please feel free to contact us at [chongjiesi@sjtu.edu.cn](mailto:chongjiesi@sjtu.edu.cn).\n\n## 🔗 Citation\n\nIf you find this repository useful, please consider giving it a star and citing it in your work:\n\n```bibtex\n@article{si2024see,\n  title={See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition},\n  author={Si, Chongjie and Yang, Xiaokang and Shen, Wei},\n  journal={arXiv preprint arXiv:2407.05417},\n  year={2024}\n}\n```\n\n**This repository also contains the code for our other projects. If you find these methods useful, please consider giving a star and citing them in your work.**\n\n#### FLoRA: Low-Rank Core Space for N-dimension\n\n```bibtex\n@inproceedings{\nsi2025maintaining,\ntitle={Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning},\nauthor={Chongjie Si and Xuehui Wang and Xue Yang and Zhengqin Xu and Qingyun Li and Jifeng Dai and Yu Qiao and Xiaokang Yang and Wei Shen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=OALIb8oNfl}\n}\n```\n\n#### Unleashing the Power of Task-Specific Directions in Parameter Efficient Fine-tuning\n\n```bibtex\n@inproceedings{\nsi2025unleashing,\ntitle={Unleashing the Power of Task-Specific Directions in Parameter Efficient Fine-tuning},\nauthor={Chongjie Si and Zhiyi Shi and Shifan Zhang and Xiaokang Yang and Hanspeter Pfister and Wei Shen},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=RYrJqz44p4}\n}\n```\n\n\u003c/details\u003e\n\n## 📄 License\n\nThis repository is licensed under the [Apache 2.0 license](./LICENSE). See the LICENSE file for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchongjie-si%2Fsubspace-tuning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchongjie-si%2Fsubspace-tuning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchongjie-si%2Fsubspace-tuning/lists"}