{"id":15040644,"url":"https://github.com/microsoft/adamix","last_synced_at":"2025-04-10T02:27:36.532Z","repository":{"id":37199527,"uuid":"496011321","full_name":"microsoft/AdaMix","owner":"microsoft","description":"This is the implementation of the paper AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning (https://arxiv.org/abs/2205.12410). ","archived":false,"fork":false,"pushed_at":"2023-08-14T22:22:39.000Z","size":17479,"stargazers_count":130,"open_issues_count":25,"forks_count":11,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-30T21:37:29.667Z","etag":null,"topics":["adapter","bert","dart","glue","gpt-2","nlg","nlp","nlu","parameter-efficient","pytorch","roberta","webnlg"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microsoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null}},"created_at":"2022-05-24T22:52:02.000Z","updated_at":"2025-03-16T13:22:18.000Z","dependencies_parsed_at":"2023-02-16T13:15:55.307Z","dependency_job_id":null,"html_url":"https://github.com/microsoft/AdaMix","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FAdaMix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FAdaMix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FAdaMix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FAdaMix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microsoft","download_url":"https://codeload.github.com/microsoft/AdaMix/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246899658,"owners_count":20851898,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adapter","bert","dart","glue","gpt-2","nlg","nlp","nlu","parameter-efficient","pytorch","roberta","webnlg"],"created_at":"2024-09-24T20:44:51.933Z","updated_at":"2025-04-02T22:08:52.124Z","avatar_url":"https://github.com/microsoft.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AdaMix (Mixture-of-Adaptations)\n\n\nThis is the implementation of the paper [AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning](https://arxiv.org/abs/2205.12410). \n\n\n## Quick links\n\n* [Overview](#overview)\n* [GLUE Benchmark](#adapting-to-the-glue-benchmark)\n  * [BERT-base](#main-results-bert-base)\n  * [RoBERTa-large](#main-results-roberta-large)\n  * [Averaging Weigths v.s. Random Routing](#comparision-between-averaging-weights-and-random-single-adapter)\n* [Checkpoints](#download-adamix-checkpoints)\n* [Run the model](#steps-to-reproduce-our-results)\n  * [Quick start](#quick-start)\n  * [Evaluate checkpoints](#evaluate-the-checkpoints)\n* [Notes and Acknowledgments](#notes-and-acknowledgments)\n* [Contact Information](#contact-information)\n* [Citation](#how-do-i-cite-adamix)\n\n\n\n## Overview\n### Training\n\u003cimg src=\"./figures/MixAdapter.png\" width=\"650\"/\u003e\n\n### Inference\n\u003cimg src=\"./figures/training_inference.png\" width=\"450\"/\u003e\n\n## Adapting to the GLUE Benchmark\nOur experiments on the GLUE benchmark are run on 16 NVIDIA Tesla V100 GPU. The results may vary due to different GPU models, drivers, CUDA SDK versions, floating-point precisions, and random seeds. \n\n### Main Results (BERT-base)\n\u003cimg src=\"./figures/bert_base.png\" width=\"750\"/\u003e\n\n\n### Main Results (RoBERTa-large)\n\u003cimg src=\"./figures/roberta_large.png\" width=\"750\"/\u003e\n\n### Comparision between Averaging Weights and Random Single Adapter \n\u003cimg src=\"./figures/violin_plot.png\" width=\"950\"/\u003e\n\n\n## Download AdaMix checkpoints\nWe release all copies of Adapter weights for users' Adapter aggregation study. \n\n|   | Dataset  | BERT base 110M \u003cbr\u003e   | RoBERTa large 355M \u003cbr\u003e  |\n|---|----------|--------------------|----------------------|\n|   | MNLI     |[8.5 MB](https://github.com/yaqingwang/MoA/releases/download/bert_base/pytorch_model_mnli_expert_soup.bin) |[11.7 MB](https://github.com/yaqingwang/MoA/releases/download/roberta_large/pytorch_model_mnli_expert_soup.bin) |\n|   | SST2     |[8.5 MB](https://github.com/yaqingwang/MoA/releases/download/bert_base/pytorch_model_sst2_expert_soup.bin)  |[11.7 MB](https://github.com/yaqingwang/MoA/releases/download/roberta_large/pytorch_model_sst2_expert_soup.bin)  |\n|   | MRPC     |[8.5 MB](https://github.com/yaqingwang/MoA/releases/download/bert_base/pytorch_model_mrpc_expert_soup.bin)  |[11.7 MB](https://github.com/yaqingwang/MoA/releases/download/roberta_large/pytorch_model_mrpc_expert_soup.bin)  |\n|   | CoLA     |[8.5 MB](https://github.com/yaqingwang/MoA/releases/download/bert_base/pytorch_model_cola_expert_soup.bin)  |[11.7 MB](https://github.com/yaqingwang/MoA/releases/download/roberta_large/pytorch_model_cola_expert_soup.bin)  |\n|   | QNLI     |[8.5 MB](https://github.com/yaqingwang/MoA/releases/download/bert_base/pytorch_model_qnli_expert_soup.bin)  |[11.7 MB](https://github.com/yaqingwang/MoA/releases/download/roberta_large/pytorch_model_qnli_expert_soup.bin)  |\n|   | QQP      |[8.5 MB](https://github.com/yaqingwang/MoA/releases/download/bert_base/pytorch_model_qqp_expert_soup.bin)  |[11.7 MB](https://github.com/yaqingwang/MoA/releases/download/roberta_large/pytorch_model_qqp_expert_soup.bin)  |\n|   | RTE      |[8.5 MB](https://github.com/yaqingwang/MoA/releases/download/bert_base/pytorch_model_rte_expert_soup.bin)  |[11.7 MB](https://github.com/yaqingwang/MoA/releases/download/roberta_large/pytorch_model_rte_expert_soup.bin)  |\n|   | STSB     |[8.5 MB](https://github.com/yaqingwang/MoA/releases/download/bert_base/pytorch_model_stsb_expert_soup.bin)  |[11.7 MB](https://github.com/yaqingwang/MoA/releases/download/roberta_large/pytorch_model_stsb_expert_soup.bin)  |\n\n## Steps to reproduce our results\n### Create and activate conda env\n```console\nconda env create -f environment.yml\n```\n### Install the pre-requisites\n```console\npip install -e .\n```\n\nWe also provide the shell scripts for bert-base and roberta-large.\n\n### Quick start\n```console\nexport num_gpus=1\nexport PYTHONHASHSEED=0\ntask_name=mnli\nmodel=roberta-large\nexport output_dir=\"./models/${model}/${task_name}\"\npython -m torch.distributed.launch --nproc_per_node=$num_gpus \\\nexamples/text-classification/run_glue.py \\\n--model_name_or_path $model \\\n--task_name $task_name \\\n--do_train \\\n--do_eval \\\n--max_seq_length 128 \\\n--per_device_train_batch_size 64 \\\n--per_device_eval_batch_size 32 \\\n--learning_rate 3e-4 \\\n--num_train_epochs 20 \\\n--output_dir $output_dir/model \\\n--overwrite_output_dir \\\n--logging_steps 1000 \\\n--logging_dir $output_dir/log \\\n--evaluation_strategy epoch \\\n--save_strategy epoch \\\n--warmup_ratio 0.06 \\\n--apply_expert_soup \\\n--adapter_size 16 \\\n--num_experts 4 \\\n--seed 0 \\\n--inference_level 3 \\\n--weight_decay 0.1 \\\n--sharing_up 1 \\\n--sharing_down 0 \\\n--use_consistency_loss 1\n\n```\nMost arguments are inherited from transformers and are easy to understand. We further explain some of the AdaMix's arguments:\n* `inference_level`: There are two suggested modes\n  * `1`: Random Routing\n  * `3`: Averaging the weights of Adapters for routing (used in AdaMix)\n\n* `num_experts`: Number of Adapters in AdaMix\n\n* `use_consistency_loss`: Two modes. \n  * `0`: No consistency loss\n  * `1`: Use consistency loss\n\n\n* `sharing_up`: There are two modes. (sharing_down is same)\n  * `0`: No weight sharing\n  * `1`: Sharing Project-up layer weights in Adapter\n\n\n\n### Evaluate the checkpoints\nCreate checkpoints directory and download checkpoints of corresponding tasks under the directory. Use MNLI as an example. Use your checkpoint path in **expert_soup_path** argument.\n```console\nexport num_gpus=1\nexport PYTHONHASHSEED=0\ntask_name=mnli\nmodel=roberta-large\nexport output_dir=\"./models/${model}/${task_name}\"\npython -m torch.distributed.launch --nproc_per_node=$num_gpus \\\nexamples/text-classification/run_glue.py \\\n--model_name_or_path $model \\\n--task_name $task_name \\\n--do_eval \\\n--expert_soup_path ./checkpoints/pytorch_model_${task_name}_expert_soup.bin \\\n--max_seq_length 128 \\\n--per_device_train_batch_size 64 \\\n--per_device_eval_batch_size 32 \\\n--learning_rate 3e-4 \\\n--num_train_epochs 20 \\\n--output_dir $output_dir/model \\\n--overwrite_output_dir \\\n--logging_steps 1000 \\\n--logging_dir $output_dir/log \\\n--evaluation_strategy epoch \\\n--save_strategy epoch \\\n--warmup_ratio 0.06 \\\n--apply_expert_soup \\\n--adapter_size 16 \\\n--num_experts 4 \\\n--seed 0 \\\n--inference_level 3 \\\n--weight_decay 0.1 \\\n--sharing_up 1 \\\n--sharing_down 0 \\\n--use_consistency_loss 1\n\n```\n\n## Notes and Acknowledgments\nThe implementation is based on https://github.com/huggingface/transformers  \u003cbr\u003e\nWe also used some code from: https://github.com/microsoft/LoRA \n\n## Contact Information\nFor personal communication related to this package, please contact [Yaqing Wang](https://yaqingwang.github.io/) (wang5075@purdue.edu), [Sahaj Agarwal](https://www.linkedin.com/in/sahaj-agarwal-89aa49174/) (sahagar@microsoft.com), [Subhabrata (Subho) Mukherjee](https://www.microsoft.com/en-us/research/people/submukhe/) (submukhe@microsoft.com) or [Xiaodong Liu](https://sites.google.com/view/buptxiaodong/home) (xiaodl@microsoft.com).\n\n\n\n\n## How do I cite AdaMix?\n\n```\n@article{wang2022adamix,\n  title={AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning},\n  author={Wang, Yaqing and Agarwal, Sahaj and Mukherjee, Subhabrata and Liu, Xiaodong and Gao, Jing and Awadallah, Ahmed Hassan and Gao, Jianfeng},\n  journal={arXiv preprint arXiv:2205.12410},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fadamix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrosoft%2Fadamix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fadamix/lists"}