{"id":23237059,"url":"https://github.com/txsun1997/Black-Box-Tuning","last_synced_at":"2025-08-19T23:31:44.563Z","repository":{"id":38385143,"uuid":"445499672","full_name":"txsun1997/Black-Box-Tuning","owner":"txsun1997","description":"ICML'2022: Black-Box Tuning for Language-Model-as-a-Service \u0026 EMNLP'2022: BBTv2: Towards a Gradient-Free Future with Large Language Models","archived":false,"fork":false,"pushed_at":"2022-11-08T04:40:43.000Z","size":67138,"stargazers_count":241,"open_issues_count":5,"forks_count":28,"subscribers_count":7,"default_branch":"main","last_synced_at":"2023-11-07T19:07:18.713Z","etag":null,"topics":["black-box-optimization","deep-learning","few-shot-learning","language-model","natural-language-processing","pytorch"],"latest_commit_sha":null,"homepage":"https://proceedings.mlr.press/v162/sun22e.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/txsun1997.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-01-07T11:33:52.000Z","updated_at":"2023-11-07T14:12:00.000Z","dependencies_parsed_at":"2022-08-28T02:00:36.008Z","dependency_job_id":null,"html_url":"https://github.com/txsun1997/Black-Box-Tuning","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/txsun1997%2FBlack-Box-Tuning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/txsun1997%2FBlack-Box-Tuning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/txsun1997%2FBlack-Box-Tuning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/txsun1997%2FBlack-Box-Tuning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/txsun1997","download_url":"https://codeload.github.com/txsun1997/Black-Box-Tuning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230374271,"owners_count":18216044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["black-box-optimization","deep-learning","few-shot-learning","language-model","natural-language-processing","pytorch"],"created_at":"2024-12-19T04:13:25.641Z","updated_at":"2024-12-19T04:13:29.824Z","avatar_url":"https://github.com/txsun1997.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["Language Models"],"readme":"# Black-Box-Tuning for Language-Model-as-a-Service\n## Updates\n\n- 2022/10/14: Release the latest version of BBTv2, check out the [updated results](https://docs.google.com/spreadsheets/d/1FA9zMW613OoskI_fBXuZNNBVo7RFV5OvdZ0YHnA2j5Q/edit?usp=sharing). :mag:\n- 2022/07/05: Release a [paper list](https://github.com/txsun1997/LMaaS-Papers) on LMaaS, check out other awesome papers! :bookmark_tabs:\n- 2022/06/05: Support T5 and GPT-2 model. :clap:\n- 2022/05/15: Support BERT and BART model. :clap:\n- 2022/05/04: Release BBTv2, check out our [paper](https://arxiv.org/abs/2205.11200) and try it with `deepbbt.py` :tada:\n- 2022/02/18: Support ONNX Runtime optimization (training speed is doubled!) :rocket:\n- 2022/01/13: Release the first version of BBT, check out our [paper](https://arxiv.org/abs/2201.03514). :tada:\n\n## Quick Links\n\n- [Introduction](#introduction)\n- [Prepare your environment](#prepare-your-environment)\n- [Using BBT](#using-bbt)\n- [Using BBTv2](#using-bbtv2)\n- [Inference Optimization](#inference-optimization)\n- [Citation](#citation)\n\n## Introduction\n\nBlack-Box Tuning (BBT) is a gradient-free method to drive large language models (LLMs) for few-shot learning. It optimizes a sequence of soft prompt tokens prepended to the input of LLMs, without requiring gradients/back-propagation of the LLMs. Therefore, pre-trained general-purposed LLMs can be viewed as black-box models and deployed efficiently on some inference servers. In such a scenario, which we call Language-Model-as-a-Service (LMaaS), BBT can achieve comparable performance to full model tuning by only accessing model inference APIs. Generally, BBT can achieve considerable results on most language understanding datasets within 8k model forward passes.\n\nMore details are provided in our ICML paper [Black-Box Tuning for Language-Model-as-a-Service](https://arxiv.org/abs/2201.03514) and our EMNLP paper [BBTv2: Towards a Gradient-Free Future with Large Language Models](https://arxiv.org/abs/2205.11200).\n\n\u003e To help reproduce results reported in the paper, we also release a [Google Sheets](https://docs.google.com/spreadsheets/d/1FA9zMW613OoskI_fBXuZNNBVo7RFV5OvdZ0YHnA2j5Q/edit?usp=sharing) recording BBTv2 performance on each dataset using each random seed. Feel free to reach out to me if you cannot obtain similar results.\n\n## Prepare your environment\n\nThe implementation of Black-Box Tuning is quite simple, you can check our code and easily implement it in your own environment. Or you can create a new environment to run our implementation based on `pycma`, `Transformers` and `FastNLP`. Optionally, you can use `fitlog` to monitor experimental results. You can uncomment the fitlog-related lines in our code to use it.\n\n```bash\nconda create --name bbt python=3.8\nconda activate bbt\npip install transformers==4.1.1\npip install fastNLP==0.6.0\npip install datasets\npip install cma\npip install sklearn\ngit clone https://github.com/txsun1997/Black-Box-Tuning\ncd Black-Box-Tuning\n```\n\n## Using BBT\n\nNow you can run Black-Box Tuning with `run.sh`:\n\n```bash\nbash run.sh\n```\n\nIn general, you will obtain the following results in ~13 minutes (tested on NVIDIA 3090 GPU):\n\n| SST-2 split | Best Accuracy |\n| ----------- | ------------- |\n| Train       | 100 %         |\n| Dev         | 96.88 %       |\n| Test        | 90.48 %       |\n\nTo reproduce other experiments in our paper, change the arguments of `bbt.py`, for example, \n\n```bash\npython bbt.py \\\n  --task_name \"sst2\" \\\n  --n_prompt_tokens 50 \\\n  --intrinsic_dim 500 \\\n  --k_shot 16 \\\n  --device \"cuda:0\" \\\n  --seed 42 \\\n  --loss_type \"ce\" \\\n  --cat_or_add \"add\" \\\n  --budget 8000 \\\n  --print_every 50 \\\n  --eval_every 100\n```\n\nTo obtain similar results as reported in the original paper, we recommend using `--loss_type \"hinge\"` for sentence-pair tasks (i.e., MRPC, SNLI, and RTE) and using `--budget 20000` for DBPedia.\n\nIn addition, black-box tuning also supports parallel evaluation. That is, you can evaluate a population of solutions in parallel by putting them into a single large batch. For example,\n\n```bash\npython bbt.py \\\n  --task_name \"sst2\" \\\n  --n_prompt_tokens 50 \\\n  --intrinsic_dim 500 \\\n  --k_shot 16 \\\n  --device \"cuda:0\" \\\n  --seed 42 \\\n  --loss_type \"ce\" \\\n  --cat_or_add \"add\" \\\n  --budget 300 \\\n  --print_every 10 \\\n  --eval_every 20 \\\n  --parallel\n```\n## Using BBTv2\n\nBBTv2 is an improved version of BBT. Instead of optimizing the prompt merely in the input layer, BBTv2 adopts a divide-and-conquer algorithm to alternately optimize prompts in every layer (i.e., deep prompt). You can simply try BBTv2 using the following command,\n\n```bash\npython deepbbt.py \\\n  --model_name \"roberta-large\"\\\n  --task_name \"agnews\" \\\n  --n_prompt_tokens 50 \\\n  --intrinsic_dim 500 \\\n  --k_shot 16 \\\n  --device \"cuda:0\" \\\n  --seed 42 \\\n  --loss_type \"ce\" \\\n  --cat_or_add \"add\" \\\n  --random_proj \"normal\" \\\n  --sigma 0.2 \\\n  --alpha 0.2 \\\n  --popsize 20 \\\n  --bound 0 \\\n  --budget 8000 \\\n  --print_every 50 \\\n  --eval_every 100\n```\n\nBBTv2 usually confers better results on many label classification tasks (e.g., DBPedia) and entailment tasks (e.g., MRPC, SNLI, RTE, etc.). Check our [Google Sheets](https://docs.google.com/spreadsheets/d/1FA9zMW613OoskI_fBXuZNNBVo7RFV5OvdZ0YHnA2j5Q/edit?usp=sharing) if you have problem reproducing the results of BBTv2.\n\n## Inference Optimization\n\nIn contrast to training with gradient descent, BBT (and BBTv2) only requires model forward computation, and therefore can be significantly accelerated using [ONNX Runtime](https://onnxruntime.ai/) or NVIDIA [TensorRT](https://developer.nvidia.com/tensorrt). \n\nHere we provide an implementation of inference optimization using ONNX Runtime. You can obtain ~2x speedup using only one line of code.\n\nSDK `onnxruntime-gpu` is required for optimization. Installation of this package can be troublesome. And there may be some environment-specific errors or unexpected performance. But in real-world scenarios, this is a part of the black box on the server side.\n\nThe following code works well to configure the environment on an NVIDIA GeForce RTX 3090 GPU with Driver Version: 470.82.00 and CUDA Version: 11.4.\n```bash\npip install transformers==4.1.1\npip install datasets\npip install fastNLP\npip install cma\npip install sklearn\npip3 install torch --extra-index-url https://download.pytorch.org/whl/cu113\npip install onnx\npip install onnxruntime-gpu==1.10.0\npip install coloredlogs\npip install sympy\n```\n\nTo export a BBT model based on `PyTorch` to an `ONNX` model, \nyou can run `export_and_optimize.py` with all arguments set to default to get a demo onnx model.\n\n```bash\npython export_and_optimize.py\n```\nTwo models will be saved to `./onnx_models/`, namely exported (not accelerated) and optimized model.\nThen you can modify `run.sh`. \nBy setting parameter `inference_framework` to `'ort'` and `onnx_model_path` to `\u003cYour model path\u003e`,\na faster version of BBT is ready. Here is an example.\n```bash\npython bbt.py \\\n  --task_name \"sst2\" \\\n  --n_prompt_tokens 50 \\\n  --intrinsic_dim 500 \\\n  --k_shot 16 \\\n  --device \"cuda:0\" \\\n  --seed 42 \\\n  --loss_type \"ce\" \\\n  --cat_or_add \"add\" \\\n  --budget 8000 \\\n  --print_every 50 \\\n  --eval_every 100 \\\n  --inference_framework 'ort' \\\n  --onnx_model_path './onnx_models/optimized_model.onnx'\n```\n\nTo add some flexibility to model optimization, we provided some options in `export_and_optimize.py`.\nYou can adjust these arguments in `export_and_optimize.sh`. Here is an example.\n```bash\npython export_and_optimize.py \\\n  --batch_size 32 \\\n  --max_seq_len 128 \\\n  --n_prompt_tokens 50 \\\n  --prompt_embed_dim 1024 \\\n  --cat_or_add \"add\" \\\n  --exported_model_name 'model' \\\n  --optimized_model_name 'optimized_model'\n```\nOnnx models are static, but to cat or to add is a branch in the model. During building phase, unused nodes in the model graph are removed for better performance. So you have to build one for each mode.\n\nYou can get the following results in 4.3 ± 0.1 minutes, compared to pytorch version of BBT whose training time is 8.9 ± 0.15 minutes (depends on hardware settings)\n\nYou can get the following results by running BBT 100 times on sst2 with random seed set from 1 to 100.\nFp16 optimization does not hurt performance on all tasks.\n\n| SST-2 split | Best Accuracy   |\n| ----------- | --------------- |\n| Test        | 88.0  %   |\n\n## Citation\n\nIf you find this work helpful, please cite:\n\n```bibtex\n@inproceedings{sun2022bbt,\n  title={Black-Box Tuning for Language-Model-as-a-Service}, \n  author={Tianxiang Sun and Yunfan Shao and Hong Qian and Xuanjing Huang and Xipeng Qiu},\n  booktitle = {Proceedings of {ICML}},\n  year={2022}\n}\n```\n\n```bibtex\n@inproceedings{sun2022bbtv2,\n  title={BBTv2: Towards a Gradient-Free Future with Large Language Models},\n  author={Tianxiang Sun and Zhengfu He and Hong Qian and Yunhua Zhou and Xuanjing Huang and Xipeng Qiu},\n  booktitle = {Proceedings of {EMNLP}},\n  year={2022}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftxsun1997%2FBlack-Box-Tuning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftxsun1997%2FBlack-Box-Tuning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftxsun1997%2FBlack-Box-Tuning/lists"}