https://github.com/amazon-science/mezo_svrg
Code the ICML 2024 paper: "Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models"
https://github.com/amazon-science/mezo_svrg
deep-learning fine-tuning language-model large-language-models llm-training llms machine-learning machine-learning-algorithms optimization optimization-algorithms svrg variance-reduction zero-order-methods
Last synced: 4 months ago
JSON representation
Code the ICML 2024 paper: "Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models"
- Host: GitHub
- URL: https://github.com/amazon-science/mezo_svrg
- Owner: amazon-science
- License: apache-2.0
- Created: 2024-05-28T16:50:58.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-25T15:50:07.000Z (about 2 years ago)
- Last Synced: 2025-09-09T05:03:35.308Z (10 months ago)
- Topics: deep-learning, fine-tuning, language-model, large-language-models, llm-training, llms, machine-learning, machine-learning-algorithms, optimization, optimization-algorithms, svrg, variance-reduction, zero-order-methods
- Language: Python
- Homepage: https://arxiv.org/abs/2404.08080
- Size: 96.7 KB
- Stars: 11
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
## MeZO-SVRG: Variance-Reduced Zero-Order Methods for fine-tuning LLMs
This repository implements the Memory-Efficient Zeroth-Order Stochastic Variance-Reduced Gradient (MeZO-SVRG) algorithm for fine-tuning pre-trained hugging face LMs. As baselines we also implement Memory-efficient ZO Optimizer (MeZO) and first-order SGD (FO-SGD). The repository is written in PyTorch and leverages the Pytorch Lightning framework.
## Installation
To install the relevant python environment use the command
```bash
conda create --name zo_opt python=3.9
conda activate zo_opt
python -m pip install -r requirements.txt
```
## File Overview
This repository implements the MeZO-SVRG algorithm and enables fine-tuning on a range on language models using the GLUE benchmark dataset. To run experiments, execute the 'finetune_llm.sh' bash script.
The script supports the following models:
1. 'distilbert-base-cased'
2. 'roberta-large'
3. 'gpt2-xl'
4. 'facebook/opt-2.7b'
5. 'facebook/opt-6.7b'
The script supports the following GLUE tasks:
1. MNLI
2. QNLI
3. SST-2
4. CoLA
Indicate the fine-tuning algorithm by passing one of the following {'FO', 'ZO', 'ZOSVRG'}. The exact hyperparameter settings used to generate the tables/figures in the paper are provided in the Appendix.
## Citation
Please consider citing our paper if you use our code:
```text
@misc{gautam2024variancereduced,
title={Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models},
author={Tanmay Gautam and Youngsuk Park and Hao Zhou and Parameswaran Raman and Wooseok Ha},
year={2024},
eprint={2404.08080},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```
## Security
See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.
## License
This project is licensed under the Apache-2.0 License.