https://github.com/git-disl/recap
Code for CVPR24 Paper - Resource-Efficient Transformer Pruning for Finetuning of Large Models
https://github.com/git-disl/recap
Last synced: 4 months ago
JSON representation
Code for CVPR24 Paper - Resource-Efficient Transformer Pruning for Finetuning of Large Models
- Host: GitHub
- URL: https://github.com/git-disl/recap
- Owner: git-disl
- Created: 2024-06-20T22:39:59.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2025-10-31T18:28:05.000Z (7 months ago)
- Last Synced: 2025-10-31T20:24:38.953Z (7 months ago)
- Language: Python
- Size: 45.9 KB
- Stars: 11
- Watchers: 3
- Forks: 1
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# CVPR24 Paper
Code for CVPR24 Paper - Resource-Efficient Transformer Pruning for Finetuning of Large Models
Fatih Ilhan, Gong Su, Selim Furkan Tekin, Tiansheng Huang, Sihao Hu, and Ling Liu, "Resource-Efficient Transformer Pruning for Finetuning of Large Models," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, Jun. 17-21, 2024.
## Setup
Python 3.10
Pytorch 2.0.1
Transformers 4.33
https://github.com/microsoft/nni
Please check requirements.txt for the list of other packages.
## Usage
CIFAR, TinyImageNet, GLUE datasets are automatically downloaded. You can download Cityscapes from https://www.cityscapes-dataset.com/ and KITTI from https://www.cvlibs.net/datasets/kitti/eval_semseg.php?benchmark=semantics2015.
### General usage for finetuning with RECAP:
```python main.py --task --data --arch --init_sparse --iter_sparse - -num_pi -num_pr ```
### Example: Finetune ViT-base at CIFAR100 with 33% pruning and 87.5% masking in 10 iterations:
```python main.py --task img_class --data cifar100 --arch vit-base --init_sparse 0.33 --iter_sparse -0.875 -num_pi 2 -num_pr 10```
### Example: Finetune Mask2Former at Cityscapes with 50% pruning and 50% masking in 20 iterations:
```python main.py --task img_seg --data cityscapes --arch m2f --init_sparse 0.5 --iter_sparse -0.5 -num_pi 3 -num_pr 20```
### Example: Finetune BERT-base at CoLA with 17% pruning and 50% masking in 5 iterations:
```python main.py --task glue --data cola --arch bert-base-uncased --init_sparse 0.17 --iter_sparse -0.5 -num_pi 1 -num_pr 5```
### Parameters
All pruning/finetuning parameters are controlled from ``config.py``.