https://github.com/vita-group/scalable-l2o
[ECCV 2022] "Scalable Learning to Optimize: A Learned Optimizer Can Train Big Models" by Xuxi Chen, Tianlong Chen, Yu Cheng, Weizhu Chen, Ahmed Awadallah, and Zhangyang Wang
https://github.com/vita-group/scalable-l2o
learning-to-optimize
Last synced: 11 months ago
JSON representation
[ECCV 2022] "Scalable Learning to Optimize: A Learned Optimizer Can Train Big Models" by Xuxi Chen, Tianlong Chen, Yu Cheng, Weizhu Chen, Ahmed Awadallah, and Zhangyang Wang
- Host: GitHub
- URL: https://github.com/vita-group/scalable-l2o
- Owner: VITA-Group
- Created: 2022-07-18T15:40:51.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2022-11-21T03:37:17.000Z (over 3 years ago)
- Last Synced: 2025-06-01T07:12:26.080Z (about 1 year ago)
- Topics: learning-to-optimize
- Language: Python
- Homepage:
- Size: 375 KB
- Stars: 6
- Watchers: 10
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Scalable-L2O
Implementation of ECCV 2022 paper: Scalable Learning to Optimize: A Learned Optimizer Can Train Big Models.
## Environment
We recommend using Anaconda to manage the virtual environment.
```bash
conda env create -f environment.yaml
conda activate sl2o
```
## Experiments
### CNNs
#### Subspaces
We provide pre-generated subspaces for models in this [link](https://drive.google.com/drive/folders/1PVfLiBXg-n_YknBAlaN8ZpRFiP5B_zdd?usp=sharing). Optionally, one can generate the subspaces by themselves. Please refer to the `subspaces` directory for more details.
#### Meta-Training
ResNet-18 (CIFAR-10)
```bash
python -u src/resnet18_ft_de.py --max_epoch 20 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --meta_train_eval_epoch 2
```
ResNet-18 (CIFAR-100)
```bash
python -u src/resnet18_ft_de.py --max_epoch 20 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --meta_train_eval_epoch 2 --dataset CIFAR100
```
ResNet8 (CIFAR-10)
```bash
python -u src/resnet8_ft_de.py --max_epoch 20 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --meta_train_eval_epoch 2
```
ResNet20 (CIFAR-10)
```bash
python -u src/resnet20_ft_de.py --max_epoch 20 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --meta_train_eval_epoch 2
```
#### Meta-Testing
ResNet-8 (CIFAR-10)
```bash
python -u src/resnet8_eval_de.py --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --max_epoch 100
```
ResNet-20 (CIFAR-10)
```bash
python -u src/resnet20_eval_de.py --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --max_epoch 100
```
### VITs
#### Meta-Training
```bash
python -u src/vit_ft.py --max_epoch 20 --lora_dim 16 --lora_alpha 32 --lora_dropout 0.1 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 64 --unroll 10 --random_seed 1 --name cifar10-100_500 --dataset cifar10 --model_type ViT-B_16 --pretrained_dir checkpoint/ViT-B_16.npz --meta_train_eval_epoch 2
```
#### Meta-Testing
```bash
python -u src/vit_ft_eval.py --max_epoch 20 --lora_dim 16 --lora_alpha 32 --lora_dropout 0.1 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 10 --training_steps 1000 --batch-size 64 --unroll 10 --random_seed 1 --name cifar10-100_500 --dataset cifar10 --model_type ViT-B_16 --pretrained_dir checkpoint/ViT-B_16.npz --random_seed 1 --eval_interval 391
```