Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/YuchenJin/autolrs

Automatic learning-rate scheduler
https://github.com/YuchenJin/autolrs

automl deep-learning hyperparameter-tuning learning-rate-scheduling machine-learning

Last synced: 3 months ago
JSON representation

Automatic learning-rate scheduler

Lists

README 

        
# AutoLRS 



This is the PyTorch code implementation for the paper [_AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly_](https://openreview.net/forum?id=SlrqM9_lyju) published at ICLR 2021.



A TensorFlow version will appear in this repo later.



## What is AutoLRS?

Finding a good learning rate schedule for a DNN model is non-trivial. The goal of AutoLRS is to automatically tune the learning rate (LR) over the course of training without human involvement. AutoLRS chops up the whole training process into a few training stages (each consists of τ steps), and its mission is to determine a constant LR for each training stage. AutoLRS treats the validation loss as a black-box function of LR, and uses Bayesian optimization (BO) to search for the best LR which can minimize the validation loss for each training stage. Because BO would require τ steps of training to evaluate the validation loss for each LR it explores, to reduce this cost, we only apply an LR to train the DNN for τ’ (τ’ << τ) steps and train an exponential time-series forecasting model to predict the loss after τ steps. In our default setting, τ’ = τ/10 and BO explores 10 LRs in each stage, so the number of steps for searching LR is equal to the number of steps for actual training.



AutoLRS does not depend on a pre-defined LR schedule, dataset, or a specified task and is compatible with almost all optimizers. The LR schedules auto-generated by AutoLRS lead to speedup over highly hand-tuned LR schedules for several state-of-the-art DNNs including ResNet-50, Transformer, and BERT.



## Setup

```sh

$ pip install --user -r requirements.txt

```

## How to use AutoLRS for your work?

`autolrs_server.py` is the brain of AutoLRS, which implements the search algorithm including BO and the exponential forecasting model.



`autolrs_callback.py` implements a callback which you can plug into your Pytorch training loop. The callback receives commands from the server via socket, adjusting the learning rate, saving/restoring model parameters and optimizer states according to commands sent from the server.



## Notes

* You need to pass two arguments `min_lr` and `max_lr` when launching `autolrs_server.py` to set the LR search interval. This interval can be found by an LR range test or simply set according to your experience. Do not set the min_lr too small (for example 1e-10), otherwise, BO will waste a lot of cycles to try exploring very small LR values.

* The current AutoLRS does not search LR for warmup steps since warmup does not have an explicit optimization objective, such as minimizing the validation loss. Warmup usually takes very few steps, and its main purpose is to prevent deeper layers in a DNN from creating training instability, especially when training using a large batch size. You can manually add a warmup stage by setting `warmup_step` and `warmup_lr` when initializing the `autolrs_callback.AutoLRS` callback.



## Example

We provide an example of using AutoLRS to train various DNNs on the CIFAR-10 dataset. The models are imported from `kuangliu`'s great and simple [pytorch-cifar](https://github.com/kuangliu/pytorch-cifar) repository.



__Prerequisites:__ 

Python 3.6+, PyTorch 1.0+



__Run the example__ 

```sh

$ bash run.sh

```



## Contact 

+ [Yuchen Jin](http://yuchenjin.github.io/)



You can contact us at . We would love to hear your questions and feedback!



## Poster