https://github.com/titu1994/keras_novograd
Keras implementation of NovoGrad
https://github.com/titu1994/keras_novograd
Last synced: about 1 month ago
JSON representation
Keras implementation of NovoGrad
- Host: GitHub
- URL: https://github.com/titu1994/keras_novograd
- Owner: titu1994
- License: mit
- Created: 2019-08-17T01:37:21.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2020-08-21T05:06:33.000Z (about 5 years ago)
- Last Synced: 2025-04-11T08:32:41.759Z (6 months ago)
- Language: Python
- Size: 378 KB
- Stars: 20
- Watchers: 3
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# NovoGrad for Keras
Keras port of [NovoGrad](https://github.com/NVIDIA/OpenSeq2Seq/blob/master/open_seq2seq/optimizers/novograd.py), from the paper [Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks.](https://arxiv.org/abs/1905.11286)
# NovoGrad
The above image is from the paper. NovoGrad makes the optimizer more resilient to choice of initial learning rate as it behaves similarly to SGD but with gradient normalization per layer. It extends ND-Adam and also decouples weight decay from regularization. Also, it has only half the memory cost as compared to Adam, and similar memory requirements to SGD with Momentum. This allows larger models to be trained without compromizing training efficiency.
## Usage
Add the `novograd.py` script to your project, and import it. Can be a dropin replacement for `Adam` Optimizer.
Note that NovoGrad also supports "AMSGrad"-like behaviour with the `amsgrad=True` flag.
```python
from novograd import NovoGrad
optm = NovoGrad(lr=1e-2)
```
# Requirements
- Keras 2.2.4+ & Tensorflow 1.14+ (Only supports TF backend for now).
- Numpy