https://github.com/ypeleg/nfnets-keras

Keras implementation of Normalizer-Free Networks and SGD - Adaptive Gradient Clipping
https://github.com/ypeleg/nfnets-keras

Last synced: 25 days ago
JSON representation

Keras implementation of Normalizer-Free Networks and SGD - Adaptive Gradient Clipping

• Host: GitHub
• URL: https://github.com/ypeleg/nfnets-keras
• Owner: ypeleg
• License: mit
• Created: 2021-02-14T23:36:21.000Z (over 3 years ago)
• Default Branch: master
• Last Pushed: 2021-07-12T11:29:10.000Z (over 3 years ago)
• Last Synced: 2024-09-28T13:40:55.522Z (about 1 month ago)
• Language: Python
• Size: 21.5 KB
• Stars: 68
• Watchers: 7
• Forks: 21
• Open Issues: 11
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

README

        

# Keras implementation of Normalizer-Free Networks and SGD - Adaptive Gradient Clipping

Paper: https://arxiv.org/abs/2102.06171.pdf

Original code: https://github.com/deepmind/deepmind-research/tree/master/nfnets

Do star this repository if it helps your work!

> Note: Huge Credit to [this comment](https://github.com/vballoli/nfnets-pytorch) for the pytorch implementation this repository is based on.

> Note: See [this comment](https://github.com/vballoli/nfnets-pytorch/issues/1#issuecomment-778853439) for a generic implementation for any optimizer as a temporary reference for anyone who needs it.

# Installation

Install from PyPi:

`pip3 install nfnets-keras`

or install the latest code using:

`pip3 install git+https://github.com/ypeleg/nfnets-keras`

# Usage

## NFNetF Model

Use any of the `NFNetF` models like any other keras Model!

```python

from nfnets_keras import NFNetF3

model = NFNetF3(include_top = True, num_classes = 10)

model.compile('adam', 'categorical_crossentropy')

model.fit(X, y)

```

## WSConv2D

Use `WSConv2D` like any other `keras.layer`.

```python

from nfnets_keras import WSConv2D

conv = Conv2D(16, 3)(l)

w_conv = WSConv2D(16, 3)(conv)

```

## SGD_AGC - Adaptive Gradient Clipping

Similarly, use `SGD_AGC` like `keras.optimizer.SGD`

```python

from nfnets_keras import SGD_AGC

model.compile( SGD_AGC(lr=1e-3), loss='categorical_crossentropy' )

```

# TODO

- [x] WSConv2D

- [x] SGD - Adaptive Gradient Clipping

- [x] Function to automatically replace Convolutions in any module with WSConv2d

- [x] Documentation

- [x] NFNets

- [ ] NF-ResNets

# Credit for the original pytroch implementation 

```

https://github.com/vballoli/nfnets-pytorch

```

# Cite Original Work

To cite the original paper, use:

```

@article{brock2021high,

  author={Andrew Brock and Soham De and Samuel L. Smith and Karen Simonyan},

  title={High-Performance Large-Scale Image Recognition Without Normalization},

  journal={arXiv preprint arXiv:},

  year={2021}

}

```