https://github.com/peterding/lamb

An Implementation for the optimizer LAMB with Tensorflow 2.0
https://github.com/peterding/lamb

deep-learning lamb nlp optimizer

Last synced: 10 months ago
JSON representation

An Implementation for the optimizer LAMB with Tensorflow 2.0

• Host: GitHub
• URL: https://github.com/peterding/lamb
• Owner: PeterDing
• Created: 2019-10-25T05:04:38.000Z (about 6 years ago)
• Default Branch: master
• Last Pushed: 2019-10-25T08:19:15.000Z (about 6 years ago)
• Last Synced: 2025-01-12T17:47:10.155Z (12 months ago)
• Topics: deep-learning, lamb, nlp, optimizer
• Language: Python
• Homepage:
• Size: 4.88 KB
• Stars: 1
• Watchers: 3
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# An Implementation for the optimizer [LAMB](https://arxiv.org/abs/1904.00962) with Tensorflow 2.0

We use pure python code inherted from `tensorflow.keras.optimizers.Adam` to implement the LAMB algorithm ( Layerwise Adaptive Large Latch optimization).

This implementation uses L2 regularization for all weights excepted for layernorm and all biases.