https://github.com/hengluchang/visualizing_momentum

Visualizing Gradient Descent with Momentum in Python
https://github.com/hengluchang/visualizing_momentum

deep-learning gradient-descent machienlearning momentum optimization visualization

Last synced: about 2 months ago
JSON representation

Visualizing Gradient Descent with Momentum in Python

• Host: GitHub
• URL: https://github.com/hengluchang/visualizing_momentum
• Owner: hengluchang
• License: mit
• Created: 2018-08-14T03:40:00.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2018-08-14T04:12:49.000Z (over 6 years ago)
• Last Synced: 2024-03-20T06:10:30.516Z (10 months ago)
• Topics: deep-learning, gradient-descent, machienlearning, momentum, optimization, visualization
• Language: Python
• Size: 7.81 KB
• Stars: 25
• Watchers: 1
• Forks: 11
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# visualizing_momentum

Visualizing Gradient Descent with Momentum in Python, check out the blog post [here](https://medium.com/@hengluchang/visualizing-gradient-descent-with-momentum-in-python-7ef904c8a847)! 

# Pre-reqs

Python 3.6

# Libraries

- matplotlib

- numpy

# How to run

`python3 loss_surface.py` to generate loss surface figure. 

`python3 momentum.py` to generate weight trajectories and velocity-iteration plots. 

# References

- [An overview of gradient descent algorithms](http://ruder.io/optimizing-gradient-descent/index.html#momentum)

- [On the importance of initialization and momentum in deep learning](http://www.cs.toronto.edu/~fritz/absps/momentum.pdf)

- [Why momentum really works](https://distill.pub/2017/momentum/)

- [Momentum and learning rate adaptation](https://www.willamette.edu/~gorr/classes/cs449/momrate.html)