https://github.com/wassname/viz_torch_optim

Videos of deep learning optimizers moving on 3D problem-landscapes
https://github.com/wassname/viz_torch_optim

3d-graphics deep-learning learning-rate machine-learning optimization-algorithms optimizer pytorch vizualisation

Last synced: 7 months ago
JSON representation

Videos of deep learning optimizers moving on 3D problem-landscapes

• Host: GitHub
• URL: https://github.com/wassname/viz_torch_optim
• Owner: wassname
• Created: 2017-11-14T12:32:55.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2024-07-25T10:12:07.000Z (almost 2 years ago)
• Last Synced: 2025-04-04T15:02:06.939Z (about 1 year ago)
• Topics: 3d-graphics, deep-learning, learning-rate, machine-learning, optimization-algorithms, optimizer, pytorch, vizualisation
• Language: Jupyter Notebook
• Homepage:
• Size: 21.1 MB
• Stars: 108
• Watchers: 4
• Forks: 13
• Open Issues: 1
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

README

          
This project generates animations of pytorch optimizers solving toy problems. Examples Below.

[Some nice animations](http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html) were posted a few years ago by Alex Radford but didn't include the Adam optimizer or landscapes with noise.  [Louis Tiao](http://louistiao.me/notes/visualizing-and-animating-optimization-algorithms-with-matplotlib/) blogged about how to make the visualizations. The [pytorch unit tests](https://github.com/pytorch/pytorch/blob/master/test/test_optim.py) show how to run the optimizers on test functions. I pulled these together and shared the result at https://github.com/wassname/viz_torch_optim.

# Examples

Please note that each optimizer has a differen't learning rate, so they are not directly comparable. This is because simpler optimizers perform better on low dimensional problems and are often given a smaller learning rate. With the same learning rate, the simpler SGD optimizer races to the finish while Adam crawls along. In that case SGD is too fast to see and Adam is too boring to watch. For visualisation purposes I used differen't learning rates for each optimizer to make them move at similar speeds in the video.

## Beales function

![](docs/videos/beales_20171117_00-02-20_2d.gif)

## Beales function (cyclic annealing)

![](docs/videos/beales_CyclicLR_20171117_04-51-12_2d.gif)

## Beales function with noise

![](docs/videos/beales_None_20171124_04-24-27_2d.gif)

## Six humped camel function

![](docs/videos/six_humped_camel_back_20171115_09-38-57.gif)

![](docs/videos/six_humped_camel_back_20171115_09-38-57_3d.gif)

# Usage:

- `git clone https://github.com/wassname/viz_torch_optim`

- `jupyter notebook`

- open main.ipynb

- install any missing dependencies with pip

- Run and wait, because rendering video is quite slow