https://github.com/shreyansh26/gradient-descent-on-neural-networks-typically-occurs-at-the-edge-of-stability

A re-implementation of the paper "Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability" by Cohen et al.
https://github.com/shreyansh26/gradient-descent-on-neural-networks-typically-occurs-at-the-edge-of-stability

Last synced: 7 months ago
JSON representation

A re-implementation of the paper "Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability" by Cohen et al.

• Host: GitHub
• URL: https://github.com/shreyansh26/gradient-descent-on-neural-networks-typically-occurs-at-the-edge-of-stability
• Owner: shreyansh26
• Created: 2022-12-29T02:54:18.000Z (almost 3 years ago)
• Default Branch: master
• Last Pushed: 2022-12-29T03:36:03.000Z (almost 3 years ago)
• Last Synced: 2025-01-14T02:14:22.316Z (9 months ago)
• Language: Jupyter Notebook
• Homepage:
• Size: 422 KB
• Stars: 3
• Watchers: 4
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1dc4Q9W4KmV5FJg8GTkpKQ-Hbnx43iNWD?usp=sharing)

A re-implementation of the paper [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://arxiv.org/abs/2103.00065v3) by Cohen et al., specifically the experiment described in Appendix L.2 of the paper.

Thanks to the Honghua Dong and Tianxing Li at University of Toronto, for their [implementation](https://colab.research.google.com/drive/1yXyJIAAqFHAV_uNoW5WGBIfyp-5gpGWu?usp=sharing). I learnt a lot from it and extended the implementation to do Gradient Flow training as well.

All the plots and the code can be found in the Jupyter notebook.