https://github.com/arn4/master-thesis

Learning transitions for one-pass stochastic gradient descent on shallow neural networks
https://github.com/arn4/master-thesis

neural-networks thesis

Last synced: about 11 hours ago
JSON representation

Learning transitions for one-pass stochastic gradient descent on shallow neural networks

• Host: GitHub
• URL: https://github.com/arn4/master-thesis
• Owner: arn4
• Created: 2022-07-18T14:18:53.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2022-10-05T12:59:58.000Z (about 2 years ago)
• Last Synced: 2023-03-21T13:07:11.389Z (over 1 year ago)
• Topics: neural-networks, thesis
• Language: TeX
• Homepage:
• Size: 11.1 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
## *Learning transitions for one-pass stochastic gradient descent on shallow neural networks*

**Supervisors**: [F. Krzakala](krzakala.org), [B. Loureiro](https://brloureiro.github.io)

**Abstract**. In recent years, neural networks have made possible great progress in several fields of artificial intelligence, but their theoretical understanding is still lacking. In this thesis, we study the high-dimensional input limit of a two-layer neural network, through statistical physic tools. Using the squared activation function we are able to derive some ODEs for the dynamics of sufficient statistics, that can then be used for estimating time of transition between learning phases. We apply this analysis to the simplest case known as phase retrieval, exploring different kinds of initial conditions. We then study the dynamics with the weights constrained on a hypersphere; we estimate the exit time from the first phase of learning, therefrom we derive an estimate of the gain that occurs by overparameterizing the network. We conclude by adding a stochastic corrective term to the equations, showing that this leads to a better estimation of the exit times.

### Resources

- **[Summary](https://uz.sns.it/~arna/static_/ext_files/master-thesis/summary.pdf)**

- **[Thesis](https://uz.sns.it/~arna/static_/ext_files/master-thesis/thesis.pdf)**

- **Slides**: 

  - [Handout](https://uz.sns.it/~arna/static_/ext_files/master-thesis/slides-handout.pdf)

  - [Timings](https://uz.sns.it/~arna/static_/ext_files/master-thesis/slides.pdf)