awesome-soms
A curated list of resources for second-order stochastic optimization
https://github.com/cor3bit/awesome-soms
Last synced: 1 day ago
JSON representation
-
Books and Lecture Notes
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Introduction to Optimization and Data Fitting
- Optimization for Machine Learning
- Topics in Machine Learning: Neural Net Training Dynamics (Winter 2022)
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
- Numerical Optimization
-
Papers
-
Overview
-
Analysis of the Hessian
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
- PyHessian: Neural Networks Through the Lens of the Hessian
- A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization
-
Diagonal Scaling
-
Hessian-free Optimization
-
Quasi-Newton
-
Gauss-Newton
- Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks
- On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNs
- Stochastic Gauss-Newton Algorithms for Nonconvex Compositional Optimization
- Nonlinear Least Squares for Large-Scale Machine Learning using Stochastic Jacobian Estimates
- Improving Levenberg-Marquardt Algorithm for Neural Networks
- Rethinking Gauss-Newton for learning over-parameterized models
- Exact Gauss-Newton Optimization for Training Deep Neural Networks
-
Fisher Information
-
Other
-
-
Implementation in JAX
-
Other
- Optax - mostly first-order accelerated methods
- Somax - second-order stochastic solvers
- JAXopt - deterministic second-order methods (e.g., Gauss-Newton, Levenberg Marquardt), stochastic first-order methods PolyakSGD, ArmijoSGD
- KFAC-JAX - implementation of KFAC from the DeepMind team
- AdaHessianJax - implementation of the AdaHessian optimizer by Nestor Demeure
-
Programming Languages
Sub Categories