Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
awesome-soms
A curated list of resources for second-order stochastic optimization
https://github.com/cor3bit/awesome-soms
- Numerical Optimization
- Introduction to Optimization and Data Fitting
- Optimization for Machine Learning
- Topics in Machine Learning: Neural Net Training Dynamics (Winter 2022)
- Optimization Methods for Large-Scale Machine Learning
- Exact and inexact subsampled Newton methods for optimization
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
- PyHessian: Neural Networks Through the Lens of the Hessian
- A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization
- AdaHessian: An Adaptive Second Order Optimizer for Machine Learning
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
- Learning Recurrent Neural Networks with Hessian-Free Optimization
- Training Neural Networks with Stochastic Hessian-Free Optimization
- A Stochastic Quasi-Newton Method for Large-Scale Optimization
- A Multi-Batch L-BFGS Method for Machine Learning
- Stochastic Quasi-Newton with Line-Search Regularization
- Practical Quasi-Newton Methods for Training Deep Neural Networks
- Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks
- On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNs
- Stochastic Gauss-Newton Algorithms for Nonconvex Compositional Optimization
- Nonlinear Least Squares for Large-Scale Machine Learning using Stochastic Jacobian Estimates
- Improving Levenberg-Marquardt Algorithm for Neural Networks
- Rethinking Gauss-Newton for learning over-parameterized models
- Exact Gauss-Newton Optimization for Training Deep Neural Networks
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Second-order optimization with lazy Hessians
- Optax - mostly first-order accelerated methods
- Somax - second-order stochastic solvers
- JAXopt - deterministic second-order methods (e.g., Gauss-Newton, Levenberg Marquardt), stochastic first-order methods PolyakSGD, ArmijoSGD
- KFAC-JAX - implementation of KFAC from the DeepMind team
- AdaHessianJax - implementation of the AdaHessian optimizer by Nestor Demeure
Programming Languages