Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/cshjin/awesome-nonconvex-optimization

A collection of papers and readings for non-convex optimization
https://github.com/cshjin/awesome-nonconvex-optimization

List: awesome-nonconvex-optimization

awesome-list machine-learning nonconvex optimization stochastic-optimization

Last synced: about 2 months ago
JSON representation

A collection of papers and readings for non-convex optimization

Awesome Lists containing this project

README 

        
# Awesome Non-convex Optimization (in order of time)



Last edit: March, 2018



## Before 2010s  

2004, Nesterov : textbook    

[Introductory Lectures on Convex Programming Volume: A Basic course](http://www.springer.com/us/book/9781402075537)  

By Yurii Nesterov



2005, Nemirovski : SIOPT  

[Prox-method with rate of convergence o(1/t)   for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems](http://epubs.siam.org/doi/pdf/10.1137/S1052623403425629)  

By Arkadi Nemirovski



2005, Nesterov : Mathematics Programming  

[Smooth minimization of non-smooth functions](https://link.springer.com/article/10.1007/s10107-004-0552-5)  

By Yurii Nesterov



2006, Nesterov : Cubic regularization  

[Cubic regularization of Newton method and its global performance](https://link.springer.com/content/pdf/10.1007%2Fs10107-006-0706-8.pdf)  

By Yurii Nesterov, B.T. Polyak



## 2011  

2011, Chambolle-Pock:   

[A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging](https://link.springer.com/article/10.1007/s10851-010-0251-1)  

By Antonin Chambolle and Thomas Pock



## 2012  

2012, LeRoux-Schmidt-Bach, : SAG  

[A stochastic gradient method with an exponential convergence rate for finite training sets](https://arxiv.org/abs/1202.6258)  

By Nicolas Le Roux, Mark Schmidt, and Francis R. Bach



2012, ShalevShwartz-Zhang, : SDCA  

[Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization](https://arxiv.org/abs/1209.1873)  

By Shai Shalev-Shwartz and Tong Zhang



2012, Nesterov : RCDM/ACDM  

[Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems](https://pdfs.semanticscholar.org/0059/cfac9c5b7811866f0729d0917b7478148fc5.pdf)  

By Yurii Nesterov.



## 2013  

2013, ShalevShwartz-Zhang, : AccSDCA  

[Accelerated Proximal Stochastic Dual Coordinate Ascent for Regularized Loss Minimization](https://arxiv.org/abs/1309.2375)  

by Shai Shalev-Shwartz, Tong Zhang



2013, BenTal-Nemirovski,   

[Lectures on Modern Convex Optimization](http://epubs.siam.org/doi/book/10.1137/1.9780898718829)  

By Aharon Ben-Tal and Arkadi Nemirovski



2013, Johnson-Zhang, : SVRG  

[Accelerating stochastic gradient descent using predictive variance reduction](http://dl.acm.org/citation.cfm?id=2999647)  

By Rie Johnson and Tong Zhang



2013, Lee-Sidford, : ACDM (better proof)    

[Efficient accelerated coordinate descent methods and faster algorithms for solving linear systems](https://arxiv.org/abs/1305.1922)  

By Yin Tat Lee and Aaron Sidford



2013, Zhang-Mahdavi-Jin,   

[Linear convergence with condition number independent access of full gradients](https://papers.nips.cc/paper/4940-linear-convergence-with-condition-number-independent-access-of-full-gradients)  

By Lijun Zhang, Mehrdad Mahdavi, and Rong Jin.



2013, Mahdavi-Zhang-Jin   

[Mixed optimization for smooth functions](https://papers.nips.cc/paper/4941-mixed-optimization-for-smooth-functions)  

By Mehrdad Mahdavi, Lijun Zhang, and Rong Jin.



2013, Ghadimi-Lan,  

[Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://arxiv.org/pdf/1309.5549.pdf)  

By Saeed Ghadimi, Guanghui Lan  



## 2014  

2014, Lin-Lu-Xiao, : APCG  

[An Accelerated Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimization](https://arxiv.org/abs/1407.1296)  

By Qihang Lin, Zhaosong Lu, Lin Xiao



2014, Defazio-Bach-LacosteJulien, : SAGA  

[SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://www.di.ens.fr/~fbach/Defazio_NIPS2014.pdf)   

By Defazio, A., Bach, F., & Lacoste-Julien, S.



2014, ShalevShwartz-Zhang, : AccSDCA  

[Accelerated Proximal Stochastic Dual Coordinate Ascent for Regularized Loss Minimization](https://arxiv.org/pdf/1309.2375.pdf)  

By Shai Shalev-Shwartz and Tong Zhang



2014, Su-Boyd-Candes,   

[A differential equation for modeling nesterovs accelerated gradient method: Theory and insights](https://arxiv.org/pdf/1503.01243.pdf)  

By Weijie Su, Stephen Boyd, Emmanuel J. Candès



2014, AllenZhu-Orecchia,   

[Linear Coupling: An Ultimate Unification of Gradient and Mirror Descent](https://arxiv.org/pdf/1407.1537.pdf)  

By Zeyuan Allen-Zhu and Lorenzo Orecchia.



2014, Nitanda,  

[Stochastic proximal gradient descent with acceleration techniques](https://papers.nips.cc/paper/5610-stochastic-proximal-gradient-descent-with-acceleration-techniques.pdf)  

By Atsushi Nitanda.



2014, Sun-Luo,   

[Guaranteed Matrix Completion via Non-convex Factorization](https://arxiv.org/pdf/1411.8003.pdf)  

By Ruoyu Sun and Zhi-Quan Luo



## 2015  

2015, Zhang-Xiao, : SPDC  

[Stochastic Primal-Dual Coordinate Method for Regularized Empirical Risk Minimization](https://arxiv.org/pdf/1409.3257.pdf)  

By Yuchen Zhang and Lin Xiao.



2015, Lan-Zhou, : RPDG  

[An optimal randomized incremental gradient method](https://arxiv.org/pdf/1507.02000.pdf)  

By Guanghui Lan and Yi Zhou.



2015, Frostig-Ge-Kakade-Sidford, : APPA  

[Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization](https://arxiv.org/pdf/1506.07512.pdf)  

By Roy Frostig, Rong Ge, Sham M. Kakade, and Aaron Sidford



2015, Lin-Mairal-Harchaoui, : Catalyst  

[A universal catalyst for first-order optimization](https://arxiv.org/pdf/1506.02186.pdf)  

By Hongzhou Lin, Julien Mairal and Zaid Harchaoui



2015, AllenZhu-Yuan, : SVRG++   

[Improved SVRG for Non-Strongly-Convex or Sum-of-Non-Convex Objectives](https://arxiv.org/pdf/1506.01972.pdf)  

By Zeyuan Allen-Zhu and Yang Yuan



2015, Bubeck-Lee-Singh,   

[A geometric alternative to Nesterov's accelerated gradient descent](https://arxiv.org/pdf/1506.08187.pdf)  

By Sébastien Bubeck, Yin Tat Lee, Mohit Singh



2015, Garber et al, : shift and invert (two papers about the same result)    

[Faster Eigenvector Computation via Shift-and-Invert Preconditioning](https://arxiv.org/pdf/1605.08754.pdf)  

By Dan Garber, Elad Hazan, Chi Jin, Sham M. Kakade, Cameron Musco, Praneeth Netrapalli, Aaron Sidford



2015, Garber-Hazan,   

[Fast and Simple PCA via Convex Optimization](https://arxiv.org/pdf/1509.05647.pdf)  

By Dan Garber and Elad Hazan



2015, Arora-Ge-Ma-Moitra,   

[Simple, Efficient, and Neural Algorithms for Sparse Coding](https://arxiv.org/pdf/1503.00778.pdf)  

By Sanjeev Arora, Rong Ge, Tengyu Ma and Ankur Moitra



2015, Chen-Candès,   

[Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems](https://arxiv.org/pdf/1505.05114.pdf)  

By Yuxin Chen, Emmanuel J. Candès



2015, ShalevShwartz,   

[SDCA without Duality](https://arxiv.org/pdf/1602.01582.pdf)  

By Shai Shalev-Shwartz



2015, Ge-Huang-Jin-Yuan,   

[Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition](https://arxiv.org/pdf/1503.02101.pdf)  

By Rong Ge, Furong Huang, Chi Jin, Yang Yuan



## 2016  

2016, AllenZhu-Qu-Richtarik-Yuan, : NUACDM  

[Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling](https://arxiv.org/pdf/1512.09103.pdf)  

By Zeyuan Allen-Zhu, Zheng Qu, Peter Richtarik and Yang Yuan.



2016b, AllenZhu-Hazan : reduction  

[Optimal Black-Box Reductions Between Optimization Objectives](http://papers.nips.cc/paper/6364-optimal-black-box-reductions-between-optimization-objectives.pdf)  

By Zeyuan Allen-Zhu and Elad Hazan



2016a, AllenZhu-Hazan : non-convex SVRG  

[Variance Reduction for Faster Non-Convex Optimization](https://arxiv.org/pdf/1603.05643.pdf)  

By Zeyuan Allen-Zhu and Elad Hazan



2016, Wibisono-Wilson-Jordan,   

[A Variational Perspective on Accelerated Methods in Optimization](https://arxiv.org/pdf/1603.04245.pdf)  

By Andre Wibisono, Ashia C. Wilson, Michael I. Jordan



2016, AllenZhu,   

[Katyusha: The First Direct Acceleration of Stochastic Gradient Methods](https://arxiv.org/pdf/1603.05953.pdf)  

By Zeyuan Allen-Zhu



2016, Woodworth-Srebro,   

[Tight Complexity Bounds for Optimizing Composite Objectives](https://arxiv.org/pdf/1605.08003.pdf)  

By Blake Woodworth and Nathan Srebro



2016, Frostig-Musco-Musco-Sidford, : PCR  

[Principal Component Projection Without Principal Component Analysis](https://arxiv.org/abs/1602.06872)  

By Roy Frostig, Cameron Musco, Christopher Musco and Aaron Sidford.



2016, AllenZhu-Li, : LazySVD  

[LazySVD: Even Faster SVD Decomposition Yet Without Agonizing Pain](https://arxiv.org/pdf/1607.03463.pdf)  

By Zeyuan Allen-Zhu and Yuanzhi Li.



2016a, Reddi et al.,   

[Stochastic variance reduction for nonconvex optimization](https://arxiv.org/pdf/1603.06160.pdf)  

By Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola



2016b, Reddi et al.,   

[Fast incremental method for nonconvex optimization](https://arxiv.org/pdf/1603.06159.pdf)  

Sashank J Reddi, Suvrit Sra, Barnabás Póczos, and Alex Smola.



2016c, Reddi et al.,   

[Fast Stochastic Methods for Nonsmooth Nonconvex Optimization](https://arxiv.org/pdf/1605.06900.pdf)  

Sashank J Reddi, Suvrit Sra, Barnabás Póczos, and Alex Smola.



2016, Carmon-Duchi-Hinder-Sidford,   

[Accelerated Methods for Non-Convex Optimization](https://arxiv.org/pdf/1611.00756.pdf)  

By Yair Carmon, John C. Duchi, Oliver Hinder and Aaron Sidford



2016, Agarwal et al.,   

[Finding Approximate Local Minima Faster Than Gradient Descent](https://arxiv.org/pdf/1611.01146.pdf)  

By Naman Agarwal, Zeyuan Allen-Zhu, Brian Bullins, Elad Hazan, Tengyu Ma



## 2017  

2017, Lei-Jordan,   

[Less than a Single Pass: Stochastically Controlled Stochastic Gradient](https://arxiv.org/pdf/1609.03261.pdf)  

By Lihua Lei and Michael I. Jordan.



2017, Lei-Ju-Chen-Jordan,   

[Nonconvex Finite-Sum Optimization Via SCSG Methods](https://arxiv.org/pdf/1706.09156.pdf)  

By Lihua Lei, Cheng Ju, Jianbo Chen, and Michael I. Jordan.



2017, AllenZhu-Li, : PCR  

[Faster Principal Component Regression and Stable Matrix Chebyshev Approximation](https://arxiv.org/pdf/1608.04773.pdf)  

By Zeyuan Allen-Zhu and Yuanzhi Li



2017, Shibagaki-Takeuchi, : mini-batch SPDC  

[Stochastic primal dual coordinate method with nonuniform sampling based on optimality violations](https://arxiv.org/pdf/1703.07056.pdf)  

By Atsushi Shibagaki and Ichiro Takeuchi.



2017, Murata-Suzuki, : DASVRDA  

[Doubly accelerated stochastic variance reduced dual averaging method for regularized empirical risk minimization](http://papers.nips.cc/paper/6663-doubly-accelerated-stochastic-variance-reduced-dual-averaging-method-for-regularized-empirical-risk-minimization.pdf)  

By Tomoya Murata and Taiji Suzuki



2017, Li-Yuan,   

[Convergence Analysis of Two-layer Neural Networks with ReLU Activation](https://arxiv.org/pdf/1705.09886.pdf)  

By Yuanzhi Li and Yang Yuan



2017, AllenZhu,   

[Natasha: Faster Non-Convex Stochastic Optimization via Strongly Non-Convex Parameter](https://arxiv.org/pdf/1702.00763.pdf)  

By Zeyuan Allen-Zhu



2017, Carmon- Duchi-Hinder-Sidford,   

["Convex Until Proven Guilty": Dimension-Free Acceleration of Gradient Descent on Non-Convex Functions](https://arxiv.org/pdf/1705.02766.pdf)  

By Yair Carmon, John C. Duchi, Oliver Hinder and Aaron Sidford



2017, Jin et al.,   

[How to Escape Saddle Points Efficiently](https://arxiv.org/pdf/1703.00887.pdf)  

By Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, Michael I. Jordan



2017, Zhou-Mertikopoulos-Bambos-Boyd-Glynn  

[Mirror descent in non-convex stochastic programming](https://arxiv.org/pdf/1706.05681.pdf)  

By Zhengyuan Zhou, Panayotis Mertikopoulos, Nicholas Bambos, Stephen Boyd, Peter Glynn  



## 2018

2018, AllenZhu  

[Natasha2: Faster Non-Convex Optimization Than SGD](https://arxiv.org/pdf/1708.08694.pdf)  

By Zeyuan Allen-Zhu



2018, AllenZhu  

[Katyusha X: Practical Momentum Method for Stochastic Sum-of-Nonconvex Optimization](https://arxiv.org/pdf/1802.03866.pdf)  

By Zeyuang Allen-Zhu



2018, AllenZhu-Li  

[Neon2: Finding Local Minima via First-Order Oracles](https://arxiv.org/pdf/1711.06673.pdf)  

By Zeyuang Allen-Zhu, Yuanzhi Li



2018, Yun-Stra-Jadbabaie  

[Global Optimality Conditions for Deep Neural Networks](https://arxiv.org/pdf/1707.02444.pdf)  

By Chulhee Yun, Suvrit Sra, Ali Jadbabaie



2018, Hong-Lee-Razaviyayn  

[Gradient Primal-Dual Algorithm Converges to Second-Order Stationary Solutions for Nonconvex Distributed Optimization](https://arxiv.org/pdf/1802.08941.pdf)  

By Mingyi Hong, Jason D. Lee, Meisam Razaviyayn



2018, Zhang-Aragam-Ravikumar-Xing  

[DAGs with NO TEARS: Smooth Optimization for Structure Learning](https://arxiv.org/pdf/1803.01422.pdf)  

By Xun Zheng, Bryon Aragam, Pradeep Ravikumar, Eric P. Xing



2018, Xu-Jin-Yang: NEON  

[First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Time](https://arxiv.org/abs/1711.01944)  

By Yi Xu, Rong Jin, Tianbao Yang



## Licenses



License



[![CC0](https://licensebuttons.net/p/zero/1.0/88x31.png)](http://creativecommons.org/publicdomain/zero/1.0/)



To the extent possible under law, [Hongwei Jin](https://www.cs.uic.edu/~hjin) has waived all copyright and related or neighboring rights to this work.