{"id":102818,"url":"https://github.com/mlpapers/optimization","name":"optimization","description":"Awesome papers on Optimization","projects_count":35,"last_synced_at":"2026-04-14T03:00:19.822Z","repository":{"id":301067502,"uuid":"253094685","full_name":"mlpapers/optimization","owner":"mlpapers","description":"Awesome papers on Optimization","archived":false,"fork":false,"pushed_at":"2026-02-14T21:38:10.000Z","size":6,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-03-30T08:12:42.447Z","etag":null,"topics":["autodiff","awesome","awesome-list","differentiation","gradient","optimization","optimization-algorithms"],"latest_commit_sha":null,"homepage":"https://mlpapers.org/optimization/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlpapers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-04-04T20:40:56.000Z","updated_at":"2026-02-14T21:38:13.000Z","dependencies_parsed_at":"2026-03-02T12:01:08.252Z","dependency_job_id":null,"html_url":"https://github.com/mlpapers/optimization","commit_stats":null,"previous_names":["mlpapers/optimization"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mlpapers/optimization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlpapers%2Foptimization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlpapers%2Foptimization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlpapers%2Foptimization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlpapers%2Foptimization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlpapers","download_url":"https://codeload.github.com/mlpapers/optimization/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlpapers%2Foptimization/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31779947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T02:24:21.117Z","status":"ssl_error","status_checked_at":"2026-04-14T02:24:20.627Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"readme":"# Optimization ([Wiki](https://en.wikipedia.org/wiki/Mathematical_optimization))\n\n- **Overview**\n  - [An overview of gradient descent optimization algorithms](https://arxiv.org/pdf/1609.04747.pdf) (2017) *Sebastian Ruder*\n  - [Derivative tests](https://en.wikipedia.org/wiki/Derivative_test#First-derivative_test)\n\n### Linear programming\n- **Simplex algorithm** ([Wiki](https://en.wikipedia.org/wiki/Simplex_algorithm))\n\n### Gradient-based optimization\n- **Batch gradient descent**\n- **Stochastic gradient descent**\n  - [An Alternative View:  When Does SGD Escape Local Minima?](https://arxiv.org/pdf/1802.06175.pdf) (2018) *Robert Kleinberg, Yuanzhi Li, Yang Yuan*\n- **Mini-batch gradient descent**\n- **Momentum**\n- **Nesterov accelerated gradient**\n- **Adagrad**\n- **Adadelta**\n- **RMSprop**\n- **Adam, AdaMax**\n  - [Adam: A Method for Stochastic Optimization](https://arxiv.org/pdf/1412.6980.pdf) (2017) *Diederik P. Kingma, Jimmy Ba*\n- **TAdam**\n  - [TAdam: A Robust Stochastic Gradient Optimizer](https://arxiv.org/pdf/2003.00179.pdf) (2020) *Wendyam Eric Lionel Ilboudo, Taisuke Kobayashi, Kenji Sugimoto*\n- **Nadam**\n- **AMSGrad**\n- **LaProp** ([Code](https://github.com/Z-T-WANG/LaProp-Optimizer))\n  - [LaProp: a Better Way to Combine Momentum with Adaptive Gradient](https://arxiv.org/pdf/2002.04839.pdf) (2020) *Liu Ziyin, Zhikang T.Wang, Masahito Ueda*\n\n### Bayesian optimization\n  - [A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning](https://arxiv.org/pdf/1012.2599.pdf) (2010) *Eric Brochu, Vlad M. Cora, Nando de Freitas*\n  - [Practical Bayesian Optimization of Machine Learning Algorithms](https://arxiv.org/pdf/1206.2944.pdf) (2012) *Jasper Snoek, Hugo Larochelle, Ryan P. Adams*\n  - [Taking the Human Out of the Loop: A Review of Bayesian Optimization](https://www.cs.ox.ac.uk/people/nando.defreitas/publications/BayesOptLoop.pdf) (2016) *Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, Nando de Freitas*\n- **REMBO** \n  - [Bayesian Optimization in High Dimensions via Random Embeddings](https://www.cs.ubc.ca/~hutter/papers/13-IJCAI-BO-highdim.pdf) (2013) *Ziyu Wang, Masrour Zoghi, Frank Hutter, David Matheson, Nando De Freitas*\n  - [On the choice of the low-dimensional domain for globaloptimization via random embedding](https://arxiv.org/pdf/1704.05318.pdf) (2018) *Mickael Binois, David Ginsbourger, Olivier Roustant*\n- **HeSBO**\n  - [A Framework for Bayesian Optimization in Embedded Subspaces](http://proceedings.mlr.press/v97/nayebi19a/nayebi19a.pdf) (2019) *Alexander Munteanu, Amin Nayebi, Matthias Poloczek*\n- **SIRBO**\n  - [High Dimensional Bayesian Optimization via Supervised Dimension Reduction](https://arxiv.org/pdf/1907.08953.pdf) (2019) *Miao Zhang, Huiqi Li, Steven Su*\n- **SI-BO**\n- **SILBO**\n  - [Semi-supervised Embedding Learning for High-dimensionalBayesian Optimization](https://arxiv.org/pdf/2005.14601.pdf) (2020) *Jingfan Chen, Guanghui Zhu, Rong Gu, Chunfeng Yuan, Yihua Huang*\n- **BOHB** - Bayesian Optimization with Hyperband\n  - [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](http://proceedings.mlr.press/v80/falkner18a/falkner18a.pdf) (2018) *Stefan Falkner, Aaron Klein, Frank Hutter*\n\n### Evolutionary algorithms ([Wiki](https://en.wikipedia.org/wiki/Evolutionary_algorithm))\n- Imitate some aspects of natural evolution\n- **GA** Genetic algorithm ([Wiki](https://en.wikipedia.org/wiki/Genetic_algorithm))\n- **MA** Memetic algorithm ([Wiki](https://en.wikipedia.org/wiki/Memetic_algorithm))\n- **GP** Genetic programming\n- **ES** Evolutionary strategies\n- **EP** Evolutionary programming\n- **LCS** Learning classifier systems\n- **Harmony search**\n- **PBT** Population-based Training\n  - [Population Based Training of Neural Networks](https://arxiv.org/pdf/1711.09846.pdf) (2017) *Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, Koray Kavukcuoglu*\n\n### Trajectory-based algorithms\n- **Simulated annealing**\n  - [Approximate Solution of Certain Types of Constrained Optimization Problems](https://pubsonline.informs.org/doi/pdf/10.1287/opre.18.6.1225) (1970) *Martin Pincus*\n  - [Optimization by Simulated Annealing](http://leonidzhukov.net/hse/2013/stochmod/papers/KirkpatrickGelattVecchi83.pdf) (1983) *S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi*\n- **Hill climbing**\n- **b-Hill climbing**\n- **Tabu search**\n- **Variable neighborhood search**\n\n### Swarm-based algorithms\n- **Artificial bee colony**\n- **Cuckoo search**\n- **Firefly algorithm**\n- **Particle swarm** ([Wiki](https://en.wikipedia.org/wiki/Particle_swarm_optimization))\n  - [Particle Swarm Optimization](https://www.cs.tufts.edu/comp/150GA/homeworks/hw3/_reading6%201995%20particle%20swarming.pdf) (1995) *James Kennedy, Russell Eberhart*\n\n### Multi-armed bandits ([Wiki](https://en.wikipedia.org/wiki/Multi-armed_bandit))\n- [Some Aspects of the Sequential Design of Experiments](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.335.3232\u0026rep=rep1\u0026type=pdf) (1952) *Herbert Robbins*\n- [Multi-armed Bandit Algorithmsand Empirical Evaluation](http://bandit.sourceforge.net/Vermorel2005poker.pdf) (2005) *Joannes Vermorel, Mehryar Mohri*\n- [A modern Bayesian look at the multi-armed bandit](http://www.economics.uci.edu/~ivan/asmb.874.pdf) (2010) *Steven L. Scott*\n\n### Heuristic algorithms\n\n### Multimodel optimization\n\n### Multiobjective optimization\n  - [Pareto set](https://en.wikipedia.org/wiki/Pareto_efficiency#Use_in_engineering)\n\n## Software\n- **Python**\n  - Sherpa ([Docs](https://parameter-sherpa.readthedocs.io/en/latest/), [Code](https://github.com/sherpa-ai/sherpa), [Paper](https://arxiv.org/pdf/2005.04048.pdf))\n  - PyMOO ([Homepage](https://pymoo.org/), [Paper](https://arxiv.org/pdf/2002.04504.pdf))\n\n- **C++**\n  - Emsmallen ([Homepage](https://ensmallen.org/))\n\n## Related Topics\n- [Neural Networks](https://mlpapers.org/neural-nets/)\n- [AutoML](https://mlpapers.org/automl/)\n- [Bayesian Inference](https://mlpapers.org/bayesian-inference/)\n- [Reinforcement Learning](https://mlpapers.org/reinforcement-learning/)\n","created_at":"2026-01-02T00:00:41.067Z","updated_at":"2026-04-14T03:00:19.822Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["Uncategorized","Software","Related Topics"],"sub_categories":["Uncategorized"],"projects_url":"https://awesome.ecosyste.ms/api/v1/lists/mlpapers%2Foptimization/projects"}