{"id":7214,"url":"https://github.com/harsh306/awesome-nn-optimization","name":"awesome-nn-optimization","description":"Awesome list for Neural Network Optimization methods.","projects_count":145,"last_synced_at":"2026-07-31T03:00:22.383Z","repository":{"id":73719658,"uuid":"200449457","full_name":"harsh306/awesome-nn-optimization","owner":"harsh306","description":"Awesome list for Neural Network Optimization methods.","archived":false,"fork":false,"pushed_at":"2025-02-17T07:57:20.000Z","size":207,"stargazers_count":83,"open_issues_count":1,"forks_count":12,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-07-12T01:03:23.196Z","etag":null,"topics":["awesome","awesome-list","bifurcation","continuation","convergence-analysis","convex-optimization","curriculum-learning","deep-learning","dynamical-systems","generalization","local-minima","loss-surface","neural-network","non-convex-optimization","optimization"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc-by-4.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/harsh306.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":"harshpathak","tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2019-08-04T04:28:29.000Z","updated_at":"2026-05-13T21:27:39.000Z","dependencies_parsed_at":"2024-01-02T03:29:28.975Z","dependency_job_id":"f5115dff-5230-4816-b094-b099416321eb","html_url":"https://github.com/harsh306/awesome-nn-optimization","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/harsh306/awesome-nn-optimization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harsh306%2Fawesome-nn-optimization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harsh306%2Fawesome-nn-optimization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harsh306%2Fawesome-nn-optimization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harsh306%2Fawesome-nn-optimization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/harsh306","download_url":"https://codeload.github.com/harsh306/awesome-nn-optimization/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harsh306%2Fawesome-nn-optimization/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36099841,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-31T02:00:06.731Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-01-12T13:37:55.652Z","updated_at":"2026-07-31T03:00:22.384Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["Content"],"sub_categories":[],"readme":"## Content\n\n#### Popular Optimization algorithms\n- SGD [[Book]](https://www.deeplearningbook.org/contents/optimization.html)\n- Momentum [[Book]](https://www.deeplearningbook.org/contents/optimization.html)\n- RMSProp [[Book]](https://www.deeplearningbook.org/contents/optimization.html)\n- AdaGrad [[Link]](http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf)\n- ADAM [[Link]](https://arxiv.org/abs/1412.6980)\n- AdaBound [[Link]](https://arxiv.org/abs/1902.09843) [[Github]](https://github.com/Luolc/AdaBound)\n- ADAMAX [[Link]](https://arxiv.org/abs/1412.6980)\n- NADAM [[Link]](https://openreview.net/pdf?id=OM0jvwB8jIp57ZJjtNEZ)\n- ADAMW [[Link]](https://openreview.net/forum?id=rk6qdGgCZ)\n- AdaLOMO [Link](https://arxiv.org/pdf/2310.10195.pdf)\n- All optimizers list [Awesome-Optimizer](https://github.com/zoq/Awesome-Optimizer)\n\n#### Normalization Methods\n- BatchNorm [[Link]](https://arxiv.org/abs/1502.03167)\n- Weight Norm [[Link]](http://papers.nips.cc/paper/6113-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural-networks)\n- Spectral Norm [[Link]](https://arxiv.org/abs/1802.05957)\n- Cosine Normalization [[Link]](https://arxiv.org/pdf/1702.05870.pdf)\n- L2 Regularization versus Batch and Weight Normalization [Link](https://arxiv.org/pdf/1706.05350.pdf) \n- WHY GRADIENT CLIPPING ACCELERATES TRAINING: A THEORETICAL JUSTIFICATION FOR ADAPTIVITY [Link](https://openreview.net/pdf?id=BJgnXpVYwS)\n\n#### On Convexity and Generalization of Neural Networks\n- Convex Neural Networks [[Link]](http://papers.nips.cc/paper/2800-convex-neural-networks.pdf)\n- Breaking the Curse of Dimensionality with Convex Neural Networks [[Link]](http://jmlr.org/papers/volume18/14-546/14-546.pdf)\n- UNDERSTANDING DEEP LEARNING REQUIRES RETHINKING GENERALIZATION [[Link]](https://arxiv.org/pdf/1611.03530.pdf)\n- Optimal Control Via Neural Networks: A Convex Approach. [[Link]](https://openreview.net/forum?id=H1MW72AcK7)\n- Input Convex Neural Networks [[Link]](https://arxiv.org/pdf/1609.07152.pdf)\n- A New Concept of Convex based Multiple Neural Networks Structure. [[Link](http://www.ifaamas.org/Proceedings/aamas2019/pdfs/p1306.pdf)\n- SGD Converges to Global Minimum in Deep Learning via Star-convex Path [[Link]](https://arxiv.org/abs/1901.00451)\n- A Convergence Theory for Deep Learning via Over-Parameterization [Link](https://arxiv.org/abs/1811.03962)\n\n#### Continuation Methods and Curriculum Learning \n- Curriculum Learning [[Link]](https://ronan.collobert.com/pub/matos/2009_curriculum_icml.pdf)\n- SOLVING RUBIK’S CUBE WITH A ROBOT HAND [Link](https://arxiv.org/pdf/1910.07113.pdf)\n- Noisy Activation Function [[Link]](http://proceedings.mlr.press/v48/gulcehre16.pdf)\n- Mollifying Networks [[Link]](https://arxiv.org/abs/1608.04980)\n- Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks [Link](https://arxiv.org/pdf/1802.03796.pdf) [Talk](https://vimeo.com/287808087)\n- Automated Curriculum Learning for Neural Networks [Link](http://proceedings.mlr.press/v70/graves17a/graves17a.pdf)\n- On The Power of Curriculum Learning in Training Deep Networks [Link](https://arxiv.org/pdf/1904.03626.pdf)\n- On-line Adaptative Curriculum Learning for GANs [Link](https://arxiv.org/abs/1808.00020)\n- Parameter Continuation with Secant Approximation for Deep Neural Networks and Step-up GAN [Link](https://digitalcommons.wpi.edu/etd-theses/1256/)\n- HashNet: Deep Learning to Hash by Continuation. [[Link]](https://arxiv.org/abs/1702.00758)\n- Learning Combinations of Activation Functions. [[Link]](https://arxiv.org/pdf/1801.09403.pdf)\n- Learning and development in neural networks: The importance of starting small (1993) [Link](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.128.4487\u0026rep=rep1\u0026type=pdf)\n- Flexible shaping: How learning in small steps helps [Link](https://www.sciencedirect.com/science/article/pii/S0010027708002850)\n- Curriculum Labeling: Self-paced Pseudo-Labeling for Semi-Supervised Learning [Link](https://arxiv.org/pdf/2001.06001.pdf)\n- RETHINKING CURRICULUM LEARNING WITH INCREMENTAL LABELS AND ADAPTIVE COMPENSATION [Link](https://arxiv.org/pdf/2001.04529.pdf)\n- Parameter Continuation Methods for the Optimization of Deep Neural Networks [Link](https://ieeexplore.ieee.org/abstract/document/8999318)\n- Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection [Link (https://www.aclweb.org/anthology/W18-6314.pdf)\n- Reinforcement Learning based Curriculum Optimization for Neural Machine Translation [Link](https://www.aclweb.org/anthology/N19-1208.pdf)\n- EVOLUTIONARY POPULATION CURRICULUM FOR SCALING MULTI-AGENT REINFORCEMENT LEARNING [Link](https://openreview.net/pdf?id=SJxbHkrKDH)\n- ENTROPY-SGD: BIASING GRADIENT DESCENT INTO WIDE VALLEYS [Link](https://arxiv.org/pdf/1611.01838.pdf)\n- NEIGHBOURHOOD DISTILLATION: ON THE BENEFITS OF NON END-TO-END DISTILLATION [Link](https://arxiv.org/abs/2010.01189)\n- LEARNING TO EXECUTE [Link](https://arxiv.org/pdf/1410.4615.pdf)\n- Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing [Link](https://arxiv.org/pdf/1903.10145.pdf)\n- Data Parameters: A New Family of Parameters for Learning a Differentiable Curriculum [Link](https://proceedings.neurips.cc/paper/2019/file/926ffc0ca56636b9e73c565cf994ea5a-Paper.pdf)\n- Breaking the Curse of Space Explosion: Towards Effcient NAS with Curriculum Search [Link](http://proceedings.mlr.press/v119/guo20b.html)\n- Continuation Methods and Curriculum Learning for Learning to Rank [Link](http://www.dei.unipd.it/~ferro/papers/2018/CIKM2018_FLMP.pdf)\n\n\n#### On Loss Surfaces and Generalization of Deep Neural Networks\n- Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape [Link](https://arxiv.org/abs/2409.14396)\n- Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape [Link](https://arxiv.org/pdf/2201.08025)\n- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks [Link](https://arxiv.org/abs/1312.6120)\n- QUALITATIVELY CHARACTERIZING NEURAL NETWORK OPTIMIZATION PROBLEMS[[Link]](https://arxiv.org/pdf/1412.6544.pdf)\n- The Loss Surfaces of Multilayer Networks [[Link]](https://arxiv.org/abs/1412.0233)\n- Visualizing the Loss Landscape of Neural Nets [[Link]](https://papers.nips.cc/paper/7875-visualizing-the-loss-landscape-of-neural-nets.pdf)\n- The Loss Surface Of Deep Linear Networks Viewed Through The Algebraic Geometry Lens [[Link]](https://arxiv.org/pdf/1810.07716.pdf)\n- How regularization affects the critical points in linear\nnetworks.[[Link]](http://papers.nips.cc/paper/6844-how-regularization-affects-the-critical-points-in-linear-networks.pdf)\n- Local minima in training of neural networks [[Link]](https://arxiv.org/abs/1611.06310)\n- Necessary and Sufficient Geometries for Gradient Methods [Link](http://papers.nips.cc/paper/9325-necessary-and-sufficient-geometries-for-gradient-methods)\n- Fine-grained Optimization of Deep Neural Networks [Link](http://papers.nips.cc/paper/8425-fine-grained-optimization-of-deep-neural-networks)\n- SCORE-BASED GENERATIVE MODELING THROUGH STOCHASTIC DIFFERENTIAL EQUATIONS [Link](https://openreview.net/pdf?id=PxTIG12RRHS)\n\n#### Dynamics, Bifurcations and  RNNs difficulty to train\n- Deep Equilibrium Models [Link](http://papers.nips.cc/paper/8358-deep-equilibrium-models.pdf)\n-  Bifurcations of Recurrent Neural Networks in Gradient Descent Learning [[Link]](https://pdfs.semanticscholar.org/b579/27b713a6f9b73c7941f99144165396483478.pdf)\n- On the difficulty of training recurrent neural networks [[Link]](http://proceedings.mlr.press/v28/pascanu13.pdf)\n- Understanding and Controlling Memory in Recurrent Neural Networks [[Link]](https://arxiv.org/pdf/1902.07275.pdf)\n- Dynamics and Bifurcation of Neural Networks [[Link]](https://pdfs.semanticscholar.org/a413/4a36fef5ef55d0ff7dae029d6b8f55140cf7.pdf)\n- Context Aware Machine Learning [[Link]](https://arxiv.org/pdf/1901.03415.pdf)\n- The trade-off between long-term memory and smoothness for recurrent networks [[Link]](https://arxiv.org/pdf/1906.08482.pdf)\n- Dynamical complexity and computation in recurrent neural networks beyond their fxed point [[Link]](https://www.nature.com/articles/s41598-018-21624-2.pdf)\n- Bifurcations in discrete-time neural networks : controlling complex network behaviour with inputs [[Links]](https://pub.uni-bielefeld.de/record/2302580)\n- Interpreting Recurrent Neural Networks Behaviour via Excitable Network Attractors [[Link]](https://link.springer.com/article/10.1007/s12559-019-09634-2#Sec11)\n- Bifurcation analysis of a neural network model [Link](https://link.springer.com/article/10.1007/BF00203668)\n- A Differentiable Physics Engine for Deep Learning in Robotics [Link](https://www.frontiersin.org/articles/10.3389/fnbot.2019.00006/full)\n- Deep learning for universal linear embeddings\nof nonlinear dynamics [Link](https://arxiv.org/pdf/1712.09707.pdf)\n- Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations [Link](http://www.jmlr.org/papers/volume19/18-046/18-046.pdf)\n- Analysis of gradient descent learning algorithms for multilayer feedforward neural networks [Link](https://ieeexplore.ieee.org/abstract/document/203921)\n- A dynamical model for the analysis and acceleration of learning in feedforward networks [Link](https://www.sciencedirect.com/science/article/abs/pii/S0893608001000521)\n- A bio-inspired bistable recurrent cell allows for long-lasting memory [Link](https://arxiv.org/abs/2006.05252)\n- Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation [Link (https://www.frontiersin.org/articles/10.3389/fncom.2017.00024/full)\n\n#### Poor Local Minima? and Sharp Minima\n- Adding One Neuron Can Eliminate All Bad\nLocal Minima [Link](https://papers.nips.cc/paper/7688-adding-one-neuron-can-eliminate-all-bad-local-minima.pdf)\n- Deep Learning without Poor Local Minima [Link](https://papers.nips.cc/paper/6112-deep-learning-without-poor-local-minima.pdf)\n- Elimination of All Bad Local Minima in Deep Learning [Link](https://arxiv.org/pdf/1901.00279.pdf)\n- How to escape saddle points efficiently. [Link](https://arxiv.org/pdf/1703.00887.pdf)\n- Depth with Nonlinearity Creates No Bad Local Minima in ResNets [Link](https://arxiv.org/abs/1810.09038)\n- Sharp Minima Can Generalize For Deep Nets [Link](https://arxiv.org/pdf/1703.04933.pdf)\n- Asymmetric Valleys: Beyond Sharp and Flat Local\nMinima [Link](https://papers.nips.cc/paper/2019/file/01d8bae291b1e4724443375634ccfa0e-Paper.pdf)\n- A Reparameterization-Invariant Flatness Measure for Deep Neural Networks [Link](https://arxiv.org/pdf/1912.00058.pdf)\n- A Simple Weight Decay Can Improve Generalization [Link](https://papers.nips.cc/paper/1991/file/8eefcfdf5990e441f0fb6f3fad709e21-Paper.pdf)\n- Finding Critical and Gradient-Flat Points of Deep Neural Network Loss Functions [Link](https://escholarship.org/content/qt4fw6x5b3/qt4fw6x5b3_noSplash_14ef3ae1644808c863f9b2eb344addcc.pdf?t=qhtt5i)\n- The Loss Surface Of Deep Linear Networks Viewed Through The Algebraic Geometry Lens [Link](https://arxiv.org/pdf/1810.07716.pdf)\n- Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization [Link](https://arxiv.org/pdf/1908.09375.pdf)\n- Flatness is a False Friend [Link](https://arxiv.org/pdf/2006.09091.pdf)\n- Are_Saddles_Good_Enough_for_Deep_Learning [Link](https://www.researchgate.net/publication/317399405_Are_Saddles_Good_Enough_for_Deep_Learning)\n\n#### Initialization of Neural Network\n- Deep learning course notes [Link](https://www.deeplearning.ai/ai-notes/initialization/)\n- On the importance of initialization and momentum in deep learning [Link](http://proceedings.mlr.press/v28/sutskever13.html)\n- The Break-Even Point on Optimization Trajectories of Deep Neural Networks [Link](https://arxiv.org/abs/2002.09572)\n- THE EARLY PHASE OF NEURAL NETWORK TRAINING [Link](https://research.fb.com/wp-content/uploads/2020/02/The-Early-Phase-of-Neural-Network-Training.pdf?)\n- One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers [Link](http://papers.nips.cc/paper/8739-one-ticket-to-win-them-all-generalizing-lottery-ticket-initializations-across-datasets-and-optimizers.pdf)\n- PCA-Initialized Deep Neural Networks Applied To Document Image Analysis [Link](https://arxiv.org/abs/1702.00177)\n- Understanding the difficulty of training deep feedforward neural networks [Link](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf?hc_location=ufi])\n- Unitary Evolution of RNNs [Link](https://arxiv.org/abs/1511.06464)\n\n#### Momentum in Optimization\n- RETHINKING THE HYPERPARAMETERS FOR FINE-TUNING [Link](https://openreview.net/pdf?id=B1g8VkHFPH)\n- Momentum Residual Neural Networks [Link](https://proceedings.mlr.press/v139/sander21a.html)\n- Smooth momentum: improving lipschitzness in gradient descent [Link](https://doi.org/10.1007/s10489-022-04207-7)\n- Momentum-based Weight Interpolation of Strong\nZero-Shot Models for Continual Learning [link](https://arxiv.org/pdf/2211.03186.pdf)\n\n#### Batch size Optimiation \n- ON LARGE-BATCH TRAINING FOR DEEP LEARNING: GENERALIZATION GAP AND SHARP MINIMA[Link](https://arxiv.org/pdf/1609.04836.pdf)\n- Revisiting Small Batch Training for Deep Neural Networks [Link](https://arxiv.org/abs/1804.07612)\n- LARGE BATCH TRAINING OF CONVOLUTIONAL NETWORKS [Link](https://arxiv.org/pdf/1708.03888.pdf)\n- Large Batch Optimization for Deep Learning: Training BERT in 76 minutes [Link](https://arxiv.org/abs/1904.00962)\n- DON’T DECAY THE LEARNING RATE, INCREASE THE BATCH SIZE [Link](https://arxiv.org/abs/1711.00489)\n\n#### Degeneracy of Neural Networks\n- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks [Link](https://arxiv.org/pdf/1312.6120.pdf)\n- Avoiding pathologies in very deep networks [Link](https://arxiv.org/abs/1402.5836)\n- Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice [Link](https://arxiv.org/abs/1711.04735)\n- SKIP CONNECTIONS ELIMINATE SINGULARITIES [Link](https://openreview.net/pdf?id=HkwBEMWCZ)\n- How degenerate is the parametrization of neural networks with the ReLU activation function? [Link](https://arxiv.org/pdf/1905.09803.pdf)\n- Theory of Deep Learning III: explaining the non-overfitting puzzle [Link](https://cbmm.mit.edu/sites/default/files/publications/CBMM-Memo-073v2_0.pdf)\n- Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks [Link](https://openreview.net/forum?id=rkgqN1SYvr)\n- Understanding Deep Learning: Expected Spanning Dimension and Controlling the Flexibility of Neural Networks [Link](https://www.frontiersin.org/articles/10.3389/fams.2020.572539/full)\n- The Loss Surface Of Deep Linear Networks Viewed Through The Algebraic Geometry Lens [Link](https://arxiv.org/pdf/1810.07716.pdf)\n- PYHESSIAN: Neural Networks Through the Lens of the Hessian [Link](https://arxiv.org/pdf/1912.07145.pdf)\n\n#### Convergencec Analysis in Deep Learning\n- A CONVERGENCE ANALYSIS OF GRADIENT DESCENT FOR DEEP LINEAR NEURAL NETWORKS [Link](https://openreview.net/pdf?id=SkMQg3C5K7)\n- A Convergence Theory for Deep Learning via Over-Parameterization [Link](http://proceedings.mlr.press/v97/allen-zhu19a/allen-zhu19a.pdf)\n- Convergence Analysis of Homotopy-SGD for Non-Convex Optimization [Link](https://openreview.net/forum?id=Twf5rUVeU-I)\n\n#### Multi-Task Learning with curricula\n- Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning. [Link](https://www.aclweb.org/anthology/P16-1013.pdf)\n- Learning a Multitask Curriculum for Neural Machine Translation. [Link](https://arxiv.org/pdf/1908.10940.pdf)\n- Self-paced Curriculum Learning. [Link](http://www.cs.cmu.edu/~lujiang/camera_ready_papers/AAAI_SPCL_2015.pdf)\n- Curriculum Learning of Multiple Tasks. [Link](http://openaccess.thecvf.com/content_cvpr_2015/papers/Pentina_Curriculum_Learning_of_2015_CVPR_paper.pdf)\n\n#### Constrained Optimization for Deep Learning\n- A Primal-Dual Formulation for Deep Learning with Constraints [Link](https://papers.nips.cc/paper/9385-a-primal-dual-formulation-for-deep-learning-with-constraints.pdf)\n\n#### Reinforcement Learning and Curriculum\n- Object-Oriented Curriculum Generation for Reinforcement Learning [Link](http://ifaamas.org/Proceedings/aamas2018/pdfs/p1026.pdf)\n- Teacher-Student Curriculum Learning [Link](https://arxiv.org/abs/1707.00183)\n\n\n#### Tutorials, Surveys and Blogs\n- Curriculum Learning: A Survey [Link](https://arxiv.org/pdf/2101.10382.pdf)\n- A Comprehensive Survey on Curriculum Learning [Link](https://arxiv.org/pdf/2010.13166.pdf)\n- https://www.offconvex.org/ \n- An overview of gradient descent optimization algorithms [[Link]](https://arxiv.org/pdf/1609.04747.pdf)\n- Review of second-order optimization techniques in artificial neural networks backpropagation [Link](https://iopscience.iop.org/article/10.1088/1757-899X/495/1/012003/pdf#:~:text=Second%2Dorder%20optimization%20technique%20is,training%20phase%20of%20neural%20network.)\n- Linear Algebra and data [Link](https://github.com/harsh306/ML_Notes/blob/master/linear_algebra.md)\n- Why Momentum really works?[[Blog]](https://distill.pub/2017/momentum/)\n- Optimization [[Book]](https://www.deeplearningbook.org/contents/optimization.html)\n- Optimization for deep learning: theory and algorithms [Link](https://arxiv.org/pdf/1912.08957.pdf)\n- Generalization Error in Deep Learning [Link](https://arxiv.org/pdf/1808.01174.pdf)\n- Automatic Differentiation in Machine Learning: a Survey [Link](https://arxiv.org/pdf/1502.05767.pdf)\n- Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey [Link](https://arxiv.org/pdf/2003.04960.pdf)\n- Automatic Curriculum Learning For Deep RL: A Short Survey [Link](https://arxiv.org/abs/2003.04664)\n- The Generalization Mystery: Sharp vs Flat Minima [Link](https://www.inference.vc/sharp-vs-flat-minima-are-still-a-mystery-to-me/)\n\n#### Contributing\nIf you've found any informative resources that you think belong here, be sure to submit a pull request or create an issue! \n\n##### If you find this helpful, I can enjoy a coffee donation :) \n\n- [![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/F1F02R7JR)\n- Or send me 2-4 dollars on my venmo account [@HARSHNILESH-PATHAK](https://venmo.com/HARSHNILESH-PATHAK)\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/harsh306%2Fawesome-nn-optimization/projects"}