{"id":13589920,"url":"https://github.com/jettify/pytorch-optimizer","last_synced_at":"2025-05-13T18:11:25.518Z","repository":{"id":39580078,"uuid":"231503549","full_name":"jettify/pytorch-optimizer","owner":"jettify","description":"torch-optimizer -- collection of optimizers for Pytorch","archived":false,"fork":false,"pushed_at":"2024-03-22T11:10:03.000Z","size":48349,"stargazers_count":3111,"open_issues_count":55,"forks_count":306,"subscribers_count":31,"default_branch":"master","last_synced_at":"2025-05-12T16:14:18.115Z","etag":null,"topics":["accsgd","adabelief","adabound","adamod","apollo","diffgrad","hacktoberfest","lamb","lookahead","novograd","optimizer","pytorch","sgdp","shampoo","swats","yogi"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jettify.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGES.rst","contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-01-03T03:16:39.000Z","updated_at":"2025-05-11T01:50:39.000Z","dependencies_parsed_at":"2024-06-18T12:27:19.761Z","dependency_job_id":null,"html_url":"https://github.com/jettify/pytorch-optimizer","commit_stats":{"total_commits":402,"total_committers":26,"mean_commits":"15.461538461538462","dds":"0.49502487562189057","last_synced_commit":"910b414565427f0a66e20040475e7e4385e066a5"},"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jettify%2Fpytorch-optimizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jettify%2Fpytorch-optimizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jettify%2Fpytorch-optimizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jettify%2Fpytorch-optimizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jettify","download_url":"https://codeload.github.com/jettify/pytorch-optimizer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254000855,"owners_count":21997442,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accsgd","adabelief","adabound","adamod","apollo","diffgrad","hacktoberfest","lamb","lookahead","novograd","optimizer","pytorch","sgdp","shampoo","swats","yogi"],"created_at":"2024-08-01T16:00:36.239Z","updated_at":"2025-05-13T18:11:25.493Z","avatar_url":"https://github.com/jettify.png","language":"Python","readme":"torch-optimizer\n===============\n.. image:: https://github.com/jettify/pytorch-optimizer/workflows/CI/badge.svg\n   :target: https://github.com/jettify/pytorch-optimizer/actions?query=workflow%3ACI\n   :alt: GitHub Actions status for master branch\n.. image:: https://codecov.io/gh/jettify/pytorch-optimizer/branch/master/graph/badge.svg\n    :target: https://codecov.io/gh/jettify/pytorch-optimizer\n.. image:: https://img.shields.io/pypi/pyversions/torch-optimizer.svg\n    :target: https://pypi.org/project/torch-optimizer\n.. image:: https://readthedocs.org/projects/pytorch-optimizer/badge/?version=latest\n    :target: https://pytorch-optimizer.readthedocs.io/en/latest/?badge=latest\n    :alt: Documentation Status\n.. image:: https://img.shields.io/pypi/v/torch-optimizer.svg\n    :target: https://pypi.python.org/pypi/torch-optimizer\n.. image:: https://static.deepsource.io/deepsource-badge-light-mini.svg\n    :target: https://deepsource.io/gh/jettify/pytorch-optimizer/?ref=repository-badge\n\n\n**torch-optimizer** -- collection of optimizers for PyTorch_ compatible with optim_\nmodule.\n\n\nSimple example\n--------------\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.DiffGrad(model.parameters(), lr=0.001)\n    optimizer.step()\n\n\nInstallation\n------------\nInstallation process is simple, just::\n\n    $ pip install torch_optimizer\n\n\nDocumentation\n-------------\nhttps://pytorch-optimizer.rtfd.io\n\n\nCitation\n--------\nPlease cite the original authors of the optimization algorithms. If you like this\npackage::\n\n    @software{Novik_torchoptimizers,\n    \ttitle        = {{torch-optimizer -- collection of optimization algorithms for PyTorch.}},\n    \tauthor       = {Novik, Mykola},\n    \tyear         = 2020,\n    \tmonth        = 1,\n    \tversion      = {1.0.1}\n    }\n\nOr use the github feature: \"cite this repository\" button.\n\n\nSupported Optimizers\n====================\n\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `A2GradExp`_  | https://arxiv.org/abs/1810.00553                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `A2GradInc`_  | https://arxiv.org/abs/1810.00553                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `A2GradUni`_  | https://arxiv.org/abs/1810.00553                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `AccSGD`_     | https://arxiv.org/abs/1803.05591                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `AdaBelief`_  | https://arxiv.org/abs/2010.07468                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `AdaBound`_   | https://arxiv.org/abs/1902.09843                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `AdaMod`_     | https://arxiv.org/abs/1910.12249                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `Adafactor`_  | https://arxiv.org/abs/1804.04235                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `Adahessian`_ | https://arxiv.org/abs/2006.00719                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `AdamP`_      | https://arxiv.org/abs/2006.08217                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `AggMo`_      | https://arxiv.org/abs/1804.00325                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `Apollo`_     | https://arxiv.org/abs/2009.13586                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `DiffGrad`_   | https://arxiv.org/abs/1909.11015                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `Lamb`_       | https://arxiv.org/abs/1904.00962                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `Lookahead`_  | https://arxiv.org/abs/1907.08610                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `MADGRAD`_    | https://arxiv.org/abs/2101.11075                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `NovoGrad`_   | https://arxiv.org/abs/1905.11286                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `PID`_        | https://www4.comp.polyu.edu.hk/~cslzhang/paper/CVPR18_PID.pdf                                                                        |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `QHAdam`_     | https://arxiv.org/abs/1810.06801                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `QHM`_        | https://arxiv.org/abs/1810.06801                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `RAdam`_      | https://arxiv.org/abs/1908.03265                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `Ranger`_     | https://medium.com/@lessw/new-deep-learning-optimizer-ranger-synergistic-combination-of-radam-lookahead-for-the-best-of-2dc83f79a48d |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `RangerQH`_   | https://arxiv.org/abs/1810.06801                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `RangerVA`_   | https://arxiv.org/abs/1908.00700v2                                                                                                   |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `SGDP`_       | https://arxiv.org/abs/2006.08217                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `SGDW`_       | https://arxiv.org/abs/1608.03983                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `SWATS`_      | https://arxiv.org/abs/1712.07628                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `Shampoo`_    | https://arxiv.org/abs/1802.09568                                                                                                     |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n|               |                                                                                                                                      |\n| `Yogi`_       | https://papers.nips.cc/paper/8186-adaptive-methods-for-nonconvex-optimization                                                        |\n+---------------+--------------------------------------------------------------------------------------------------------------------------------------+\n\n\nVisualizations\n--------------\nVisualizations help us see how different algorithms deal with simple\nsituations like: saddle points, local minima, valleys etc, and may provide\ninteresting insights into the inner workings of an algorithm. Rosenbrock_ and Rastrigin_\nbenchmark_ functions were selected because:\n\n* Rosenbrock_ (also known as banana function), is non-convex function that has\n  one global minimum  `(1.0. 1.0)`. The global minimum is inside a long,\n  narrow, parabolic shaped flat valley. Finding the valley is trivial. \n  Converging to the global minimum, however, is difficult. Optimization\n  algorithms might pay a lot of attention to one coordinate, and struggle\n  following the valley which is relatively flat.\n\n .. image::  https://upload.wikimedia.org/wikipedia/commons/3/32/Rosenbrock_function.svg\n\n* Rastrigin_ is a non-convex function  and has one global minimum in `(0.0, 0.0)`.\n  Finding the minimum of this function is a fairly difficult problem due to\n  its large search space and its large number of local minima.\n\n  .. image::  https://upload.wikimedia.org/wikipedia/commons/8/8b/Rastrigin_function.png\n\nEach optimizer performs `501` optimization steps. Learning rate is the best one found\nby a hyper parameter search algorithm, the rest of the tuning parameters are default. It\nis very easy to extend the script and tune other optimizer parameters.\n\n\n.. code::\n\n    python examples/viz_optimizers.py\n\n\nWarning\n-------\nDo not pick an optimizer based on visualizations, optimization approaches\nhave unique properties and may be tailored for different purposes or may\nrequire explicit learning rate schedule etc. The best way to find out is to try \none on your particular problem and see if it improves scores.\n\nIf you do not know which optimizer to use, start with the built in SGD/Adam. Once\nthe training logic is ready and baseline scores are established, swap the optimizer \nand see if there is any improvement.\n\n\nA2GradExp\n---------\n\n+--------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_A2GradExp.png   |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_A2GradExp.png  |\n+--------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.A2GradExp(\n        model.parameters(),\n        kappa=1000.0,\n        beta=10.0,\n        lips=10.0,\n        rho=0.5,\n    )\n    optimizer.step()\n\n\n**Paper**: *Optimal Adaptive and Accelerated Stochastic Gradient Descent* (2018) [https://arxiv.org/abs/1810.00553]\n\n**Reference Code**: https://github.com/severilov/A2Grad_optimizer\n\n\nA2GradInc\n---------\n\n+--------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_A2GradInc.png   |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_A2GradInc.png  |\n+--------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.A2GradInc(\n        model.parameters(),\n        kappa=1000.0,\n        beta=10.0,\n        lips=10.0,\n    )\n    optimizer.step()\n\n\n**Paper**: *Optimal Adaptive and Accelerated Stochastic Gradient Descent* (2018) [https://arxiv.org/abs/1810.00553]\n\n**Reference Code**: https://github.com/severilov/A2Grad_optimizer\n\n\nA2GradUni\n---------\n\n+--------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_A2GradUni.png   |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_A2GradUni.png  |\n+--------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.A2GradUni(\n        model.parameters(),\n        kappa=1000.0,\n        beta=10.0,\n        lips=10.0,\n    )\n    optimizer.step()\n\n\n**Paper**: *Optimal Adaptive and Accelerated Stochastic Gradient Descent* (2018) [https://arxiv.org/abs/1810.00553]\n\n**Reference Code**: https://github.com/severilov/A2Grad_optimizer\n\n\nAccSGD\n------\n\n+-----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_AccSGD.png   |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_AccSGD.png  |\n+-----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.AccSGD(\n        model.parameters(),\n        lr=1e-3,\n        kappa=1000.0,\n        xi=10.0,\n        small_const=0.7,\n        weight_decay=0\n    )\n    optimizer.step()\n\n\n**Paper**: *On the insufficiency of existing momentum schemes for Stochastic Optimization* (2019) [https://arxiv.org/abs/1803.05591]\n\n**Reference Code**: https://github.com/rahulkidambi/AccSGD\n\n\nAdaBelief\n---------\n\n+-------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_AdaBelief.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_AdaBelief.png |\n+-------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.AdaBelief(\n        m.parameters(),\n        lr= 1e-3,\n        betas=(0.9, 0.999),\n        eps=1e-3,\n        weight_decay=0,\n        amsgrad=False,\n        weight_decouple=False,\n        fixed_decay=False,\n        rectify=False,\n    )\n    optimizer.step()\n\n\n**Paper**: *AdaBelief Optimizer, adapting stepsizes by the belief in observed gradients* (2020) [https://arxiv.org/abs/2010.07468]\n\n**Reference Code**: https://github.com/juntang-zhuang/Adabelief-Optimizer\n\n\nAdaBound\n--------\n\n+------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_AdaBound.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_AdaBound.png |\n+------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.AdaBound(\n        m.parameters(),\n        lr= 1e-3,\n        betas= (0.9, 0.999),\n        final_lr = 0.1,\n        gamma=1e-3,\n        eps= 1e-8,\n        weight_decay=0,\n        amsbound=False,\n    )\n    optimizer.step()\n\n\n**Paper**: *Adaptive Gradient Methods with Dynamic Bound of Learning Rate* (2019) [https://arxiv.org/abs/1902.09843]\n\n**Reference Code**: https://github.com/Luolc/AdaBound\n\nAdaMod\n------\nThe AdaMod method restricts the adaptive learning rates with adaptive and momental\nupper bounds. The dynamic learning rate bounds are based on the exponential\nmoving averages of the adaptive learning rates themselves, which smooth out\nunexpected large learning rates and stabilize the training of deep neural networks.\n\n+------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_AdaMod.png    |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_AdaMod.png   |\n+------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.AdaMod(\n        m.parameters(),\n        lr= 1e-3,\n        betas=(0.9, 0.999),\n        beta3=0.999,\n        eps=1e-8,\n        weight_decay=0,\n    )\n    optimizer.step()\n\n**Paper**: *An Adaptive and Momental Bound Method for Stochastic Learning.* (2019) [https://arxiv.org/abs/1910.12249]\n\n**Reference Code**: https://github.com/lancopku/AdaMod\n\n\nAdafactor\n---------\n+------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_Adafactor.png |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_Adafactor.png |\n+------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.Adafactor(\n        m.parameters(),\n        lr= 1e-3,\n        eps2= (1e-30, 1e-3),\n        clip_threshold=1.0,\n        decay_rate=-0.8,\n        beta1=None,\n        weight_decay=0.0,\n        scale_parameter=True,\n        relative_step=True,\n        warmup_init=False,\n    )\n    optimizer.step()\n\n**Paper**: *Adafactor: Adaptive Learning Rates with Sublinear Memory Cost.* (2018) [https://arxiv.org/abs/1804.04235]\n\n**Reference Code**: https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py\n\n\nAdahessian\n----------\n+-------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_Adahessian.png |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_Adahessian.png  |\n+-------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.Adahessian(\n        m.parameters(),\n        lr= 1.0,\n        betas= (0.9, 0.999),\n        eps= 1e-4,\n        weight_decay=0.0,\n        hessian_power=1.0,\n    )\n\t  loss_fn(m(input), target).backward(create_graph = True) # create_graph=True is necessary for Hessian calculation\n    optimizer.step()\n\n\n**Paper**: *ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning* (2020) [https://arxiv.org/abs/2006.00719]\n\n**Reference Code**: https://github.com/amirgholami/adahessian\n\n\nAdamP\n------\nAdamP propose a simple and effective solution: at each iteration of the Adam optimizer\napplied on scale-invariant weights (e.g., Conv weights preceding a BN layer), AdamP\nremoves the radial component (i.e., parallel to the weight vector) from the update vector.\nIntuitively, this operation prevents the unnecessary update along the radial direction\nthat only increases the weight norm without contributing to the loss minimization.\n\n+------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_AdamP.png     |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_AdamP.png    |\n+------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.AdamP(\n        m.parameters(),\n        lr= 1e-3,\n        betas=(0.9, 0.999),\n        eps=1e-8,\n        weight_decay=0,\n        delta = 0.1,\n        wd_ratio = 0.1\n    )\n    optimizer.step()\n\n**Paper**: *Slowing Down the Weight Norm Increase in Momentum-based Optimizers.* (2020) [https://arxiv.org/abs/2006.08217]\n\n**Reference Code**: https://github.com/clovaai/AdamP\n\n\nAggMo\n-----\n\n+------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_AggMo.png     |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_AggMo.png    |\n+------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.AggMo(\n        m.parameters(),\n        lr= 1e-3,\n        betas=(0.0, 0.9, 0.99),\n        weight_decay=0,\n    )\n    optimizer.step()\n\n**Paper**: *Aggregated Momentum: Stability Through Passive Damping.* (2019) [https://arxiv.org/abs/1804.00325]\n\n**Reference Code**: https://github.com/AtheMathmo/AggMo\n\n\nApollo\n------\n\n+------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_Apollo.png    |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_Apollo.png   |\n+------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.Apollo(\n        m.parameters(),\n        lr= 1e-2,\n        beta=0.9,\n        eps=1e-4,\n        warmup=0,\n        init_lr=0.01,\n        weight_decay=0,\n    )\n    optimizer.step()\n\n**Paper**: *Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization.* (2020) [https://arxiv.org/abs/2009.13586]\n\n**Reference Code**: https://github.com/XuezheMax/apollo\n\n\nDiffGrad\n--------\nOptimizer based on the difference between the present and the immediate past\ngradient, the step size is adjusted for each parameter in such\na way that it should have a larger step size for faster gradient changing\nparameters and a lower step size for lower gradient changing parameters.\n\n+------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_DiffGrad.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_DiffGrad.png  |\n+------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.DiffGrad(\n        m.parameters(),\n        lr= 1e-3,\n        betas=(0.9, 0.999),\n        eps=1e-8,\n        weight_decay=0,\n    )\n    optimizer.step()\n\n\n**Paper**: *diffGrad: An Optimization Method for Convolutional Neural Networks.* (2019) [https://arxiv.org/abs/1909.11015]\n\n**Reference Code**: https://github.com/shivram1987/diffGrad\n\nLamb\n----\n\n+--------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_Lamb.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_Lamb.png  |\n+--------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.Lamb(\n        m.parameters(),\n        lr= 1e-3,\n        betas=(0.9, 0.999),\n        eps=1e-8,\n        weight_decay=0,\n    )\n    optimizer.step()\n\n\n**Paper**: *Large Batch Optimization for Deep Learning: Training BERT in 76 minutes* (2019) [https://arxiv.org/abs/1904.00962]\n\n**Reference Code**: https://github.com/cybertronai/pytorch-lamb\n\nLookahead\n---------\n\n+-----------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_LookaheadYogi.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_LookaheadYogi.png  |\n+-----------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    # base optimizer, any other optimizer can be used like Adam or DiffGrad\n    yogi = optim.Yogi(\n        m.parameters(),\n        lr= 1e-2,\n        betas=(0.9, 0.999),\n        eps=1e-3,\n        initial_accumulator=1e-6,\n        weight_decay=0,\n    )\n\n    optimizer = optim.Lookahead(yogi, k=5, alpha=0.5)\n    optimizer.step()\n\n\n**Paper**: *Lookahead Optimizer: k steps forward, 1 step back* (2019) [https://arxiv.org/abs/1907.08610]\n\n**Reference Code**: https://github.com/alphadl/lookahead.pytorch\n\n\nMADGRAD\n---------\n\n+-----------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_MADGRAD.png        |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_MADGRAD.png        |\n+-----------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.MADGRAD(\n        m.parameters(),\n        lr=1e-2,\n        momentum=0.9,\n        weight_decay=0,\n        eps=1e-6,\n    )\n    optimizer.step()\n\n\n**Paper**: *Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization* (2021) [https://arxiv.org/abs/2101.11075]\n\n**Reference Code**: https://github.com/facebookresearch/madgrad\n\n\nNovoGrad\n--------\n\n+------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_NovoGrad.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_NovoGrad.png  |\n+------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.NovoGrad(\n        m.parameters(),\n        lr= 1e-3,\n        betas=(0.9, 0.999),\n        eps=1e-8,\n        weight_decay=0,\n        grad_averaging=False,\n        amsgrad=False,\n    )\n    optimizer.step()\n\n\n**Paper**: *Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks* (2019) [https://arxiv.org/abs/1905.11286]\n\n**Reference Code**: https://github.com/NVIDIA/DeepLearningExamples/\n\n\nPID\n---\n\n+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_PID.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_PID.png  |\n+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.PID(\n        m.parameters(),\n        lr=1e-3,\n        momentum=0,\n        dampening=0,\n        weight_decay=1e-2,\n        integral=5.0,\n        derivative=10.0,\n    )\n    optimizer.step()\n\n\n**Paper**: *A PID Controller Approach for Stochastic Optimization of Deep Networks* (2018) [http://www4.comp.polyu.edu.hk/~cslzhang/paper/CVPR18_PID.pdf]\n\n**Reference Code**: https://github.com/tensorboy/PIDOptimizer\n\n\nQHAdam\n------\n\n+----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_QHAdam.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_QHAdam.png  |\n+----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.QHAdam(\n        m.parameters(),\n        lr= 1e-3,\n        betas=(0.9, 0.999),\n        nus=(1.0, 1.0),\n        weight_decay=0,\n        decouple_weight_decay=False,\n        eps=1e-8,\n    )\n    optimizer.step()\n\n\n**Paper**: *Quasi-hyperbolic momentum and Adam for deep learning* (2019) [https://arxiv.org/abs/1810.06801]\n\n**Reference Code**: https://github.com/facebookresearch/qhoptim\n\n\nQHM\n---\n\n+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_QHM.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_QHM.png  |\n+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.QHM(\n        m.parameters(),\n        lr=1e-3,\n        momentum=0,\n        nu=0.7,\n        weight_decay=1e-2,\n        weight_decay_type='grad',\n    )\n    optimizer.step()\n\n\n**Paper**: *Quasi-hyperbolic momentum and Adam for deep learning* (2019) [https://arxiv.org/abs/1810.06801]\n\n**Reference Code**: https://github.com/facebookresearch/qhoptim\n\n\nRAdam\n-----\n\n+---------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_RAdam.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_RAdam.png  |\n+---------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+\n\nDeprecated, please use version provided by PyTorch_.\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.RAdam(\n        m.parameters(),\n        lr= 1e-3,\n        betas=(0.9, 0.999),\n        eps=1e-8,\n        weight_decay=0,\n    )\n    optimizer.step()\n\n\n**Paper**: *On the Variance of the Adaptive Learning Rate and Beyond* (2019) [https://arxiv.org/abs/1908.03265]\n\n**Reference Code**: https://github.com/LiyuanLucasLiu/RAdam\n\n\nRanger\n------\n\n+----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_Ranger.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_Ranger.png  |\n+----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.Ranger(\n        m.parameters(),\n        lr=1e-3,\n        alpha=0.5,\n        k=6,\n        N_sma_threshhold=5,\n        betas=(.95, 0.999),\n        eps=1e-5,\n        weight_decay=0\n    )\n    optimizer.step()\n\n\n**Paper**: *New Deep Learning Optimizer, Ranger: Synergistic combination of RAdam + LookAhead for the best of both* (2019) [https://medium.com/@lessw/new-deep-learning-optimizer-ranger-synergistic-combination-of-radam-lookahead-for-the-best-of-2dc83f79a48d]\n\n**Reference Code**: https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer\n\n\nRangerQH\n--------\n\n+------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_RangerQH.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_RangerQH.png  |\n+------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.RangerQH(\n        m.parameters(),\n        lr=1e-3,\n        betas=(0.9, 0.999),\n        nus=(.7, 1.0),\n        weight_decay=0.0,\n        k=6,\n        alpha=.5,\n        decouple_weight_decay=False,\n        eps=1e-8,\n    )\n    optimizer.step()\n\n\n**Paper**: *Quasi-hyperbolic momentum and Adam for deep learning* (2018) [https://arxiv.org/abs/1810.06801]\n\n**Reference Code**: https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer\n\n\nRangerVA\n--------\n\n+------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_RangerVA.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_RangerVA.png  |\n+------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.RangerVA(\n        m.parameters(),\n        lr=1e-3,\n        alpha=0.5,\n        k=6,\n        n_sma_threshhold=5,\n        betas=(.95, 0.999),\n        eps=1e-5,\n        weight_decay=0,\n        amsgrad=True,\n        transformer='softplus',\n        smooth=50,\n        grad_transformer='square'\n    )\n    optimizer.step()\n\n\n**Paper**: *Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM* (2019) [https://arxiv.org/abs/1908.00700v2]\n\n**Reference Code**: https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer\n\n\nSGDP\n----\n\n+--------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_SGDP.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_SGDP.png  |\n+--------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.SGDP(\n        m.parameters(),\n        lr= 1e-3,\n        momentum=0,\n        dampening=0,\n        weight_decay=1e-2,\n        nesterov=False,\n        delta = 0.1,\n        wd_ratio = 0.1\n    )\n    optimizer.step()\n\n\n**Paper**: *Slowing Down the Weight Norm Increase in Momentum-based Optimizers.* (2020) [https://arxiv.org/abs/2006.08217]\n\n**Reference Code**: https://github.com/clovaai/AdamP\n\n\nSGDW\n----\n\n+--------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_SGDW.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_SGDW.png  |\n+--------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.SGDW(\n        m.parameters(),\n        lr= 1e-3,\n        momentum=0,\n        dampening=0,\n        weight_decay=1e-2,\n        nesterov=False,\n    )\n    optimizer.step()\n\n\n**Paper**: *SGDR: Stochastic Gradient Descent with Warm Restarts* (2017) [https://arxiv.org/abs/1608.03983]\n\n**Reference Code**: https://github.com/pytorch/pytorch/pull/22466\n\n\nSWATS\n-----\n\n+---------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_SWATS.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_SWATS.png  |\n+---------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.SWATS(\n        model.parameters(),\n        lr=1e-1,\n        betas=(0.9, 0.999),\n        eps=1e-3,\n        weight_decay= 0.0,\n        amsgrad=False,\n        nesterov=False,\n    )\n    optimizer.step()\n\n\n**Paper**: *Improving Generalization Performance by Switching from Adam to SGD* (2017) [https://arxiv.org/abs/1712.07628]\n\n**Reference Code**: https://github.com/Mrpatekful/swats\n\n\nShampoo\n-------\n\n+-----------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_Shampoo.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_Shampoo.png  |\n+-----------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.Shampoo(\n        m.parameters(),\n        lr=1e-1,\n        momentum=0.0,\n        weight_decay=0.0,\n        epsilon=1e-4,\n        update_freq=1,\n    )\n    optimizer.step()\n\n\n**Paper**: *Shampoo: Preconditioned Stochastic Tensor Optimization* (2018) [https://arxiv.org/abs/1802.09568]\n\n**Reference Code**: https://github.com/moskomule/shampoo.pytorch\n\n\nYogi\n----\n\nYogi is optimization algorithm based on ADAM with more fine grained effective\nlearning rate control, and has similar theoretical guarantees on convergence as ADAM.\n\n+--------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_Yogi.png  |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_Yogi.png  |\n+--------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n\n.. code:: python\n\n    import torch_optimizer as optim\n\n    # model = ...\n    optimizer = optim.Yogi(\n        m.parameters(),\n        lr= 1e-2,\n        betas=(0.9, 0.999),\n        eps=1e-3,\n        initial_accumulator=1e-6,\n        weight_decay=0,\n    )\n    optimizer.step()\n\n\n**Paper**: *Adaptive Methods for Nonconvex Optimization* (2018) [https://papers.nips.cc/paper/8186-adaptive-methods-for-nonconvex-optimization]\n\n**Reference Code**: https://github.com/4rtemi5/Yogi-Optimizer_Keras\n\n\nAdam (PyTorch built-in)\n-----------------------\n\n+---------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_Adam.png   |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_Adam.png  |\n+---------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+\n\nSGD (PyTorch built-in)\n----------------------\n\n+--------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+\n| .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rastrigin_SGD.png   |  .. image:: https://raw.githubusercontent.com/jettify/pytorch-optimizer/master/docs/rosenbrock_SGD.png  |\n+--------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+\n\n.. _Python: https://www.python.org\n.. _PyTorch: https://github.com/pytorch/pytorch\n.. _Rastrigin: https://en.wikipedia.org/wiki/Rastrigin_function\n.. _Rosenbrock: https://en.wikipedia.org/wiki/Rosenbrock_function\n.. _benchmark: https://en.wikipedia.org/wiki/Test_functions_for_optimization\n.. _optim: https://pytorch.org/docs/stable/optim.html\n","funding_links":[],"categories":["Other Resources","Python","Pytorch \u0026 related libraries｜Pytorch \u0026 相关库","Deep Learning Framework","Pytorch \u0026 related libraries","Optimizations and fine-tuning","Pytorch实用程序"],"sub_categories":["Other libraries｜其他库:","High-Level DL APIs","Other libraries:"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjettify%2Fpytorch-optimizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjettify%2Fpytorch-optimizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjettify%2Fpytorch-optimizer/lists"}