{"id":14970641,"url":"https://github.com/fabian-sp/momo","last_synced_at":"2025-10-26T13:31:01.509Z","repository":{"id":165461874,"uuid":"630007528","full_name":"fabian-sp/MoMo","owner":"fabian-sp","description":"MoMo: Momentum Models for Adaptive Learning Rates","archived":false,"fork":false,"pushed_at":"2024-06-12T09:38:29.000Z","size":2304,"stargazers_count":15,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-11T04:40:25.293Z","etag":null,"topics":["deep-learning","optimization","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fabian-sp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-19T13:26:52.000Z","updated_at":"2024-07-23T01:18:48.000Z","dependencies_parsed_at":"2023-11-12T14:44:27.862Z","dependency_job_id":"d5ea2e55-dd34-4a1f-9d88-dc339453275d","html_url":"https://github.com/fabian-sp/MoMo","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabian-sp%2FMoMo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabian-sp%2FMoMo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabian-sp%2FMoMo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabian-sp%2FMoMo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fabian-sp","download_url":"https://codeload.github.com/fabian-sp/MoMo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219862887,"owners_count":16555951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","optimization","pytorch"],"created_at":"2024-09-24T13:43:54.161Z","updated_at":"2025-10-26T13:31:01.151Z","avatar_url":"https://github.com/fabian-sp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MoMo: Momentum Models for Adaptive Learning Rates [[ICML2024](https://icml.cc/virtual/2024/poster/33842)]\n[![arXiv](https://img.shields.io/badge/arXiv-2305.07583-b31b1b.svg)](https://arxiv.org/abs/2305.07583)\n[![PyPI version fury.io](https://badge.fury.io/py/momo-opt.svg)](https://pypi.python.org/pypi/momo-opt/)\n\nPytorch implementation of MoMo optimizer. Adaptive learning rates for SGD with momentum (SGD-M) and Adam. \n\n*Authors: Fabian Schaipp, Ruben Ohana, Michael Eickenberg, Aaron Defazio, Robert M. Gower*\n\n**MoMo and MoMo-Adam are drop-in replacements for SGD-M and Adam in order to reduce learning-rate tuning.**\nWe list some recommendations for hyperparamter setting below. \n\nNote that MoMo is also available in [`optax`](https://optax.readthedocs.io/en/latest/api/contrib.html#optax.contrib.momo).\n\n## Installation\n\nYou can install the package with\n\n```\npip install momo-opt\n```\n\n## Usage\n\nImport the optimizers in Python with\n\n``` python\nfrom momo import Momo\nopt = Momo(model.parameters(), lr=1)\n```\nor\n\n``` python\nfrom momo import MomoAdam\nopt = MomoAdam(model.parameters(), lr=1e-2)\n```\n\n**Important: you only need to adapt one line of your training script, described below.** \nMomo needs access to the value of the (mini-batch) loss in the ``.step()`` method. The easiest way to do this is\n\n``` python\nloss = YOUR_LOSS_FUNCTION()\nloss.backward()\nopt.step(loss=loss)           # the line you need to change\n```\n\nNote that we have passed the loss as `torch.tensor`, you do not need to pass `loss.item()`.\n\n**See [a full example script](example.py).**\n\n\nAlternatively, you can pass a callable ``closure`` to the argument `closure` that computes gradients and returns the loss. \n\nFor example:\n\n``` python\ndef compute_loss(output, labels):\n  loss = criterion(output, labels)\n  loss.backward()\n  return loss\n\n# in each training step, use:\nclosure = lambda: compute_loss(output,labels)\nopt.step(closure=closure)\n```\n\n\n\n## Examples\n\nFor a complete description of our experiments, please see the paper. Below are a few exemplary results. In the right-hand side plot, we plot the training curves for the best three LR values of each method.\n\n### ResNet110 for CIFAR100\n\n\u003cp float=\"left\"\u003e\n    \u003cimg src=\"png/cifar100_resnet.png\" width=\"320\" /\u003e\n    \u003cimg src=\"png/cifar100_resnet_training.png\" width=\"290\" /\u003e\n\u003c/p\u003e\n\n### ViT for Imagenet\n\n\n\u003cp float=\"left\"\u003e\n    \u003cimg src=\"png/imagenet_vit.png\" width=\"320\" /\u003e\n    \u003cimg src=\"png/imagenet_vit_training.png\" width=\"290\" /\u003e\n\u003c/p\u003e\n\n\n### ViT for CIFAR10\n\n\n\u003cp float=\"left\"\u003e\n    \u003cimg src=\"png/cifar10_vit.png\" width=\"320\" /\u003e\n    \u003cimg src=\"png/cifar10_vit_training.png\" width=\"290\" /\u003e\n\u003c/p\u003e\n\n\n\n## Recommendations\n\nIn general, if you expect SGD-M to work well on your task, then use Momo. If you expect Adam to work well on your problem, then use MomoAdam.\n\n* The option `lr` and `weight_decay` are the same as in standard optimizers. As Momo and MomoAdam automatically adapt the learning rate, you should get good performance without heavy tuning of the `lr` value and its schedule. Across many of our experiments, Momo with constant lerning rate set to `lr=1` performed very well (see plots above); for MomoAdam we recommend `lr=1e-2` (or slightly smaller).\n\n* You can still combine Momo(Adam) with your favourite schedule, if you want to (and if you know a good schedule for your problem).\n\n\n* The option `lb` refers to a lower bound of your loss function. In many cases, `lb=0` will be a good enough estimate (for instance for all of the above plots we used `lb=0`). If your loss converges to a large positive number (and you roughly know the value), then set `lb` to this value (or slightly smaller). \n\n* If you can not estimate a lower bound before training, use the option `use_fstar=True`. This will activate an online estimation of the lower bound.\n\n* For Momo, the argument `beta` refers to the momentum parameter. The default is `beta=0.9`. For MomoAdam, `(beta1,beta2)` have the same role as in Adam and default to `(0.9,0.999)`.\n\n\n\n\n## Citation\n\nIf you use MoMo, please cite\n\n```\n@Article{Schaipp2023,\n  author        = {Fabian Schaipp and Ruben Ohana and Michael Eickenberg and Aaron Defazio and Robert M. Gower},\n  title         = {Mo{M}o: {M}omentum Models for Adaptive Learning Rates},\n  year          = {2023},\n  month         = may,\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabian-sp%2Fmomo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffabian-sp%2Fmomo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabian-sp%2Fmomo/lists"}