{"id":15600932,"url":"https://github.com/lucidrains/adan-pytorch","last_synced_at":"2025-04-05T13:08:58.250Z","repository":{"id":57870147,"uuid":"528687234","full_name":"lucidrains/Adan-pytorch","owner":"lucidrains","description":"Implementation of the Adan (ADAptive Nesterov momentum algorithm) Optimizer in Pytorch","archived":false,"fork":false,"pushed_at":"2022-09-01T15:38:45.000Z","size":135,"stargazers_count":251,"open_issues_count":1,"forks_count":9,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-03-29T12:09:06.741Z","etag":null,"topics":["artificial-intelligence","deep-learning","optimizer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-25T04:00:22.000Z","updated_at":"2025-03-20T07:44:38.000Z","dependencies_parsed_at":"2022-09-02T08:51:19.561Z","dependency_job_id":null,"html_url":"https://github.com/lucidrains/Adan-pytorch","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FAdan-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FAdan-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FAdan-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FAdan-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/Adan-pytorch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247339158,"owners_count":20923014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","deep-learning","optimizer"],"created_at":"2024-10-03T02:09:34.912Z","updated_at":"2025-04-05T13:08:58.234Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"./adan-pseudocode.png\" width=\"450px\"\u003e\u003c/img\u003e\n\n## Adan - Pytorch\n\nImplementation of the \u003ca href=\"https://arxiv.org/abs/2208.06677\"\u003eAdan\u003c/a\u003e (ADAptive Nesterov momentum algorithm) Optimizer in Pytorch.\n\nExplanation from \u003ca href=\"https://twitter.com/davisblalock/status/1561976182567870465\"\u003eDavis Blalock\u003c/a\u003e\n\n\u003ca href=\"https://github.com/sail-sg/Adan\"\u003eOfficial Adan code\u003c/a\u003e\n\n## Install\n\n```bash\n$ pip install adan-pytorch\n```\n\n## Usage\n\n```python\nfrom adan_pytorch import Adan\n\n# mock model\n\nimport torch\nfrom torch import nn\n\nmodel = torch.nn.Sequential(\n    nn.Linear(16, 16),\n    nn.GELU()\n)\n\n# instantiate Adan with model parameters\n\noptim = Adan(\n    model.parameters(),\n    lr = 1e-3,                  # learning rate (can be much higher than Adam, up to 5-10x)\n    betas = (0.02, 0.08, 0.01), # beta 1-2-3 as described in paper - author says most sensitive to beta3 tuning\n    weight_decay = 0.02         # weight decay 0.02 is optimal per author\n)\n\n# train\n\nfor _ in range(10):\n    loss = model(torch.randn(16)).sum()\n    loss.backward()\n    optim.step()\n    optim.zero_grad()\n\n```\n\n## Citations\n\n```bibtex\n@article{Xie2022AdanAN,\n    title   = {Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models},\n    author  = {Xingyu Xie and Pan Zhou and Huan Li and Zhouchen Lin and Shuicheng Yan},\n    journal = {ArXiv},\n    year    = {2022},\n    volume  = {abs/2208.06677}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fadan-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fadan-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fadan-pytorch/lists"}