{"id":13564808,"url":"https://github.com/Luolc/AdaBound","last_synced_at":"2025-04-03T21:31:50.726Z","repository":{"id":41258127,"uuid":"170907532","full_name":"Luolc/AdaBound","owner":"Luolc","description":"An optimizer that trains as fast as Adam and as good as SGD.","archived":false,"fork":false,"pushed_at":"2023-07-23T10:44:20.000Z","size":2714,"stargazers_count":2910,"open_issues_count":19,"forks_count":336,"subscribers_count":70,"default_branch":"master","last_synced_at":"2025-03-22T09:06:50.015Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.luolc.com/publications/adabound/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Luolc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-02-15T18:05:20.000Z","updated_at":"2025-03-14T14:47:05.000Z","dependencies_parsed_at":"2022-07-16T23:46:16.583Z","dependency_job_id":"abad3390-0e3d-45bb-a01f-7968b7780784","html_url":"https://github.com/Luolc/AdaBound","commit_stats":{"total_commits":27,"total_committers":2,"mean_commits":13.5,"dds":0.07407407407407407,"last_synced_commit":"2e928c3007a2fc44af0e4c97e343e1fed6986e44"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Luolc%2FAdaBound","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Luolc%2FAdaBound/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Luolc%2FAdaBound/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Luolc%2FAdaBound/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Luolc","download_url":"https://codeload.github.com/Luolc/AdaBound/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247083734,"owners_count":20880900,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:36.377Z","updated_at":"2025-04-03T21:31:48.597Z","avatar_url":"https://github.com/Luolc.png","language":"Python","funding_links":[],"categories":["Python","Pytorch \u0026 related libraries","Pytorch实用程序","Pytorch \u0026 related libraries｜Pytorch \u0026 相关库","DeepCNN"],"sub_categories":["Other libraries:","Other libraries｜其他库:","Optimization"],"readme":"# AdaBound\n[![PyPI - Version](https://img.shields.io/pypi/v/adabound.svg?style=flat)](https://pypi.org/project/adabound/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/adabound.svg)](https://pypi.org/project/adabound/)\n[![PyPI - Wheel](https://img.shields.io/pypi/wheel/adabound.svg?style=flat)](https://pypi.org/project/adabound/)\n[![GitHub - LICENSE](https://img.shields.io/github/license/Luolc/AdaBound.svg?style=flat)](./LICENSE)\n\nAn optimizer that trains as fast as Adam and as good as SGD, for developing state-of-the-art \ndeep learning models on a wide variety of popular tasks in the field of CV, NLP, and etc.\n\nBased on Luo et al. (2019). \n[Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://openreview.net/forum?id=Bkg3g2R9FX).\nIn *Proc. of ICLR 2019*.\n\n\u003cp align='center'\u003e\n  \u003cimg src='https://www.luolc.com/assets/research/adabound/adabound-banner.png' width=\"60%\"/\u003e\n\u003c/p\u003e\n\n## Quick Links\n\n- [Website](https://www.luolc.com/publications/adabound/)\n- [Demos](./demos)\n\n## Installation\n\nAdaBound requires Python 3.6.0 or later.\nWe currently provide PyTorch version and AdaBound for TensorFlow is coming soon.\n\n### Installing via pip\n\nThe preferred way to install AdaBound is via `pip` with a virtual environment.\nJust run \n```bash\npip install adabound\n```\nin your Python environment and you are ready to go!\n\n### Using source code\n\nAs AdaBound is a Python class with only 100+ lines, an alternative way is directly downloading\n[adabound.py](./adabound/adabound.py) and copying it to your project.\n\n## Usage\n\nYou can use AdaBound just like any other PyTorch optimizers.\n\n```python3\noptimizer = adabound.AdaBound(model.parameters(), lr=1e-3, final_lr=0.1)\n```\n\nAs described in the paper, AdaBound is an optimizer that behaves like Adam at the beginning of\ntraining, and gradually transforms to SGD at the end.\nThe `final_lr` parameter indicates AdaBound would transforms to an SGD with this learning rate.\nIn common cases, a default final learning rate of `0.1` can achieve relatively good and stable\nresults on unseen data.\nIt is not very sensitive to its hyperparameters.\nSee Appendix G of the paper for more details.\n\nDespite of its robust performance, we still have to state that, **there is no silver bullet**.\nIt does not mean that you will be free from tuning hyperparameters once using AdaBound.\nThe performance of a model depends on so many things including the task, the model structure,\nthe distribution of data, and etc.\n**You still need to decide what hyperparameters to use based on your specific situation,\nbut you may probably use much less time than before!**\n\n## Demos\n\nThanks to the awesome work by the GitHub team and the Jupyter team, the Jupyter notebook (`.ipynb`)\nfiles can render directly on GitHub.\nWe provide several notebooks (like [this one](./demos/cifar10/visualization.ipynb)) for better\nvisualization.\nWe hope to illustrate the robust performance of AdaBound through these examples.\n\nFor the full list of demos, please refer to [this page](./demos).\n\n## Citing\nIf you use AdaBound in your research, please cite [Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://openreview.net/forum?id=Bkg3g2R9FX).\n```text\n@inproceedings{Luo2019AdaBound,\n  author = {Luo, Liangchen and Xiong, Yuanhao and Liu, Yan and Sun, Xu},\n  title = {Adaptive Gradient Methods with Dynamic Bound of Learning Rate},\n  booktitle = {Proceedings of the 7th International Conference on Learning Representations},\n  month = {May},\n  year = {2019},\n  address = {New Orleans, Louisiana}\n}\n```\n\n## Contributors\n\n[@kayuksel](https://github.com/kayuksel)\n\n## License\n[Apache 2.0](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLuolc%2FAdaBound","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLuolc%2FAdaBound","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLuolc%2FAdaBound/lists"}