{"id":21529585,"url":"https://github.com/sooftware/pytorch-lr-scheduler","last_synced_at":"2025-04-09T23:51:34.763Z","repository":{"id":47310748,"uuid":"369285396","full_name":"sooftware/pytorch-lr-scheduler","owner":"sooftware","description":"PyTorch implementation of some learning rate schedulers for deep learning researcher.","archived":false,"fork":false,"pushed_at":"2022-09-06T00:07:36.000Z","size":126,"stargazers_count":89,"open_issues_count":2,"forks_count":17,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-09T23:51:30.908Z","etag":null,"topics":["learning-rate","lr","plateau","reduce","scheduler","transformer","tri-stage","warmup"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sooftware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-20T17:19:27.000Z","updated_at":"2025-02-09T06:54:38.000Z","dependencies_parsed_at":"2023-01-17T21:02:37.432Z","dependency_job_id":null,"html_url":"https://github.com/sooftware/pytorch-lr-scheduler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sooftware%2Fpytorch-lr-scheduler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sooftware%2Fpytorch-lr-scheduler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sooftware%2Fpytorch-lr-scheduler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sooftware%2Fpytorch-lr-scheduler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sooftware","download_url":"https://codeload.github.com/sooftware/pytorch-lr-scheduler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248131454,"owners_count":21052819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["learning-rate","lr","plateau","reduce","scheduler","transformer","tri-stage","warmup"],"created_at":"2024-11-24T01:58:14.066Z","updated_at":"2025-04-09T23:51:34.740Z","avatar_url":"https://github.com/sooftware.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pytorch-lr-scheduler\nPyTorch implementation of some learning rate schedulers for deep learning researcher.\n  \n## Usage\n  \n### [`WarmupReduceLROnPlateauScheduler`](https://github.com/sooftware/pytorch-lr-scheduler/blob/main/lr_scheduler/warmup_reduce_lr_on_plateau_scheduler.py)\n  \n- Visualize\n  \n\u003cimg src=\"https://github.com/sooftware/pytorch-lr-scheduler/blob/main/images/WarmupReduceLROnPlateauScheduler.png\" width=400\u003e\n  \n- Example code\n```python\nimport torch\n\nfrom lr_scheduler.warmup_reduce_lr_on_plateau_scheduler import WarmupReduceLROnPlateauScheduler\n\nif __name__ == '__main__':\n    max_epochs, steps_in_epoch = 10, 10000\n\n    model = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]\n    optimizer = torch.optim.Adam(model, 1e-10)\n\n    scheduler = WarmupReduceLROnPlateauScheduler(\n        optimizer, \n        init_lr=1e-10, \n        peak_lr=1e-4, \n        warmup_steps=30000, \n        patience=1,\n        factor=0.3,\n    )\n\n    for epoch in range(max_epochs):\n        for timestep in range(steps_in_epoch):\n            ...\n            ...\n            if timestep \u003c warmup_steps:\n                scheduler.step()\n                \n        val_loss = validate()\n        scheduler.step(val_loss)\n```\n  \n### [`TransformerLRScheduler`](https://github.com/sooftware/pytorch-lr-scheduler/blob/main/lr_scheduler/transformer_lr_scheduler.py)\n  \n- Visualize\n  \n\u003cimg src=\"https://github.com/sooftware/pytorch-lr-scheduler/blob/main/images/TransformerLRScheduler.png\" width=400\u003e\n  \n- Example code\n  \n```python\nimport torch\n\nfrom lr_scheduler.transformer_lr_scheduler import TransformerLRScheduler\n\nif __name__ == '__main__':\n    max_epochs, steps_in_epoch = 10, 10000\n\n    model = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]\n    optimizer = torch.optim.Adam(model, 1e-10)\n\n    scheduler = TransformerLRScheduler(\n        optimizer=optimizer, \n        init_lr=1e-10, \n        peak_lr=0.1,\n        final_lr=1e-4, \n        final_lr_scale=0.05,\n        warmup_steps=3000, \n        decay_steps=17000,\n    )\n\n    for epoch in range(max_epochs):\n        for timestep in range(steps_in_epoch):\n            ...\n            ...\n            scheduler.step()\n```\n  \n### [`TriStageLRScheduler`](https://github.com/sooftware/pytorch-lr-scheduler/blob/main/lr_scheduler/tri_stage_lr_scheduler.py)\n  \n- Visualize\n  \n\u003cimg src=\"https://github.com/sooftware/pytorch-lr-scheduler/blob/main/images/TriStageLRScheduler.png\" width=400\u003e\n  \n- Example code\n  \n```python\nimport torch\n\nfrom lr_scheduler.tri_stage_lr_scheduler import TriStageLRScheduler\n\nif __name__ == '__main__':\n    max_epochs, steps_in_epoch = 10, 10000\n\n    model = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]\n    optimizer = torch.optim.Adam(model, 1e-10)\n\n    scheduler = TriStageLRScheduler(\n        optimizer, \n        init_lr=1e-10, \n        peak_lr=1e-4, \n        final_lr=1e-7, \n        init_lr_scale=0.01, \n        final_lr_scale=0.05,\n        warmup_steps=30000, \n        hold_steps=70000, \n        decay_steps=100000,\n        total_steps=200000,\n    )\n\n    for epoch in range(max_epochs):\n        for timestep in range(steps_in_epoch):\n            ...\n            ...\n            scheduler.step()\n```\n  \n### [`ReduceLROnPlateauScheduler`](https://github.com/sooftware/pytorch-lr-scheduler/blob/main/lr_scheduler/reduce_lr_on_plateau_lr_scheduler.py)  \n  \n- Visualize\n  \n\u003cimg src=\"https://github.com/sooftware/pytorch-lr-scheduler/blob/main/images/ReduceLROnPlateauScheduler.png\" width=400\u003e\n  \n- Example code\n  \n```python\nimport torch\n\nfrom lr_scheduler.reduce_lr_on_plateau_lr_scheduler import ReduceLROnPlateauScheduler\n\nif __name__ == '__main__':\n    max_epochs, steps_in_epoch = 10, 10000\n\n    model = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]\n    optimizer = torch.optim.Adam(model, 1e-4)\n\n    scheduler = ReduceLROnPlateauScheduler(optimizer, patience=1, factor=0.3)\n\n    for epoch in range(max_epochs):\n        for timestep in range(steps_in_epoch):\n            ...\n            ...\n        \n        val_loss = validate()\n        scheduler.step(val_loss)\n```\n  \n  \n  \n### [`WarmupLRScheduler`](https://github.com/sooftware/pytorch-lr-scheduler/blob/main/lr_scheduler/warmup_lr_scheduler.py)\n  \n- Visualize\n  \n\u003cimg src=\"https://github.com/sooftware/pytorch-lr-scheduler/blob/main/images/WarmupLRScheduler.png\" width=400\u003e\n  \n- Example code\n  \n```python\nimport torch\n\nfrom lr_scheduler.warmup_lr_scheduler import WarmupLRScheduler\n\nif __name__ == '__main__':\n    max_epochs, steps_in_epoch = 10, 10000\n\n    model = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]\n    optimizer = torch.optim.Adam(model, 1e-10)\n\n    scheduler = WarmupLRScheduler(\n        optimizer, \n        init_lr=1e-10, \n        peak_lr=1e-4, \n        warmup_steps=4000,\n    )\n\n    for epoch in range(max_epochs):\n        for timestep in range(steps_in_epoch):\n            ...\n            ...\n            scheduler.step()\n```\n  \n  \n## Installation\n```bash\ngit clone git@github.com:sooftware/pytorch-lr-scheduler.git\ncd pytorch-lr-scheduler\npip install .\n```\n\n## Troubleshoots and Contributing\nIf you have any questions, bug reports, and feature requests, please [open an issue](https://github.com/sooftware/pytorch-lr-scheduler/issues) on Github.   \n  \nI appreciate any kind of feedback or contribution.  Feel free to proceed with small issues like bug fixes, documentation improvement.  For major contributions and new features, please discuss with the collaborators in corresponding issues.\n  \n## Code Style\nI follow [PEP-8](https://www.python.org/dev/peps/pep-0008/) for code style. Especially the style of docstrings is important to generate documentation. \n  \n## License\nThis project is licensed under the MIT LICENSE - see the [LICENSE.md](https://github.com/sooftware/pytorch-lr-scheduler/blob/master/LICENSE) file for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsooftware%2Fpytorch-lr-scheduler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsooftware%2Fpytorch-lr-scheduler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsooftware%2Fpytorch-lr-scheduler/lists"}