{"id":13633324,"url":"https://github.com/speediedan/finetuning-scheduler","last_synced_at":"2025-09-01T12:13:19.003Z","repository":{"id":37519283,"uuid":"455666112","full_name":"speediedan/finetuning-scheduler","owner":"speediedan","description":"A PyTorch Lightning extension that accelerates and enhances foundation model experimentation with flexible fine-tuning schedules.","archived":false,"fork":false,"pushed_at":"2025-08-15T00:13:44.000Z","size":2773,"stargazers_count":66,"open_issues_count":0,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-08-15T01:16:19.499Z","etag":null,"topics":["artificial-intelligence","fine-tuning","finetuning","machine-learning","neural-networks","pytorch","pytorch-lightning","superglue","transfer-learning"],"latest_commit_sha":null,"homepage":"https://finetuning-scheduler.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/speediedan.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-02-04T19:08:54.000Z","updated_at":"2025-07-29T12:30:45.000Z","dependencies_parsed_at":"2023-02-14T00:16:34.269Z","dependency_job_id":"e50627f9-6fea-4597-9a88-4e0e9d18e317","html_url":"https://github.com/speediedan/finetuning-scheduler","commit_stats":{"total_commits":278,"total_committers":2,"mean_commits":139.0,"dds":"0.010791366906474864","last_synced_commit":"f358e564a8a363ba1eb7f5c25460c7fb7d3cef91"},"previous_names":[],"tags_count":42,"template":false,"template_full_name":null,"purl":"pkg:github/speediedan/finetuning-scheduler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/speediedan%2Ffinetuning-scheduler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/speediedan%2Ffinetuning-scheduler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/speediedan%2Ffinetuning-scheduler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/speediedan%2Ffinetuning-scheduler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/speediedan","download_url":"https://codeload.github.com/speediedan/finetuning-scheduler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/speediedan%2Ffinetuning-scheduler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273122129,"owners_count":25049539,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","fine-tuning","finetuning","machine-learning","neural-networks","pytorch","pytorch-lightning","superglue","transfer-learning"],"created_at":"2024-08-01T23:00:33.577Z","updated_at":"2025-09-01T12:13:18.988Z","avatar_url":"https://github.com/speediedan.png","language":"Python","funding_links":[],"categories":["Training"],"sub_categories":["Foundation Model Fine Tuning"],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"docs/source/_static/images/logos/logo_fts.png\" width=\"401px\"\u003e\n\n**A PyTorch Lightning extension that enhances model experimentation with flexible fine-tuning schedules.**\n\n______________________________________________________________________\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://finetuning-scheduler.readthedocs.io/en/stable/\"\u003eDocs\u003c/a\u003e •\n  \u003ca href=\"#Setup\"\u003eSetup\u003c/a\u003e •\n  \u003ca href=\"#examples\"\u003eExamples\u003c/a\u003e •\n  \u003ca href=\"#community\"\u003eCommunity\u003c/a\u003e\n\u003c/p\u003e\n\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/finetuning-scheduler)](https://pypi.org/project/finetuning-scheduler/)\n[![PyPI Status](https://badge.fury.io/py/finetuning-scheduler.svg)](https://badge.fury.io/py/finetuning-scheduler)\\\n[![codecov](https://codecov.io/gh/speediedan/finetuning-scheduler/branch/main/graph/badge.svg?flag=gpu)](https://codecov.io/gh/speediedan/finetuning-scheduler)\n[![ReadTheDocs](https://readthedocs.org/projects/finetuning-scheduler/badge/?version=latest)](https://finetuning-scheduler.readthedocs.io/en/stable/)\n[![DOI](https://zenodo.org/badge/455666112.svg)](https://zenodo.org/badge/latestdoi/455666112)\n[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/speediedan/finetuning-scheduler/blob/master/LICENSE)\n\n\u003c/div\u003e\n\n______________________________________________________________________\n\n\u003cimg width=\"300px\" src=\"docs/source/_static/images/fts/fts_explicit_loss_anim.gif\" alt=\"FinetuningScheduler explicit loss animation\" align=\"right\"/\u003e\n\n[FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) is simple to use yet powerful, offering a number of features that facilitate model research and exploration:\n\n- easy specification of flexible fine-tuning schedules with explicit or regex-based parameter selection\n  - implicit schedules for initial/naive model exploration\n  - explicit schedules for performance tuning, fine-grained behavioral experimentation and computational efficiency\n- automatic restoration of best per-phase checkpoints driven by iterative application of early-stopping criteria to each fine-tuning phase\n- composition of early-stopping and manually-set epoch-driven fine-tuning phase transitions\n\n______________________________________________________________________\n\n## Setup\n\n### Step 0: Install from PyPI\n\n```bash\npip install finetuning-scheduler\n```\n\n\u003c!-- following section will be skipped from PyPI description --\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eAdditional installation options\u003c/summary\u003e\n    \u003c!-- following section will be skipped from PyPI description --\u003e\n\n#### *Install Optional Packages*\n\n#### To install additional packages required for examples:\n\n```bash\npip install finetuning-scheduler['examples']\n```\n\n#### or to include packages for examples, development and testing:\n\n```bash\npip install finetuning-scheduler['all']\n```\n\n#### *Source Installation Examples*\n\n#### To install from (editable) source (includes docs as well):\n\n```bash\n# FTS pins Lightning to a specific commit for CI and development\n# This is similar to PyTorch's approach with Triton.\nexport USE_CI_COMMIT_PIN=\"1\"\n\ngit clone https://github.com/speediedan/finetuning-scheduler.git\ncd finetuning-scheduler\npython -m pip install -e \".[all]\" -r requirements/docs.txt\n```\n\n#### Install a specific FTS version from source using the standalone `pytorch-lighting` package:\n\n```bash\nexport FTS_VERSION=2.6.0\nexport PACKAGE_NAME=pytorch\ngit clone -b v${FTS_VERSION} https://github.com/speediedan/finetuning-scheduler\ncd finetuning-scheduler\npython -m pip install -e \".[all]\" -r requirements/docs.txt\n```\n\n#### *Latest Docker Image*\n\nNote, publishing of new `finetuning-scheduler` version-specific docker images was paused after the `2.0.2` patch release. If new version-specific images are required, please raise an issue.\n\n![Docker Image Version (tag latest semver)](https://img.shields.io/docker/v/speediedan/finetuning-scheduler/latest?color=%23000080\u0026label=docker)\n\n\u003c/details\u003e\n\n\u003c!-- end skipping PyPI description --\u003e\n\n### Step 1: Import the FinetuningScheduler callback and start fine-tuning!\n\n```python\nimport lightning as L\nfrom finetuning_scheduler import FinetuningScheduler\n\ntrainer = L.Trainer(callbacks=[FinetuningScheduler()])\n```\n\nGet started by following [the Fine-Tuning Scheduler introduction](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) which includes a [CLI-based example](https://finetuning-scheduler.readthedocs.io/en/stable/index.html#example-scheduled-fine-tuning-for-superglue) or by following the [notebook-based](https://pytorch-lightning.readthedocs.io/en/stable/notebooks/lightning_examples/finetuning-scheduler.html) Fine-Tuning Scheduler tutorial.\n\n______________________________________________________________________\n\n### Installation Using the Standalone `pytorch-lightning` Package\n\n*applicable to versions \u003e= `2.0.0`*\n\nNow that the core Lightning package is `lightning` rather than `pytorch-lightning`, Fine-Tuning Scheduler (FTS) by default depends upon the `lightning` package rather than the standalone `pytorch-lightning`. If you would like to continue to use FTS with the standalone `pytorch-lightning` package instead, you can still do so as follows:\n\nInstall a given FTS release (for example v2.0.0) using standalone `pytorch-lightning`:\n\n```bash\nexport FTS_VERSION=2.0.0\nexport PACKAGE_NAME=pytorch\nwget https://github.com/speediedan/finetuning-scheduler/releases/download/v${FTS_VERSION}/finetuning-scheduler-${FTS_VERSION}.tar.gz\npip install finetuning-scheduler-${FTS_VERSION}.tar.gz\n```\n\n### Dynamic Versioning\n\nFTS (as of version `2.6.0`) now enables dynamic versioning both at installation time and via CLI post-installation. Initially, the dynamic versioning system allows toggling between Lightning unified and standalone imports. The two conversion operations are individually idempotent and mutually reversible.\n\n#### Toggling Between Unified and Standalone Lightning Imports\n\nFTS provides a simple CLI tool to easily toggle between unified and standalone import installation versions post-installation:\n\n```bash\n# Toggle from unified to standalone Lightning imports\ntoggle-lightning-mode --mode standalone\n\n# Toggle from standalone to unified Lightning imports (default)\ntoggle-lightning-mode --mode unified\n```\n\n\u003e **Note:** If you have the standalone package (`pytorch-lightning`) installed but not the unified package (`lightning`), toggling to unified mode will be prevented. You must install the `lightning` package first before toggling.\n\nThis can be useful when:\n\n- You need to adapt existing code to work with a different Lightning package\n- You're switching between projects using different Lightning import styles\n- You want to test compatibility with both import styles\n\n______________________________________________________________________\n\n## Examples\n\n### Scheduled Fine-Tuning For SuperGLUE\n\n- [Notebook-based Tutorial](https://pytorch-lightning.readthedocs.io/en/stable/notebooks/lightning_examples/finetuning-scheduler.html)\n- [CLI-based Tutorial](https://finetuning-scheduler.readthedocs.io/en/stable/#example-scheduled-fine-tuning-for-superglue)\n- [FSDP Scheduled Fine-Tuning](https://finetuning-scheduler.readthedocs.io/en/stable/advanced/fsdp_scheduled_fine_tuning.html)\n- [LR Scheduler Reinitialization](https://finetuning-scheduler.readthedocs.io/en/stable/advanced/lr_scheduler_reinitialization.html) (advanced)\n- [Optimizer Reinitialization](https://finetuning-scheduler.readthedocs.io/en/stable/advanced/optimizer_reinitialization.html) (advanced)\n\n______________________________________________________________________\n\n## Continuous Integration\n\nFine-Tuning Scheduler is rigorously tested across multiple CPUs, GPUs and against major Python and PyTorch versions. Each Fine-Tuning Scheduler minor release (major.minor.patch) is paired with a Lightning minor release (e.g. Fine-Tuning Scheduler 2.0 depends upon Lightning 2.0).\n\nTo ensure maximum stability, the latest Lightning patch release fully tested with Fine-Tuning Scheduler is set as a maximum dependency in Fine-Tuning Scheduler's requirements.txt (e.g. \\\u003c= 1.7.1). If you'd like to test a specific Lightning patch version greater than that currently in Fine-Tuning Scheduler's requirements.txt, it will likely work but you should install Fine-Tuning Scheduler from source and update the requirements.txt as desired.\n\n\u003cdetails\u003e\n  \u003csummary\u003eCurrent build statuses for Fine-Tuning Scheduler \u003c/summary\u003e\n\n| System / (PyTorch/Python ver) |                                                                                                        2.3.1/3.9                                                                                                         |                                                                                                              2.9.0/3.9, 2.9.0/3.12                                                                                                               |\n| :---------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |\n|      Linux \\[GPUs\\*\\*\\]       |                                                                                                            -                                                                                                             | [![Build Status](https://dev.azure.com//speediedan/finetuning-scheduler/_apis/build/status/Multi-GPU%20\u0026%20Example%20Tests?branchName=main)](https://dev.azure.com/speediedan/finetuning-scheduler/_build/latest?definitionId=1\u0026branchName=main) |\n|     Linux (Ubuntu 22.04)      | [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?branch=main\u0026event=push)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml) |             [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?branch=main\u0026event=push)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml)             |\n|           OSX (14)            | [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?branch=main\u0026event=push)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml) |             [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?branch=main\u0026event=push)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml)             |\n|        Windows (2022)         | [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?branch=main\u0026event=push)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml) |             [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?branch=main\u0026event=push)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml)             |\n\n- \\*\\* tests run on one RTX 4090 and one RTX 2070\n\n\u003c/details\u003e\n\n## Community\n\nFine-Tuning Scheduler is developed and maintained by the community in close communication with the [Lightning team](https://pytorch-lightning.readthedocs.io/en/stable/governance.html). Thanks to everyone in the community for their tireless effort building and improving the immensely useful core Lightning project.\n\nPR's welcome! Please see the [contributing guidelines](https://finetuning-scheduler.readthedocs.io/en/stable/generated/CONTRIBUTING.html) (which are essentially the same as Lightning's).\n\n______________________________________________________________________\n\n## Citing Fine-Tuning Scheduler\n\nPlease cite:\n\n```tex\n@misc{Dan_Dale_2022_6463952,\n    author       = {Dan Dale},\n    title        = {{Fine-Tuning Scheduler}},\n    month        = Feb,\n    year         = 2022,\n    doi          = {10.5281/zenodo.6463952},\n    publisher    = {Zenodo},\n    url          = {https://zenodo.org/record/6463952}\n    }\n```\n\nFeel free to star the repo as well if you find it useful or interesting. Thanks 😊!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspeediedan%2Ffinetuning-scheduler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspeediedan%2Ffinetuning-scheduler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspeediedan%2Ffinetuning-scheduler/lists"}