{"id":15065065,"url":"https://github.com/adamvvu/snapshot_ensemble","last_synced_at":"2025-10-28T05:31:59.095Z","repository":{"id":57750810,"uuid":"525707881","full_name":"adamvvu/snapshot_ensemble","owner":"adamvvu","description":"Train TensorFlow Keras models with cosine annealing and save an ensemble of models with no additional computational expense.","archived":false,"fork":false,"pushed_at":"2023-02-02T10:48:20.000Z","size":279,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-23T01:50:03.194Z","etag":null,"topics":["data-science","deep-learning","keras","machine-learning","python","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adamvvu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-17T08:43:23.000Z","updated_at":"2023-03-05T13:23:59.000Z","dependencies_parsed_at":"2023-02-17T18:35:16.690Z","dependency_job_id":null,"html_url":"https://github.com/adamvvu/snapshot_ensemble","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/adamvvu/snapshot_ensemble","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamvvu%2Fsnapshot_ensemble","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamvvu%2Fsnapshot_ensemble/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamvvu%2Fsnapshot_ensemble/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamvvu%2Fsnapshot_ensemble/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adamvvu","download_url":"https://codeload.github.com/adamvvu/snapshot_ensemble/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adamvvu%2Fsnapshot_ensemble/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280983652,"owners_count":26424807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-25T02:00:06.499Z","response_time":81,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","deep-learning","keras","machine-learning","python","tensorflow"],"created_at":"2024-09-25T00:30:21.746Z","updated_at":"2025-10-28T05:31:59.068Z","avatar_url":"https://github.com/adamvvu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build](https://img.shields.io/github/actions/workflow/status/adamvvu/snapshot_ensemble/snapshot_ensemble_tests.yml?style=for-the-badge)](https://github.com/adamvvu/snapshot_ensemble/actions/workflows/snapshot_ensemble_tests.yml)\n[![PyPi](https://img.shields.io/pypi/v/snapshot_ensemble?style=for-the-badge)](https://pypi.org/project/snapshot_ensemble/)\n[![Downloads](https://img.shields.io/pypi/dm/snapshot_ensemble?style=for-the-badge)](https://pypi.org/project/snapshot_ensemble/)\n[![License](https://img.shields.io/pypi/l/snapshot_ensemble?style=for-the-badge)](https://github.com/adamvvu/snapshot_ensemble/blob/master/LICENSE)\n\nTrain TensorFlow Keras models with cosine annealing and save an ensemble of models with no additional computational expense.\n\n------------------------------------------------------------------------\n\n## **snapshot_ensemble**\n\nEnsembles of machine learning models have empirically demonstrated\nstate-of-the-art results in many regression and classification tasks.\nDeep neural networks are popular models given their flexibility and\ntheoretical properties, but ensembling several independent neural networks \nis often impractical due to the computational expense.\n\nHuang et al. (2017) proposes the simple idea of *Snapshot Ensembling*, where\na single neural network is trained via cyclic learning rate schedules such as\ncosine annealing (Loshchilov and Hutter, 2017). At the end of each annealing cycle,\nthe model parameters are saved and thus we obtain an ensemble of trained neural\nnetworks at the cost of training a single one.\n\nConceptually, we may think of this as letting the neural network quickly converge \nby using a decaying learning rate, and then saving the model at several \nlocal minima of the loss surface. We may then used the saved models as part of\nan ensemble for prediction or inference.\n\nThis simple library is an implementation of their ideas as a TensorFlow 2 Keras Callback\nto be used during training.\n\n[Documentation](https://adamvvu.github.io/snapshot_ensemble/docs/)\n\n## Getting Started\n\n### Installation\n\n`pip install snapshot-ensemble`\n\n#### Dependencies:\n\n    # Required\n    python \u003e= 3.6\n    numpy\n    tensorflow \u003e= 2.0\n\n    # Suggested\n    matplotlib\n\n### Usage\n\n``` python\nfrom snapshot_ensemble import SnapshotEnsembleCallback\n\nmodel = # Compiled TensorFlow 2 Keras model\n\n# Train the Keras model with Cosine Annealing + Snapshot Ensembling\nsnapshotCB = SnapshotEnsembleCallback()\nmodel.fit(*args,\n          callbacks = [ snapshotCB ]\n        )\n\n# Snapshotted models are then automatically saved (default: `Ensemble/`)\n# and may be loaded in for ensembled predictions or inference\n```\n\n### Dynamic Learning Rate Schedule\n\nThe learning rate schedule inside `SnapshotEnsembleCallback` takes the following parameters:  \n    -`cycle_length` : Initial number of epochs per cycle  \n    -`cycle_length_multiplier` : Multiplier on number of epochs per cycle  \n    -`lr_init` : Initial maximum learning rate  \n    -`lr_min` : Initial minimum learning rate  \n    -`lr_multiplier` : Multiplier on learning rate per cycle  \n\nThe `cycle_length`, `lr_init`, and `lr_min` parameters control the initial length and learning rate bounds of each cycle. \nThe `*_multiplier` parameters allow for dynamically adjusting the length and/or learning rate bounds as training\nprogresses. It is very likely that the default parameters are suboptimal for your task, so these hyperparameters\nwill need to be tuned. There is a helper function `VisualizeLR()` to visualize the learning rate schedule.\n\n\u003cp align=\"middle\"\u003e\n    \u003cimg src=\"assets/LR0.png\" width=\"32%\" /\u003e\n    \u003cimg src=\"assets/LR1.png\" width=\"32%\" /\u003e\n    \u003cimg src=\"assets/LR2.png\" width=\"32%\" /\u003e\n\u003c/p\u003e\n\n\u003cp style=\"text-align: center;\"\u003e\n    \u003cem\u003e\n    (Left) Standard Cosine Annealing (Middle) Dynamic length (Right) Dynamic length and learning rate bounds\n    \u003c/em\u003e\n\u003c/p\u003e\n\n### Example\n\nFor a simple example, see this\n[notebook](https://adamvvu.github.io/snapshot_ensemble/examples/Example.html).\n\n## References\n\nHuang, G., Li, Y., \u0026 Pleiss, G. (2017). Snapshot Ensembles: Train 1, Get M for Free. \n   International Conference on Learning Representations. https://doi.org/https://doi.org/10.48550/arXiv.1704.00109\n\nLoshchilov, I., \u0026 Hutter, F. (2017). SGDR: Stochastic Gradient Descent with Warm Restarts. \n    International Conference on Learning Representations. https://doi.org/https://doi.org/10.48550/arXiv.1608.03983\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadamvvu%2Fsnapshot_ensemble","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadamvvu%2Fsnapshot_ensemble","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadamvvu%2Fsnapshot_ensemble/lists"}