{"id":15290462,"url":"https://github.com/andreped/gradientaccumulator","last_synced_at":"2025-04-13T09:32:22.935Z","repository":{"id":37460844,"uuid":"498288524","full_name":"andreped/GradientAccumulator","owner":"andreped","description":":dart: Accumulated Gradients for TensorFlow 2","archived":false,"fork":false,"pushed_at":"2024-02-11T06:01:24.000Z","size":5557,"stargazers_count":50,"open_issues_count":4,"forks_count":10,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-10-14T20:21:14.919Z","etag":null,"topics":["accumulated-batch-normalization","accumulated-gradients","adaptive-gradient-clipping","batch-size","deep-learning","distributed-training","float16","gpu","gradient-accumulation","hacktoberfest","huggingface","keras","memory-constraints","mixed-precision","multi-gpu","tensorflow","tensorflow2","tf2","tpu"],"latest_commit_sha":null,"homepage":"https://gradientaccumulator.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andreped.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-05-31T10:27:18.000Z","updated_at":"2024-07-01T13:52:03.000Z","dependencies_parsed_at":"2023-12-19T17:56:09.924Z","dependency_job_id":null,"html_url":"https://github.com/andreped/GradientAccumulator","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreped%2FGradientAccumulator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreped%2FGradientAccumulator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreped%2FGradientAccumulator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreped%2FGradientAccumulator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andreped","download_url":"https://codeload.github.com/andreped/GradientAccumulator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219846731,"owners_count":16556426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accumulated-batch-normalization","accumulated-gradients","adaptive-gradient-clipping","batch-size","deep-learning","distributed-training","float16","gpu","gradient-accumulation","hacktoberfest","huggingface","keras","memory-constraints","mixed-precision","multi-gpu","tensorflow","tensorflow2","tf2","tpu"],"created_at":"2024-09-30T16:08:16.589Z","updated_at":"2024-10-14T20:21:28.225Z","avatar_url":"https://github.com/andreped.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"assets/accum_grad_v5_reduced.png\" width=\"35%\" alt='gradient-accumulator'\u003e\n\u003ch1 align=\"center\"\u003eGradientAccumulator\u003c/h1\u003e\n\u003ch3 align=\"center\"\u003eSeemless gradient accumulation for TensorFlow 2\u003c/h3\u003e\n\n[![Pip Downloads](https://img.shields.io/pypi/dm/gradient-accumulator?label=pip%20downloads\u0026logo=python)](https://pypi.org/project/gradient-accumulator/)\n[![PyPI version](https://badge.fury.io/py/gradient-accumulator.svg)](https://badge.fury.io/py/gradient-accumulator)\n[![License](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6615018.svg)](https://doi.org/10.5281/zenodo.6615018)\n\n**GradientAccumulator** was developed by SINTEF Health due to the lack of an easy-to-use method for gradient accumulation in TensorFlow 2.\n\nThe package is available on PyPI and is compatible with and have been tested against `TensorFlow 2.2-2.15` and `Python 3.6-3.11`, and works cross-platform (Ubuntu, Windows, macOS).\n\u003c/div\u003e\n\n\n## [Continuous integration](https://github.com/andreped/GradientAccumulator#continuous-integration)\n\n| Build Type | Status |\n| - | - |\n| **Code coverage** | [![codecov](https://codecov.io/gh/andreped/GradientAccumulator/branch/main/graph/badge.svg?token=MWLK71V750)](https://codecov.io/gh/andreped/GradientAccumulator) |\n| **Documentations** | [![Documentation Status](https://readthedocs.org/projects/gradientaccumulator/badge/?version=latest)](https://gradientaccumulator.readthedocs.io/en/latest/?badge=latest) |\n| **Unit tests** | [![CI](https://github.com/andreped/GradientAccumulator/workflows/CI/badge.svg)](https://github.com/andreped/GradientAccumulator/actions) |\n\n\n## [Install](https://github.com/andreped/GradientAccumulator#install)\n\nStable release from PyPI:\n```\npip install gradient-accumulator\n```\n\nOr from source:\n```\npip install git+https://github.com/andreped/GradientAccumulator\n```\n\n**Disclaimer:** Note that different `TensorFlow` versions supports different Python versions. Therefore, be sure to check which Python versions are supported when you install a specific `TensorFlow` version. `GradientAccumulator` does not support Python `\u003e3.11`, as `TensorFlow` has yet to add support for them.\n\n\n## [Getting started](https://github.com/andreped/GradientAccumulator#getting-started)\n\nA simple example to add gradient accumulation to an existing model is by:\n```\nfrom gradient_accumulator import GradientAccumulateModel\nfrom tensorflow.keras.models import Model\n\nmodel = Model(...)\nmodel = GradientAccumulateModel(accum_steps=4, inputs=model.input, outputs=model.output)\n```\n\nThen simply use the `model` as you normally would!\n\nIn practice, using gradient accumulation with a custom pipeline might require some extra overhead and tricks to get working.\n\nFor more information, see documentations which are hosted at [gradientaccumulator.readthedocs.io](https://gradientaccumulator.readthedocs.io/en/latest/)\n\n\n## [What?](https://github.com/andreped/GradientAccumulator#what)\nGradient accumulation (GA) enables reduced GPU memory consumption through dividing a batch into smaller reduced batches, and performing gradient computation either in a distributing setting across multiple GPUs or sequentially on the same GPU. When the full batch is processed, the gradients are then _accumulated_ to produce the full batch gradient.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/grad_accum.png\" width=\"70%\"\u003e\n\u003c/p\u003e\n\nNote that how we implemented gradient accumulation is slightly different from this illustration, as our design does not require having the entire batch in CPU memory. More information on what goes under the hood can be seen in the [documentations](https://gradientaccumulator.readthedocs.io/en/latest/background/gradient_accumulation.html).\n\n\n## [Why?](https://github.com/andreped/GradientAccumulator#why)\nIn TensorFlow 2, there did not exist a plug-and-play method to use gradient accumulation with any custom pipeline. Hence, we have implemented two generic TF2-compatible approaches:\n\n| Method | Usage |\n| - | - |\n| `GradientAccumulateModel` | `model = GradientAccumulateModel(accum_steps=4, inputs=model.input, outputs=model.output)` |\n| `GradientAccumulateOptimizer` | `opt = GradientAccumulateOptimizer(accum_steps=4, optimizer=tf.keras.optimizers.SGD(1e-2))` |\n\nBoth approaches control how frequently the weigths are updated but in their own way. Approach (1) overrides the `train_step` method of a given Model, whereas approach (2) wraps the optimizer. (1) is only compatible with single-GPU usage, whereas (2) also supports distributed training (multi-GPU).\n\nOur implementations enable theoretically **infinitely large batch size**, with **identical memory consumption** as for a regular mini batch. If a single GPU is used, this comes at the cost of increased training runtime. Multiple GPUs could be used to improve runtime performance.\n\n| Technique | Usage |\n| - | - |\n| `Batch Normalization` | `layer = AccumBatchNormalization(accum_steps=4)` |\n| `Adaptive Gradient Clipping` | `model = GradientAccumulateModel(accum_steps=4, agc=True, inputs=model.input, outputs=model.output)` |\n| `Mixed precision` | `model = GradientAccumulateModel(accum_steps=4, mixed_precision=True, inputs=model.input, outputs=model.output)` |\n\n* As batch normalization (BN) is not natively compatible with GA, we have implemented a custom BN layer which can be used as a drop-in replacement.\n* Support for adaptive gradient clipping has been added as an alternative to BN.\n* Mixed precision can also be utilized on both GPUs and TPUs.\n* Multi-GPU distributed training using generic optimizer wrapper.\n\nFor more information on usage, supported techniques, and examples, refer to [the documentations](https://gradientaccumulator.readthedocs.io/en/latest/).\n\n\n## [Applications](https://github.com/andreped/GradientAccumulator#applications)\n* Støverud et al., AeroPath: An airway segmentation benchmark dataset with challenging pathology (2023), arXiv (preprint), https://doi.org/10.48550/arXiv.2311.01138\n* Bouget et al., Raidionics: an open software for pre- and postoperative central nervous system tumor segmentation and standardized reporting (2023), Scientific Reports, https://doi.org/10.1038/s41598-023-42048-7\n* Helland et al., Segmentation of glioblastomas in early post-operative multi-modal MRI with deep neural networks (2023), Scientific Reports, https://doi.org/10.1038/s41598-023-45456-x\n* Pérez de Frutos et al., Learning deep abdominal CT registration through adaptive loss weighting and synthetic data generation (2023), PLOS ONE, https://doi.org/10.1371/journal.pone.0282110\n* Bouget et al., Preoperative Brain Tumor Imaging: Models and Software for Segmentation and Standardized Reporting, (2022) Frontiers in Neurology, https://doi.org/10.3389/fneur.2022.932219\n* Pedersen et al., H2G-Net: A multi-resolution refinement approach for segmentation of breast cancer region in gigapixel histopathological images (2022), Frontiers in Medicine, https://doi.org/10.3389/fmed.2022.971873\n\n\n## [Acknowledgements](https://github.com/andreped/GradientAccumulator#acknowledgements)\nThe gradient accumulator model wrapper is based on the implementation presented in [this thread](https://stackoverflow.com/a/66524901) on stack overflow. The adaptive gradient clipping method is based on [the implementation by @sayakpaul](https://github.com/sayakpaul/Adaptive-Gradient-Clipping).\nThe optimizer wrapper is derived from [the implementation by @fsx950223 and @stefan-falk](https://github.com/tensorflow/addons/pull/2525).\n\nThe documentations hosted [here](https://gradientaccumulator.readthedocs.io/en/latest/index.html) was made possible by the incredible [Read The Docs team](https://readthedocs.org/) which offer free documentation hosting!\n\n  \n## [How to cite?](https://github.com/andreped/GradientAccumulator#how-to-cite)\nIf you used this package or found the project relevant in your research, please, include the following citation:\n\n```\n@software{andre_pedersen_2023_7905351,\n  author       = {André Pedersen and Tor-Arne Schmidt Nordmo and Javier Pérez de Frutos and David Bouget},\n  title        = {andreped/GradientAccumulator: v0.5.0},\n  month        = may,\n  year         = 2023,\n  publisher    = {Zenodo},\n  version      = {v0.5.0},\n  doi          = {10.5281/zenodo.7905351},\n  url          = {https://doi.org/10.5281/zenodo.7905351}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreped%2Fgradientaccumulator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandreped%2Fgradientaccumulator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreped%2Fgradientaccumulator/lists"}