{"id":13784161,"url":"https://github.com/mapillary/inplace_abn","last_synced_at":"2025-05-14T00:05:13.908Z","repository":{"id":37734873,"uuid":"111797719","full_name":"mapillary/inplace_abn","owner":"mapillary","description":"In-Place Activated BatchNorm for Memory-Optimized Training of DNNs","archived":false,"fork":false,"pushed_at":"2025-04-18T17:37:32.000Z","size":314,"stargazers_count":1325,"open_issues_count":63,"forks_count":186,"subscribers_count":38,"default_branch":"main","last_synced_at":"2025-05-10T17:16:05.327Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mapillary.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-23T10:47:45.000Z","updated_at":"2025-05-06T14:04:27.000Z","dependencies_parsed_at":"2024-01-18T18:15:02.069Z","dependency_job_id":"106403c5-f0eb-4c88-a170-f84bcd98a474","html_url":"https://github.com/mapillary/inplace_abn","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mapillary%2Finplace_abn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mapillary%2Finplace_abn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mapillary%2Finplace_abn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mapillary%2Finplace_abn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mapillary","download_url":"https://codeload.github.com/mapillary/inplace_abn/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254043342,"owners_count":22004932,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T19:00:36.471Z","updated_at":"2025-05-14T00:05:13.860Z","avatar_url":"https://github.com/mapillary.png","language":"Python","readme":"# In-Place Activated BatchNorm\n\n[**In-Place Activated BatchNorm for Memory-Optimized Training of DNNs**](https://arxiv.org/abs/1712.02616)\n\nIn-Place Activated BatchNorm (InPlace-ABN) is a novel approach to reduce the memory required for training deep networks.\nIt allows for up to 50% memory savings in modern architectures such as ResNet, ResNeXt and Wider ResNet by redefining\nBN + non linear activation as a single in-place operation, while smartly dropping or recomputing intermediate buffers as\nneeded.\n\nThis repository contains a [PyTorch](http://pytorch.org/) implementation of the InPlace-ABN layer, as well as some\ntraining scripts to reproduce the ImageNet classification results reported in our paper.\n\n- [Overview](#overview)\n- [Installation](#installation)\n- [Training on ImageNet](#training-on-imagenet)\n\nWe have now also released the inference code for semantic segmentation, together with the Mapillary Vistas trained model leading to [#1 position on the Mapillary Vistas Semantic Segmentation leaderboard](https://eval-vistas.mapillary.com/featured-challenges/1/leaderboard/1). More information can be found at the bottom of this page.\n\n## Citation\n\nIf you use In-Place Activated BatchNorm in your research, please cite:\n```bibtex\n@inproceedings{rotabulo2017place,\n  title={In-Place Activated BatchNorm for Memory-Optimized Training of DNNs},\n  author={Rota Bul\\`o, Samuel and Porzi, Lorenzo and Kontschieder, Peter},\n  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},\n  year={2018}\n}\n```\n\n## Overview\n\n\u003cp align=\"center\"\u003e\u003cimg width=\"70%\" src=\"inplace_abn.png\" /\u003e\u003c/p\u003e\n\nWhen processing a BN-Activation-Convolution sequence in the forward pass, most deep learning frameworks need to store\ntwo big buffers, _i.e._ the input `x` of BN and the input `z` of Conv.\nThis is necessary because the standard implementations of the backward passes of BN and Conv depend on their inputs to\ncalculate the gradients.\nUsing Inplace-ABN to replace the BN-Activation sequence, we can safely discard `x`, thus saving up to 50% GPU memory at\ntraining time.\nTo achieve this, we rewrite the backward pass of BN in terms of its output `y`, which is in turn reconstructed from `z`\nby inverting the activation function.\n\nThe parametrization for the scaling factor of BN changed compared to standard BN, in order to ensure an invertible transformation. Specifically, the scaling factor becomes\n\u003cimg src=\"./equation.svg\"\u003e.\n\n## Requirements\n\nTo install PyTorch, please refer to https://github.com/pytorch/pytorch#installation.\n\n**NOTE 1: our code _requires_ PyTorch v1.1 or later**\n\n**NOTE 2: we are only able to provide support for Linux platforms and CUDA versions \u003e= 10.0**\n\n**NOTE 3: in general, it is not possible to load weights from a network trained with standard BN into an InPlace-ABN network without severe performance degradation, due to the different handling of BN scaling parameters**\n\nTo install the package containing the iABN layers:\n```bash\npip install inplace-abn\n```\nNote that some parts of InPlace-ABN have native C++/CUDA implementations, meaning that the command above will need to\ncompile them.\n\nAlternatively, to download and install the latest version of our library, also obtaining a copy of the Imagenet / Vistas\nscripts:\n```bash\ngit clone https://github.com/mapillary/inplace_abn.git\ncd inplace_abn\npython setup.py install\ncd scripts\npip install -r requirements.txt\n```\nThe last of the commands above will install some additional libraries required by the Imagenet / Vistas scripts.\n\n### Force compiling with CUDA\n\nIn order to force the compilation of the native CUDA functions on systems that do not\nhave access to a GPU (e.g. Docker containers), two environment variables have to be set:\n```bash\nexport TORCH_CUDA_ARCH_LIST=\"{archs}\"\nexport IABN_FORCE_CUDA=1\n```\nwhere `{archs}` is a list of target CUDA architectures, e.g. `Pascal;Volta`, `6.0;6.5` etc.\n\n## Training on ImageNet-1k\n\nHere you can find the results from our arXiv paper (top-1 / top-5 scores) with corresponding, trained models and md5 checksums, respectively. The model files provided below are made available under the [license attached to ImageNet](http://www.image-net.org/download-faq).\n\n| Network                           | Batch | 224            | 224, 10-crops  | 320           |       Trained models (+md5)      |\n|-----------------------------------|-------|----------------|----------------|---------------|----------------------------------|\n| [ResNeXt101, Std-BN][1]           | 256   | 77.04 / 93.50  | 78.72 / 94.47  | 77.92 / 94.28 | [`448438885986d14db5e870b95f814f91`][6] |\n| [ResNeXt101, InPlace-ABN][2]      | 512   | 78.08 / 93.79  | 79.52 / 94.66  | 79.38 / 94.67 | [`3b7a221cbc076410eb12c8dd361b7e4e`][7] |\n| [ResNeXt152, InPlace-ABN][3]      | 256   | 78.28 / 94.04  | 79.73 / 94.82  | 79.56 / 94.67 | [`2c8d572587961ed74611d534c5b2e9ce`][8] |\n| [WideResNet38, InPlace-ABN][4]    | 256   | 79.72 / 94.78  | 81.03 / 95.43  | 80.69 / 95.27 | [`1c085ab70b789cc1d6c1594f7a761007`][9] |\n| [ResNeXt101, InPlace-ABN sync][5] | 256   | 77.70 / 93.78  | 79.18 / 94.60  | 78.98 / 94.56 | [`0a85a21847b15e5a242e17bf3b753849`][10] |\n| [DenseNet264, InPlace-ABN][11]    | 256   | 78.57 / 94.17  | 79.72 / 94.93  | 79.49 / 94.89 | [`0b413d67b725619441d0646d663865bf`][12] |\n| [ResNet50v1, InPlace-ABN sync][13]  | 512   | 75.53 / 92.59  | 77.04 / 93.57  | 76.60 / 93.49 | [`2522ca639f7fdfd7c0089ba1f5f6c2e8`][14] |\n| [ResNet34v1, InPlace-ABN sync][15]  | 512   | 73.27 / 91.34  | 75.19 / 92.66  | 74.87 / 92.42 | [`61515c1484911c3cc753d405131e1dda`][16] |\n| [ResNet101v1, InPlace-ABN sync][17]  | 512   | 77.07 / 93.45  | 78.58 / 94.40  | 78.25 / 94.19 | [`1552ae0f3d610108df702135f56bd27b`][18] |\n\n[1]: scripts/experiments/resnext101_stdbn_lr_256.json\n[2]: scripts/experiments/resnext101_ipabn_lr_512.json\n[3]: scripts/experiments/resnext152_ipabn_lr_256.json\n[4]: scripts/experiments/wider_resnet38_ipabn_lr_256.json\n[5]: scripts/experiments/resnext101_ipabn-sync_lr_256.json\n[6]: https://drive.google.com/file/d/1qT8qCSZzUHorai1EP6Liywa28ASac_G_/view\n[7]: https://drive.google.com/file/d/1rQd-NoZuCsGZ7_l_X9GO1GGiXeXHE8CT/view\n[8]: https://drive.google.com/file/d/1RmHK3tdVTVsHiyNO14bYLkMC0XUjenIn/view\n[9]: https://drive.google.com/file/d/1Y0McSz9InDSxMEcBylAbCv1gvyeaz8Ij/view\n[10]: https://drive.google.com/file/d/1v2gmUPBMDKf0wZm9r1JwCQLGAig0DdXJ/view\n[11]: scripts/experiments/densenet264_ipabn_lr_256.json\n[12]: https://drive.google.com/file/d/1J2wp59bzzEd6zttM6oMa1KgbmCL1MS0k/view\n[13]: scripts/experiments/resnet50_ipabn-sync_lr_512.json\n[14]: https://drive.google.com/file/d/1N7kjWrnUbD_aBOUNi9ZLGnI3E_1ATH8U/view\n[15]: scripts/experiments/resnet34_ipabn-sync_lr_512.json\n[16]: https://drive.google.com/file/d/1V5dCIZeRCfnZi9krNaQNhXNDHyXz9JR8/view\n[17]: scripts/experiments/resnet101_ipabn-sync_lr_512.json\n[18]: https://drive.google.com/file/d/1oFVSIUYAxa_uNDq2OLkbhyiFmKwnYzpt/view\n\n### Data preparation\n\nOur script uses [torchvision.datasets.ImageFolder](http://pytorch.org/docs/master/torchvision/datasets.html#torchvision.datasets.ImageFolder)\nfor loading ImageNet data, which expects folders organized as follows:\n```\nroot/train/[class_id1]/xxx.{jpg,png,jpeg}\nroot/train/[class_id1]/xxy.{jpg,png,jpeg}\nroot/train/[class_id2]/xxz.{jpg,png,jpeg}\n...\n\nroot/val/[class_id1]/asdas.{jpg,png,jpeg}\nroot/val/[class_id1]/123456.{jpg,png,jpeg}\nroot/val/[class_id2]/__32_.{jpg,png,jpeg}\n...\n```\nImages can have any name, as long as the extension is that of a recognized image format.\nClass ids are also free-form, but they are expected to match between train and validation data.\nNote that the training data in the standard ImageNet distribution is already given in the required format, while\nvalidation images need to be split into class sub-folders as described above.\n\n### Training\n\nThe main training script is `scripts/train_imagenet.py`: this supports training on ImageNet, or any other dataset\nformatted as described above, while keeping a log of relevant metrics in Tensorboard format and periodically saving\nsnapshots.\nMost training parameters can be specified as a `json`-formatted configuration file (look\n[here](scripts/imagenet/config.py) for a complete list of configurable parameters).\nAll parameters not explicitly specified in the configuration file are set to their defaults, also available in\n[scripts/imagenet/config.py](scripts/imagenet/config.py).\n\nOur arXiv results can be reproduced by running `scripts/train_imagenet.py` with the configuration files in\n`scripts/experiments`.\nAs an example, the command to train `ResNeXt101` with InPlace-ABN, Leaky ReLU and `batch_size = 512` is:\n```bash\ncd scripts\npython -m torch.distributed.launch --nproc_per_node \u003cn. GPUs per node\u003e train_imagenet.py --log-dir /path/to/tensorboard/logs experiments/resnext101_ipabn_lr_512.json /path/to/imagenet/root\n```\n\n### Validation\n\nValidation is run by `scripts/train_imagenet.py` at the end of every training epoch.\nTo validate a trained model, you can use the `scripts/test_imagenet.py` script, which allows for 10-crops validation and\ntransferring weights across compatible networks (_e.g._ from `ResNeXt101` with ReLU to `ResNeXt101` with Leaky\nReLU).\nThis script accepts the same configuration files as `scripts/train_imagenet.py`, but note that the `scale_val` and\n`crop_val` parameters are ignored in favour of the `--scale` and `--crop` command-line arguments.\n\nAs an example, to validate the `ResNeXt101` trained above using 10-crops of size `224` from images scaled to `256`\npixels, you can run:\n```bash\ncd scripts\npython -m torch.distributed.launch --nproc_per_node \u003cn. GPUs per node\u003e test_imagenet.py --crop 224 --scale 256 --ten_crops experiments/resnext101_ipabn_lr_512.json /path/to/checkpoint /path/to/imagenet/root\n```\n\n## Usage for Semantic Segmentation on Cityscapes and Mapillary Vistas\n\nWe have successfully used InPlace-ABN with a DeepLab3 segmentation head that was trained on top of the WideResNet38\nmodel above.\nDue to InPlace-ABN, we can significantly increase the amount of input data to this model, which eventually allowed us to\nobtain #1 positions on [Cityscapes](https://www.cityscapes-dataset.com/benchmarks/#scene-labeling-task),\n[Mapillary Vistas](https://eval-vistas.mapillary.com/featured-challenges/1/leaderboard/1), [AutoNUE](http://cvit.iiit.ac.in/scene-understanding-challenge-2018/benchmarks.php),\n[Kitti](http://www.cvlibs.net/datasets/kitti/eval_semseg.php?benchmark=semantics2015) and\n[ScanNet](http://dovahkiin.stanford.edu/adai/semantic_label) segmentation leaderboards.\nThe training settings mostly follow the description in our [paper](https://arxiv.org/abs/1712.02616).\n\n### Mapillary Vistas pre-trained model\n\nWe release our WideResNet38 + DeepLab3 segmentation model trained on the Mapillary Vistas research set.\nThis is the model used to reach #1 position on the MVD semantic segmentation leaderboard.\nThe segmentation model file provided below is made available under a\n[CC BY-NC-SA 4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/).\n\n| Network                       | mIOU  | Trained model (+md5)                   |\n|-------------------------------|-------|----------------------------------------|\n| [WideResNet38 + DeepLab3][19] | 53.42 | [913f78486a34aa1577a7cd295e8a33bb][20] |\n\n[19]: scripts/test_vistas.py\n[20]: https://drive.google.com/file/d/1SJJx5-LFG3J3M99TrPMU-z6ZmgWynxo-/view\n\nTo use this, please download the `.pth.tar` model file linked above and run the `test_vistas.py` script as follows:\n```bash\ncd scripts\npython test_vistas.py /path/to/model.pth.tar /path/to/input/folder /path/to/output/folder\n```\n\nThe script will process all `.png`, `.jpg` and `.jpeg` images from the input folder and write the predictions in the\noutput folder as `.png` images.\nFor additional options, _e.g._ test time augmentation, please consult the script's help message.\n\nThe results on the test data written above were obtained by employing only scale 1.0 + flipping.\n\n## Changelog\n\n**Update 04 Jul. 2019: version 1.0.0**\n- Complete rewrite of the CUDA code following the most recent native BN implementation from Pytorch\n- Improved synchronized BN implementation, correctly handling different per-GPU batch sizes and Pytorch distributed groups\n- The iABN layers are now packaged in an installable python library to simplify use in other projects\n- The Imagenet / Vistas scripts are still available in the `scripts` folder\n- Requires now PyTorch 1.1\n\n**Update 08 Jan. 2019:**\n- Enabled multiprocessing and inplace ABN synchronization over multiple processes (previously using threads). It now requires to use DistributedDataParallel instead of DataParallel\n- Added compatibility with fp16 (currently allows fp16 input but requires the module to stay in fp32 mode)\n- Requires now PyTorch 1.0\n\n**Update Feb. 2019:**\n- Added ResNet34v1, ResNet50v1 and ResNet101v1 ImageNet-1k pre-trained models\n\nWe have modified the imagenet training code and BN synchronization in order to work with multiple processes. We have also added compatibility of our Inplace ABN module with fp16.\n","funding_links":[],"categories":["Paper implementations｜论文实现","Paper implementations","2018"],"sub_categories":["Other libraries｜其他库:","Other libraries:"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmapillary%2Finplace_abn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmapillary%2Finplace_abn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmapillary%2Finplace_abn/lists"}