https://github.com/vita-group/bnn_nobn
[CVPRW 21] "BNN - BN = ? Training Binary Neural Networks without Batch Normalization", Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, Zhangyang Wang
https://github.com/vita-group/bnn_nobn
adaptive-gradient-clipping batch-normalization binary-neural-networks normalization-free-training normalizer-free weight-standardization
Last synced: about 2 months ago
JSON representation
[CVPRW 21] "BNN - BN = ? Training Binary Neural Networks without Batch Normalization", Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, Zhangyang Wang
- Host: GitHub
- URL: https://github.com/vita-group/bnn_nobn
- Owner: VITA-Group
- License: mit
- Created: 2021-04-14T20:12:35.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2021-12-30T06:52:11.000Z (over 3 years ago)
- Last Synced: 2024-04-16T07:18:15.211Z (about 1 year ago)
- Topics: adaptive-gradient-clipping, batch-normalization, binary-neural-networks, normalization-free-training, normalizer-free, weight-standardization
- Language: Python
- Homepage: https://tianlong-chen.github.io/about/
- Size: 310 KB
- Stars: 54
- Watchers: 8
- Forks: 10
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# BNN - BN = ? Training Binary Neural Networks without Batch Normalization
[](https://opensource.org/licenses/MIT)
Codes for this paper [BNN - BN = ? Training Binary Neural Networks without Batch Normalization](https://arxiv.org/pdf/2104.08215.pdf). [CVPR BiVision Workshop 2021]
Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, Zhangyang Wang.
## Overview
Batch normalization (BN) is a key facilitator and considered essential for state-of-the-art binary neural networks (BNN). However, the BN layer is costly to calculate and is typically implemented with non-binary parameters, leaving a hurdle for the efficient implementation of BNN training. It also introduces undesirable dependence between samples within each batch.
Inspired by the latest advance on Batch Normalization Free (BN-Free) training, we extend their framework to training BNNs, and for the first time demonstrate that BNs can be completely removed from BNN training and inference regimes. By plugging in and customizing techniques including adaptive gradient clipping, scale weight standardization, and specialized bottleneck block, a **BN-free BNN** is capable of maintaining competitive accuracy compared to its BN-based counterpart. Experimental results can be found in [our paper](https://arxiv.org/pdf/2104.08215.pdf).
## BN-Free Binary Neural Networks
## Reproduce
### Environment
```
pytorch == 1.5.0
torchvision == 0.6.0
timm == 0.4.5
```
### Training on ImageNet
```
./script/imagenet_reactnet_A_bf.sh (BN-Free ReActNet-A)
./script/imagenet_reactnet_A_bn.sh (with BN ReActNet-A)
./script/imagenet_reactnet_A_none.sh (without BN ReActNet-A)
```
## Citation
```
@misc{chen2021bnn,
title={"BNN - BN = ?": Training Binary Neural Networks without Batch Normalization},
author={Tianlong Chen and Zhenyu Zhang and Xu Ouyang and Zechun Liu and Zhiqiang Shen and Zhangyang Wang},
year={2021},
eprint={2104.08215},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```
## Acknowledgement
https://github.com/liuzechun/ReActNet
https://github.com/liuzechun/Bi-Real-net
https://github.com/vballoli/nfnets-pytorch
https://github.com/deepmind/deepmind-research/tree/master/nfnets