Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tczhangzhi/awesome-normalization-techniques
Papers for normalization techniques, released codes collections.
https://github.com/tczhangzhi/awesome-normalization-techniques
List: awesome-normalization-techniques
Last synced: about 1 month ago
JSON representation
Papers for normalization techniques, released codes collections.
- Host: GitHub
- URL: https://github.com/tczhangzhi/awesome-normalization-techniques
- Owner: tczhangzhi
- Created: 2020-07-30T09:00:01.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2020-08-05T13:00:28.000Z (over 4 years ago)
- Last Synced: 2024-05-19T21:59:24.992Z (7 months ago)
- Language: Python
- Size: 29.3 KB
- Stars: 222
- Watchers: 19
- Forks: 25
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- ultimate-awesome - awesome-normalization-techniques - Papers for normalization techniques, released codes collections. (Other Lists / Monkey C Lists)
README
# Awesome-Normalization-Techniques [](https://github.com/sindresorhus/awesome)
Papers for normalization techniques, released codes collections.
Any addition or bug feel free to open an issue or pull requests.
[2020](https://github.com/tczhangzhi/awesome-normalization-techniques#2020) - [2019](https://github.com/tczhangzhi/awesome-normalization-techniques#2019) - [2018](https://github.com/tczhangzhi/awesome-normalization-techniques#2018) - [2017](https://github.com/tczhangzhi/awesome-normalization-techniques#2017) - [2016](https://github.com/tczhangzhi/awesome-normalization-techniques#2016)
## 2020
- **A New Look at Ghost Normalization**
- Neofytos Dimitriou, Ognjen Arandjelovic
- [[Paper](https://arxiv.org/abs/2007.08554)]
- **TaskNorm: Rethinking Batch Normalization for Meta-Learning (ICML 2020)**
- John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, Richard E Turner
- [[Paper](https://icml.cc/virtual/2020/poster/6200)]
- [[Python Reference](https://github.com/cambridge-mlg/cnaps)]
- **Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise (AAAI 2020)**
- Senwei Liang, Zhongzhan Huang, Mingfu Liang, Haizhao Yang
- [[Paper](https://arxiv.org/abs/1908.04008)]
- [[Python Reference](https://github.com/gbup-group/IEBN)]
- **Towards Stabilizing Batch Statistics in BackWard Propagation of Batch Normalization (ICLR 2020)**
- Junjie Yan, Ruosi Wan, Xiangyu Zhang, Wei Zhang, Yichen Wei, Jian Sun
- [[Paper](https://arxiv.org/abs/2001.06838)]
- **Rethinking Spatially-Adaptive Normalization**
- Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He, Lu Yuan, Nenghai Yu
- [[Paper](https://arxiv.org/abs/2004.02867)]
- **Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks (CVPR 2020)**
- Saurabh Singh, Shankar Krishnan
- [[Paper](https://arxiv.org/abs/1911.09737)]
- [[Python Reference](https://github.com/philipperemy/keras-frn)]
- **Extended Batch Normalization**
- Chunjie Luo, Jianfeng Zhan, Lei Wang, and Wanling Gao
- [[Paper](https://arxiv.org/abs/2003.05569)]
- [[C++ Reference](https://github.com/layumi/Batch-Normal-For-Caffe)]
- **Knowledge Distillation via Adaptive Instance Normalization**
- Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos
- [[Paper](https://arxiv.org/abs/2003.04289)]
- **Four Things Everyone Should Know to Improve Batch Normalization**
- Cecilia Summers, Michael J. Dinneen
- [[Paper](https://arxiv.org/abs/1906.03548)]
- **Region Normalization for Image Inpainting (AAAI 2020)**
- Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping Li, Zhizheng Zhang, Sen Liu
- [[Paper](https://arxiv.org/pdf/1911.10375)]
- [[Python Reference](https://github.com/geekyutao/RN)]
- **Local Context Normalization: Revisiting Local Normalization (CVPR 2020)**
- Anthony Ortiz , Caleb Robinson, Mahmudulla Hassan, Dan Morris2, Olac Fuentes1, Christopher Kiekintveld, and Nebojsa Jojic
- [[Paper](https://arxiv.org/abs/1912.05845)]
- **Attentive Normalization for Conditional Image Generation (CVPR 2020)**
- Yi Wang, Ying-Cong Chen, Xiangyu Zhang, Jian Sun, Jiaya Jia
- [[Paper](https://arxiv.org/abs/2004.03828)]
- [[Python Reference](https://github.com/shepnerd/AttenNorm)]
- **Cross-Iteration Batch Normalization**
- Zhuliang Yao, Yue Cao, Shuxin Zheng, Gao Huang, Stephen Lin
- [[Paper](https://arxiv.org/abs/2002.05712)]
- [[Python Reference](https://github.com/hlld/cross-iteration-batch_normalization)]
- **SEAN: Image Synthesis with Semantic Region-Adaptive Normalization (CVPR 2020)**
- Peihao Zhu, Rameen Abdal, Yipeng Qin, Peter Wonka
- [[Paper](https://arxiv.org/abs/1911.12861)]
- [[Python Reference](https://github.com/ZPdesu/SEAN)]
- **Evolving Normalization-Activation Layers**
- Hanxiao Liu, Andrew Brock, Karen Simonyan, Quoc V. Le
- [[Paper](https://arxiv.org/abs/2004.02967)]
## 2019
- **Differentiable Dynamic Normalization for Learning Deep Representation (PMLR 2019/ICML 2019)**
- Ping Luo, Peng Zhanglin, Shao Wenqi, Zhang Ruimao, Ren Jiamin, Wu Lingyun
- [[Paper](http://proceedings.mlr.press/v97/luo19a.html)]
- **Mean Spectral Normalization of Deep Neural Networks for Embedded Automation**
- Anand Krishnamoorthy Subramanian, Nak Young Chong
- [[Paper](https://arxiv.org/abs/1907.04003)]
- [[Python Reference](https://github.com/AntixK/mean-spectral-norm)]
- **An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation**
- Vincent Michalski, Vikram Voleti, Samira Ebrahimi Kahou, Anthony Ortiz, Pascal Vincent, Chris Pal, Doina Precup
- [[Paper](https://arxiv.org/pdf/1908.00061.pdf)]
- **Generalized Batch Normalization: Towards Accelerating Deep Neural Networks (AAAI 2019)**
- Xiaoyong Yuan, Zheng Feng, Matthew Norton, Xiaolin Li
- [[Paper](https://arxiv.org/abs/1812.03271)]
- **Split Batch Normalization: Improving Semi-Supervised Learning under Domain Shift**
- Michał Zając, Konrad Zolna, Stanisław Jastrzębski
- [[Paper](https://arxiv.org/abs/1904.03515)]
- **Channel Normalization in Convolutional Neural Network avoids Vanishing Gradients (ICML 2019)**
- Zhenwei Dai, Reinhard Heckel
- [[Paper](https://openreview.net/pdf?id=rJxBv4r22V)]
- [[Python Reference](https://github.com/MLI-lab/channel_normalization)]
- **Unpaired Image Translation via Adaptive Convolution-based Normalization**
- Wonwoong Cho, Kangyeol Kim, Eungyeup Kim, Hyunwoo J. Kim, Jaegul Choo
- [[Paper](https://arxiv.org/abs/1911.13271)]
- **EvalNorm: Estimating Batch Normalization Statistics for Evaluation (ICCV 2019)**
- Saurabh Singh, Abhinav Shrivastava
- [[Paper](https://arxiv.org/abs/1904.06031)]
- **Online Normalization for Training Neural Networks (NIPS 2019)**
- Vitaliy Chiley, Ilya Sharapov, Atli Kosson Urs Koster
- [[Paper](https://arxiv.org/abs/1905.05894)]
- [[Python Reference](https://github.com/Cerebras/online-normalization)]
- **Transferable Normalization: Towards Improving Transferability of Deep Neural Networks (NIPS 2019)**
- Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael I. Jordan
- [[Paper](https://papers.nips.cc/paper/8470-transferable-normalization-towards-improving-transferability-of-deep-neural-networks)]
- [[Python Reference](https://github.com/thuml/TransNorm)]
- **Iterative Normalization: Beyond Standardization Towards Efficient Whitening (CVPR 2019)**
- Lei Huang, Yi Zhou, Fan Zhu, Li Liu, Ling Shao
- [[Paper](https://arxiv.org/abs/1904.03441)]
- [[Python Reference](https://github.com/huangleiBuaa/IterNorm)]
- **Domain-Specific Batch Normalization for Unsupervised Domain Adaptation (CVPR 2019)**
- Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, Bohyung Han
- [[Paper](https://arxiv.org/abs/1906.03950)]
- [[Python Reference](https://github.com/wgchang/DSBN)]
- **Attentive Normalization**
- Xilai Li, Wei Sun and Tianfu Wu
- [[Paper](https://arxiv.org/abs/1908.01259)]
- [[Python Reference](https://github.com/Cyril9227/Keras_AttentiveNormalization)]
- **Rethinking Normalization and Elimination Singularity in Neural Networks**
- Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, Alan Yuille
- [[Paper](https://arxiv.org/abs/1911.09738k)]
- **Dynamic Instance Normalization for Arbitrary**
- Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, Shilei Wen
- [[Paper](https://arxiv.org/abs/1911.06953)]
- **Semantic Image Synthesis with Spatially-Adaptive Normalization (CVPR 2019)**
- Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu
- [[Paper](https://openaccess.thecvf.com/content_CVPR_2019/papers/Park_Semantic_Image_Synthesis_With_Spatially-Adaptive_Normalization_CVPR_2019_paper.pdf)]
- [[Python Reference](https://github.com/taki0112/SPADE-Tensorflow)]
- **Differentiable Learning-to-Normalize via Switchable Normalization (ICLR 2019)**
- Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, Jingyu Li
- [[Paper](https://arxiv.org/abs/1806.10779)]
- [[Python Reference](https://github.com/switchablenorms/Switchable-Normalization)]
- **SSN: Learning Sparse Switchable Normalization via Sparsestmax (CVPR 2019)**
- Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, Ping Luo
- [[Paper](https://openaccess.thecvf.com/content_CVPR_2019/papers/Shao_SSN_Learning_Sparse_Switchable_Normalization_via_SparsestMax_CVPR_2019_paper.pdf)]
- **A Mean Field Theory of Batch Normalization**
- Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz
- [[Paper](https://arxiv.org/abs/1902.08129)]
- **Restructuring Batch Normalization to Accelerate CNN Training**
- Wonkyung Jung, Daejin Jung, Byeongho Kim, Sunjung Lee, Wonjong Rhee, Jung Ho Ahn
- [[Paper](https://arxiv.org/abs/1807.01702)]
- **A Novel Convolutional Neural Network for Image Steganalysis with Shared Normalization (TMM 2019)**
- Songtao Wu, Sheng-hua Zhong, and Yan Liu
- [[Paper](https://arxiv.org/abs/1711.07306)]
- **Training Faster by Separating Modes of Variation in Batch-normalized Models (TPAMI 2019)**
- Mahdi M. Kalayeh, Mubarak Shah
- [[Paper](https://arxiv.org/abs/1806.02892)]
- **Uncertainty Estimation via Stochastic Batch Normalization (ISNN 2019)**
- Andrei Atanov, Arsenii Ashukha, Dmitry Molchanov, Kirill Neklyudov, Dmitry Vetrov
- [[Paper](https://arxiv.org/abs/1802.04893)]
- [[Python Reference](https://github.com/AndrewAtanov/stochastic-batch-normalization)]
## 2018
- **Decorrelated Batch Normalization (CVPR 2018)**
- Lei Huang, Dawei Yang, Bo Lang, Jia Deng
- [[Paper](https://arxiv.org/abs/1804.08450)]
- [[Lua Reference](https://github.com/princeton-vl/DecorrelatedBN)]
- **MegDet: A Large Mini-Batch Object Detector (CVPR 2018)**
- Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, Jian Sun
- [[Paper](https://arxiv.org/abs/1711.07240)]
- **Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks (NIPS 2018)**
- Hyeonseob Nam, Hyo-Eun Kim
- [[Paper](https://arxiv.org/abs/1805.07925)]
- **Kalman Normalization: Normalizing Internal Representations Across Network Layers (NIPS 2018)**
- Guangrun Wang, Jiefeng Peng, Ping Luo, Xinjiang Wang, Liang Lin
- [[Paper](https://arxiv.org/abs/1802.03133)]
- **Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?**
- Ping Luo, Zhanglin Peng, Jiamin Ren, Ruimao Zhang
- [[Paper](https://arxiv.org/abs/1811.07727)]
- **L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks (TNNLS)**
- Shuang Wu , Guoqi Li, Lei Deng, Liu Liu, Dong Wu, Yuan Xie, Luping Shi
- [[Paper](https://arxiv.org/abs/1802.09769)]
- **In-Place Activated BatchNorm for Memory-Optimized Training of DNNs (CVPR 2018)**
- Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder
- [[Paper](https://openaccess.thecvf.com/content_cvpr_2018/html/Bulo_In-Place_Activated_BatchNorm_CVPR_2018_paper.html9)]
- [[Python Reference](https://github.com/mapillary/inplace_abn)]
- **Group Normalization (ECCV 2018)**
- Yuxin Wu, Kaiming He
- [[Paper](https://arxiv.org/abs/1803.08494)]
- [[Python Reference](https://github.com/kuangliu/pytorch-groupnorm)]
- **Spectral Normalization for Generative Adversarial**
- Takeru Miyato, Toshiki Kataoka, Masanori Koyama, Yuichi Yoshida
- [[Paper](https://openreview.net/pdf?id=B1QRgziT-)]
- [[Python Reference](https://github.com/godisboy/SN-GAN)]
## 2017
- **Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models (NIPS 2017)**
- Sergey Ioffe
- [[Paper](https://arxiv.org/abs/1702.03275)]
- [[Python Reference](https://github.com/chainer/chainer/blob/b65bb350d6f23e1ad24020d589899388af03933f/chainer/links/normalization/batch_renormalization.py)]
- **Modulating Early Visual Processing by Language (NIPS 2017)**
- Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, Aaron Courville
- [[Paper](https://arxiv.org/abs/1707.00683)]
- [[Python Reference](https://github.com/ap229997/Conditional-Batch-Norm)]
- **Instance Normalization: the Missing Ingredient for Fast Stylization**
- Dmitry Ulyanov, Andrea Vedaldi
- [[Paper](https://arxiv.org/abs/1607.08022)]
- [[PyTorch Reference](https://github.com/pytorch/pytorch/blob/eace0533985641d9c2f36e43e3de694aca886bd9/torch/nn/modules/instancenorm.pyh)]
- **Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks**
- Luo Chunjie, Zhan Jianfeng, Wang Lei, Yang Qiang
- [[Paper](https://arxiv.org/pdf/1702.05870)]
- **Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization (ICCV 2017)**
- Xun Huang, Serge Belongie
- [[Paper](https://arxiv.org/abs/1703.06868)]
- [[Python Reference](https://github.com/naoto0804/pytorch-AdaIN)]
- **Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes**
- Mengye Ren, Renjie Liao, Raquel Urtasun, Fabian H. Sinz, Richard S. Zemel
- [[Paper](https://arxiv.org/abs/1611.04520)]
- **Recurrent Batch Normalization**
- Tim Cooijmans, Nicolas Ballas, César Laurent, Çag ̆lar Gülçehre, Aaron Courville
- [[Paper](https://arxiv.org/abs/1603.09025)]
## 2016
- **Layer Normalization**
- Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton
- [[Paper](https://arxiv.org/abs/1607.06450)]
- [[PyTorch Reference](https://github.com/pytorch/pytorch/blob/9fe3b1857d8315d66f8a39b3d4661237640f2f74/benchmarks/tensorexpr/normalization.py)]
- **Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks**
- Devansh Arpit, Yingbo Zhou, Bhargava U. Kota, Venu Govindaraju, SUNY Buffalo
- [[Paper](https://arxiv.org/abs/1603.01431)]
- **Weight Normalization: a Simple Reparameterization to Accelerate Training of Deep Neural Networks (NIPS 2016)**
- Tim Salimans, Diederik P. Kingma
- [[Paper](https://arxiv.org/abs/1602.07868)]
## 2015
- **Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift**
- Sergey Ioffe, Christian Szegedy
- [[Paper](https://arxiv.org/abs/1502.03167)]
- [[PyTorch Reference](https://github.com/pytorch/pytorch/blob/881c1adfcd916b6cd5de91bc343eb86aff88cc80/torch/nn/modules/batchnorm.py)]