https://github.com/Cyril9227/Keras_AttentiveNormalization

Unofficial Keras implementation of the paper Attentive Normalization.
https://github.com/Cyril9227/Keras_AttentiveNormalization

attention-mechanism batch-normalization deep-learning keras keras-tensorflow

Last synced: 2 months ago
JSON representation

Unofficial Keras implementation of the paper Attentive Normalization.

• Host: GitHub
• URL: https://github.com/Cyril9227/Keras_AttentiveNormalization
• Owner: Cyril9227
• License: mit
• Created: 2019-08-22T08:22:24.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2020-02-12T05:52:30.000Z (almost 5 years ago)
• Last Synced: 2024-08-03T19:08:30.004Z (6 months ago)
• Topics: attention-mechanism, batch-normalization, deep-learning, keras, keras-tensorflow
• Language: Jupyter Notebook
• Homepage:
• Size: 239 KB
• Stars: 29
• Watchers: 5
• Forks: 5
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-normalization-techniques - [Python Reference

README

        
# Attentive Normalization

This repository is an unofficial Keras implementation of the paper [Attentive Normalization](https://arxiv.org/abs/1908.01259) by [Xilai Li](https://github.com/xilaili), [Wei Sun](https://github.com/WillSuen) and [Tianfu Wu](https://github.com/tfwu).

The official implementation will be released here : https://github.com/ivMCL/

## Introduction

Attentive Normalization (AN) is an attention-based version of BN which recalibrates channel information of BN. AN absorbs the [Squeeze-and-Excitation (SE) mechanism](https://arxiv.org/abs/1709.01507) into the affine transformation of BN. AN learns a small number of scale and offset parameters per channel (i.e., different affine transformations). Their weighted sums (i.e., mixture) are used in the final affine transformation. The weights are instance-specific and learned in a way that channel-wise attention is considered, similar in spirit to the squeeze module in the SE unit. This can be used as a drop-in replacement of standard BatchNormalization layer. 



  



## Usage

Please refer to the notebook for an usage example.