https://github.com/unnir/cancelout
CancelOut is a special layer for deep neural networks that can help identify a subset of relevant input features for streaming or static data.
https://github.com/unnir/cancelout
deep-learning feature-importance feature-selection
Last synced: 9 months ago
JSON representation
CancelOut is a special layer for deep neural networks that can help identify a subset of relevant input features for streaming or static data.
- Host: GitHub
- URL: https://github.com/unnir/cancelout
- Owner: unnir
- License: mit
- Created: 2018-12-14T16:13:25.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2020-11-09T10:45:49.000Z (over 5 years ago)
- Last Synced: 2025-06-22T01:38:20.969Z (about 1 year ago)
- Topics: deep-learning, feature-importance, feature-selection
- Language: Jupyter Notebook
- Homepage:
- Size: 213 KB
- Stars: 26
- Watchers: 4
- Forks: 13
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# CancelOut
# TL;DR
**CancelOut** is a layer for deep neural networks, that can help identify a subset of relevant input features for streaming or static data.
TAGS: Feature Importance, Feature Ranking, Feature Selection, Deep Learning Sensitivity Analysis.
# Intuition
The main idea is to update weights (W) of CancelOut during a training stage, so that ''noisy'' or less essential features will be canceled out with a negative weight. Otherwise, the best features, which contribute more to a learning process is going to be passed through with a positive weight. One can see CancelOut is a "gate" input, there NN decides who is going to pass through (see the equation below).
&space;=&space;\boldsymbol{X}&space;\odot&space;g&space;({W_{CO}}))
$&space;denotes&space;here&space;elementwise&space;application,&space;e.g.&space;$&space;\boldsymbol{X}&space;=\begin{bmatrix}&space;a&space;\\&space;b&space;\\&space;c&space;\\&space;\end{bmatrix}&space;$,&space;then&space;$g(\boldsymbol{X})&space;=&space;g\biggl(\begin{bmatrix}&space;a&space;\\&space;b&space;\\&space;c&space;\\&space;\end{bmatrix}\biggl)&space;=&space;\biggl(\begin{bmatrix}&space;g(a)&space;\\&space;g(b)&space;\\&space;g(c)&space;\\&space;\end{bmatrix}\bigg)$.)
# Example
For examples, please refer to the `_example.ipynb` files.
Or just copy the code:
## PyTorch implementation:
```python
class CancelOut(nn.Module):
'''
CancelOut Layer
x - an input data (vector, matrix, tensor)
'''
def __init__(self,inp, *kargs, **kwargs):
super(CancelOut, self).__init__()
self.weights = nn.Parameter(torch.zeros(inp,requires_grad = True) + 4)
def forward(self, x):
return (x * torch.sigmoid(self.weights.float()))
```
## Keras/TensorFlow implementation:
```python
class CancelOut(keras.layers.Layer):
'''
CancelOut Layer
'''
def __init__(self, activation='sigmoid', cancelout_loss=True, lambda_1=0.002, lambda_2=0.001):
super(CancelOut, self).__init__()
self.lambda_1 = lambda_1
self.lambda_2 = lambda_2
self.cancelout_loss = cancelout_loss
if activation == 'sigmoid': self.activation = tf.sigmoid
if activation == 'softmax': self.activation = tf.nn.softmax
def build(self, input_shape):
self.w = self.add_weight(
shape=(input_shape[-1],),
initializer=tf.keras.initializers.Constant(1),
trainable=True)
def call(self, inputs):
if self.cancelout_loss:
self.add_loss( self.lambda_1 * tf.norm(self.w, ord=1) + self.lambda_2 * tf.norm(self.w, ord=2))
return tf.math.multiply(inputs, self.activation(self.w))
def get_config(self):
return {"activation": self.activation}
```
# * Work in progress. *