https://github.com/americast/algo_bias
Detecting bias in ML models using heat maps
https://github.com/americast/algo_bias
algorithmic-bias deep-learning heatmaps machine-learning pytorch tensorflow torch
Last synced: 6 months ago
JSON representation
Detecting bias in ML models using heat maps
- Host: GitHub
- URL: https://github.com/americast/algo_bias
- Owner: americast
- License: gpl-3.0
- Created: 2018-05-19T12:06:57.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2022-02-06T09:50:04.000Z (over 3 years ago)
- Last Synced: 2025-03-27T18:50:21.574Z (6 months ago)
- Topics: algorithmic-bias, deep-learning, heatmaps, machine-learning, pytorch, tensorflow, torch
- Language: Python
- Homepage:
- Size: 17.8 MB
- Stars: 2
- Watchers: 4
- Forks: 7
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Algorithmic Bias
[](https://gitter.im/KWoC-americast/algo-bias?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
The aim of this project is to classify the income category people in a binary fashion as per their attributes. Next, we try to find out the factors which have contributed to the output of our model, at every layer, using a technique called [Layerwise Relevance Propagation (LRP)](http://heatmapping.org/). Three variants of the implementation have been presented in this project. `main.py` is PyTorch based and `tmain.py` is TensorFlow based. LRP is supported by `kmain.py` and `interprettensor/` which are in Keras (preferably through theano) and interprettonsor TF wrapper respectively. We also generate heatmaps for various layers against the input using the Keras variant.
## Setup
```
pip install theano innvestigate
```
The package `innvestigate` installs a variant of keras which needs a version of tensorflow which is currently unsupported. Hence, we use `theano` for this project. We adjust the `backend` parameter to Theano instead of Tensorflow accordingly at `~/.keras/keras.json`. For more information on how to do this, visit [here](https://stackoverflow.com/questions/47153429/how-to-change-the-backend-of-keras-to-theano).
## Understanding the Code
The Keras variant of this code is used to generate heatmap as PyTorch and Tensorflow are not supported yet. So we will describe the kmain.py here until then. The first step is to import all the necessary libraries and functions. Next we define Batch Size, Epochs and Learning Rate to aid us in the training process. Then we use the Keras Sequential function to define model layers. Why we chose this particular model architecture? We will provide an ablation study later in this README listing accuracies of various models.
Next, we ask user for the choice to load previous saved model. Depending on which we load the model or train from beginning. We then ask user the choice to train or test. Depending on the selection, we load a csv file containing data to train or test. We then seperate attributes with target values, namely x_train and y_train. We then compile the model using Adam optimizer and Categorical Crossentropy as loss function. We use Keras' checkpoints to define early stopping (for training) if accuracy does not improve after certain epochs. Then depending on the user choice we train or test the model. If we train the model, then the best performing weights are saved to use them to load the model next time.
If the user chose to test, we print the model accuracy on the test data. Next, we do LRP (Layer-wise Relevance Propagation) which is a method for understanding deep neural networks by running a backward pass to observe how the individual layers of the program work. This technique brings brings explainability to Deep Neural Networks. More information on LRP is given [here](https://link.springer.com/chapter/10.1007/978-3-030-28954-6_10).
Source: heatmapping.org
Let us understand the code used to implement LRP using Innvestigate:
```python
# we use lrp.z here out of various methods available like
# Saliency, Deconvnet, GuidedBackprop, SmoothGrad, and IntergratedGradient
# then we pass our model here
analyzer = innvestigate.create_analyzer("lrp.z", model)
# then we pass model inputs, that is x_train
analysis = analyzer.analyze(x_train)
```
Then we perform the layer wise analysis by blocking off the layers and then performing analysis on them.
```python
# Here we cutoff the layers and define a new model. This will remove the last Dense layer.
new_model_1 = Model(model.inputs, model.layers[-3].output)
# model.get_weights() return a numpy list of weights
new_model_1.set_weights(model.get_weights())
# summarize the model
new_model_1.summary()
```
Then we perform the analysis on this new model:
```python
analyzer = innvestigate.create_analyzer("lrp.z", new_model_1)
analysis = analyzer.analyze(x_train)
# We save the results as numpy vectors
np.save("out_lrp", analysis)
```
We perform this method on every layers, meanwhile saving the numpy vectors.
Using this, we get layer by layer heatmaps in form of numpy vectors. In summary, we are cutting off layers in the network to obtain heatmap vectors for every layer and we are then merging them together to form a heatmap matrix/tensor. We use lrp_matrix.py to get complete numpy vectors then we plot them.
## Choosing Appropriate Model
We perforemed experiments with various model architecture to choose the right one. We present a table below stating the model architecture and accuracy.
Dense layer : 'D', Batch Normalization: 'BN', Dropout: 'DR'
| Model Architecture | Train acurracy | test accuracy |
|:-:|:-:|:-:|
| D512-SELU-D256-BN-SELU-D128-BN-SELU-D64-BN-SELU-D64-BN-SELU-D32-BN-SELU-D16-BN-SELU-D8-BN-SELU-D2-SIGMOID | 79.64 | 42 |
| D512-SELU-D512-BN-SELU-D128-BN-SELU-D64-BN-SELU-D64-BN-SELU-D32-BN-SELU-D16-BN-SELU-D8-BN-SELU-D2-SIGMOID | 81.19 | 78.643 |
| D512-SELU-D512-BN-SELU-D128-BN-SELU-D128-BN-SELU-D64-BN-SELU-D32-BN-SELU-D16-BN-SELU-D8-BN-SELU-D2-SIGMOID | 80.37 | 59.787 |
| D512-SELU-D512-BN-SELU-D512-BN-SELU-D128-BN-SELU-D64-BN-SELU-D32-BN-SELU-D16-BN-SELU-D8-BN-SELU-D2-SIGMOID | 80.38 | 57.14 |
| D256-RELU-D256-BN-RELU-D128-BN-RELU-D64-BN-RELU-D64-BN-RELU-D32-BN-RELU-D16-BN-RELU-D8-BN-RELU-D2-SIGMOID | 80.20 | 77.25 |
| D512-RELU-D512-BN-RELU-D256-BN-RELU-D64-BN-RELU-D64-BN-RELU-D32-BN-RELU-D16-BN-RELU-D8-BN-RELU-D2-SIGMOID | 80.04 | 61.16 |
| SAME PREVIOUS MODEL, EPOCHS=20000 | 92.477 | 78.84 |
| D256-RELU-D256-BN-RELU-D128-BN-RELU-D128-BN-RELU-D64-BN-RELU-D64-BN-RELU-D32-BN-RELU-D16-BN-RELU-D8-BN-RELU-D2-SIGMOID | 91.545 | 78.68 |
| **D512-RELU-D512-BN-RELU-DR(0.4)-D256-BN-RELU-DR(0.4)-D256-BN-RELU-DR(0.4)-D128-BN-RELU-DR(0.3)-D128-BN-RELU-DR(0.3)-D64-BN-RELU-DR(0.3)-D64-BN-RELU-DR(0.3)-D64-BN-RELU-DR(0.3)-D64-BN-RELU-DR(0.3)-D8-BN-RELU-DR(0.3)-D2-SIGMOID, EPOCHS=30000** | **88.23** | **81.78** |
| D512-RELU-D256-BN-RELU-D256-BN-RELU-D256-BN-RELU-D128-BN-RELU-D64-BN-RELU-D64-BN-RELU-D16-BN-RELU-D8-BN-RELU-D2-SIGMOID | 91.361 | 77.86 |
## Generating heatmaps
Next, we generate heatmaps using
```
python kmain.py
python lrp_matrix.py
python plot.py
```
Please make sure you run `kmain.py` in inference mode by answering the questions prompted accordingly.
## Sample heatmap
