https://github.com/nna301023/stackconvprun

DenseNet & Resnet as Backbone to Perform Lahan Gambut Classification with Self-Attention Pruning Mechanism
https://github.com/nna301023/stackconvprun

attention-mechanism pruning-optimization streamlit-webapp

Last synced: 12 months ago
JSON representation

DenseNet & Resnet as Backbone to Perform Lahan Gambut Classification with Self-Attention Pruning Mechanism

• Host: GitHub
• URL: https://github.com/nna301023/stackconvprun
• Owner: NnA301023
• License: apache-2.0
• Created: 2023-12-03T01:01:49.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-01-22T04:36:18.000Z (about 2 years ago)
• Last Synced: 2024-01-22T05:34:49.609Z (about 2 years ago)
• Topics: attention-mechanism, pruning-optimization, streamlit-webapp
• Language: Python
• Homepage: https://stackconvprun-erika.streamlit.app/
• Size: 1.17 MB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# StackConvPrun

DenseNet & Resnet as Backbone to Perform Lahan Gambut Classification with Self-Attention Pruning Mechanism

# Initial Research Problem

The utilization of a concatenation layer based on two backbone architectures, such as DenseNet and ResNet, introduces additional computational overhead during both training and inference. In this study, we propose the implementation of pruning based on a self-attention mechanism scoring. This pruning mechanism aims to selectively remove layers in the backbone/head section, reducing computational requirements. The decision to prune a layer is determined by comparing its attention mechanism score to the standard deviation of the total attention mechanism scores.

*note: This approach introduces variability in the pruned layers across different use cases, and we address this by initializing random variables for each layer.

# Conclusion

Based on the evaluation results, we successfully reduced the training time of the baseline model from 30 minutes per epoch to just 1 minute. While the evaluation results show a trade-off, the achieved improvements in computational efficiency are deemed acceptable.

# Evaluation Result

## Before Pruning

![](images/Evaluation%20Metrics%20Before%20Pruning%20Attention.png)

## After Pruning

![](images/Evaluation%20Metrics%20After%20Pruning%20Attention.png)