https://github.com/mahdihosseini/rmsgd
Exploiting Explainable Metrics for Augmented SGD [CVPR2022]
https://github.com/mahdihosseini/rmsgd
convolutional-neural-networks deep-learning pytorch stochastic-gradient-descent
Last synced: 7 months ago
JSON representation
Exploiting Explainable Metrics for Augmented SGD [CVPR2022]
- Host: GitHub
- URL: https://github.com/mahdihosseini/rmsgd
- Owner: mahdihosseini
- License: mit
- Created: 2020-06-04T14:11:02.000Z (about 6 years ago)
- Default Branch: main
- Last Pushed: 2022-04-03T00:13:06.000Z (over 4 years ago)
- Last Synced: 2025-03-24T12:21:44.263Z (over 1 year ago)
- Topics: convolutional-neural-networks, deep-learning, pytorch, stochastic-gradient-descent
- Language: Python
- Homepage:
- Size: 19.5 MB
- Stars: 45
- Watchers: 4
- Forks: 13
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# RMSGD: Augmented SGD Optimizer
Official PyTorch implementation of the **RMSGD** optimizer from:
> [**Exploiting Explainable Metrics for Augmented SGD**](https://arxiv.org/abs/2203.16723)
> Mahdi S. Hosseini, Mathieu Tuli, Konstantinos N. Plataniotis
> *Accepted in IEEE/CVF Conference on Computer Vision and Pattern Recognition ([CVPR2022](https://cvpr2022.thecvf.com/))*
>
---
We propose new explainability metrics that measure the redundant information in a network's layers and exploit this information to augment the Stochastic Gradient Descent (SGD) optimizer by adaptively adjusting the learning rate in each layer. We call this new optimizer **RMSGD**. RMSGD is fast, performs better than existing sota, and generalizes well across experimental configurations.
## Contents
This repository + branch contains the standalone optimizer, which is pip installable. Equally, you could copy the contents of [src/rmsgd](src/rmsgd) into your local repository and use the optimizer as is.
For all code relating to our paper and to replicate those experiments, see the [paper](https://github.com/mahdihosseini/RMSGD/tree/paper) branch
## Installation
You can install rmsgd using `pip install rmsgd`, or equally:
```console
git clone https://github.com/mahdihosseini/RMSGD.git
cd RMSGD
pip install .
```
## Usage
RMSGD can be used like any other optimizer, with one additional step:
```python
from rmsgd import RMSGD
...
optimizer = RMSGD(...)
...
for input in data_loader:
optimizer.zero_grad()
output = network(input)
optimizer.step()
optimizer.epoch_step()
```
Simply, you must call `.epoch_step()` at the end of each epoch to update the analysis of the network layers.
## Citation
```
@Article{hosseini2022rmsgd,
author = {Hosseini, Mahdi S. and Tuli, Mathieu and Plataniotis, Konstantinos N.},
title = {Exploiting Explainable Metrics for Augmented SGD},
journal = {Accepted in IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2022},
}
```
## License
This project is released under the MIT license. Please see the [LICENSE](LICENSE) file for more information.