Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/frgfm/torch-cam
Class activation maps for your PyTorch models (CAM, Grad-CAM, Grad-CAM++, Smooth Grad-CAM++, Score-CAM, SS-CAM, IS-CAM, XGrad-CAM, Layer-CAM)
https://github.com/frgfm/torch-cam
activation-maps class-activation-map cnn deep-learning grad-cam gradcam gradcam-plus-plus interpretability interpretable-deep-learning python pytorch saliency-map score-cam smoothgrad
Last synced: 22 days ago
JSON representation
Class activation maps for your PyTorch models (CAM, Grad-CAM, Grad-CAM++, Smooth Grad-CAM++, Score-CAM, SS-CAM, IS-CAM, XGrad-CAM, Layer-CAM)
- Host: GitHub
- URL: https://github.com/frgfm/torch-cam
- Owner: frgfm
- License: apache-2.0
- Created: 2020-03-23T18:34:51.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2024-10-07T04:18:11.000Z (about 1 month ago)
- Last Synced: 2024-10-15T10:05:39.301Z (29 days ago)
- Topics: activation-maps, class-activation-map, cnn, deep-learning, grad-cam, gradcam, gradcam-plus-plus, interpretability, interpretable-deep-learning, python, pytorch, saliency-map, score-cam, smoothgrad
- Language: Python
- Homepage: https://frgfm.github.io/torch-cam/
- Size: 10.5 MB
- Stars: 2,005
- Watchers: 11
- Forks: 206
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Citation: CITATION.cff
Awesome Lists containing this project
- Awesome-explainable-AI - https://github.com/frgfm/torch-cam - cam?style=social) (Python Libraries(sort in alphabeta order) / Evaluation methods)
- awesome-yolo-object-detection - TorchCAM - cam?style=social"/> : Class activation maps for your PyTorch models (CAM, Grad-CAM, Grad-CAM++, Smooth Grad-CAM++, Score-CAM, SS-CAM, IS-CAM, XGrad-CAM, Layer-CAM). (Applications)
- awesome-yolo-object-detection - TorchCAM - cam?style=social"/> : Class activation maps for your PyTorch models (CAM, Grad-CAM, Grad-CAM++, Smooth Grad-CAM++, Score-CAM, SS-CAM, IS-CAM, XGrad-CAM, Layer-CAM). (Applications)
README
TorchCAM: class activation explorer
Simple way to leverage the class-specific activation of convolutional layers in PyTorch.
Source: image from woopets (activation maps created with a pretrained Resnet-18)
## Quick Tour
### Setting your CAM
TorchCAM leverages [PyTorch hooking mechanisms](https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html#forward-and-backward-function-hooks) to seamlessly retrieve all required information to produce the class activation without additional efforts from the user. Each CAM object acts as a wrapper around your model.
You can find the exhaustive list of supported CAM methods in the [documentation](https://frgfm.github.io/torch-cam/methods.html), then use it as follows:
```python
# Define your model
from torchvision.models import resnet18
model = resnet18(pretrained=True).eval()
# Set your CAM extractor
from torchcam.methods import SmoothGradCAMpp
cam_extractor = SmoothGradCAMpp(model)
```
*Please note that by default, the layer at which the CAM is retrieved is set to the last non-reduced convolutional layer. If you wish to investigate a specific layer, use the `target_layer` argument in the constructor.*
### Retrieving the class activation map
Once your CAM extractor is set, you only need to use your model to infer on your data as usual. If any additional information is required, the extractor will get it for you automatically.
```python
from torchvision.io.image import read_image
from torchvision.transforms.functional import normalize, resize, to_pil_image
from torchvision.models import resnet18
from torchcam.methods import SmoothGradCAMpp
model = resnet18(pretrained=True).eval()
# Get your input
img = read_image("path/to/your/image.png")
# Preprocess it for your chosen model
input_tensor = normalize(resize(img, (224, 224)) / 255., [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
with SmoothGradCAMpp(model) as cam_extractor:
# Preprocess your data and feed it to the model
out = model(input_tensor.unsqueeze(0))
# Retrieve the CAM by passing the class index and the model output
activation_map = cam_extractor(out.squeeze(0).argmax().item(), out)
```
If you want to visualize your heatmap, you only need to cast the CAM to a numpy ndarray:
```python
import matplotlib.pyplot as plt
# Visualize the raw CAM
plt.imshow(activation_map[0].squeeze(0).numpy()); plt.axis('off'); plt.tight_layout(); plt.show()
```
Or if you wish to overlay it on your input image:
```python
import matplotlib.pyplot as plt
from torchcam.utils import overlay_mask
# Resize the CAM and overlay it
result = overlay_mask(to_pil_image(img), to_pil_image(activation_map[0].squeeze(0), mode='F'), alpha=0.5)
# Display it
plt.imshow(result); plt.axis('off'); plt.tight_layout(); plt.show()
```
## Setup
Python 3.8 (or higher) and [pip](https://pip.pypa.io/en/stable/)/[conda](https://docs.conda.io/en/latest/miniconda.html) are required to install TorchCAM.
### Stable release
You can install the last stable release of the package using [pypi](https://pypi.org/project/torchcam/) as follows:
```shell
pip install torchcam
```
or using [conda](https://anaconda.org/frgfm/torchcam):
```shell
conda install -c frgfm torchcam
```
### Developer installation
Alternatively, if you wish to use the latest features of the project that haven't made their way to a release yet, you can install the package from source:
```shell
git clone https://github.com/frgfm/torch-cam.git
pip install -e torch-cam/.
```
## CAM Zoo
This project is developed and maintained by the repo owner, but the implementation was based on the following research papers:
- [Learning Deep Features for Discriminative Localization](https://arxiv.org/abs/1512.04150): the original CAM paper
- [Grad-CAM](https://arxiv.org/abs/1610.02391): GradCAM paper, generalizing CAM to models without global average pooling.
- [Grad-CAM++](https://arxiv.org/abs/1710.11063): improvement of GradCAM++ for more accurate pixel-level contribution to the activation.
- [Smooth Grad-CAM++](https://arxiv.org/abs/1908.01224): SmoothGrad mechanism coupled with GradCAM.
- [Score-CAM](https://arxiv.org/abs/1910.01279): score-weighting of class activation for better interpretability.
- [SS-CAM](https://arxiv.org/abs/2006.14255): SmoothGrad mechanism coupled with Score-CAM.
- [IS-CAM](https://arxiv.org/abs/2010.03023): integration-based variant of Score-CAM.
- [XGrad-CAM](https://arxiv.org/abs/2008.02312): improved version of Grad-CAM in terms of sensitivity and conservation.
- [Layer-CAM](http://mftp.mmcheng.net/Papers/21TIP_LayerCAM.pdf): Grad-CAM alternative leveraging pixel-wise contribution of the gradient to the activation.
Source: YouTube video (activation maps created by Layer-CAM with a pretrained ResNet-18)
## What else
### Documentation
The full package documentation is available [here](https://frgfm.github.io/torch-cam/) for detailed specifications.
### Demo app
A minimal demo app is provided for you to play with the supported CAM methods! Feel free to check out the live demo on [](https://huggingface.co/spaces/frgfm/torch-cam)
If you prefer running the demo by yourself, you will need an extra dependency ([Streamlit](https://streamlit.io/)) for the app to run:
```
pip install -e ".[demo]"
```
You can then easily run your app in your default browser by running:
```
streamlit run demo/app.py
```
### Example script
An example script is provided for you to benchmark the heatmaps produced by multiple CAM approaches on the same image:
```shell
python scripts/cam_example.py --arch resnet18 --class-idx 232 --rows 2
```
*All script arguments can be checked using `python scripts/cam_example.py --help`*
### Latency benchmark
You crave for beautiful activation maps, but you don't know whether it fits your needs in terms of latency?
In the table below, you will find a latency benchmark (forward pass not included) for all CAM methods:
| CAM method | Arch | GPU mean (std) | CPU mean (std) |
| ------------------------------------------------------------ | ------------------ | ------------------ | -------------------- |
| [CAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.CAM) | resnet18 | 0.11ms (0.02ms) | 0.14ms (0.03ms) |
| [GradCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.GradCAM) | resnet18 | 3.71ms (1.11ms) | 40.66ms (1.82ms) |
| [GradCAMpp](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.GradCAMpp) | resnet18 | 5.21ms (1.22ms) | 41.61ms (3.24ms) |
| [SmoothGradCAMpp](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.SmoothGradCAMpp) | resnet18 | 33.67ms (2.51ms) | 239.27ms (7.85ms) |
| [ScoreCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.ScoreCAM) | resnet18 | 304.74ms (11.54ms) | 6796.89ms (415.14ms) |
| [SSCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.SSCAM) | resnet18 | | |
| [ISCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.ISCAM) | resnet18 | | |
| [XGradCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.XGradCAM) | resnet18 | 3.78ms (0.96ms) | 40.63ms (2.03ms) |
| [LayerCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.LayerCAM) | resnet18 | 3.65ms (1.04ms) | 40.91ms (1.79ms) |
| [CAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.CAM) | mobilenet_v3_large | N/A* | N/A* |
| [GradCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.GradCAM) | mobilenet_v3_large | 8.61ms (1.04ms) | 26.64ms (3.46ms) |
| [GradCAMpp](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.GradCAMpp) | mobilenet_v3_large | 8.83ms (1.29ms) | 25.50ms (3.10ms) |
| [SmoothGradCAMpp](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.SmoothGradCAMpp) | mobilenet_v3_large | 77.38ms (3.83ms) | 156.25ms (4.89ms) |
| [ScoreCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.ScoreCAM) | mobilenet_v3_large | 35.19ms (2.11ms) | 679.16ms (55.04ms) |
| [SSCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.SSCAM) | mobilenet_v3_large | | |
| [ISCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.ISCAM) | mobilenet_v3_large | | |
| [XGradCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.XGradCAM) | mobilenet_v3_large | 8.41ms (0.98ms) | 24.21ms (2.94ms) |
| [LayerCAM](https://frgfm.github.io/torch-cam/latest/methods.html#torchcam.methods.LayerCAM) | mobilenet_v3_large | 8.02ms (0.95ms) | 25.14ms (3.17ms) |
**The base CAM method cannot work with architectures that have multiple fully-connected layers*
This benchmark was performed over 100 iterations on (224, 224) inputs, on a laptop to better reflect performances that can be expected by common users. The hardware setup includes an [Intel(R) Core(TM) i7-10750H](https://ark.intel.com/content/www/us/en/ark/products/201837/intel-core-i710750h-processor-12m-cache-up-to-5-00-ghz.html) for the CPU, and a [NVIDIA GeForce RTX 2070 with Max-Q Design](https://www.nvidia.com/fr-fr/geforce/graphics-cards/rtx-2070/) for the GPU.
You can run this latency benchmark for any CAM method on your hardware as follows:
```bash
python scripts/eval_latency.py SmoothGradCAMpp
```
*All script arguments can be checked using `python scripts/eval_latency.py --help`*
### Example notebooks
Looking for more illustrations of TorchCAM features?
You might want to check the [Jupyter notebooks](notebooks) designed to give you a broader overview.
## Citation
If you wish to cite this project, feel free to use this [BibTeX](http://www.bibtex.org/) reference:
```bibtex
@misc{torcham2020,
title={TorchCAM: class activation explorer},
author={François-Guillaume Fernandez},
year={2020},
month={March},
publisher = {GitHub},
howpublished = {\url{https://github.com/frgfm/torch-cam}}
}
```
## Contributing
Feeling like extending the range of possibilities of CAM? Or perhaps submitting a paper implementation? Any sort of contribution is greatly appreciated!
You can find a short guide in [`CONTRIBUTING`](CONTRIBUTING.md) to help grow this project!
## License
Distributed under the Apache 2.0 License. See [`LICENSE`](LICENSE) for more information.
[](https://app.fossa.com/projects/git%2Bgithub.com%2Ffrgfm%2Ftorch-cam?ref=badge_large&issueType=license)
