{"id":18556662,"url":"https://github.com/ahmdtaha/constrained_attention_filter","last_synced_at":"2025-08-31T13:40:34.991Z","repository":{"id":98612044,"uuid":"278173105","full_name":"ahmdtaha/constrained_attention_filter","owner":"ahmdtaha","description":"(ECCV2020) Tensorflow implementation of A Generic Visualization Approach for Convolutional Neural Networks","archived":false,"fork":false,"pushed_at":"2021-10-09T23:18:36.000Z","size":65140,"stargazers_count":40,"open_issues_count":0,"forks_count":8,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-10T16:54:08.817Z","etag":null,"topics":["attention-visualization","autonomous-driving","autonomous-vehicles","classification","convolutional-neural-networks","deep-neural-networks","feature-embedding","imagenet","python-3","python3","retrieval","tensorflow","weakly-supervised-localization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ahmdtaha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-08T19:08:47.000Z","updated_at":"2025-01-10T00:05:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"bf4a241e-6671-4e31-a0b9-1e398fefc76c","html_url":"https://github.com/ahmdtaha/constrained_attention_filter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ahmdtaha/constrained_attention_filter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmdtaha%2Fconstrained_attention_filter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmdtaha%2Fconstrained_attention_filter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmdtaha%2Fconstrained_attention_filter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmdtaha%2Fconstrained_attention_filter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ahmdtaha","download_url":"https://codeload.github.com/ahmdtaha/constrained_attention_filter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahmdtaha%2Fconstrained_attention_filter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272988781,"owners_count":25026959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-31T02:00:09.071Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-visualization","autonomous-driving","autonomous-vehicles","classification","convolutional-neural-networks","deep-neural-networks","feature-embedding","imagenet","python-3","python3","retrieval","tensorflow","weakly-supervised-localization"],"created_at":"2024-11-06T21:32:41.660Z","updated_at":"2025-08-31T13:40:34.983Z","avatar_url":"https://github.com/ahmdtaha.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Constrained Attention Filter (CAF)\n(ECCV2020) Tensorflow implementation of **A Generic Visualization Approach for Convolutional Neural Networks**\n\nAn official [PyTorch implementation of L2-CAF](https://github.com/ahmdtaha/l2_caf_pytorch) is now available :star: :star: :star:\n\n[Paper](https://arxiv.org/abs/2007.09748) | [1 Min Video](https://youtu.be/W4xaKQlPEl0) | [10 Mins Video](https://youtu.be/Wpw3ewSvnFE)\n\n### Qualitative Evaluation -- L2-CAF Slow Motion Convergence\n\n|                   | One Object | Two Objects |\n|-------------------|------------|-------------|\n|     Last Conv     |![](https://github.com/ahmdtaha/constrained_attention_filter/blob/master/gif/ILSVRC2012_val_00000003_cls_230_dense_block4_conv_block24_0.gif)|![](https://github.com/ahmdtaha/constrained_attention_filter/blob/master/gif/ILSVRC2012_val_00000021_cls_334_dense_block4_conv_block24_0.gif)|\n|Intermediate Conv |![](https://github.com/ahmdtaha/constrained_attention_filter/blob/master/gif/ILSVRC2012_val_00000003_cls_230_dense_block4_conv_block10_0.gif)|![](https://github.com/ahmdtaha/constrained_attention_filter/blob/master/gif/ILSVRC2012_val_00000021_cls_334_dense_block4_conv_block10_0.gif)|\n\n### TL;DR\nL2-CAF has three core components:\n\n1- [TF filter](https://github.com/ahmdtaha/constrained_attention_filter/blob/035f0880baae6a12540dd0b4cc0830cef243c1af/nets/attention_filter.py#L19) This is the function that inserts L2-CAF inside a network (E.g, inside a [DenseNet](https://github.com/ahmdtaha/constrained_attention_filter/blob/035f0880baae6a12540dd0b4cc0830cef243c1af/nets/densenet161.py#L90)). L2-CAF is by default disabled; it is passive during classification.\nTo active/de-activate L2-CAF (turn on and off the filter), I use the bool `atten_var_gate`. False deactivate L2-CAF, while True activates the filter.\n\n2- [Optimization loop](https://github.com/ahmdtaha/constrained_attention_filter/blob/1d45e121fa56b131e94dbb72c22c169589bb679f/visualize_attention.py#L160)  In this loop, we computes the class-oblivious and class-specific loss and leverage gradient descent to minimize it. When the loss stabilize (loss - prev_loss\u003c 10e-5), break out of the loop.\n\n3- [Finalize filter before saving](https://github.com/ahmdtaha/constrained_attention_filter/blob/1d45e121fa56b131e94dbb72c22c169589bb679f/visualize_attention.py#L18) After convergence, the output filter is normalized (L2-Norm|Softmaxed|Gauss-ed) before generating the heatmap.\n\n## Requirements\n\n* Python 3+ [Tested on 3.7]\n* Tensorflow 1.X [Tested on 1.14]\n\n## ImageNet Pretrained Models\nI used the following\n* [DenseNet](https://github.com/pudae/tensorflow-densenet)\n* [InceptionV1](https://github.com/tensorflow/models/tree/master/research/slim)\n* [ResNet](https://github.com/tensorflow/models/tree/master/research/slim)\n\n## Usage example\n\nUpdate [`base_config._load_user_setup`](https://github.com/ahmdtaha/constrained_attention_filter/blob/f95afd6c547a24122b8f182427fa4191ce5cb86c/config/base_config.py#L74) with your configurations.\nMainly, set the location of pre-trained model (e.g, densenet). The released code optimizes the constrained attention filter on samples images from the \"input_imgs\" directory. However, if you plan to run the code on a whole dataset (e.g, ImageNet), you shoud set the `local_datasets_dir` in _load_user_setup  \n\nThe unit L2-Norm constrained attention filter has two operating modes. \n* `visualize_attention.py` is the script for the vanilla \"slow\" (4 seconds) mode. I recommend running this first before experimenting with the fast L2-CAF version. The code of this mode is much easier to understand. The script's main function sets all the hyper-parameters needed. I will ellaborate more on each hyper-parameter soon.\n\n* `visualize_attention_fast.py` is the script for the fast (0.3 seconds) mode. The script only supports denseNet. I will add support to Inception and ResNet soon.\n This script only works for visualizing attention is the last conv layer. I only use it for quantitative evaluation experiments, for instance, when I evaluate L2-CAF using ImageNet validation split.\n\n    \n### TODO LIST\n\n* ~~Add Fast L2-CAF on DenseNet~~\n* ~~Add InceptionNet and ResNet support~~\n* Document to use the code\n* Document the intermediate layer visualization\n* Document extra technical tricks not mentioned in the paper \n\n### Contributing\n**It would be great if someone re-implement this in pytorch. Let me know and I will add a link to your Pytorch implementation here**\n\n\n### MISC Notes\n* We did not write localization evaluation code. We used the matlab code released by [CAM](https://github.com/zhoubolei/CAM) in Tables 1  and 3.\nWe used the python code released by [ADL](https://github.com/junsukchoe/ADL) in Table 2. \nFeel free to evaluate L2-CAF localization with other evaluation codes.\n* The softmax and Gaussian filters are released upon a reviewer request. The current Gaussian filter implementation is hard-coded to support only 7x7 attention filter.\n It is straight forward to extend it for any odd filter-size (e.g., 13x13). However, for even filter-size I think more changes are required. The last conv layer in standard architectures is 7x7. So the current configuration should cover most case-scenarios.\n \n* I used modules of this code (especially the nets package) in multiple projects, so there is a lot of code that is not related to L2-CAF. I will iteratively clean the code. The TL;DR section, at the top of the readme file, highlights the core functions related to L2-CAF.\n\n## Release History\n* 1.0.0\n    * First commit Vanilla L2-CAF on DenseNet, InceptionV1, and ResNet50V2 on 12, 15,18 July 2020\n    * Add Fast L2-CAF on DenseNet 21 July 2020\n    * Add Fast L2-CAF on Inception 22 July 2020\n    * Add Fast L2-CAF on ResNet 23 July 2020\n\n### Citation\n```\n@inproceedings{taha2020generic,\ntitle={A Generic Visualization Approach for Convolutional Neural Networks},\nauthor={Taha, Ahmed and Yang, Xitong and Shrivastava, Abhinav and Davis, Larry},\nbooktitle={Proceedings of the European Conference on Computer Vision (ECCV)},\nyear={2020}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahmdtaha%2Fconstrained_attention_filter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahmdtaha%2Fconstrained_attention_filter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahmdtaha%2Fconstrained_attention_filter/lists"}