{"id":15034069,"url":"https://github.com/hila-chefer/transformer-explainability","last_synced_at":"2025-05-15T18:00:24.123Z","repository":{"id":37300128,"uuid":"315439501","full_name":"hila-chefer/Transformer-Explainability","owner":"hila-chefer","description":"[CVPR 2021] Official PyTorch implementation for Transformer Interpretability Beyond Attention Visualization, a novel method to visualize classifications by Transformer based networks.","archived":false,"fork":false,"pushed_at":"2024-01-24T05:59:39.000Z","size":3943,"stargazers_count":1867,"open_issues_count":15,"forks_count":248,"subscribers_count":20,"default_branch":"main","last_synced_at":"2025-04-07T22:11:12.402Z","etag":null,"topics":["attention-matrix","attention-visualization","bert","bert-model","cvpr2021","deep-learning","explainability","perturbation","transformer-interpretability","vision-transformer","visualize-classifications","vit"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hila-chefer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":"DeiT.PNG","publiccode":null,"codemeta":null}},"created_at":"2020-11-23T21:00:00.000Z","updated_at":"2025-04-07T14:30:45.000Z","dependencies_parsed_at":"2024-09-20T16:20:40.336Z","dependency_job_id":null,"html_url":"https://github.com/hila-chefer/Transformer-Explainability","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hila-chefer%2FTransformer-Explainability","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hila-chefer%2FTransformer-Explainability/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hila-chefer%2FTransformer-Explainability/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hila-chefer%2FTransformer-Explainability/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hila-chefer","download_url":"https://codeload.github.com/hila-chefer/Transformer-Explainability/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254394718,"owners_count":22063984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-matrix","attention-visualization","bert","bert-model","cvpr2021","deep-learning","explainability","perturbation","transformer-interpretability","vision-transformer","visualize-classifications","vit"],"created_at":"2024-09-24T20:23:49.176Z","updated_at":"2025-05-15T18:00:24.082Z","avatar_url":"https://github.com/hila-chefer.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PyTorch Implementation of [Transformer Interpretability Beyond Attention Visualization](https://arxiv.org/abs/2012.09838) [CVPR 2021]\n\n#### Check out our new advancements- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](https://github.com/hila-chefer/Transformer-MM-Explainability)!\nFaster, more general, and can be applied to *any* type of attention!\nAmong the features:\n* We remove LRP for a simple and quick solution, and prove that the great results from our first paper still hold!\n* We expand our work to *any* type of Transformer- not just self-attention based encoders, but also co-attention encoders and encoder-decoders!\n* We show that VQA models can actually understand both image and text and make connections!\n* We use a DETR object detector and create segmentation masks from our explanations!\n* We provide a colab notebook with all the examples. You can very easily add images and questions of your own!\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"400\" height=\"450\" src=\"new_work.jpg\"\u003e\n\u003c/p\u003e\n\n---\n## ViT explainability notebook:\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/hila-chefer/Transformer-Explainability/blob/main/Transformer_explainability.ipynb)\n\n## BERT explainability notebook:\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/hila-chefer/Transformer-Explainability/blob/main/BERT_explainability.ipynb)\n---\n\n## Updates\nApril 5 2021: Check out this new [post](https://analyticsindiamag.com/compute-relevancy-of-transformer-networks-via-novel-interpretable-transformer/) about our paper! A great resource for understanding the main concepts behind our work.\n\nMarch 15 2021: [A Colab notebook for BERT for sentiment analysis added!](https://colab.research.google.com/github/hila-chefer/Transformer-Explainability/blob/main/BERT_explainability.ipynb)\n\nFeb 28 2021: Our paper was accepted to CVPR 2021! \n\nFeb 17 2021: [A Colab notebook with all examples added!](https://github.com/hila-chefer/Transformer-Explainability/blob/main/Transformer_explainability.ipynb)\n\nJan 5 2021: [A Jupyter notebook for DeiT added!](https://github.com/hila-chefer/Transformer-Explainability/blob/main/DeiT_example.ipynb)\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"300\" height=\"460\" src=\"https://github.com/hila-chefer/Transformer-Explainability/blob/main/DeiT.PNG\"\u003e\n\u003c/p\u003e\n\n\n## Introduction\nOfficial implementation of [Transformer Interpretability Beyond Attention Visualization](https://arxiv.org/abs/2012.09838).\n\nWe introduce a novel method which allows to visualize classifications made by a Transformer based model for both vision and NLP tasks.\nOur method also allows to visualize explanations per class.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"600\" height=\"200\" src=\"https://github.com/hila-chefer/Transformer-Explainability/blob/main/method-page-001.jpg\"\u003e\n\u003c/p\u003e\nMethod consists of 3 phases:\n\n1. Calculating relevance for each attention matrix using our novel formulation of LRP.\n\n2. Backpropagation of gradients for each attention matrix w.r.t. the visualized class. Gradients are used to average attention heads.\n\n3. Layer aggregation with rollout.\n\nPlease notice our [Jupyter notebook](https://github.com/hila-chefer/Transformer-Explainability/blob/main/example.ipynb) where you can run the two class specific examples from the paper.\n\n\n![alt text](https://github.com/hila-chefer/Transformer-Explainability/blob/main/example.PNG)\n\nTo add another input image, simply add the image to the [samples folder](https://github.com/hila-chefer/Transformer-Explainability/tree/main/samples), and use the `generate_visualization` function for your selected class of interest (using the `class_index={class_idx}`), not specifying the index will visualize the top class.\n\n## Credits\nViT implementation is based on:\n- https://github.com/rwightman/pytorch-image-models\n- https://github.com/lucidrains/vit-pytorch\n- pretrained weights from: https://github.com/google-research/vision_transformer\n\nBERT implementation is taken from the huggingface Transformers library:\nhttps://huggingface.co/transformers/\n\nERASER benchmark code adapted from the ERASER GitHub implementation: https://github.com/jayded/eraserbenchmark\n\nText visualizations in supplementary were created using TAHV heatmap generator for text: https://github.com/jiesutd/Text-Attention-Heatmap-Visualization\n\n## Reproducing results on ViT\n\n### Section A. Segmentation Results\n\nExample:\n```\nCUDA_VISIBLE_DEVICES=0 PYTHONPATH=./:$PYTHONPATH python3 baselines/ViT/imagenet_seg_eval.py --method transformer_attribution --imagenet-seg-path /path/to/gtsegs_ijcv.mat\n\n```\n[Link to download dataset](http://calvin-vision.net/bigstuff/proj-imagenet/data/gtsegs_ijcv.mat).\n\nIn the exmaple above we run a segmentation test with our method. Notice you can choose which method you wish to run using the `--method` argument. \nYou must provide a path to imagenet segmentation data in `--imagenet-seg-path`.\n\n### Section B. Perturbation Results\n\nExample:\n```\nCUDA_VISIBLE_DEVICES=0 PYTHONPATH=./:$PYTHONPATH python3 baselines/ViT/generate_visualizations.py --method transformer_attribution --imagenet-validation-path /path/to/imagenet_validation_directory\n```\n\nNotice that you can choose to visualize by target or top class by using the `--vis-cls` argument.\n\nNow to run the perturbation test run the following command:\n```\nCUDA_VISIBLE_DEVICES=0 PYTHONPATH=./:$PYTHONPATH python3 baselines/ViT/pertubation_eval_from_hdf5.py --method transformer_attribution\n```\n\nNotice that you can use the `--neg` argument to run either positive or negative perturbation.\n\n## Reproducing results on BERT\n\n1. Download the pretrained weights:\n\n- Download `classifier.zip` from https://drive.google.com/file/d/1kGMTr69UWWe70i-o2_JfjmWDQjT66xwQ/view?usp=sharing\n- mkdir -p `./bert_models/movies`\n- unzip classifier.zip -d ./bert_models/movies/\n\n2. Download the dataset pkl file:\n\n- Download `preprocessed.pkl` from https://drive.google.com/file/d/1-gfbTj6D87KIm_u1QMHGLKSL3e93hxBH/view?usp=sharing\n- mv preprocessed.pkl ./bert_models/movies\n\n3. Download the dataset:\n\n- Download `movies.zip` from https://drive.google.com/file/d/11faFLGkc0hkw3wrGTYJBr1nIvkRb189F/view?usp=sharing\n- unzip movies.zip -d ./data/\n\n4. Now you can run the model.\n\nExample:\n```\nCUDA_VISIBLE_DEVICES=0 PYTHONPATH=./:$PYTHONPATH python3 BERT_rationale_benchmark/models/pipeline/bert_pipeline.py --data_dir data/movies/ --output_dir bert_models/movies/ --model_params BERT_params/movies_bert.json\n```\nTo control which algorithm to use for explanations change the `method` variable in `BERT_rationale_benchmark/models/pipeline/bert_pipeline.py` (Defaults to 'transformer_attribution' which is our method).\nRunning this command will create a directory for the method in `bert_models/movies/\u003cmethod_name\u003e`.\n\nIn order to run f1 test with k, run the following command:\n```\nPYTHONPATH=./:$PYTHONPATH python3 BERT_rationale_benchmark/metrics.py --data_dir data/movies/ --split test --results bert_models/movies/\u003cmethod_name\u003e/identifier_results_k.json\n```\n\nAlso, in the method directory there will be created `.tex` files containing the explanations extracted for each example. This corresponds to our visualizations in the supplementary.\n\n## Citing our paper\nIf you make use of our work, please cite our paper:\n```\n@InProceedings{Chefer_2021_CVPR,\n    author    = {Chefer, Hila and Gur, Shir and Wolf, Lior},\n    title     = {Transformer Interpretability Beyond Attention Visualization},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2021},\n    pages     = {782-791}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhila-chefer%2Ftransformer-explainability","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhila-chefer%2Ftransformer-explainability","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhila-chefer%2Ftransformer-explainability/lists"}