{"id":26103678,"url":"https://github.com/aleph-alpha/atman","last_synced_at":"2025-03-09T20:42:02.112Z","repository":{"id":205825929,"uuid":"702936489","full_name":"Aleph-Alpha/AtMan","owner":"Aleph-Alpha","description":null,"archived":false,"fork":false,"pushed_at":"2023-11-07T15:44:31.000Z","size":19039,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2023-11-08T04:33:45.951Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aleph-Alpha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-legalcode.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-10-10T09:38:37.000Z","updated_at":"2023-11-06T16:32:16.000Z","dependencies_parsed_at":"2023-11-07T01:09:18.834Z","dependency_job_id":null,"html_url":"https://github.com/Aleph-Alpha/AtMan","commit_stats":null,"previous_names":["aleph-alpha/atman"],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2FAtMan","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2FAtMan/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2FAtMan/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2FAtMan/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aleph-Alpha","download_url":"https://codeload.github.com/Aleph-Alpha/AtMan/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242750781,"owners_count":20179256,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-09T20:42:01.523Z","updated_at":"2025-03-09T20:42:02.094Z","avatar_url":"https://github.com/Aleph-Alpha.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AtMan- XAI on generative models\n\nAtMan is an explainability method designed for multi-modal generative transformer models. It correlates the relevance of the input tokens to the generated output by exhaustive perturbation. To obtain the score values, it applies ATtention MANipulation throughout all layers, and measures the difference in the resulting logprobs on the target tokens.\nIt further encorporates embedding similarity to surppress the entire entropy found at once.\nAs depicted in following examples, one is able to highlight various discriminative features on the same input, i.p. on text as well as image-modality.\n\n\n![Demonstrating AtMan XAI on model generation](figs/title.png)\n\n[Paper Link](https://arxiv.org/abs/2301.08110)\n\n## roadmap\n - continue to cleanup repo\n - i.p. remove Explainer class and other overhead\n - more examples\n - hf integration?\n\n## prelim\nThis repo includes the XAI methods AtMan, Chefer, and a Captum interface for IG, GradCam etc. for the language-model GPT-J and vision-language model [MAGMA](https://github.com/Aleph-Alpha/magma) and [BLIP](https://colab.research.google.com/github/salesforce/BLIP). (Big props to Mayukh Deb.)\n\nTo install all required dependencies, run the following command, e.g. in a conda environment with python3.8:\n```\nbash startup-hook.sh\n```\nNote: further model-checkpoints will be downloaded when executing for the first time. Sometimes CLIP fails to verify on the first execution -\u003e running again works usually.\n\nThe main folders are atman-magma, for all XAI implementations on the MAGMA model, and BLIP for all XAI implementations on the BLIP model.\n\n# examples with MAGMA\n```\ncd atman-magma\n```\n\n## image-text/ MAGMA x AtMan\nrequires 1 RTX 3090\n\n```\npython example_explain_panda_atman.py\n```\n\n## image-text/ MAGMA x Chefer\nrequires 1 A100\n\n```\npython example_explain_panda_chefer.py\n```\n\n## image-text/ MAGMA x Captum IxG, ...\nrequires 1 A100\n\n```\npython example_explain_panda_captum.py\n```\n\n## image-text/ rollout\nrequires 1 RTX 3090\n\n```\npython example_explain_attention_rollout.py\n```\n\n## text/ GPT-J\n```\npython example_steering.py\npython example_document_qa_sentence_level_explain.py\n```\n\n# examples with BLIP\n```\ncd BLIP\n```\n\n## image-text/ BLIP x AtMan\n```\npython explain_vqa_run.py\n```\n\n## image-text/ BLIP x Chefer\n```\npython explain_vqa_chefer.py\n```\n\n\n# Method and Evaluation\n\n![steering and measuring](figs/fig2.png)\n\n![embed similarity and squas](figs/fig3.png)\n\n![vqa](figs/fig4.png)\n\n![performance](figs/fig5.png)\n\n![quantitative](figs/tab1.png)\n\n\n# cite\n```\n@inproceedings{\ndeiseroth2023atman,\ntitle={{ATMAN}: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation},\nauthor={Bj{\\\"o}rn Deiseroth and Mayukh Deb and Samuel Weinbach and Manuel Brack and Patrick Schramowski and Kristian Kersting},\nbooktitle={Thirty-seventh Conference on Neural Information Processing Systems},\nyear={2023},\nurl={https://openreview.net/forum?id=PBpEb86bj7}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Fatman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faleph-alpha%2Fatman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Fatman/lists"}