{"id":74270,"url":"https://github.com/wassname/awesome-interpretability","name":"awesome-interpretability","description":"Awesome tools for interpreting, manipulating the internals of of deep neural networks. ","projects_count":54,"last_synced_at":"2026-08-03T20:00:18.785Z","repository":{"id":237904755,"uuid":"795464480","full_name":"wassname/awesome-interpretability","owner":"wassname","description":"Awesome tools for interpreting, manipulating the internals of of deep neural networks. ","archived":false,"fork":false,"pushed_at":"2026-04-25T23:26:23.000Z","size":27,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-07-15T15:31:25.353Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wassname.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-05-03T10:41:03.000Z","updated_at":"2026-05-08T19:25:50.000Z","dependencies_parsed_at":"2026-06-27T07:00:22.643Z","dependency_job_id":null,"html_url":"https://github.com/wassname/awesome-interpretability","commit_stats":null,"previous_names":["wassname/awesome-interpretability"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/wassname/awesome-interpretability","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wassname%2Fawesome-interpretability","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wassname%2Fawesome-interpretability/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wassname%2Fawesome-interpretability/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wassname%2Fawesome-interpretability/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wassname","download_url":"https://codeload.github.com/wassname/awesome-interpretability/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wassname%2Fawesome-interpretability/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36249873,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-08-03T02:00:06.975Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-10-15T19:31:41.682Z","updated_at":"2026-08-03T20:00:18.786Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["Structured output","Explainability, counterfactuals and probing","Mechanistic interpretability libraries","Mechanistic interpretability","See more","Adapters"],"sub_categories":[],"readme":"# awesome-interpretability\n\n\n## Mechanistic interpretability libraries\n- [BauKit](https://github.com/davidbau/baukit) ![](https://img.shields.io/github/stars/davidbau/baukit?style=social) - light, simple, and well loved\n\n- [TransformerLens](https://github.com/neelnanda-io/TransformerLens) ![](https://img.shields.io/github/stars/neelnanda-io/TransformerLens?style=social)\n  - uses jaxtyping, aliases models into a common interface, not as huggingface compatible as other libs\n  - \u003e [an extremely opinionated toolkit for doing whatever you want to specific models, ](https://twitter.com/NeelNanda5/status/1786146027659280430)\n- [Tuned Lens](https://github.com/AlignmentResearch/tuned-lens) ![](https://img.shields.io/github/stars/AlignmentResearch/tuned-lens?style=social) - tools for looking at how transformer predictions are built layer-by-layer\n- [vgel/repeng](https://github.com/vgel/repeng) - A library for making RepE control vectors \n- [nnsight](https://github.com/ndif-team/nnsight) ![](https://img.shields.io/github/stars/ndif-team/nnsight?style=social) \n  - \u003e [To customize a model, instead of running it as a function, you run it as a \"with\" context. Inside \"with\" you can write regular pytorch to modify the computation.](https://twitter.com/davidbau/status/1785991660197015827)\n  - aim to keep it as simple as bauket eventually, and support remote mechinterp. HuggingFace compatible\n- [Pyvene (intervention focused)](https://github.com/stanfordnlp/pyvene)  ![](https://img.shields.io/github/stars/stanfordnlp/pyvene?style=social)\n  - \u003e [pyvene tries to be HuggingFace-native, supporting pre-defined interventions or customized interventions (below).](https://twitter.com/ZhengxuanZenWu/status/1768356269470191842)\n- [penzai](https://github.com/google-deepmind/penzai) ![](https://img.shields.io/github/stars/google-deepmind/penzai?style=social) - jax-based, not HuggingFace-native\n- [ViT-Prisma](https://github.com/Prisma-Multimodal/ViT-Prisma) ![](https://img.shields.io/github/stars/Prisma-Multimodal/ViT-Prisma?style=social) - mechanistic interpretability for vision and video transformers\n- [Transformer Debugger (OpenAI)](https://github.com/openai/transformer-debugger) ![](https://img.shields.io/github/stars/openai/transformer-debugger?style=social) - not HuggingFace-native \n- [Graphpatch](https://github.com/evan-lloyd/graphpatch) ![](https://img.shields.io/github/stars/evan-lloyd/graphpatch?style=social) - promising but abandoned\n- [NeuroX](https://github.com/fdalvi/NeuroX)\n- [A tutorial on doing it manually](https://github.com/annahdo/implementing_activation_steering)\n- [cupbearer](https://github.com/ejnnr/cupbearer) A library for mechanistic anomaly detection \n- [Overcomplete](https://github.com/KempnerInstitute/overcomplete) ![](https://img.shields.io/github/stars/KempnerInstitute/overcomplete?style=social) - vision SAE toolbox\n- [vLLM-Hook](https://github.com/IBM/vLLM-Hook) ![](https://img.shields.io/github/stars/IBM/vLLM-Hook?style=social) - program internal states of vLLM-served models\n- [vllm-lens](https://github.com/UKGovernmentBEIS/vllm-lens) ![](https://img.shields.io/github/stars/UKGovernmentBEIS/vllm-lens?style=social) - extract residual stream activations and apply steering vectors in vLLM\n- [Neuronpedia](http://neuronpedia.org/) - public feature/neuron browser\n- [Docent](https://transluce.org/introducing-docent) - interactive model explanation and steering interface\n  \n## Explainability, counterfactuals and probing\n\n- [captum](https://github.com/pytorch/captum)\n- [inseq](https://github.com/inseq-team/inseq)\n- [Explabox](https://github.com/MarcelRobeer/explabox) (2022)\n- [IBM: AIX360](https://github.com/Trusted-AI/AIX360) (2019)\n- [Microsoft: Responsible AI Toolbox](https://responsibleaitoolbox.ai/) (2021)\n    - Dashboard that integrates: Error analysis, Fairlearn, InterpretML, DiCE, EconML and Data Balance\n- [InterpretML](https://github.com/interpretml/interpret-community)\n    - SHAP, Mimic and LIME explainers. Permutation feature importance.\n- [MI2.ai](Ihttps://www.mi2.ai/)\n    - [DrWhy](https://github.com/ModelOriented/DrWhy/tree/master) (2019)\n        - DALEX, survex, Arena, fairmodels,\n    - Currently working on: ARES, xSurvival, Large Model Analysis\n- [XAI](https://github.com/EthicalML/xai) (2018)\n- [ELI5](https://eli5.readthedocs.io/en/latest/overview.html)\n- [NN-SVG](https://alexlenail.me/NN-SVG/)\n- [Neptune-AI blog](https://neptune.ai/blog/ml-model-interpretation-tools)\n- [Neptune-AI blog](https://neptune.ai/blog/explainability-auditability-ml-definitions-techniques-tools)\n- [AI Ethics tool landscape](https://edwinwenink.github.io/ai-ethics-tool-landscape/)\n\n## Adapters\n\nSee [this lit review of Adapter intervention types](https://github.com/wassname/adapters_as_hypotheses)\n\n## Steering\n\nTODO format\nhttps://github.com/vgel/repeng\nhttps://github.com/IBM/AISteer360\nhttps://github.com/wassname/ssteer-eval-aware\nhttps://github.com/IBM/activation-steering\nhttps://github.com/chili-lab/Spherical-Steering\nhttps://github.com/safety-research/weight-steering\n\n\n## Structured output \n\n- [jsonformer](https://github.com/1rgs/jsonformer)\n  - doesn't do enums. huggingface only\n- [prob_jsonformer](https://github.com/wassname/prob_jsonformer) - Jsonformer, but it can output the probability of each choice in a single pass. Has enum\n- [outlines](https://github.com/outlines-dev/outlines) \n- [Microsoft Guidance](https://github.com/guidance-ai/guidance)\n- [lmql.ai](https://lmql.ai/)\n- [llama.cpp grammar](https://github.com/ggerganov/llama.cpp/pull/1773)\n- [langchain output_parsers](https://python.langchain.com/docs/modules/model_io/output_parsers/)\n- [salute](https://github.com/LevanKvirkvelia/salute) - typescript\n- [TypeChat](https://github.com/microsoft/TypeChat) - typescript\n- [guardrails](https://github.com/ShreyaR/guardrails)\n- [clownfish](https://github.com/newhouseb/clownfish) - 2023 Modifying Transformers to Follow a JSON Schema - not updated\n- [relm](https://github.com/mkuchnik/relm) - 2023 Regular Expression engine for Language Models  - not updated\n- [Constrained-Text-Generation-Studio](https://github.com/Hellisotherpeople/Constrained-Text-Generation-Studio)\n- [kor](https://github.com/eyurtsev/kor)\n- [lm-format-enforcer](https://github.com/noamgat/lm-format-enforcer) - remote api's\n- [instructor](https://github.com/jxnl/instructor/) - for remote api's without logits\n- [Promptify](https://github.com/promptslab/Promptify)\n\n## See more\n\n- [s list that inspired this one](https://github.com/dweprinz/dweprinz.github.io/blob/905db3fe5bd0d3ca0ddd2b201382c2a25accc00b/_pages/resources/responsible-ai/ai-safety.md?plain=1#L48)\n- https://mechinterpworkshop.com/cfp/\n- [the github interpretability topic](https://github.com/topics/interpretability)\n- https://github.com/wangyongjie-ntu/Awesome-explainable-AI\n- https://twitter.com/davidbau/status/1785991694279913617\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/wassname%2Fawesome-interpretability/projects"}