{"id":13472279,"url":"https://github.com/openai/automated-interpretability","last_synced_at":"2025-04-12T17:39:16.939Z","repository":{"id":163158398,"uuid":"637994750","full_name":"openai/automated-interpretability","owner":"openai","description":null,"archived":false,"fork":false,"pushed_at":"2024-03-06T18:40:46.000Z","size":204,"stargazers_count":1000,"open_issues_count":15,"forks_count":116,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-04-03T18:15:22.600Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-08T21:24:11.000Z","updated_at":"2025-03-29T05:51:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"d6d84a61-6f54-4698-a600-c4f7785dff9e","html_url":"https://github.com/openai/automated-interpretability","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fautomated-interpretability","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fautomated-interpretability/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fautomated-interpretability/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fautomated-interpretability/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openai","download_url":"https://codeload.github.com/openai/automated-interpretability/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248606866,"owners_count":21132430,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T16:00:53.503Z","updated_at":"2025-04-12T17:39:16.917Z","avatar_url":"https://github.com/openai.png","language":"Python","funding_links":[],"categories":["Table of Contents","NLP","Tools","Python"],"sub_categories":["LLM Interpretability Tools","Interpretability/Explicability"],"readme":"# Automated interpretability\n\n## Code and tools\n\nThis repository contains code and tools associated with the [Language models can explain neurons in\nlanguage models](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html) paper, specifically:\n\n* Code for automatically generating, simulating, and scoring explanations of neuron behavior using\nthe methodology described in the paper. See the\n[neuron-explainer README](neuron-explainer/README.md) for more information.\n\nNote: if you run into errors of the form \"Error: Could not find any credentials that grant access to storage account: 'openaipublic' and container: 'neuron-explainer'\".\" you might be able to fix this by signing up for an azure account and specifying the credentials as described in the error message. \n\n* A tool for viewing neuron activations and explanations, accessible\n[here](https://openaipublic.blob.core.windows.net/neuron-explainer/neuron-viewer/index.html). See\nthe [neuron-viewer README](neuron-viewer/README.md) for more information.\n\n## Public datasets\n\nTogether with this code, we're also releasing public datasets of GPT-2 XL neurons and explanations.\nHere's an overview of those datasets.  \n\n* Neuron activations: `az://openaipublic/neuron-explainer/data/collated-activations/{layer_index}/{neuron_index}.json`\n    - Tokenized text sequences and their activations for the neuron. We\n    provide multiple sets of tokens and activations: top-activating ones, random\n    samples from several quantiles; and a completely random sample. We also provide\n    some basic statistics for the activations.\n    - Each file contains a JSON-formatted\n    [`NeuronRecord`](neuron-explainer/neuron_explainer/activations/activations.py#L89) dataclass.\n* Neuron explanations: `az://openaipublic/neuron-explainer/data/explanations/{layer_index}/{neuron_index}.jsonl`\n    - Scored model-generated explanations of the behavior of the neuron, including simulation results.\n    - Each file contains a JSON-formatted\n    [`NeuronSimulationResults`](neuron-explainer/neuron_explainer/explanations/explanations.py#L146)\n    dataclass.\n* Related neurons: `az://openaipublic/neuron-explainer/data/related-neurons/weight-based/{layer_index}/{neuron_index}.json`\n    - Lists of the upstream and downstream neurons with the most positive and negative connections (see below for definition).\n    - Each file contains a JSON-formatted dataclass whose definition is not included in this repo.\n* Tokens with high average activations:\n`az://openaipublic/neuron-explainer/data/related-tokens/activation-based/{layer_index}/{neuron_index}.json`\n    - Lists of tokens with the highest average activations for individual neurons, and their average activations.\n    - Each file contains a JSON-formatted [`TokenLookupTableSummaryOfNeuron`](neuron-explainer/neuron_explainer/activations/token_connections.py#L36)\n    dataclass.\n* Tokens with large inbound and outbound weights:\n`az://openaipublic/neuron-explainer/data/related-tokens/weight-based/{layer_index}/{neuron_index}.json`\n    - List of the most-positive and most-negative input and output tokens for individual neurons,\n    as well as the associated weight (see below for definition). \n    - Each file contains a JSON-formatted [`WeightBasedSummaryOfNeuron`](neuron-explainer/neuron_explainer/activations/token_connections.py#L17)\n    dataclass.\n\nUpdate (July 5, 2023):\nWe also released a set of explanations for GPT-2 Small. The methodology is slightly different from the methodology used for GPT-2 XL so the results aren't directly comparable.\n* Neuron activations: `az://openaipublic/neuron-explainer/gpt2_small_data/collated-activations/{layer_index}/{neuron_index}.json`\n* Neuron explanations: `az://openaipublic/neuron-explainer/gpt2_small_data/explanations/{layer_index}/{neuron_index}.jsonl`\n\nUpdate (August 30, 2023): We recently discovered a bug in how we performed inference on the GPT-2 series models used for the paper and for these datasets. Specifically, we used an optimized GELU implementation rather than the original GELU implementation associated with GPT-2. While the model’s behavior is very similar across these two configurations, the post-MLP activation values we used to generate and simulate explanations differ from the correct values by the following amounts for GPT-2 small:\n\n- Median: 0.0090\n- 90th percentile: 0.0252\n- 99th percentile: 0.0839\n- 99.9th percentile: 0.1736\n\n### Definition of connection weights\n\nRefer to [GPT-2 model code](https://github.com/openai/gpt-2/blob/master/src/model.py) for\nunderstanding of model weight conventions.\n\n*Neuron-neuron*: For two neurons `(l1, n1)` and `(l2, n2)` with `l1 \u003c l2`, the connection strength is defined as\n`h{l1}.mlp.c_proj.w[:, n1, :] @ diag(h{l2}.ln_2.g) @ h{l2}.mlp.c_fc.w[:, :, n2]`.\n\n*Neuron-token*: For token `t` and neuron `(l, n)`, the input weight is computed as\n`wte[t, :] @ diag(h{l}.ln_2.g) @ h{l}.mlp.c_fc.w[:, :, n]`\nand the output weight is computed as\n`h{l}.mlp.c_proj.w[:, n, :] @ diag(ln_f.g) @ wte[t, :]`.\n\n### Misc Lists of Interesting Neurons\nLists of neurons we thought were interesting according to different criteria, with some preliminary descriptions.\n* [Interesting Neurons (external)](https://docs.google.com/spreadsheets/d/1p7fYs31NU8sJoeKyUx4Mn2laGx8xXfHg_KcIvYiKPpg/edit#gid=0)\n* [Neurons that score high on random, possibly monosemantic? (external)](https://docs.google.com/spreadsheets/d/1TqKFcz-84jyIHLU7VRoTc8BoFBMpbgac-iNBnxVurQ8/edit?usp=sharing)\n* [Clusters of neurons well explained by activation explanation but not by tokens](https://docs.google.com/document/d/1lWhKowpKDdwTMALD_K541cdwgGoQx8DFUSuEe1U2AGE/edit?usp=sharing)\n* [Neurons sensitive to truncation](https://docs.google.com/document/d/1x89TWBvuHcyC2t01EDbJZJ5LQYHozlcS-VUmr5shf_A/edit?usp=sharing)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Fautomated-interpretability","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenai%2Fautomated-interpretability","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Fautomated-interpretability/lists"}