{"id":30838807,"url":"https://github.com/eleutherai/attention-probes","last_synced_at":"2025-09-06T18:12:24.541Z","repository":{"id":301032763,"uuid":"995053873","full_name":"EleutherAI/attention-probes","owner":"EleutherAI","description":"Linear probes with attention weighting","archived":false,"fork":false,"pushed_at":"2025-08-02T01:54:08.000Z","size":216502,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-02T03:56:46.887Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EleutherAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-02T22:36:40.000Z","updated_at":"2025-08-02T01:54:12.000Z","dependencies_parsed_at":"2025-08-02T03:34:52.337Z","dependency_job_id":null,"html_url":"https://github.com/EleutherAI/attention-probes","commit_stats":null,"previous_names":["eleutherai/attention-probes"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/EleutherAI/attention-probes","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EleutherAI%2Fattention-probes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EleutherAI%2Fattention-probes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EleutherAI%2Fattention-probes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EleutherAI%2Fattention-probes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EleutherAI","download_url":"https://codeload.github.com/EleutherAI/attention-probes/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EleutherAI%2Fattention-probes/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273941527,"owners_count":25195104,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-06T02:00:13.247Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-06T18:12:22.950Z","updated_at":"2025-09-06T18:12:24.530Z","avatar_url":"https://github.com/EleutherAI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Attention probes\n\nBased on https://github.com/shan23chen/MOSAIC (unlicensed). The original README can be found in `README_MOSAIC.md`.\n\n## Install\n\n```bash\n# install\nuv pip install git+https://github.com/EleutherAI/attention-probes.git\n\n# development\ngit clone https://github.com/EleutherAI/attention-probes \u0026\u0026 cd attention-probes\nuv sync \u0026\u0026 uv pip install -e .\n```\n\n## Experiments (from blog post)\n\n### Collect data\n\n```bash\n\n# caches to /output\nuv run scripts_mosaic/run_experiments.py --extract-only\n\n# for Neurons in a Haystack paper data:\n# step 1: download https://www.dropbox.com/scl/fo/14oxabm2eq47bkw2u0oxo/AKFTcikvAB8-GVdoBztQHxE?rlkey=u9qny1tsza6lqetzzua3jr8xn\u0026e=1\u0026dl=0\n# step 2: unzip into data/gurnee_data\n# step 3: uv run python scripts/gurnee_data.py\n# step 4 (caches to /output_haystack):\n# uv run scripts/run_haystack\n```\n\n### Run experiments\n\n```bash\nuv run -m attention_probe --run_set v1\nuv run -m attention_probe --run_set v1-last --last_only\nuv run -m attention_probe --run_set v1-mean --take_mean\nuv run -m attention_probe --run_set v1-tanh --use_tanh\n```\n\n### Analyze\n\nOpen `scripts/analyze_cache.py` as Jupytext and run. Edit `configs[]` to change which experiments to compare. Update `names{}` for custom experiment names.\n\n## API\n\n```python\n# Example usage\n# Overfit an attention probe on a small dataset\nfrom attention_probe import AttentionProbe, AttentionProbeTrainConfig, TrainingData, train_probe, evaluate_probe, compute_metrics\nimport torch\n\ndataset_size = 1024\nseq_len = 16\nhidden_dim = 256\nnum_classes = 2\nn_heads = 2\n\ndata = TrainingData(\n    x=torch.randn(dataset_size, seq_len, hidden_dim),\n    y=torch.randint(0, num_classes, (dataset_size,)),\n    mask=torch.ones(dataset_size, seq_len),\n    position=torch.arange(seq_len),\n    n_classes=num_classes,\n    class_mapping={0: \"class 0\", 1: \"class 1\"},\n)\n\nconfig = AttentionProbeTrainConfig(\n    n_heads=n_heads,\n    hidden_dim=hidden_dim,\n)\nprobe, _loss = train_probe(data, config, device=\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprobs = evaluate_probe(probe, data, config)\nmetrics = compute_metrics(probs, data)\nprint(metrics)\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feleutherai%2Fattention-probes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feleutherai%2Fattention-probes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feleutherai%2Fattention-probes/lists"}