{"id":27400849,"url":"https://github.com/butanium/llm-lang-agnostic","last_synced_at":"2025-04-14T03:43:30.463Z","repository":{"id":252400909,"uuid":"840332209","full_name":"Butanium/llm-lang-agnostic","owner":"Butanium","description":"minimal code to reproduce results from Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers","archived":false,"fork":false,"pushed_at":"2024-12-08T11:12:24.000Z","size":14986,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-08T12:20:20.697Z","etag":null,"topics":["llm","mechanistic-interpretability","research"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Butanium.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-09T13:18:46.000Z","updated_at":"2024-11-19T06:12:05.000Z","dependencies_parsed_at":"2024-12-08T12:20:01.730Z","dependency_job_id":"bf88a581-a329-4094-af88-2d677e656b56","html_url":"https://github.com/Butanium/llm-lang-agnostic","commit_stats":null,"previous_names":["butanium/llm-lang-agnostic"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Butanium%2Fllm-lang-agnostic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Butanium%2Fllm-lang-agnostic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Butanium%2Fllm-lang-agnostic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Butanium%2Fllm-lang-agnostic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Butanium","download_url":"https://codeload.github.com/Butanium/llm-lang-agnostic/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248819114,"owners_count":21166470,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","mechanistic-interpretability","research"],"created_at":"2025-04-14T03:43:29.941Z","updated_at":"2025-04-14T03:43:30.457Z","avatar_url":"https://github.com/Butanium.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"This repo contains minimal code to reproduce results from [Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers](https://arxiv.org/abs/2411.08745)\n previously published under the title [How Do Llamas Process Multilingual Text? A Latent Exploration through Activation Patching](https://openreview.net/forum?id=0ku2hIm4BS), spotlight at the ICML 2024 Mechanistic interpretability workshop. \n# Setup\nCreate a python environment and `pip install -r requirements.txt`. We think the code probably work with `python\u003e=3.7` but only tested it with `python=3.11.x`.\n\nIf this happens to not work you can use the versions specified in `pip.freeze` which contains all the package we have installed with their version in our local `conda` environment.\n\n# Reproducing results\n```bash\t\nchmod +x compute_results.sh\n./compute_results.sh\n```\n\n# Datasets used in the paper\nThe code to rebuild the datasets is located in the `build_datasets` folder but are provided in the repo because you'd need to ask for extra Babelnet API credits / ask for the local index (and make them run 👻) which you don't want to.\n\n## `word_translation.csv`\nThose files are computed using the `main_translation_dataset` function of `build_dataset/build_bn_dataset.py`.\n\nFor a lang $\\ell$, we first generated a single word translation of `word_original`. \nThis translated word is the first of each list in the $\\ell$ column of the csv.\n\nThen, using babelnet we find all meanings or \"senses\" related to this word, and then collect all the words or \"lemma\" that expresse those senses, for each language (including $\\ell$)\n\n**Disclaimer:** For some languages we didn't compute the `word_translation.csv` file so you can only use them as output language. If you need one of those, shoot us an email and we'll add it.\n\n# Extra datasets that did not make it yet in the paper\n## `synset_dataset.csv`\nThose files are computed using the `main_synset_dataset` function of `build_dataset/build_bn_dataset.py`.\n\nWe took 200 words from https://en.wiktionary.org/wiki/Appendix:Basic_English_word_list#Things_-_200_picturable_words and found their canonical concept or \"synset\" in babelnet.\n\n## `cloze_dataset.csv`\nThose files are computed using the `main_cloze_dataset` function of `build_dataset/build_bn_dataset.py`.\n\nWe took the synsets from `synset_dataset.csv` and for each language we collected the different definitions available in babelnet. \n\nThe dataset contains several columns:\n- `original_definitions` (`str`): the original definitions from babelnet\n- `clozes` (`cloze: str, acceptable_sense: tuple[str]`): definitions where we replaced one of the sense by a placeholder `____`. The `acceptable_sense` is the list of senses that don't appear in the cloze.\n- `clozes_with_start_of_word` (`cloze: str, acceptable_sense: tuple[str]`): same as `clozes` but we search both for the sense alone and for words that start with the sense.\n- `definitions_wo_ref` (`str`): definitions without the reference to any sense. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbutanium%2Fllm-lang-agnostic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbutanium%2Fllm-lang-agnostic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbutanium%2Fllm-lang-agnostic/lists"}