{"id":19551864,"url":"https://github.com/ysymyth/ec-nl","last_synced_at":"2025-07-23T14:07:29.101Z","repository":{"id":72165419,"uuid":"456327552","full_name":"ysymyth/ec-nl","owner":"ysymyth","description":"[ICLR 2022] Linking Emergent and Natural Languages via Corpus Transfer","archived":false,"fork":false,"pushed_at":"2024-06-02T01:22:23.000Z","size":22,"stargazers_count":31,"open_issues_count":0,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-03T21:18:32.536Z","etag":null,"topics":["emergent-communication","iclr","iclr2022","language-model","machine-learning","natural-language-processing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ysymyth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-02-07T02:04:37.000Z","updated_at":"2025-05-21T15:56:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"50857400-31a6-48e5-a282-51e2840c9267","html_url":"https://github.com/ysymyth/ec-nl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ysymyth/ec-nl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysymyth%2Fec-nl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysymyth%2Fec-nl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysymyth%2Fec-nl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysymyth%2Fec-nl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ysymyth","download_url":"https://codeload.github.com/ysymyth/ec-nl/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysymyth%2Fec-nl/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266691580,"owners_count":23969182,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["emergent-communication","iclr","iclr2022","language-model","machine-learning","natural-language-processing"],"created_at":"2024-11-11T04:15:41.462Z","updated_at":"2025-07-23T14:07:29.075Z","avatar_url":"https://github.com/ysymyth.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# EC-NL\n\nCode and data for paper [Linking Emergent and Natural Languages via Cospus Transfer]() at ICLR 2022 (spotlight).\n```bibtex\n@inproceedings{yao2022linking,\n  title = {Linking Emergent and Natural Languages via Corpus Transfer},\n  author = {Yao, Shunyu and Yu, Mo and Zhang, Yang and Narasimhan, Karthik and Tenenbaum, Joshua and Gan, Chuang},\n  booktitle = {International Conference on Learning Representations (ICLR)},\n  year = {2022},\n  html = {https://openreview.net/pdf?id=49A1Y6tRhaq},\n}\n```\n\n## Dependencies\n\n- PyTorch 1.8\n- SciPy 1.4\n- Transformers 4.4.2\n- (Optional) Wandb\n\n## Data\n\n[Google Drive](https://drive.google.com/drive/folders/1dBdGaZzvQ4yn-RMpDMxLlFNLzSSbkOWF?usp=sharing) includes\n\n- ```image_features```: Image features of coco-2014 (``coco.pt``) and Conceptual Captions (``cc.pt``) datasets from a pre-trained ResNet, to be used in EC pre-training.\n\n- ```lm_corpora```: Corpora used for language modeling transfer experiments. \n\n| Name   | Usage | Comment      |\n|--------------|-----------|---------|\n| cc.pt    | pre-train         | Emergent language       |\n| paren-zipf.pt    | pre-train         | Regular language of nesting parentheses  |\n| wiki-es.pt    | pre-train         | Spanish (IE-Romance) Wikipedia       |\n| wiki-da.pt    | fine-tune         | Danish (IE-Germanic) Wikipedia       |\n| wiki-eu.pt    | fine-tune         | Basque (Basque) Wikipedia       |\n| wiki-ja.pt    | fine-tune         | Japanese (Japanese) Wikipedia       |\n| wiki-ro.pt    | fine-tune         | Romanian (IE-Romance) Wikipedia       |\n| wiki-fi.pt    | fine-tune         | Finnish (Uralic) Wikipedia       |\n| wiki-id.pt    | fine-tune         | Indonesian (Austronesian) Wikipedia       |\n| wiki-kk.pt    | fine-tune         | Kazakh (Turkic) Wikipedia       |\n| wiki-he.pt    | fine-tune         | Hebrew (Afro-Asiatic) Wikipedia       |\n| wiki-ur.pt    | fine-tune         | Urdu (IE-Indic) Wikipedia       |\n| wiki-fa.pt    | fine-tune         | Persian (IE-Iranian) Wikipedia       |\n\n\n\n\n\n## Experiments\n\n### Emergent Communication (EC) Game\nThis part aims to generate emergent langauge corpus for downstream tasks.\nDownload ```image_features``` from Google Drive to ```./ec-pretrain/data```.\nTo run the emergent communication training, \n```bash\ncd ec-game\npython train.py\n```\n\n\nSome major options:\n- ```--dataset```: use Conceptual Captions (```cc```) or MS-COCO (```coco_2014```) dataset.\n- ```--vocab_size```: Vocabulary size (default ```4035```).\n- ```--seq_len```: Sequence length limit (default ```15```).\n\nSuch a game training automatically stores EC agents (e.g. ```./ckpt/cc_vocab_4035_seq_15_reset_-1_nlayers_1/run77926/model_90.6_1000_4035.pt```) and emergent language corpora (e.g. ```./ckpt/cc_vocab_4035_seq_15_reset_-1_nlayers_1/run77926/model_90.6_1000_4035.pt-cc.pt```, which can be used in place of ```lm_corpora/cc.pt``` from Google Drive)  from different training steps. In the example, ```90.6_1000_4035``` represents game accuracy, game training steps, and game vocabulary size respectively.\n\n\n\n### Language Modeling Transfer\nThis part aims to reproduce Figure 2 of the paper. \nDownload ```lm_corpora``` from Google Drive to ```./ec-pretrain/data```.\n\nTo run the pre-training, \n```bash\nexport size=2 # 2,5,10,15,30\nexport pt_name=\"wiki-es\" # \"paren-zipf\", \"cc\"\n. pretrain.sh\n```\n\nTo run the fine-tuning,\n```bash\nexport size=2 # 2,5,10,15,30\nexport pt_name=\"wiki-es\" # \"paren-zipf\", \"cc\"\nexport ft_name=\"wiki-ro\"\nexport ckpt=3000\n. finetune.sh\n```\n\nMeaning of variables above:\n- ```size```: Token size (million) of pre-training corpus (```[2, 5, 10, 15, 30]```).\n- ```pt_name```: Name of pre-training corpus (```[\"wiki-es\", \"paren-zipf\", \"cc\"]```).\n- ```ft_name```: Name of fine-tuning corpus (```[\"wiki-ro\", \"wiki-da.pt]```).\n- ```ckpt```: Which pre-training checkpoint to use for fine-tuning (default ```3000```).\n\n   \n\n## Acknowledgements\n\nThe EC part of the code is based on [ECNMT](https://github.com/cambridgeltl/ECNMT), which was partly based on [Translagent](https://github.com/facebookresearch/translagent). \n\nThe LM part of the code is based on [Huggingface run_clm.py](https://github.com/huggingface/transformers/blob/v4.4.2/examples/language-modeling/run_clm.py).\n\nThe datasets for our EC experiments include [MS COCO](http://cocodataset.org/#home) and [Conceptual Captions](https://ai.google.com/research/ConceptualCaptions).\n\nThe datasets for our LM experiments derive from [tilt-transfer](https://github.com/toizzy/tilt-transfer).\n\nPlease cite these resources accordingly. For any question, contact [Shunyu](mailto:shunyuyao.cs@gmail.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fysymyth%2Fec-nl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fysymyth%2Fec-nl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fysymyth%2Fec-nl/lists"}