{"id":31660366,"url":"https://github.com/internlm/sim-cot","last_synced_at":"2025-10-07T17:06:23.737Z","repository":{"id":316534531,"uuid":"1062448655","full_name":"InternLM/SIM-CoT","owner":"InternLM","description":"An official implementation of \"SIM-CoT: Supervised Implicit Chain-of-Thought\"","archived":false,"fork":false,"pushed_at":"2025-09-28T02:34:54.000Z","size":3336,"stargazers_count":74,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-10-04T02:21:44.843Z","etag":null,"topics":["chain-of-thought","continuous-thought","implicit-chain-of-thought","large-language-models","llm"],"latest_commit_sha":null,"homepage":"https://arxiv.org/pdf/2509.20317","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/InternLM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-23T09:14:44.000Z","updated_at":"2025-10-04T01:59:13.000Z","dependencies_parsed_at":"2025-09-26T01:16:00.115Z","dependency_job_id":null,"html_url":"https://github.com/InternLM/SIM-CoT","commit_stats":null,"previous_names":["internlm/sim-cot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/InternLM/SIM-CoT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InternLM%2FSIM-CoT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InternLM%2FSIM-CoT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InternLM%2FSIM-CoT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InternLM%2FSIM-CoT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/InternLM","download_url":"https://codeload.github.com/InternLM/SIM-CoT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InternLM%2FSIM-CoT/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278811851,"owners_count":26050183,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chain-of-thought","continuous-thought","implicit-chain-of-thought","large-language-models","llm"],"created_at":"2025-10-07T17:06:23.011Z","updated_at":"2025-10-07T17:06:23.726Z","avatar_url":"https://github.com/InternLM.png","language":"Python","readme":"\u003c!-- \u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"./docs/static/images/logo_resize.png\"  width=\"80%\"\u003e\n\u003c/p\u003e --\u003e\n\n\u003cdiv align=\"center\"\u003e\n    \u003ch1 align=\"center\"\u003e SIM-CoT: Supervised Implicit Chain-of-Thought\n    \u003c/h1\u003e\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/coconut_teaser.png\"\u003e\n\u003c/p\u003e\n\n\n- **Authors**: [Xilin Wei](https://github.com/Wiselnn570), [Xiaoran Liu](https://scholar.google.de/citations?user=Qe6F4J4AAAAJ\u0026hl=en), [Yuhang Zang](https://yuhangzang.github.io), [Xiaoyi Dong](https://lightdxy.github.io), [Yuhang Cao](https://scholar.google.com/citations?user=sJkqsqkAAAAJ\u0026hl=en), [Jiaqi Wang](https://myownskyw7.github.io/), [Xipeng Qiu](https://xpqiu.github.io/en.html), [Dahua Lin](http://dahua.site/)\n- **Institutes**: Fudan University; Shanghai AI Laboratory; The Chinese University of Hong Kong; Shanghai Innovation Institute; \n- **Resources**: [📖[Paper](https://arxiv.org/pdf/2509.20317)] [[🏠Project Page]()] [[🤗Huggingface](https://huggingface.co/collections/Wiselnn/sim-cot-supervised-implicit-chain-of-thought-68d895b00576f6166c19ab4f)]\n## 💡 Highlights\n\n- 🔥 **Latent Instability in Implicit CoT:** We systematically analyze the limitations of implicit Chain-of-Thought methods and reveal a **latent instability issue**—as the number of implicit tokens increases, models tend to collapse into homogeneous latent states that lose operator semantics.  \n\n- 🔥 **Step-Level Supervision with SIM-CoT:** We propose **S**upervised **IM**plicit-CoT (**SIM-CoT**), a plug-and-play module that introduces **step-level supervision** via an auxiliary decoder. This stabilizes optimization, prevents collapse, and ensures that latent tokens capture meaningful reasoning steps.\n\n- 🔥 **Strong and Consistent Performance:** SIM-CoT consistently outperforms both explicit and implicit baselines. On GPT-2, it exceeds supervised CoT by **+2.1%**, Coconut by **+8.2%**, and CODI by **+4.3%**. Across larger LLaMA models (1B/3B/8B), it delivers **+1.5% to +9.0%** gains, and remains stable even with **8–16 implicit tokens**, where prior methods collapse.  \n\n- 🔥 **Efficiency and Interpretability:** SIM-CoT adds **no extra inference cost** since the auxiliary decoder is discarded after training. It also provides **interpretability**, allowing each latent token to be decoded into a human-readable reasoning step.  \n\n## 📜 News\n\n**[2025/9/24]** [Code]() and [Paper](https://arxiv.org/pdf/2509.20317) are released!\n\n## 👨‍💻 Todo\n\n- [x] Code Release\n- [x] Checkpoint Release\n- [x] Usage Instructions Release\n\n\n## 🛠️ Usage\n\n### 1. Clone the repository\n```bash\ngit clone https://github.com/InternLM/SIM-CoT.git\ncd SIM-CoT\n```\n\n### 2. Install dependencies\n```bash\npip install -r requirements.txt\n```\n\n---\n\n### 3. Training with Coconut + SIM-CoT\n\n#### Step 1: Train the Coconut baseline\n```bash\ncd Coconut\ntorchrun --nnodes 1 --nproc_per_node 8 run.py args/gsm_coconut.yaml\n```\n\n#### Step 2: Continue training with SIM-CoT\nSelect a checkpoint that has been expanded to predefined implicit tokens, then continue training with SIM-CoT:\n```bash\ntorchrun --nnodes 1 --nproc_per_node 8 run.py args/gsm_simcot.yaml\n```\n\n---\n\n### 4. Evaluation with Coconut + SIM-CoT\n```bash\ntorchrun --nnodes 1 --nproc_per_node 8 run.py args/gsm_simcot_eval.yaml\n```\n\n---\n\n### 5. Training with CODI + SIM-CoT\n```bash\ncd CODI\nbash scripts/train_llama3b_gsm8k-aug-decoder-2.sh\n```\n\n---\n\n### 6. Evaluation with CODI + SIM-CoT\n```bash\nbash CODI/scripts/test_llama3b-copy.sh\n```\n\n\n\n## ✒️ Citation\n\nIf you find our work helpful for your research, please consider giving a star ⭐ and citation 📝\n\n```bibtex\n@article{wei2025simcot,\n  title={{SIM-COT}: Supervised Implicit Chain-of-Thought},\n  author={Wei, Xilin and Liu, Xiaoran and Zang, Yuhang and Dong, Xiaoyi and Cao, Yuhang and Wang, Jiaqi and Qiu, Xipeng and Lin, Dahua},\n  journal={arXiv preprint arXiv:2509.20317},\n  year={2025}\n}\n```\n\n## ❤️ Acknowledgments\n\n- [Coconut](https://github.com/facebookresearch/coconut): The codebase we built upon. Thanks for their wonderful work.\n- [CODI](https://github.com/zhenyi4/codi): Our work is based on this codebase; we are grateful for their valuable contribution.\n- [LLaMA series](https://huggingface.co/meta-llama/collections): The amazing open-sourced large language model!\n- [GPT2](https://huggingface.co/openai-community/gpt2): An impressive open-source large language model!\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finternlm%2Fsim-cot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finternlm%2Fsim-cot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finternlm%2Fsim-cot/lists"}