{"id":31944230,"url":"https://github.com/zjunlp/lightthinker","last_synced_at":"2025-10-14T10:22:25.684Z","repository":{"id":317936677,"uuid":"901162780","full_name":"zjunlp/LightThinker","owner":"zjunlp","description":"[EMNLP 2025] LightThinker: Thinking Step-by-Step Compression","archived":false,"fork":false,"pushed_at":"2025-04-12T19:05:55.000Z","size":5812,"stargazers_count":104,"open_issues_count":1,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-10-04T02:21:39.988Z","etag":null,"topics":["artificial-intelligence","compression","efficiency","generation","large-language-model","lightthinker","natural-language-processing","reasoning","slow-thinking","step-by-step"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zjunlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-10T06:42:44.000Z","updated_at":"2025-09-28T09:11:42.000Z","dependencies_parsed_at":"2025-10-04T02:21:43.270Z","dependency_job_id":"32e11ad7-9aa4-4c79-99c8-b1bafbbf3134","html_url":"https://github.com/zjunlp/LightThinker","commit_stats":null,"previous_names":["zjunlp/lightthinker"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/zjunlp/LightThinker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FLightThinker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FLightThinker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FLightThinker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FLightThinker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zjunlp","download_url":"https://codeload.github.com/zjunlp/LightThinker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FLightThinker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279018780,"owners_count":26086451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","compression","efficiency","generation","large-language-model","lightthinker","natural-language-processing","reasoning","slow-thinking","step-by-step"],"created_at":"2025-10-14T10:22:24.714Z","updated_at":"2025-10-14T10:22:25.676Z","avatar_url":"https://github.com/zjunlp.png","language":"Python","readme":"\u003cdiv align=\"center\" style=\"width: 100%;\"\u003e\n  \u003cimg src=\"assets/gif.gif\" alt=\"Case GIF\" style=\"width: 100%; max-width: 100%; height: auto;\"\u003e\n\u003c/div\u003e\n\n\n\n\u003cdiv align=\"center\"\u003e\n\u003ch1 align=\"center\"\u003e 👉 LightThinker 👈 \u003c/h1\u003e\n\u003cb\u003eLightThinker: Thinking Step-by-Step Compression\u003c/b\u003e\n\n[![Awesome](https://awesome.re/badge.svg)](https://github.com/zjunlp/LightThinker) \n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n![](https://img.shields.io/github/last-commit/zjunlp/LightThinker?color=green) \n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://arxiv.org/abs/2502.15589\"\u003e📄arXiv\u003c/a\u003e •\n  \u003ca href=\"https://x.com/zxlzr/status/1894729164609208338\"\u003e𝕏 Blog\u003c/a\u003e •\n  \u003ca href=\"https://huggingface.co/collections/zjunlp/lightthinker-67f9faaaa518f2e00b17386b\"\u003e🤗 Huggingface\u003c/a\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n## Table of Contents\n\n- 👀[Overview](#overview)\n- 🔧[Installation](#installation)\n- 🏃[Quick Start](#quick-start)\n- 🎁[Acknowledgement](#acknowledgement)\n- 🚩[Citation](#citation)\n\n## 👀Overview\n\nLLMs have shown remarkable performance in complex reasoning tasks, but their efficiency is hindered by the substantial memory and computational costs associated with generating lengthy tokens. In this paper, we propose LightThinker, a novel method that enables LLMs to dynamically compress intermediate thoughts during reasoning. Inspired by human cognitive processes, LightThinker compresses verbose thought steps into compact representations and discards the original reasoning chains, thereby significantly reducing the number of tokens stored in the context window. This is achieved by training the model on when and how to perform compression through data construction, mapping hidden states to condensed gist tokens, and creating specialized attention masks.\n\n\n## 🔧Installation\n\n```bash\ngit clone https://github.com/zjunlp/LightThinker\ncd LightThinker\nconda create -n lightthinker python=3.9 -y\nconda activate lightthinker\npip install -r requirements.txt\ncd data \u0026\u0026 unzip data.zip \u0026\u0026 cd ..\n```\n\n\n## 🏃Quick Start\n\n\u003e First, we train the model to learn how to compress (step 1). Then, we perform inference on the test set to obtain output results (step 2). Finally, we evaluate the output results (step 3).\n\n### Step 1. Training\n\nTo execute the training, run the following command:\n\n```bash\nbash train.sh\n```\n\nCurrently, the script's parameters are set to run on a machine with 4 A800 GPUs. If you encounter OOM (Out Of Memory) issues, please reduce the `micro_batch_size` and `max_length`. For other parameters in the script, please refer to the [documentation](./ARGS.md).\n\n### Step 2. Inference\n\n\u003cdetails\u003e \n\u003csummary\u003e\u003cb\u003eInference with a downloaded model\u003c/b\u003e\u003c/summary\u003e\n\nIf you are downloading a trained model from Huggingface, please set the `model_path` parameter in `inference.sh` to the absolute path of the model. The values of other parameters `ckpt` and `model_tag` will be ignored.\n\u003c/details\u003e\n\nTo execute the inference, run the following command:\n\n```bash\nbash inference.sh\n```\n\nHere, you need to modify the script file's `model_tag`, `model_short_tag`, `ckpt`, `output_tag`, and `split_size`. For details regarding the script's parameters, please refer to the [documentation](./ARGS.md).\n\n\n### Step 3. Evaluation\n\n\u003e [!NOTE]\n\u003e If this is your **first time** conducting an evaluation, please execute the following code first:\n\u003e ```bash\n\u003e python evaluation/init.py\n\u003e ```\n\nTo execute the evaluation, run the following command:\n\n```bash\nmethod=\"\"\ntokenizer_path=\"\"\ncomp_config=\"\"\nmodel_type=\"\"\ndataset=\"\"\nbos_token=\"\"\neos_token=\"\"\ncache_size=1024\nfile1=\"\"\nfile2=\"\"\nfile3=\"\"\nfile4=\"\"\npython evaluation/eval_file.py \\\n  --method $method \\\n  --tokenizer_path $tokenizer_path \\\n  --comp_config $comp_config \\\n  --model_type $model_type \\\n  --dataset $dataset \\\n  --files $file1 $file2 $file3 $file4 \\\n  --cache_size $cache_size \\\n  --bos_token $bos_token \\\n  --eos_token $eos_token \\\n  --interaction \n```\n\nPlease note that if you set `split_size\u003e1` in the second step, the number of file i here should match the value of `split_size`. It should be noted that manual evaluation was conducted during the assessment. Use the `--interaction` flag to enable manual evaluation. The `cache_size` parameter is used for `H2O` and `SepLLM`, but not for `LightThinker` or `AnLLM`.\n\n\u003cdetails\u003e \n\u003csummary\u003e\u003cb\u003eEvaluation Script Example\u003c/b\u003e\u003c/summary\u003e\n\n```bash\n# The optional values for the method argument are 'anchor-token', 'normal', 'kvcache', and 'anchor-thought'.\nmethod=\"anchor-thought\"\ntokenizer_path=\"Qwen/Qwen2.5-7B-Instruct\"\ncomp_config=\"configs/LightThinker/qwen/v1.json\"\nmodel_type=\"qwen\"\ndataset=\"gpqa\"\nbos_token=\"\u003c|im_start|\u003e\"\neos_token=\"\u003c|im_end|\u003e\"\ncache_size=1024\nfolder=\"\"\nckpt=1045\nfile1=\"inference_results/${folder}/${dataset}/${ckpt}/1-4qwen_7b.jsonl\"\nfile2=\"inference_results/${folder}/${dataset}/${ckpt}/2-4qwen_7b.jsonl\"\nfile3=\"inference_results/${folder}/${dataset}/${ckpt}/3-4qwen_7b.jsonl\"\nfile4=\"inference_results/${folder}/${dataset}/${ckpt}/4-4qwen_7b.jsonl\"\npython evaluation/eval_file.py \\\n  --method $method \\\n  --tokenizer_path $tokenizer_path \\\n  --comp_config $comp_config \\\n  --model_type $model_type \\\n  --dataset $dataset \\\n  --files $file1 $file2 $file3 $file4 \\\n  --cache_size $cache_size \\\n  --bos_token $bos_token \\\n  --eos_token $eos_token \\\n  --interaction \n```\n\u003c/details\u003e\n\n\u003cdetails\u003e \n\u003csummary\u003e\u003cb\u003eManual Evaluation Instructions\u003c/b\u003e\u003c/summary\u003e\n\nWhen string matching fails, the output will be displayed in the format \"Model Answer\" \u003c=\u003e \"Standard Answer\". At this point, you can input \"y\" or \"n\" to evaluate this case. If you believe the model's answer extraction is incorrect, you can input \"e\" to print the model's complete output, and then input \"y\" or \"n\" to evaluate this case.\n\u003c/details\u003e\n\n\n\n## 🎁Acknowledgement\n\nOur training dataset is derived from [Bespoke-Stratos-17k](https://huggingface.co/datasets/bespokelabs/Bespoke-Stratos-17k). We utilized the baseline code for H2O from the [Meta-llama](https://github.com/meta-llama/llama-cookbook)'s repository, the baseline code for SepLLM from the [HKUDS](https://github.com/HKUDS/SepLLM)'s repository. We extend our gratitude to the contributors for their outstanding work!\n\n\n## 🚩Citation\n\nIf this work is helpful, please kindly cite as:\n\n```bibtex\n@article{DBLP:journals/corr/abs-2502-15589,\n  author       = {Jintian Zhang and\n                  Yuqi Zhu and\n                  Mengshu Sun and\n                  Yujie Luo and\n                  Shuofei Qiao and\n                  Lun Du and\n                  Da Zheng and\n                  Huajun Chen and\n                  Ningyu Zhang},\n  title        = {LightThinker: Thinking Step-by-Step Compression},\n  journal      = {CoRR},\n  volume       = {abs/2502.15589},\n  year         = {2025},\n  url          = {https://doi.org/10.48550/arXiv.2502.15589},\n  doi          = {10.48550/ARXIV.2502.15589},\n  eprinttype    = {arXiv},\n  eprint       = {2502.15589},\n  timestamp    = {Thu, 20 Mar 2025 13:28:42 +0100},\n  biburl       = {https://dblp.org/rec/journals/corr/abs-2502-15589.bib},\n  bibsource    = {dblp computer science bibliography, https://dblp.org}\n}\n```\n\n\n\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Flightthinker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzjunlp%2Flightthinker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Flightthinker/lists"}