{"id":31911616,"url":"https://github.com/cyberagentailab/cheat-sheet-icl","last_synced_at":"2025-10-13T17:21:06.507Z","repository":{"id":316678567,"uuid":"1063866951","full_name":"CyberAgentAILab/cheat-sheet-icl","owner":"CyberAgentAILab","description":"[EMNLP 2025 Findings] Code for \"Distilling Many-Shot In-Context Learning into a Cheat Sheet\"","archived":false,"fork":false,"pushed_at":"2025-09-26T02:07:27.000Z","size":1975,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-26T04:10:52.696Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CyberAgentAILab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-25T08:16:55.000Z","updated_at":"2025-09-26T03:40:48.000Z","dependencies_parsed_at":"2025-09-26T04:10:54.753Z","dependency_job_id":"4bcfbadc-b193-4618-a85b-c42d6fd8b4ac","html_url":"https://github.com/CyberAgentAILab/cheat-sheet-icl","commit_stats":null,"previous_names":["cyberagentailab/cheat-sheet-icl"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/CyberAgentAILab/cheat-sheet-icl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Fcheat-sheet-icl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Fcheat-sheet-icl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Fcheat-sheet-icl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Fcheat-sheet-icl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CyberAgentAILab","download_url":"https://codeload.github.com/CyberAgentAILab/cheat-sheet-icl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Fcheat-sheet-icl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279016299,"owners_count":26085828,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-13T17:21:03.063Z","updated_at":"2025-10-13T17:21:06.498Z","avatar_url":"https://github.com/CyberAgentAILab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Distilling Many-Shot In-Context Learning into a Cheat Sheet\n\nThis repository implements the main experiments of our paper, [Distilling Many-Shot In-Context Learning into a Cheat Sheet](http://arxiv.org/abs/2509.20820) (EMNLP 2025 Findings).\nWe introduce **cheat-sheet ICL**, which distills the information from many-shot ICL into a concise textual summary (cheat sheet) used as the context at inference time.\nThis approach achieves comparable or better performance than many-shot ICL with far fewer tokens in challenging reasoning tasks, and matches retrieval-based ICL without requiring test-time retrieval.\nThe code is intended solely for reproducing the experiments.\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"image/overview_csicl.png\" height=\"220\" alt=\"Overview: cheat-sheet ICL\"\u003e\n  \u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003cimg src=\"image/bbh_gpt-4.1.png\" height=\"220\" alt=\"Results\"\u003e\n\u003c/p\u003e\n\n\n## Environment\n\nWe tested the code in the following environment.\n\n* OS: Ubuntu 22.04.5 LTS\n* Python: 3.12\n\n\n## Setup\n\n### Python and Packages\n\nWe used [uv](https://github.com/astral-sh/uv) to manage Python and its packages.\n```bash\ngit clone https://github.com/CyberAgentAILab/cheat-sheet-icl.git\ncd cheat-sheet-icl\n\n# (Optional) If this version of Python is not installed\nuv python pin 3.12\nuv python install\n\n# Create a virtual environment and install dependencies\nuv venv --python 3.12 --python-preference managed\nuv pip install --no-cache --force-reinstall -r requirements.txt\n```\n\n### API Keys\n\nThe code assumes that the API keys are stored in environment variables.\nFill in the `.envrc` file and load it.  \n`GEMINI_API_KEY` is optional; it is only required when running the cheat-sheet transferability experiments.\n\n```bash\ncp .envrc_example .envrc\n\n`Fill in the .envrc file with your API keys`\n\n# If you are using direnv\ndirenv allow\n# Otherwise\nsource .envrc\n```\n\n### Data\n\nRun the following command to download the rationale-augmented BBH data.\nThe cheat sheets used in our experiments can also be found in [`data/cheat_prompt`](data/cheat_prompt).\nThe files in [`data/cheat_prompt`](data/cheat_prompt) and [`data/metaprompt`](data/metaprompt) are adapted from [BIG-Bench-Hard](https://github.com/suzgunmirac/BIG-Bench-Hard).\n\n```bash\ncd data\ncurl -L -O https://storage.googleapis.com/ailab-public/cheat-sheet-icl/data/aug_data_bbh.zip\nunzip -q aug_data_bbh.zip\nrm aug_data_bbh.zip\n```\n\n**(Optional)** If you want to run rationale augmentation and cheat-sheet creation from scratch, download [BBH data](https://github.com/suzgunmirac/BIG-Bench-Hard/archive/refs/heads/main.zip) and place them under `data`.\nThe other datasets can be downloaded automatically with the Hugging Face Datasets library, but you will need to request access to [GPQA](https://huggingface.co/datasets/Idavidrein/gpqa) using your Hugging Face account and then run `uv run huggingface-cli login`.\nThen run [`script/preproc.sh`](script/preproc.sh) for BBH and [`script/preproc_acad.sh`](script/preproc_acad.sh) for the academic benchmarks.\n\n```bash\ncd data\ncurl -L -O https://github.com/suzgunmirac/BIG-Bench-Hard/archive/refs/heads/main.zip\nunzip -q main.zip\nrm main.zip\n```\n\n\n## Run\n\nBelow are sample commands for running the experiments. See [`script`](script) for more examples.  \n\n\u003e [!NOTE]\n\u003e * We used Azure OpenAI API for GPT-4.1.\n\u003e * We used the specific version `gpt-4.1-2025-04-14`, and the code assumes that the deployment name is the same.\n\u003e   - If you want to use a different deployment name, modify `model_libs` in [`src/args_utils.py`](src/args_utils.py).\n\u003e * The Azure OpenAI content filters were disabled upon our request.\n\n```bash\nMETA_MODEL=\"gpt-4.1-2025-04-14\"\nMODEL=\"gpt-4.1-2025-04-14\"\nTASK=\"bbh_salient_translation_error_detection\"\nREASON=\"gen\"\nSEED=1000\n\n# Few-shot ICL (8-shot)\nuv run src/run_main_api.py \\\n    --task ${TASK} \\\n    --meta_model ${META_MODEL} \\\n    --model ${MODEL} \\\n    --reason_type ${REASON} \\\n    --prompt_type shot \\\n    --shot 8 \\\n    --seed ${SEED}\n\n# Many-shot ICL (n-shot)\nuv run src/run_main_api.py \\\n    --task ${TASK} \\\n    --meta_model ${META_MODEL} \\\n    --model ${MODEL} \\\n    --reason_type ${REASON} \\\n    --prompt_type shot \\\n    --shot 0 \\\n    --seed ${SEED}\n\n# Cheat-sheet ICL\nuv run src/run_main_api.py \\\n    --task ${TASK} \\\n    --meta_model ${META_MODEL} \\\n    --model ${MODEL} \\\n    --reason_type ${REASON} \\\n    --prompt_type cheat \\\n    --shot 0 \\\n    --seed ${SEED}\n```\n\n\n## Citation\n\n```bibtex\n@inproceedings{honda-etal-2025-distilling,\n    title = \"Distilling Many-Shot In-Context Learning into a Cheat Sheet\",\n    author = \"Honda, Ukyo and Murakami, Soichiro and Zhang, Peinan\",\n    booktitle = \"Findings of the Association for Computational Linguistics: EMNLP 2025\",\n    year = \"2025\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyberagentailab%2Fcheat-sheet-icl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcyberagentailab%2Fcheat-sheet-icl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyberagentailab%2Fcheat-sheet-icl/lists"}