{"id":28653893,"url":"https://github.com/tiger-ai-lab/longiclbench","last_synced_at":"2025-06-13T07:08:02.598Z","repository":{"id":231246020,"uuid":"780966271","full_name":"TIGER-AI-Lab/LongICLBench","owner":"TIGER-AI-Lab","description":"Code and Data for \"Long-context LLMs Struggle with Long In-context Learning\" [TMLR2025]","archived":false,"fork":false,"pushed_at":"2025-02-20T18:48:20.000Z","size":4706,"stargazers_count":100,"open_issues_count":0,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-20T19:40:09.285Z","etag":null,"topics":["large-language-models"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2404.02060","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TIGER-AI-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-02T13:55:08.000Z","updated_at":"2025-02-20T18:49:09.000Z","dependencies_parsed_at":"2024-05-18T02:27:16.830Z","dependency_job_id":"706a660b-33ef-4386-8447-41c336427859","html_url":"https://github.com/TIGER-AI-Lab/LongICLBench","commit_stats":null,"previous_names":["tiger-ai-lab/longiclbench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TIGER-AI-Lab/LongICLBench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FLongICLBench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FLongICLBench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FLongICLBench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FLongICLBench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TIGER-AI-Lab","download_url":"https://codeload.github.com/TIGER-AI-Lab/LongICLBench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FLongICLBench/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259599331,"owners_count":22882357,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["large-language-models"],"created_at":"2025-06-13T07:08:01.903Z","updated_at":"2025-06-13T07:08:02.578Z","avatar_url":"https://github.com/TIGER-AI-Lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🐍 LongICLBench\nThis repo contains the official code and data for our TMLR25 paper [Long-context LLMs Struggle with Long In-context Learning](http://arxiv.org/abs/2404.02060)\n\n[**🤗 HuggingFace Dataset**](https://huggingface.co/datasets/TIGER-Lab/LongICLBench)\n\n[**🌟 Leaderboard**](https://huggingface.co/spaces/TIGER-Lab/LongICL-Leaderboard)\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"assets/curve-1.png\" width=\"90%\"\u003e\n\u003c/div\u003e\n\nWe created LongICLBench to conduct comprehensive evaluations of Large Language Models (LLMs) on extreme-label classification challenges with in-context learning. We compiled six datasets that encompass a broad spectrum of labels, ranging from 28 to 174 categories, and varied the lengths of input (from few-shot demonstrations) between 2K and 50K tokens to ensure thorough testing\nOur principal discoveries reveal: \n* 1️⃣ A notable decline in the performance of even the most advanced models as the complexity of the tasks increases. \n* 2️⃣ In the most challenging scenarios, especially those with 174 classes, models find it difficult to discern subtle differences. \n* 3️⃣ The performance of certain models is sensitive to the position of the instances in the demonstrations.\n\n## 🛠️ Set up\n```\ngit clone https://github.com/TIGER-AI-Lab/LongICLBench.git\ncd LongICLBench\npip install -r requirements.txt\n```\n\n## 🌈 Benchmarking\nWe collect six extreme-label classification datasets with details as below:\n\n|             **Dataset**             |              Task Type               |              #Classes               |                #Tokens/Shot                | #Total Tokens |\n|:-----------------------------------:|:------------------------------------:|:-----------------------------------:|:------------------------------------------:|:-------------:|\n|            **GoEmotion**            |        Emotion Classification        |                 28                  |                     28                     |   [1K, 4K]    |\n|            **BANKING77**            |        Intent Classification         |                 77                  |                     28                     |   [2K, 11K]   |\n|             **TecRED**              |         Relation Extraction          |                 41                  |                     80                     |   [4K, 18K]   |\n|            **Few-NERD**             |          Entity Recognition          |                 66                  |                     61                     |   [5K, 23K]   |\n|            **DialogRE**             |         Relation Extraction          |                 36                  |                    226                     |   [8K, 32K]   |\n|            **Discovery**            |   Discourse Marker Classification    |                 174                 |                     61                     |  [10K, 50K]   |\n\n## 🎖️ Model Evaluation\nWe evaluate the performance of 13 long-context large language models. Their information are presented as follows:\n\n|        Model         |  Size  | Initialization |          Strategy          |  Train  | Support |\n|:--------------------:|:------:|:--------------:|:--------------------------:|:-------:|:-------:|\n|    Gemma-7B-base     |   7B   |     Gemma      |         RoPE + LF          |   8K    |   8K    |\n|    LLaMA-2-7B-32K    |   7B   |    LLaMA-2     |   Position Interpolation   |   32K   |   32K   |\n|   ChatGLM3-6B-32K    |   6B   |    ChatGLM     |  Position Encoding Scheme  |   32K   |   32K   |\n|   Qwen-1.5-7B-base   |   7B   |      Qwen      |            RoPE            |   32K   |   32K   |\n| Mistral-7B-v0.2-base |   7B   |    Mistral     |             LF             |   32K   |   32K   |\n| LLaMA-2-7B-LongLora  |   7B   |    LLaMA-2     |  Shifted Short Attention   |  100K   |  100K   |\n|      Yi-6B-200K      |   6B   |       Yi       | Position Interpolation +LF |  200K   |  200K   |\n|  InternLM2-7B-base   |   7B   |    InternLM    |        Dynamic NTK         |   32K   |  200K   |\n|  Long-LLaMA-code-7B  |   7B   |    LLaMA-2     |    Focused Transformer     |   8K    |  256K   |\n|     RWKV-5-World     |   3B   |      RWKV      |    Attention-free Model    |   4K    | \u0026infin; |\n|      Mamba-2.8B      |  2.8B  |     Mamba      |     State Space Model      |   2K    | \u0026infin; |\n|      GPT4-turbo      |   -    |     GPT-4      |             -              |    -    |  128K   |\n|        GPT4o         |   -    |     GPT-4      |             -              |    -    |  128K   |\n|     Cluade3-Opus     |   -    |    Cluade3     |             -              |    -    |  200K   |\n|    Gemini1.5-Pro     |   -    |     Gemini     |             -              |    -    |   10M   |\n\nThe evaluation results for some of the datasets is presented as follows: (Please check more details in our paper)\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"assets/figure1_demo-1.png\" width=\"90%\"\u003e\n\u003c/div\u003e\n\n\n🕹️\tRun the script to re-obtain the demonstrated results: (You can just set [api_key] to '' for the locally deployed models)\n```\nsh script_goemotion.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]\nsh script_bank77.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]\nsh script_fewnerd.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]\nsh script_tacred.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]\nsh script_dialogue.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]\nsh script_discovery.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]\n```\n\nTo evaluate the results for Few-NERD and DialogRE with F1 score:\n```\npython fewnerd_evaluate.py --model [model_short_name] --round [number_of_rounds]\npython dialogueRE_evaluate.py --model [model_short_name] --round [number_of_rounds]\n```\n\n## 🖊️ Citation\n\nPlease kindly cite our paper if you find our study useful:\n\n```bibtex\n@misc{li2024longcontext,\n      title={Long-context LLMs Struggle with Long In-context Learning}, \n      author={Tianle Li and Ge Zhang and Quy Duc Do and Xiang Yue and Wenhu Chen},\n      year={2024},\n      eprint={2404.02060},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiger-ai-lab%2Flongiclbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftiger-ai-lab%2Flongiclbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiger-ai-lab%2Flongiclbench/lists"}