{"id":19053580,"url":"https://github.com/open-compass/ada-leval","last_synced_at":"2025-08-14T12:40:23.566Z","repository":{"id":232510535,"uuid":"784519262","full_name":"open-compass/Ada-LEval","owner":"open-compass","description":"The official implementation of \"Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks\"","archived":false,"fork":false,"pushed_at":"2024-04-22T09:36:27.000Z","size":1803,"stargazers_count":53,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-24T02:53:47.012Z","etag":null,"topics":["gpt4","llm","long-context"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/open-compass.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-10T02:18:27.000Z","updated_at":"2025-02-03T21:24:30.000Z","dependencies_parsed_at":"2024-04-10T04:59:44.078Z","dependency_job_id":"a1dc488d-2673-49a6-922b-b051d7e037ad","html_url":"https://github.com/open-compass/Ada-LEval","commit_stats":null,"previous_names":["open-compass/ada-leval"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FAda-LEval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FAda-LEval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FAda-LEval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-compass%2FAda-LEval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/open-compass","download_url":"https://codeload.github.com/open-compass/Ada-LEval/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250552038,"owners_count":21449162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpt4","llm","long-context"],"created_at":"2024-11-08T23:32:25.286Z","updated_at":"2025-08-14T12:40:23.539Z","avatar_url":"https://github.com/open-compass.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ada-LEval\n\n**The official implementation of [\"Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks\"](https://arxiv.org/abs/2404.06480)**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/BestAnswer.png\" width=\"80%\"\u003e\n\u003c/p\u003e\n\n**Ada-LEval** is a pioneering benchmark to assess the long-context capabilities with length-adaptable questions. It comprises two challenging tasks: **TSort**, which involves arranging text segments into the correct order, and **BestAnswer**, which requires choosing the best answer of a question among multiple candidates.\n\nBoth tasks feature the following advantages:\n1. **Controllable Test Cases**: The length of each test case can be finely tuned - by adjusting the number and length of text segments in TSort and altering the number of distractor options in BestAnswer. \n2. **Necessity for Full-Text Comprehension**: Successful completion of both tasks mandates complete reading and understanding of the provided text.\n3. **Precise Accuracy Measurement**: The design of these tasks allows for unambiguous accuracy calculation. TSort has a definitive 'correct' order, while in BestAnswer, the annotated responses by the questioner serve as definitive answers.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/AdaLEval.png\" width=\"60%\"\u003e\n\u003c/p\u003e\n\n## 🛠️QuickStart\n\nIn this repo, we implement the evaluation of Ada-LEval on GPT-4-Turbo-0125 (an example for APIs) and internlm2-[7b/20b] (an example for opensource LLMs). You can follow our implementation to evaluate Ada-LEval on your custom LLMs. \n\n1. **Preparation**\n\n   1. Installation and data preparation\n\n      ```bash\n      cd Ada-LEval\n      pip install -e . \n      bash fetch_data.sh\n      ```\n\n   2. For evaluating GPT-4, please set the environment variable: `export OPENAI_API_KEY=sk-xxxxx`\n\n      - Cost Estimation for GPT-4-Turbo-0125: `setting (2k, 4k, etc.) * n_samples * $0.01 / 1000`\n\n   3. For evaluating InternLM2-7B, please follow the [official guide](https://github.com/InternLM/lmdeploy) to install LMDeploy. \n\n2. **Evaluate GPT-4-Turbo-0125**: `python run.py --data {dataset_name} --model gpt-4-0125`\n\n3. **Evaluate InternLM2-7B**: `bash run.sh --data {dataset_name} --model internlm2-7b`\n\n\\* `dataset_name` can be `stackselect_{setting}` (for **BestAnswer**) or `textsort_{setting}` (for **TSort**). For example, `stackselect_16k`, `textsort_2k`, etc.\n\n\\** `run.sh` detect the number of available GPUs and do the data parallel. \n\n## 📊Evaluation Result\nHere is the evaluation result of TSort and BestAnswer benchmark under **long-context** \u0026 **ultra-long-context** settings. We also provide a 'random guess' baseline for each task. \n\n**Definition:** long-context  -\u003e  context window \u003c 32k; ultra-long-context: context-window \u003e= 32k\n\n**The Number of Evaluation Samples:** 1. API models on long-context: 200; 2. API models on ultra-long-context: 50; 3. Open-source models on long-context: 1000; 4. Open-source models on ultra-long-context: 200. \n\n#### TL;DR: \n\n1. **TSort is an extremely challenging benchmark:** We observe positive results (significantly better than random guess) only when evaluating SOTA API models (GPT-4 series) under short context settings (\u003c 8k).\n2. **BestAnswer is a challenging long-context benchmark with discrimination:** With 32k long-context, GPT-4-Turbo-0125 still obtains a decent 30% accuracy, while other models significantly lag behind. When the context window is 64k or even longer, models failed to solve almost all of the questions. \n\n#### TSort Evaluation Results\n\nBlanks indicate the result under the corresponding setting is not evaluated. \n\n| TSort                | 2k   | 4k   | 8k   | 16k  | 32k  | 64k  | 128k |\n| -------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| GPT-4-Turbo-0125     | 15.5 | 16.5 | 8.5  | 5.5  | 2.0  | 4.0  | 2.0  |\n| GPT-4-Turbo-1106     | 18.5 | 15.5 | 7.5  | 3.5  | 6.0  | 6.0  | 6.0  |\n| GPT-3.5-Turbo-1106   | 4.0  | 4.5  | 4.5  | 5.5  |      |      |      |\n| Claude-2             | 5.0  | 5.0  | 4.5  | 3.0  | 0.0  | 0.0  |      |\n| LongChat-7b-v1.5-32k | 5.3  | 5.0  | 3.1  | 2.5  |      |      |      |\n| ChatGLM2-6B-32k      | 0.9  | 0.7  | 0.2  | 0.9  |      |      |      |\n| ChatGLM3-6B-32k      | 2.3  | 2.4  | 2.0  | 0.7  |      |      |      |\n| Vicuna-7b-v1.5-16k   | 5.3  | 2.2  | 2.3  | 1.7  |      |      |      |\n| Vicuna-13b-v1.5-16k  | 5.4  | 5.0  | 2.4  | 3.1  |      |      |      |\n| InternLM2-7b         | 5.1  | 3.9  | 5.1  | 4.3  |      |      |      |\n| Random Guess         | 4.2  | 4.2  | 4.2  | 4.2  | 4.2  | 4.2  | 4.2  |\n\n#### BestAnswer Evaluation Results\n\nBlanks indicate the result under the corresponding setting is not evaluated. \n\n| BestAnswer           | 1k   | 2k   | 4k   | 6k   | 8k   | 12k  | 16k  | 32k  | 64k  | 128k |\n| -------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| GPT-4-Turbo-0125     | 73.5 | 73.5 | 65.5 | 63.0 | 56.5 | 52.0 | 44.5 | 30.0 | 0.0  | 0.0  |\n| GPT-4-Turbo-1106     | 74.0 | 73.5 | 67.5 | 59.5 | 53.5 | 49.5 | 44.0 | 16.0 | 0.0  | 0.0  |\n| GPT-3.5-Turbo-1106   | 61.5 | 48.5 | 41.5 | 29.5 | 17.0 | 2.5  | 2.5  |      |      |      |\n| Claude-2             | 65.0 | 43.5 | 23.5 | 15.0 | 17.0 | 12.0 | 11.0 | 4.0  | 0.0  |      |\n| LongChat-7b-v1.5-32k | 32.4 | 10.7 | 5.7  | 3.1  | 1.9  | 1.6  | 0.8  |      |      |      |\n| ChatGLM2-6B-32k      | 31.2 | 10.9 | 4.5  | 1.6  | 1.6  | 0.0  | 0.3  |      |      |      |\n| ChatGLM3-6B-32k      | 39.8 | 18.8 | 9.0  | 5.0  | 3.4  | 0.9  | 0.5  |      |      |      |\n| Vicuna-7b-v1.5-16k   | 37.0 | 11.1 | 5.8  | 3.2  | 1.8  | 1.9  | 1.0  |      |      |      |\n| Vicuna-13b-v1.5-16k  | 53.4 | 29.2 | 13.1 | 4.3  | 2.2  | 1.4  | 0.9  |      |      |      |\n| InternLM2-7b         | 58.6 | 49.5 | 33.9 | 12.3 | 13.4 | 2.0  | 0.8  | 0.5  | 0.5  | 0.0  |\n| Random Guess         | 26.7 | 10.1 | 4.5  | 3.0  | 2.3  | 1.4  | 1.1  | 0.6  | 0.3  | 0.1  |\n\n## 🖊️Citation\n\n```bib\n@inproceedings{wang2024ada,\n  title={Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks},\n  author={Wang, Chonghua and Duan, Haodong and Zhang, Songyang and Lin, Dahua and Chen, Kai},\n  booktitle={Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},\n  pages={3712--3724},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-compass%2Fada-leval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopen-compass%2Fada-leval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-compass%2Fada-leval/lists"}