{"id":29688657,"url":"https://github.com/pprp/acbench","last_synced_at":"2026-03-07T11:02:39.530Z","repository":{"id":295418384,"uuid":"900615677","full_name":"pprp/ACBench","owner":"pprp","description":"[ICML25] Agentic Compression Benchmark (ACBench)","archived":false,"fork":false,"pushed_at":"2025-07-02T04:27:45.000Z","size":17442,"stargazers_count":6,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-02T05:28:52.641Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pprp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-09T06:34:00.000Z","updated_at":"2025-07-02T04:27:53.000Z","dependencies_parsed_at":"2025-05-25T13:33:24.975Z","dependency_job_id":null,"html_url":"https://github.com/pprp/ACBench","commit_stats":null,"previous_names":["pprp/acbench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pprp/ACBench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2FACBench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2FACBench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2FACBench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2FACBench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pprp","download_url":"https://codeload.github.com/pprp/ACBench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pprp%2FACBench/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266625267,"owners_count":23958305,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-23T05:37:59.134Z","updated_at":"2026-03-07T11:02:34.492Z","avatar_url":"https://github.com/pprp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e🤖 ACBench 🔬\u003c/h1\u003e\n\u003ch3 align=\"center\"\u003e🤔 Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression 📊\u003c/h3\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://arxiv.org/abs/2505.19433\"\u003e📄arXiv\u003c/a\u003e •\n  \u003ca href=\"https://github.com/pprp/ACBench\"\u003e🌐GitHub\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/pprp/ACBench\"\u003e\u003cimg src=\"https://awesome.re/badge.svg\" alt=\"Awesome\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-green.svg\" alt=\"License: MIT\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Abstract](#abstract)\n- [Installation](#installation)\n- [Evaluation](#evaluation)\n- [Citation](#citation)\n\n![](./assets/main_figure_v2.png)\n\n## Overview\n\nPost-training compression reduces the computational and memory costs of large language models (LLMs), enabling resource-efficient deployment. However, existing compression benchmarks focus narrowly on language modeling (e.g., perplexity) and natural language understanding tasks (e.g., GLUE accuracy), ignoring the agentic capabilities—workflow, tool use/function call, long-context understanding and real-world application.\n\nWe introduce the Agent Compression Benchmark (ACBench), the first comprehensive benchmark for evaluating how compression impacts LLMs' agentic abilities. ACBench spans:\n\n- 12 tasks across 4 capabilities (e.g., WorfBench for workflow generation, Needle-in-Haystack for long-context retrieval)\n- 4-bit quantization (GPTQ, AWQ) and 50% pruning (Wanda, SparseGPT)\n- 15 models, including small (Gemma-2B), standard (Qwen2.5-7B), and distilled reasoning LLMs (DeepSeek-R1-Distill)\n\n## Abstract\n\nPost-training compression reduces the computational and memory costs of large language models (LLMs), enabling resource-efficient deployment. However, existing compression benchmarks focus narrowly on language modeling (e.g., perplexity) and natural language understanding tasks (e.g., GLUE accuracy), ignoring the agentic capabilities—_workflow_, _tool use/function call_, _long-context understanding_ and _real-world application_. We introduce the **Agent Compression Benchmark (ACBench)**, the first comprehensive benchmark for evaluating how compression impacts LLMs' agentic abilities. ACBench spans (1) 12 tasks across 4 capabilities (e.g., WorfBench for workflow generation, Needle-in-Haystack for long-context retrieval), (2) 4-bit quantization (GPTQ, AWQ) and 50% pruning (Wanda, SparseGPT), and (3) 15 models, including small (Gemma-2B), standard (Qwen2.5-7B), and distilled reasoning LLMs (DeepSeek-R1-Distill). Our experiments reveal compression tradeoffs: 4-bit quantization preserves workflow generation and tool use (1%--3% drop) but degrades real-world application accuracy by 10%--15%. We introduce _ERank_, _Top-k Ranking Correlation_ and _Energy_ to systematize analysis. ACBench provides actionable insights for optimizing LLM compression in agentic scenarios, bridging the gap between algorithmic efficiency and real-world applicability.\n\n## Installation\n\n```bash\ngit clone https://github.com/pprp/ACBench\ncd ACBench\npip install -r requirements.txt\npip install -e .\n```\n\n## Evaluation\n\nACBench builds upon and extends several excellent agentic benchmarks and compression toolkits. We integrate these benchmarks into our evaluation pipeline while preserving their original settings. For efficient model serving and evaluation, we utilize VLLM to deploy the compressed language models.\n\nFor detailed implementation and usage instructions, please refer to the corresponding subfolders in the `thirdpartys` directory. Each subfolder contains the original benchmark code along with our modifications to support compressed model evaluation. For experiment result on WorfBench, we have integrated it in acbench.\n\n### A Demo for WorfBench\n\nTaking WorfBench as example, we can run using the following scripts:\n\n```\n#!/bin/bash\n\nMODEL=$1\nTEMP=$2\nQUANT=$3\n\nDEVICE=${4:-6}\n\nexport CUDA_VISIBLE_DEVICES=$DEVICE\n\ntasks=(wikihow toolbench toolalpaca lumos alfworld webshop os)\n\nMODEL_NAME=$(basename $MODEL)\n\nfor task in ${tasks[@]}; do\n    python acbench/node_eval.py \\\n        --task gen_workflow \\\n        --model_name ${MODEL} \\\n        --gold_path ./data/gold_traj/${task}/graph_eval.json \\\n        --pred_path ./data/pred_traj/${MODEL_NAME}/${task}/${MODEL_NAME}/graph_eval_two_shot.json \\\n        --task_type ${task} \\\n        --few_shot \\\n        --temperature ${TEMP} \\\n        --quantization ${QUANT}\ndone\n```\n\n### Agentic Tasks\n\nFor Agentic Tasks:\n\n- WorfBench: Benchmarking Agentic Workflow Generation [![arXiv](https://img.shields.io/badge/arXiv-2410.07869-b31b1b.svg)](https://arxiv.org/abs/2410.07869) [![GitHub](https://img.shields.io/github/stars/zjunlp/WorfBench?style=social)](https://github.com/zjunlp/WorfBench)\n- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents [![arXiv](https://img.shields.io/badge/arXiv-2401.13178-b31b1b.svg)](https://arxiv.org/abs/2401.13178) [![GitHub](https://img.shields.io/github/stars/hkust-nlp/agentboard?style=social)](https://github.com/hkust-nlp/agentboard)\n- KVCache-Factory: Unified KV Cache Compression Methods for Auto-Regressive Models [![arXiv](https://img.shields.io/badge/arXiv-2406.02069-b31b1b.svg)](https://arxiv.org/abs/2406.02069) [![GitHub](https://img.shields.io/github/stars/Zefan-Cai/KVCache-Factory?style=social)](https://github.com/Zefan-Cai/KVCache-Factory)\n- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding [![arXiv](https://img.shields.io/badge/arXiv-2308.14508-b31b1b.svg)](https://arxiv.org/abs/2308.14508) [![GitHub](https://img.shields.io/github/stars/THUDM/LongBench?style=social)](https://github.com/THUDM/LongBench)\n- SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation [![arXiv](https://img.shields.io/badge/arXiv-2412.13649-b31b1b.svg)](https://arxiv.org/pdf/2412.13649) [![GitHub](https://img.shields.io/github/stars/Linking-ai/SCOPE?style=social)](https://github.com/Linking-ai/SCOPE)\n- T-Eval: Evaluating Tool Utilization Capability of LLMs Step by Step [![arXiv](https://img.shields.io/badge/arXiv-2312.14033-b31b1b.svg)](https://arxiv.org/abs/2312.14033) [![GitHub](https://img.shields.io/github/stars/open-compass/T-Eval?style=social)](https://github.com/open-compass/T-Eval)\n\n### Compression\n\nFor Compression:\n\n- Wanda: A Simple and Effective Pruning Approach for Large Language Models [![arXiv](https://img.shields.io/badge/arXiv-2306.11695-b31b1b.svg)](https://arxiv.org/abs/2306.11695) [![GitHub](https://img.shields.io/github/stars/locuslab/wanda?style=social)](https://github.com/locuslab/wanda)\n- LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit [![arXiv](https://img.shields.io/badge/arXiv-2405.06001-b31b1b.svg)](https://arxiv.org/abs/2405.06001) [![GitHub](https://img.shields.io/github/stars/ModelTC/llmc?style=social)](https://github.com/ModelTC/llmc)\n- QLLM-Eval: Evaluating Quantized Large Language Models [![arXiv](https://img.shields.io/badge/arXiv-2402.18158-b31b1b.svg)](https://arxiv.org/abs/2402.18158) [![GitHub](https://img.shields.io/github/stars/thu-nics/qllm-eval?style=social)](https://github.com/thu-nics/qllm-eval)\n\nFor fast serving, we employ [![Github](https://img.shields.io/github/stars/vllm-project/vllm?style=social)](https://github.com/vllm-project/vllm) for evaluation.\n\n## Visualization\n\nEnergy-based analysis:\n\n| ![Combined Position 1](./assets/energy_vis/combined_position_01.png)  | ![Combined Position 2](./assets/energy_vis/combined_position_64.png)  | ![Combined Position 3](./assets/energy_vis/combined_position_117.png) |\n| :-------------------------------------------------------------------: | :-------------------------------------------------------------------: | :-------------------------------------------------------------------: |\n| ![Combined Position 1](./assets/energy_vis/combined_position_175.png) | ![Combined Position 2](./assets/energy_vis/combined_position_244.png) | ![Combined Position 3](./assets/energy_vis/combined_position_252.png) |\n\nLogits Visualization:\n\n| ![Logits Visualization 1](./assets/logits_vis/combined_position_02.png)  | ![Logits Visualization 2](./assets/logits_vis/combined_position_82.png)  | ![Logits Visualization 3](./assets/logits_vis/combined_position_134.png) |\n| :----------------------------------------------------------------------: | :----------------------------------------------------------------------: | :----------------------------------------------------------------------: |\n| ![Logits Visualization 1](./assets/logits_vis/combined_position_184.png) | ![Logits Visualization 2](./assets/logits_vis/combined_position_225.png) | ![Logits Visualization 3](./assets/logits_vis/combined_position_246.png) |\n\nNeedle Visualization:\n\n### InternLM Models\n\n|     ![Needle Vis 1](./assets/needle/InternLM2.5_7b_AWQ_full_64_40k.png)      |   ![Needle Vis 2](./assets/needle/InternLM2.5_7b_GPTQ_full_64_40k.png)    |   ![Needle Vis 3](./assets/needle/InternLM2.5_7b_Instruct_full_64_40k.png)    |\n| :--------------------------------------------------------------------------: | :-----------------------------------------------------------------------: | :---------------------------------------------------------------------------: |\n|   ![Needle Vis 4](./assets/needle/InternLM2.5_7b_mag_2_4_full_64_40k.png)    |  ![Needle Vis 5](./assets/needle/InternLM2.5_7b_mag_un_full_64_40k.png)   | ![Needle Vis 6](./assets/needle/InternLM2.5_7b_sparsegpt_2_4_full_64_40k.png) |\n| ![Needle Vis 7](./assets/needle/InternLM2.5_7b_sparsegpt_un_full_64_40k.png) | ![Needle Vis 8](./assets/needle/InternLM2.5_7b_wanda_2_4_full_64_40k.png) |   ![Needle Vis 9](./assets/needle/InternLM2.5_7b_wanda_un_full_64_40k.png)    |\n\n### Qwen Models\n\n|     ![Needle Vis 10](./assets/needle/Qwen2.5_7b_AWQ_full_64_40k.png)      |     ![Needle Vis 11](./assets/needle/Qwen2.5_7b_GPTQ_full_64_40k.png)     |      ![Needle Vis 12](./assets/needle/Qwen2.5_7b_RTN_full_64_40k.png)      |\n| :-----------------------------------------------------------------------: | :-----------------------------------------------------------------------: | :------------------------------------------------------------------------: |\n|   ![Needle Vis 13](./assets/needle/Qwen2.5_7b_mag_2_4_full_64_40k.png)    |    ![Needle Vis 14](./assets/needle/Qwen2.5_7b_mag_un_full_64_40k.png)    | ![Needle Vis 15](./assets/needle/Qwen2.5_7b_sparsegpt_2_4_full_64_40k.png) |\n| ![Needle Vis 16](./assets/needle/Qwen2.5_7b_sparsegpt_un_full_64_40k.png) |  ![Needle Vis 17](./assets/needle/Qwen2.5_7b_wanda_2_4_full_64_40k.png)   |   ![Needle Vis 18](./assets/needle/Qwen2.5_7b_wanda_un_full_64_40k.png)    |\n|    ![Needle Vis 24](./assets/needle/qwen-1.5b-awq_full_64_40k_slm.png)    | ![Needle Vis 25](./assets/needle/qwen-1.5b-gptq-int4_full_64_40k_slm.png) | ![Needle Vis 26](./assets/needle/qwen-1.5b-gptq-int8_full_64_40k_slm.png)  |\n|     ![Needle Vis 27](./assets/needle/qwen-3b-awq_full_64_40k_slm.png)     |  ![Needle Vis 28](./assets/needle/qwen-3b-gptq-int4_full_64_40k_slm.png)  |  ![Needle Vis 29](./assets/needle/qwen-3b-gptq-int8_full_64_40k_slm.png)   |\n\n### Distilled/Megrez Models\n\n| ![Needle Vis 20](./assets/needle/deepseek-llama-8b_full_64_40k_slm.png) | ![Needle Vis 21](./assets/needle/deepseek-qwen-1.5b_full_64_40k_slm.png) |\n| :---------------------------------------------------------------------: | :----------------------------------------------------------------------: |\n| ![Needle Vis 22](./assets/needle/deepseek-qwen-7b_full_64_40k_slm.png)  |     ![Needle Vis 23](./assets/needle/megrez-3b_full_64_40k_slm.png)      |\n\n## Citation\n\nIf you use our work, please cite:\n\n```\n@inproceedings{dong2025compressed,\n  title     = {Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression},\n  author    = {Peijie Dong and Zhenheng Tang and Xiang Liu and Lujun Li and Xiaowen Chu and Bo Li},\n  booktitle = {International Conference on Machine Learning (ICML)},\n  year      = {2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpprp%2Facbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpprp%2Facbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpprp%2Facbench/lists"}