Awesome-Efficient-LLM

A curated list for Efficient Large Language Models
https://github.com/horseee/Awesome-Efficient-LLM

Last synced: 3 days ago
JSON representation

Efficient Architecture of LLM
- Tandem Transformers for Inference Efficient LLMs
- Scaling Efficient LLMs
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
- Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding
- ![Star - oryx/MobiLlama) [MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT](https://arxiv.org/abs/2402.16840) Omkar Thawakar, Ashmal Vayani, Salman Khan, Hisham Cholakal, Rao M. Anwer, Michael Felsberg, Tim Baldwin, Eric P. Xing, Fahad Shahbaz Khan |<img width="402" alt="image" src="https://github.com/mbzuai-oryx/MobiLlama/raw/main/images/mobillama_generation.gif"> |[Github](https://github.com/mbzuai-oryx/MobiLlama) [Paper](https://arxiv.org/abs/2402.16840) [Model](https://huggingface.co/MBZUAI/MobiLlama-05B) |
- Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models - Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, Caglar Gulcehre |<img width="1002" alt="image" src="https://arxiv.org/html/2402.19427v1/x3.png"> |[Paper](https://arxiv.org/abs/2402.19427)|
- ![Star
- ![Star
- ![Star - ICLR'24-blue)]() [Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs](https://arxiv.org/abs/2404.10308) Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung-Woo Ha, Jinwoo Shin |<img width="1002" alt="image" src="figures/homer.png"> |[Github](https://github.com/alinlab/HOMER) [Paper](https://arxiv.org/abs/2404.10308)|
Efficient MOE
- SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models
- ![Star - of-Experts Attention](https://arxiv.org/abs/2312.07987) Róbert Csordás, Piotr Piękos, Kazuki Irie, Jürgen Schmidhuber |<img width="1002" alt="image" src="figures/switchhead.png"> |[Github](https://github.com/robertcsordas/moe_attention) [Paper](https://arxiv.org/abs/2312.07987)|
- ![Star - Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference](https://arxiv.org/abs/2401.08383) Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. (DK)Panda |<img width="1002" alt="image" src="figures/exflow.png"> |[Github](https://github.com/YJHMITWEB/ExFlow) [Paper](https://arxiv.org/abs/2401.08383)|
- ![Star - Infinity) [MoE-Infinity: Activation-Aware Expert Offloading for Efficient MoE Serving](https://arxiv.org/abs/2401.14361) Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, Mahesh Marina |<img width="1002" alt="image" src="figures/MOE-Infinity.png"> |[Github](https://github.com/TorchMoE/MoE-Infinity) [Paper](https://arxiv.org/abs/2401.14361)|
- ![Star - GPU Orchestration for Fast Inference of Mixture-of-Experts Models](https://arxiv.org/abs/2402.07033) Keisuke Kamahori, Yile Gu, Kan Zhu, Baris Kasikci |<img width="1002" alt="image" src="https://github.com/efeslab/fiddler/blob/main/asset/key-idea.png"> |[Github](https://github.com/efeslab/fiddler) [Paper](https://arxiv.org/abs/2402.07033)|
- ![Star - Lance/Expert_Sparsity) [Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models](https://arxiv.org/abs/2402.14800) Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, Hongsheng Li |<img width="1002" alt="image" src="https://arxiv.org/html/2402.14800v1/x2.png"> |[Github](https://github.com/Lucky-Lance/Expert_Sparsity) [Paper](https://arxiv.org/abs/2402.14800)|
- Enhancing Efficiency in Sparse Models with Sparser Selection
- ![Star - prompted Mixture of Experts for Efficient LLM Generation](https://arxiv.org/abs/2404.01365) Harry Dong, Beidi Chen, Yuejie Chi |<img width="1002" alt="image" src="https://arxiv.org/html/2404.01365v1/extracted/5509263/figures/algorithm.png"> |[Github](https://github.com/hdong920/GRIFFIN) [Paper](https://arxiv.org/abs/2404.01365)|
- Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
- SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
- Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
- ![Publish - of-Experts Training via Whole Graph Computation-Communication Overlapping](https://arxiv.org/abs/2404.19429) Chenyu Jiang, Ye Tian, Zhen Jia, Shuai Zheng, Chuan Wu, Yida Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2404.19429v1/x4.png"> |[Paper](https://arxiv.org/abs/2404.19429)|
Full List
- Please check out all the papers by selecting the sub-area you're interested in. On this main page, only papers released in the past 90 days are shown.
  - curated list
- Please check out all the papers by selecting the sub-area you're interested in. On this main page, we're showing papers released in the past 90 days.
  - Publish
Hardware
- ![Star - AILab/flash-attention) [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://arxiv.org/abs/2307.08691). Tri Dao. [[Paper]](https://arxiv.org/abs/2307.08691)[[Github]](https://github.com/Dao-AILab/flash-attention)
- ![Star - Kelley. [[Paper]](https://arxiv.org/abs/2311.09431)[[Github]](https://github.com/exists-forall/striped_attention/)
- ![Star - IPADS/PowerInfer) [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://arxiv.org/abs/2312.12456). Yixin Song, Zeyu Mi, Haotong Xie, Haibo Chen. [[Paper]](https://arxiv.org/abs/2312.12456)[[Github]](https://github.com/SJTU-IPADS/PowerInfer)
Hardware/System
- ![Publish - AILab/flash-attention.svg?style=social&label=Star)](https://github.com/Dao-AILab/flash-attention) [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/abs/2205.14135). Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré. [[Paper]](https://arxiv.org/abs/2205.14135)[[Github]](https://github.com/Dao-AILab/flash-attention)
- ![Publish
- ![Publish - Throughput Generative Inference of Large Language Models with a Single GPU](https://arxiv.org/abs/2303.06865). Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang. [[Paper]](https://arxiv.org/abs/2303.06865)[[Github]](https://github.com/FMInference/FlexGen)
- ![Publish - project/vllm.svg?style=social&label=Star)](https://github.com/vllm-project/vllm) [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180). Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica. [[Paper]](https://arxiv.org/abs/2309.06180)[[Github]](https://github.com/vllm-project/vllm)
- ![Publish - extension-for-transformers.svg?style=social&label=Star)](https://github.com/intel/intel-extension-for-transformers) [Efficient LLM Inference on CPUs](https://arxiv.org/abs/2311.00502). Haihao Shen, Hanwen Chang, Bo Dong, Yu Luo, Hengyu Meng. [[Paper]](https://arxiv.org/abs/2311.00502)[[Github]](https://github.com/intel/intel-extension-for-transformers)
- EdgeMoE: Fast On-Device Inference of MoE-based Large Language Models
- ![Publish - Generation AI Accelerator Design Automation via Large Language Models](https://arxiv.org/abs/2309.10730). Yonggan Fu, Yongan Zhang, Zhongzhi Yu, Sixu Li, Zhifan Ye, Chaojian Li, Cheng Wan, Yingyan Lin. [[Paper]](https://arxiv.org/abs/2309.10730)
- Rethinking Memory and Communication Cost for Efficient Large Language Model Training
- Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models
- FlashDecoding++: Faster Large Language Model Inference on GPUs
- LLM in a flash: Efficient Large Language Model Inference with Limited Memory
- FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGA
- Efficient LLM inference solution on Intel GPU - extension-for-pytorch/tree/v2.1.10%2Bxpu/examples/gpu/inference/python/llm)
- ![Star
- ![Star - MII) [DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference](https://arxiv.org/abs/2401.08671). Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, Yuxiong He. [[Paper]](https://arxiv.org/abs/2401.08671)[[Github]](https://github.com/microsoft/DeepSpeed-MII)
- ![Star - aware Interleaving and Conflict-free Kernel for efficient LLM inference](https://arxiv.org/abs/2402.10076). Taesu Kim, Jongho Lee, Daehyun Ahn, Sarang Kim, Jiwoong Choi, Minkyu Kim, Hyungjun Kim. [[Paper]](https://arxiv.org/abs/2402.10076)[[Github]](https://github.com/SqueezeBits/QUICK)
- BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences
- ![Star - project/sglang/tree/main) [Efficiently Programming Large Language Models using SGLang](https://arxiv.org/abs/2312.07104). Lianmin Zheng*, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng*. [[Paper]](https://arxiv.org/abs/2312.07104) [[Github]](https://github.com/sgl-project/sglang/tree/main)
- MELTing point: Mobile Evaluation of Language Transformers
- DeFT: Flash Tree-attention with IO-Awareness for Efficient Tree-search-based LLM Inference
- Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs
- LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism
- ![Star - griggs/melange-release) [Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity](https://arxiv.org/abs/2404.14527). Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, Ion Stoica. [[Paper]](https://arxiv.org/abs/2404.14527)[[Github]](https://github.com/tyler-griggs/melange-release)
- Expert Router: Orchestrating Efficient Language Model Inference through Prompt Classification
- Efficient and Economic Large Language Model Inference with Attention Offloading
Inference Acceleration
- ![Publish - LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction](https://arxiv.org/abs/2310.15556) Junyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, Yiming Qian |<img width="1002" alt="image" src="figures/TCRA-LLM.png"> |[Paper](https://arxiv.org/abs/2310.15556)|
- ![Publish - Guided Early Exiting Method for Accelerating Language Models Inference](https://arxiv.org/abs/2312.11882) Ziqian Zeng, Yihuai Hong, Hongliang Dai, Huiping Zhuang, Cen Chen |<img width="1002" alt="image" src="figures/ConsistentEE.png"> |[Paper](https://arxiv.org/abs/2312.11882)|
- ![Publish
- ![Star - ICML'23%20Oral-blue)]() [Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://openreview.net/forum?id=wIPIhHd00i) Zichang Liu, Jue WANG, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, Beidi Chen |<img width="202" alt="image" src="figures/DajeVu.png"> |[Github](https://github.com/FMInference/DejaVu) [Paper](https://openreview.net/forum?id=wIPIhHd00i)|
- ![Star - Yew Lin, Yuqing Yang, Lili Qiu |<img width="1002" alt="image" src="figures/longllmlingua.png"> |[Github](https://github.com/microsoft/LLMLingua) [Paper](https://arxiv.org/abs/2310.06839)|
- ![Star - EMNLP'23-blue)]() [Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding](https://arxiv.org/abs/2310.05424) Sangmin Bae, Jongwoo Ko, Hwanjun Song, Se-Young Yun |<img width="1002" alt="image" src="figures/FREE.png"> |[Github](https://github.com/raymin0223/fast_robust_early_exit) [Paper](https://arxiv.org/abs/2310.05424)|
- ![Star - EMNLP'23-blue)]() [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://arxiv.org/abs/2310.06201) Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin |<img width="1002" alt="image" src="figures/selective_context.png"> |[Github](https://github.com/liyucheng09/Selective_Context) [Paper](https://arxiv.org/abs/2310.06201)|
- Inference with Reference: Lossless Acceleration of Large Language Models
- ![Star
- SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference
- Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding
- ![Star - zju/self-speculative-decoding) [Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding](https://arxiv.org/abs/2309.08168) Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, Sharad Mehrotra |<img width="1002" alt="image" src="https://github.com/dilab-zju/self-speculative-decoding/blob/main/assets/intro.png"> |[Github](https://github.com/dilab-zju/self-speculative-decoding) [Paper](https://arxiv.org/abs/2309.08168)|
- ![Star - han-lab/streaming-llm) [Efficient Streaming Language Models with Attention Sinks](https://arxiv.org/abs/2309.17453) Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis |<img width="1002" alt="image" src="https://github.com/mit-han-lab/streaming-llm/blob/main/figures/schemes.png"> |[Github](https://github.com/mit-han-lab/streaming-llm) [Paper](https://arxiv.org/abs/2309.17453)|
- (Dynamic) Prompting might be all you need to repair Compressed LLMs
- ![Star - efficient Reasoning](https://arxiv.org/abs/2310.03094) Murong Yue, Jie Zhao, Min Zhang, Liang Du, Ziyu Yao |<img width="1002" alt="image" src="figures/LLM_MoT_cascade.png"> |[Github](https://github.com/MurongYue/LLM_MoT_cascade) [Paper](https://arxiv.org/abs/2310.03094)|
- CacheGen: Fast Context Loading for Language Model Applications
- ![Star - EMNLP'23-blue)]() [Context Compression for Auto-regressive Transformers with Sentinel Tokens](https://arxiv.org/abs/2310.08152) Siyu Ren, Qi Jia, Kenny Q. Zhu |<img width="1002" alt="image" src="figures/KV_compression.png"> |[Github](https://github.com/DRSY/KV_Compression) [Paper](https://arxiv.org/abs/2310.08152)|
- ![Star - rankers) [A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models](https://arxiv.org/abs/2310.09497) Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, Guido Zuccon |<img width="1002" alt="image" src="figures/Setwise.png"> |[Github](https://github.com/ielab/llm-rankers) [Paper](https://arxiv.org/abs/2310.09497)|
- SPEED: Speculative Pipelined Execution for Efficient Decoding
- Accelerating LLM Inference by Enabling Intermediate Layer Decoding
- Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster
- ![Star - mllab/context-memory) [Compressed Context Memory For Online Language Model Interaction](https://arxiv.org/abs/2312.03414) Jang-Hyun Kim, Junyoung Yeom, Sangdoo Yun, Hyun Oh Song |<img width="1002" alt="image" src="https://github.com/snu-mllab/Context-Memory/blob/main/image/main.png"> |[Github](https://github.com/snu-mllab/context-memory) [Paper](https://arxiv.org/abs/2312.03414)|
- SparQ Attention: Bandwidth-Efficient LLM Inference - Galley, Charlie Blake, Carlo Luschi, Douglas Orr |<img width="1002" alt="image" src="figures/SparQ.png"> |[Paper](https://arxiv.org/abs/2312.04985)|
- Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy
- Cascade Speculative Drafting for Even Faster LLM Inference - Chuan Chang |<img width="1002" alt="image" src="figures/CSDrafting.png"> |[Paper](https://arxiv.org/abs/2312.11462)|
- ![Star - offloading) [Fast Inference of Mixture-of-Experts Language Models with Offloading](https://arxiv.org/abs/2312.17238) Artyom Eliseev, Denis Mazur |<img width="1002" alt="image" src="figures/mixtral_offloading.png"> |[Github](https://github.com/dvmazur/mixtral-offloading) [Paper](https://arxiv.org/abs/2312.17238)|
- ![Star - NeurIPS'23-blue)]() [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://arxiv.org/abs/2306.14048) Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen |<img width="1002" alt="image" src="https://github.com/FMInference/H2O/blob/main/Figs/h2o.jpg"> |[Github](https://github.com/FMInference/H2O) [Paper](https://arxiv.org/abs/2306.14048)|
- ![Star - llm) Yuhui Li, Chao Zhang, and Hongyang Zhang |<img width="302" alt="image" src="https://github.com/SafeAILab/EAGLE/blob/main/figs/fig1.png"> |[Github](https://github.com/SafeAILab/EAGLE) [Blog](https://sites.google.com/view/eagle-llm)|
- LoMA: Lossless Compressed Memory Attention
- ![Star
- APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding
- ![Star - Directional Tuning for Lossless Acceleration in Large Language Models](https://arxiv.org/abs/2401.12522) Feng Lin, Hanling Yi, Hongbin Li, Yifan Yang, Xiaotian Yu, Guangming Lu, Rong Xiao |<img width="1002" alt="image" src="figures/BiTA.png"> |[Github](https://github.com/linfeng93/BiTA) [Paper](https://arxiv.org/abs/2401.12522)|
- ![Star
- Speculative Streaming: Fast LLM Inference without Auxiliary Models
- RelayAttention for Efficient Large Language Model Serving with Long System Prompts
- Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
- ![Star
- ![Star - Timmy/GEAR) [GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM](https://arxiv.org/abs/2403.05527) Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, Tuo Zhao |<img width="1002" alt="image" src="https://github.com/HaoKang-Timmy/GEAR/raw/main/Fig/overview.png"> |[Github](https://github.com/HaoKang-Timmy/GEAR) [Paper](https://arxiv.org/abs/2403.05527)|
- CHAI: Clustered Head Attention for Efficient LLM Inference - Jean Wu |<img width="1002" alt="image" src="figures/chai.png"> |[Paper](https://arxiv.org/abs/2403.08058)|
- Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference
- Recurrent Drafter for Fast Speculative Decoding in Large Language Models
- Optimal Block-Level Draft Verification for Accelerating Speculative Decoding
- Hierarchical Skip Decoding for Efficient Autoregressive Text Generation
- ![Star
- ![Star - zhao/prepacking) [Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models](https://arxiv.org/abs/2404.09529) Siyan Zhao, Daniel Israel, Guy Van den Broeck, Aditya Grover |<img width="302" alt="image" src="https://github.com/siyan-zhao/prepacking/raw/main/figures/prepacking_gif_final.gif"> |[Github](https://github.com/siyan-zhao/prepacking) [Paper](https://arxiv.org/abs/2404.09529)|
- Towards Fast Inference: Exploring and Improving Blockwise Parallel Drafts
- ![Star - gram Parallel Decoding](https://arxiv.org/abs/2404.08698) Jie Ou, Yueming Chen, Wenhong Tian |<img width="602" alt="image" src="figures/ANPD.png"> |[Github](https://github.com/oujieww/ANPD) [Paper](https://arxiv.org/abs/2404.08698)|
- Self-Selected Attention Span for Accelerating Large Language Model Inference
- Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration
- XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference - André Noël, Valentina Zantedeschi, David Vázquez, Nicolas Chapados, Christopher Pal, Perouz Taslakian |<img width="1002" alt="image" src="https://arxiv.org/html/2404.15420v1/x2.png"> |[Paper](https://arxiv.org/abs/2404.15420)|
- ![Star - core/hybrid_llm_routing)[![Publish](https://img.shields.io/badge/Conference-ICLR'24-blue)]() [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://arxiv.org/abs/2404.14618) Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V.S. Lakshmanan, Ahmed Hassan Awadallah |<img width="1002" alt="image" src="figures/hybridLLM.png"> |[Github](https://github.com/m365-core/hybrid_llm_routing) [Paper](https://arxiv.org/abs/2404.14618)|
- Efficient LLM Inference with Kcache
- Better & Faster Large Language Models via Multi-token Prediction - Paz, Gabriel Synnaeve |<img width="1002" alt="image" src="figures/MBPP.png"> |[Paper](https://arxiv.org/abs/2404.19737)|
- You Only Cache Once: Decoder-Decoder Architectures for Language Models
- ![Star - Speculative Decoding via Double Early Exiting](https://arxiv.org/abs/2404.18911) Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, Yunhe Wang |<img width="1002" alt="image" src="https://github.com/Equationliu/Kangaroo/blob/main/imgs/kangaroo.png"> |[Github](https://github.com/Equationliu/Kangaroo) [Paper](https://arxiv.org/abs/2404.18911)|
- Accelerating Speculative Decoding using Dynamic Speculation Length
- Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge
- ![Star - SD) [EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models](https://arxiv.org/abs/2405.07542) Yunsheng Ni, Chuanjian Liu, Yehui Tang, Kai Han, Yunhe Wang |<img width="202" alt="image" src="https://github.com/niyunsheng/EMS-SD/raw/main/assets/fig2-method.png"> |[Github](https://github.com/niyunsheng/EMS-SD) [Paper](https://arxiv.org/abs/2405.07542)|
- Distributed Speculative Inference of Large Language Models
- ![Star
Knowledge Distillation
- ![Star - CoT-Specialization)[![Publish](https://img.shields.io/badge/Conference-ICML'23-blue)]() [Specializing Smaller Language Models towards Multi-Step Reasoning](https://arxiv.org/abs/2301.12726) Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, Tushar Khot |<img width="1002" alt="image" src="figures/ModelSpecialization.png"> |[Github](https://github.com/FranxYao/FlanT5-CoT-Specialization) [Paper](https://arxiv.org/abs/2301.12726)|
- ![Star - ACL'23%20Outstanding-blue)]() [Distilling Script Knowledge from Large Language Models for Constrained Language Planning](https://arxiv.org/abs/2305.05252) Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, Soham Shah, Charles Robert Jankowski, Yanghua Xiao, Deqing Yang |<img width="302" alt="image" src="figures/CoScript.png"> |[Github](https://github.com/siyuyuan/coscript) [Paper](https://arxiv.org/abs/2305.05252)|
- ![Publish - Consistent Chain-of-Thought Distillation](https://arxiv.org/abs/2305.01879) Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, Xiang Ren |<img width="1002" alt="image" src="figures/SCOTT.png"> |[Paper](https://arxiv.org/abs/2305.01879)|
- ![Star - ACL'23-blue)]() [DISCO: Distilling Counterfactuals with Large Language Models](https://arxiv.org/abs/2212.10534) Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, Kyle Richardson |<img width="1002" alt="image" src="figures/disco.png"> |[Github](https://github.com/eric11eca/disco) [Paper](https://arxiv.org/abs/2212.10534)|
- ![Star - ACL'23-blue)]() [I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation](https://arxiv.org/abs/2212.09246) Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Lianhui Qin, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, Yejin Choi |<img width="1002" alt="image" src="https://i2d2.allen.ai/i2d2-fig1.png"> |[Github](https://github.com/allenai/i2d2) [Paper](https://arxiv.org/abs/2212.09246) [Project](https://i2d2.allen.ai/) |
- ![Star - ACL'23-blue)]() [Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step](https://arxiv.org/abs/2306.14050) Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, Yejin Choi |<img width="202" alt="image" src="figures/SCoTD.png"> |[Github](https://github.com/allenai/cot_distillation) [Paper](https://arxiv.org/abs/2306.14050)|
- ![Star - NeurIPS'23-blue)]() [Can Language Models Teach? Teacher Explanations Improve Student Performance via Theory of Mind](https://arxiv.org/abs/2306.09299) Swarnadeep Saha, Peter Hase, and Mohit Bansal |<img width="302" alt="image" src="https://github.com/swarnaHub/ExplanationIntervention/blob/main/assets/main_fig.png"> |[Github](https://github.com/swarnaHub/ExplanationIntervention) [Paper](https://arxiv.org/abs/2306.09299)|
- ![Star - EMNLP-2023)[![Publish](https://img.shields.io/badge/Conference-EMNLP'23-blue)]() [PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation](https://arxiv.org/abs/2310.14192) Gaurav Sahu, Olga Vechtomova, Dzmitry Bahdanau, Issam H. Laradji |<img width="1002" alt="image" src="figures/PromptMix.png"> |[Github](https://github.com/ServiceNow/PromptMix-EMNLP-2023) [Paper](https://arxiv.org/abs/2310.14192)|
- ![Star - AAAI'24-blue)]() [Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data](https://arxiv.org/abs/2312.12832) Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Bin Sun, Xinglin Wang, Heda Wang, Kan Li |<img width="1002" alt="image" src="https://github.com/Yiwei98/TDG/blob/main/img.png"> |[Github](https://github.com/Yiwei98/TDG) [Paper](https://arxiv.org/abs/2312.12832)|
- ![Star - ACL'23%20Industry%20Track-blue)]() [GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model](https://arxiv.org/abs/2306.06629) Shicheng Tan, Weng Lam Tam, Yuanchun Wang, Wenwen Gong, Yang Yang, Hongyin Tang, Keqing He, Jiahao Liu, Jingang Wang, Shu Zhao, Peng Zhang, Jie Tang |<img width="1002" alt="image" src="figures/GKD.png"> |[Github](https://github.com/aitsc/GLMKD) [Paper](https://arxiv.org/abs/2306.06629)|
- ![Star - research/distilling-step-by-step) [![Publish](https://img.shields.io/badge/Conference-ACL'23%20Findings-blue)]() [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://arxiv.org/abs/2305.02301) Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister | <img width="2000" alt="image" src="figures/Distill_step_by_step.png">| [Github](https://github.com/google-research/distilling-step-by-step) [Paper](https://arxiv.org/abs/2305.02301) |
- ![Star - EMNLP'23%20Findings-blue)]() [Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models](https://arxiv.org/abs/2310.13395) Ilias Stogiannidis, Stavros Vassos, Prodromos Malakasiotis, Ion Androutsopoulos |<img width="252" alt="image" src="figures/OCaTS.png"> |[Github](https://github.com/stoyian/OCaTS) [Paper](https://arxiv.org/abs/2310.13395)|
- ![Star - nlp/LaMini-LM) [LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions](https://github.com/mbzuai-nlp/LaMini-LM) Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, Alham Fikri Aji | <img width="1002" alt="image" src="https://github.com/mbzuai-nlp/LaMini-LM/blob/main/images/lamini-pipeline.drawio.png"> | [Github](https://github.com/mbzuai-nlp/LaMini-LM) [paper](https://arxiv.org/abs/2304.14402) |
- Teaching Small Language Models to Reason
- ![Star - distill) [Large Language Model Distillation Doesn't Need a Teacher](https://arxiv.org/abs/2305.14864) Ananya Harsh Jha, Dirk Groeneveld, Emma Strubell, Iz Beltagy | <img width="2000" alt="image" src="figures/TeacherFreeLLM.png"> | [Github](https://github.com/ananyahjha93/llm-distill) [paper](https://arxiv.org/abs/2305.14864) |
- The False Promise of Imitating Proprietary LLMs
- ![Star - distillation) [Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing](https://arxiv.org/abs/2305.16635) Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, Yejin Choi |<img width="1002" alt="image" src="figures/impossible_distillation.png"> |[Github](https://github.com/jaehunjung1/impossible-distillation) [paper](https://arxiv.org/abs/2305.16635) |
- PaD: Program-aided Distillation Specializes Large Models in Reasoning
- RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment
- Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA - COT.png"> |[Paper](https://arxiv.org/abs/2308.04679)|
- ![Star - ner/universal-ner) [UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition](https://arxiv.org/abs/2308.03279) Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, Hoifung Poon |<img width="302" alt="image" src="figures/UniversalNER.png"> |[Github](https://github.com/universal-ner/universal-ner) [Paper](https://arxiv.org/abs/2308.03279) [Project](https://universal-ner.github.io) |
- ![Star - Loup Tastet |<img width="302" alt="image" src="figures/BabyLLaMA.png"> |[Github](https://github.com/timinar/BabyLlama) [Paper](https://arxiv.org/abs/2308.02019) | [Model](https://huggingface.co/timinar/baby-llama-58m) |
- DistillSpec: Improving Speculative Decoding via Knowledge Distillation - François Kagy, Rishabh Agarwal |<img width="1002" alt="image" src="figures/DistillSpec.png"> |[Paper](https://arxiv.org/abs/2310.08461)|
- ![Star - handbook) [Zephyr: Direct Distillation of LM Alignment](https://arxiv.org/abs/2310.16944) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, Thomas Wolf |<img width="1002" alt="image" src="figures/zephyr.png"> |[Github](https://github.com/huggingface/alignment-handbook) [Paper](https://arxiv.org/abs/2310.16944)|
- ![Star
- Unlock the Power: Competitive Distillation for Multi-Modal Large Language Models
- Mixed Distillation Helps Smaller Language Model Better Reasoning
- Distilling Event Sequence Knowledge From Large Language Models
- Knowledge Distillation for Closed-Source Language Models
- Improving Small Language Models' Mathematical Reasoning via Equation-of-Thought Distillation
- Scavenging Hyena: Distilling Transformers into Long Convolution Models - Transfer-HD.png"> |[Paper](https://arxiv.org/abs/2401.17574)|
- ![Star - Young Yun |<img width="1002" alt="image" src="https://arxiv.org/html/2402.03898v1/x4.png"> |[Github](https://github.com/jongwooko/distillm) [Paper](https://arxiv.org/abs/2402.03898)|
- Large Language Model Meets Graph Neural Network in Knowledge Distillation
- ![Star - river/LLM_unlearning) [Unmemorization in Large Language Models via Self-Distillation and Deliberate Imagination](https://arxiv.org/abs/2402.10052) Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, Ivan Vulić |<img width="1002" alt="image" src="https://arxiv.org/html/2402.10052v1/x1.png"> |[Github](https://github.com/dong-river/LLM_unlearning) [Paper](https://arxiv.org/abs/2402.10052)|
- ![Star - BZRD/llm-recipes) [Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs](https://arxiv.org/abs/2402.12030) Nicolas Boizard, Kevin El-Haddad, Céline Hudelot, Pierre Colombo |<img width="1002" alt="image" src="figures/CrossTokenizer.png"> |[Github](https://github.com/Nicolas-BZRD/llm-recipes) [Github](https://github.com/Nicolas-BZRD/llm-distillation) [Paper](https://arxiv.org/abs/2402.12030) [Model](https://huggingface.co/collections/Nicolas-BZRD/llms-distillation-65cfa07f1e4ed7404502a9eb)|
- Revisiting Knowledge Distillation for Autoregressive Language Models
- PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning
- Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
- Wisdom of Committee: Distilling from Foundation Model to Specialized Application Model
- Divide-or-Conquer? Which Part Should You Distill Your LLM?
- ![Star - cd) [Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation](https://arxiv.org/abs/2402.14874) Phuc Phan, Hieu Tran, Long Phan |<img width="1002" alt="image" src="https://github.com/pphuc25/distil-cd/blob/main/assets/figure1-method.jpg"> |[Github](https://github.com/pphuc25/distil-cd) [Paper](https://arxiv.org/abs/2402.14874)|
- ![Star - to-Reason)[![Publish](https://img.shields.io/badge/Conference-EMNLP'23-blue)]() [Democratizing Reasoning Ability: Tailored Learning from Large Language Model](https://aclanthology.org/2023.emnlp-main.120.pdf) Zhaoyang Wang, Shaohan Huang, Yuxuan Liu, Jiahai Wang, Minghui Song, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang |<img width="1002" alt="image" src="figures/learn-to-reason.png"> |[Github](https://github.com/Raibows/Learn-to-Reason) [Paper](https://aclanthology.org/2023.emnlp-main.120.pdf)|
- Leveraging Zero-Shot Prompting for Efficient Language Model Distillation
- ![Publish
- ![Star - large-metaie)|
- Gecko: Versatile Text Embeddings Distilled from Large Language Models
- Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models - AKL.png"> |[Paper](https://arxiv.org/abs/2404.02657) [Blog-Eng](https://zhuanlan.zhihu.com/p/690804722) [Blog-中](https://zhuanlan.zhihu.com/p/690748958)|
- Post-Semantic-Thinking: A Robust Strategy to Distill Reasoning Capacity from Large Language Models
- Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation
- Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning
- ![Star - ACL'24-blue)]() [RDRec: Rationale Distillation for LLM-based Recommendation](https://arxiv.org/abs/2405.10587) Xinfeng Wang, Jin Cui, Yoshimi Suzuki, Fumiyo Fukumoto |<img width="1002" alt="image" src="figures/RDRec.png"> |[Github](https://github.com/WangXFng/RDRec) [Paper](https://arxiv.org/abs/2405.10587)|
KV Cache Compression
- WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More
- DB-LLM: Accurate Dual-Binarization for Efficient LLMs - LLM.png"> |[Paper](https://arxiv.org/abs/2402.11960)|
- No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
- ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition
- Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference
- ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
- ![Star - free Low-bit Quantization with Matrix Decomposition for KV Cache Compression](https://arxiv.org/abs/2405.12591) Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Yipeng Ma, Tao Wang, Ji-Rong Wen |<img width="1002" alt="image" src="figures/DecoQuant.png"> |[Github](https://github.com/lpyhdzx/DecoQuant_code) [Paper](https://arxiv.org/abs/2405.12591)|
- ![Star - ACL'24-blue)]() [PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference](https://arxiv.org/abs/2405.12532) Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, Hai Zhao |<img width="1002" alt="image" src="figures/PyramidInfer.png"> |[Github](https://github.com/mutonix/pyramidinfer) [Paper](https://arxiv.org/abs/2405.12532)|
- Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
- ![Star - ACL'24-blue)]() [Layer-Condensed KV Cache for Efficient Inference of Large Language Models](https://arxiv.org/abs/2405.10637) Haoyi Wu, Kewei Tu |<img width="1002" alt="image" src="figures/LCKV.png"> |[Github](https://github.com/whyNLP/LCKV) [Paper](https://arxiv.org/abs/2405.10637)|
Leaderboard
- [Source
- [Source
- [Source
- [Source
- [Source
- [Source
Low-Rank Decomposition
- ![Star - ICML'23-blue)]() [LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation](https://arxiv.org/abs/2306.11222) Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, Tuo Zhao |<img width="302" alt="image" src="figures/LoSparse.png"> |[Github](https://github.com/yxli2123/LoSparse) [Paper](https://arxiv.org/abs/2306.11222)|
- ![Star - compressor)[![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]() [Matrix Compression via Randomized Low Rank and Low Precision Factorization](https://arxiv.org/abs/2310.11028) Rajarshi Saha, Varun Srivastava, Mert Pilanci |<img width="1002" alt="image" src="figures/LPLR.png"> |[Github](https://github.com/pilancilab/matrix-compressor) [Paper](https://arxiv.org/abs/2310.11028)|
- TensorGPT: Efficient Compression of the Embedding Layer in LLMs based on the Tensor-Train Decomposition - SVD.png"> |[Paper](https://arxiv.org/abs/2307.00526)|
- LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression
- ![Star - rom) [Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models](https://arxiv.org/abs/2312.07046) Arnav Chavan, Nahush Lele, Deepak Gupta |<img width="1002" alt="image" src="figures/LLM-ROM.png"> |[Github](https://github.com/transmuteAI/trailmet/tree/main/trailmet/algorithms/llm-rom) [Paper](https://arxiv.org/abs/2312.07046)|
- Data-free Weight Compress and Denoise for Large Language Models
- ![Star - MLSys-Lab/SVD-LLM) [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://arxiv.org/abs/2403.07378) Xin Wang, Yu Zheng, Zhongwei Wan, Mi Zhang |<img width="1002" alt="image" src="https://github.com/AIoT-MLSys-Lab/SVD-LLM/raw/main/figures/framework.png"> |[Github](https://github.com/AIoT-MLSys-Lab/SVD-LLM) [Paper](https://arxiv.org/abs/2403.07378)|
- ![Star - ACL'24%20Findings-blue)]() [Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization](https://arxiv.org/abs/2405.10616) Yixin Ji, Yang Xiang, Juntao Li, Wei Chen, Zhongyi Liu, Kehai Chen, Min Zhang |<img width="1002" alt="image" src="figures/bolaco.png"> |[Github](https://github.com/Dereck0602/Bolaco) [Paper](https://arxiv.org/abs/2405.10616)|
- ![Star - LLM)[![Publish](https://img.shields.io/badge/Conference-ACL'24-blue)]() [Surgical Feature-Space Decomposition of LLMs: Why, When and How?](https://arxiv.org/abs/2405.13039) Arnav Chavan, Nahush Lele, Deepak Gupta |<img width="1002" alt="image" src="figures/SFSD-LLM.png"> |[Github](https://github.com/nyunAI/SFSD-LLM) [Paper](https://arxiv.org/abs/2405.13039)|
Network Pruning
- ![Star - DASLab/sparsegpt) [![Publish](https://img.shields.io/badge/Conference-ICML'23-blue)]() [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://github.com/IST-DASLab/sparsegpt) Elias Frantar, Dan Alistarh| <img width="522" alt="image" src="figures/sparsegpt.png"> |[Github](https://github.com/IST-DASLab/sparsegpt) [paper](https://arxiv.org/abs/2301.00774) |
- ![Star - Pruner) [![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]() [![Type](https://img.shields.io/badge/Structural-C2A4A6)]() [LLM-Pruner: On the Structural Pruning of Large Language Models](https://arxiv.org/abs/2305.11627) Xinyin Ma, Gongfan Fang, Xinchao Wang | <img width="561" alt="image" src="figures/llm_pruner.png">| [Github](https://github.com/horseee/LLM-Pruner) [paper](https://arxiv.org/abs/2305.11627)|
- ![Star - Group/essential_sparsity) [![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]() [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() [The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter](https://arxiv.org/abs/2306.03805) Ajay Jaiswal, Shiwei Liu, Tianlong Chen, Zhangyang Wang |<img width="1002" alt="image" src="https://user-images.githubusercontent.com/6660499/243539825-ca3b1dbe-bc1c-45d9-a6ea-d1d0c991e997.png"> |[Github](https://github.com/VITA-Group/essential_sparsity) [Paper](https://arxiv.org/abs/2306.03805)|
- ![Star - llm)[![Publish](https://img.shields.io/badge/Conference-VLDB'24-blue)]() [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() [Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity](https://arxiv.org/abs/2309.10285) Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, Shuaiwen Leon Song |<img width="602" alt="image" src="figures/FlashLLM.png"> |[Github](https://github.com/AlibabaResearch/flash-llm) [Paper](https://arxiv.org/abs/2309.10285)|
- ![Star - Pruning-Official)[![Publish](https://img.shields.io/badge/Conference-EMNLP'23%20Findings-blue)]() [![Type](https://img.shields.io/badge/Structural-C2A4A6)]() [NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models](https://arxiv.org/abs/2310.10054) Jongwoo Ko, Seungjoon Park, Yujin Kim, Sumyeong Ahn, Du-Seong Chang, Euijai Ahn, Se-Young Yun |<img width="402" alt="image" src="figures/NASH.png"> |[Github](https://github.com/jongwooko/NASH-Pruning-Official) [Paper](https://arxiv.org/abs/2310.10054)|
- ![Star - C2A4A6)]() [A Simple and Effective Pruning Approach for Large Language Models](https://arxiv.org/abs/2306.11695) Mingjie Sun, Zhuang Liu, Anna Bair, J. Zico Kolter |<img width="1002" alt="image" src="https://user-images.githubusercontent.com/20168304/245999360-f951de47-269d-491d-826a-8e6d85627849.png"> |[Github](https://github.com/locuslab/wanda) [Paper](https://arxiv.org/abs/2306.11695)|
- ![Type
- ![Type - KICK.png"> |[Paper](https://arxiv.org/abs/2310.01382)|
- ![Star - Group/Junk_DNA_Hypothesis)[![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() [Junk DNA Hypothesis: A Task-Centric Angle of LLM Pre-trained Weights through Sparsity](https://arxiv.org/abs/2310.02277) Lu Yin, Shiwei Liu, Ajay Jaiswal, Souvik Kundu, Zhangyang Wang |<img width="1002" alt="image" src="figures/junk_DNA.png"> |[Github](https://github.com/VITA-Group/Junk_DNA_Hypothesis) [Paper](https://arxiv.org/abs/2310.02277)|
- ![Star - C2A4A6)]() [Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity](https://arxiv.org/abs/2310.05175) Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, Shiwei Liu |<img width="1002" alt="image" src="https://github.com/luuyin/OWL/blob/main/Images/Layer_wise_sparsity.png"> |[Github](https://github.com/luuyin/OWL) [Paper](https://arxiv.org/abs/2310.05175)|
- ![Star - nlp/LLM-Shearing) [![Type](https://img.shields.io/badge/Structural-C2A4A6)]() [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://arxiv.org/abs/2310.06694) Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi Chen |<img width="1002" alt="image" src="figures/LLM-shearing.png"> |[Github](https://github.com/princeton-nlp/LLM-Shearing) [Paper](https://arxiv.org/abs/2310.06694)|
- ![Star - DASLab/SparseFinetuning) [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() [Sparse Finetuning for Inference Acceleration of Large Language Models](https://arxiv.org/abs/2310.06927) Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goin, Dan Alistarh |<img width="1002" alt="image" src="figures/SquareHead.png"> |[Github](https://github.com/IST-DASLab/SparseFinetuning) [Paper](https://arxiv.org/abs/2310.06927)|

Programming Languages

Python 1

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

Awesome-Efficient-LLM

Efficient Architecture of LLM

Efficient MOE

Full List

Please check out all the papers by selecting the sub-area you're interested in. On this main page, only papers released in the past 90 days are shown.

Please check out all the papers by selecting the sub-area you're interested in. On this main page, we're showing papers released in the past 90 days.

Hardware

Hardware/System

Inference Acceleration

Knowledge Distillation

KV Cache Compression

Leaderboard

Low-Rank Decomposition

Network Pruning