Awesome-Efficient-LLM
A curated list for Efficient Large Language Models
https://github.com/horseee/Awesome-Efficient-LLM
Last synced: 12 days ago
JSON representation
Efficient Architecture of LLM
- Tandem Transformers for Inference Efficient LLMs
- Scaling Efficient LLMs
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
- Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding
- ![Star - oryx/MobiLlama)MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT Omkar Thawakar, Ashmal Vayani, Salman Khan, Hisham Cholakal, Rao M. Anwer, Michael Felsberg, Tim Baldwin, Eric P. Xing, Fahad Shahbaz Khan | |Github Paper Model |
- Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models - Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, Caglar Gulcehre | |Paper|
- ![Star
- ![Star
- ![Star - ICLR'24-blue)]()Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung-Woo Ha, Jinwoo Shin | |Github Paper|
Efficient MOE
- SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models
-  Róbert Csordás, Piotr Piękos, Kazuki Irie, Jürgen Schmidhuber | |Github Paper|
-  Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. (DK)Panda | |Github Paper|
-  Keisuke Kamahori, Yile Gu, Kan Zhu, Baris Kasikci | |Github Paper|
-  Harry Dong, Beidi Chen, Yuejie Chi | |Github Paper|
- Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
- SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
- Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
-  Chenyu Jiang, Ye Tian, Zhen Jia, Shuai Zheng, Chuan Wu, Yida Wang | |Paper|
Full List
Please check out all the papers by selecting the sub-area you're interested in. On this main page, only papers released in the past 90 days are shown.
Please check out all the papers by selecting the sub-area you're interested in. On this main page, we're showing papers released in the past 90 days.
Hardware
- ![Star - AILab/flash-attention) FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. Tri Dao. [Paper][Github]
- ![Star - Kelley. [Paper][Github]
- ![Star - IPADS/PowerInfer) PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU. Yixin Song, Zeyu Mi, Haotong Xie, Haibo Chen. [Paper][Github]
Hardware/System
-  FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré. [Paper][Github]
- . Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang. [Paper][Github]
-  Efficient Memory Management for Large Language Model Serving with PagedAttention. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica. [Paper][Github]
-  Efficient LLM Inference on CPUs. Haihao Shen, Hanwen Chang, Bo Dong, Yu Luo, Hengyu Meng. [Paper][Github]
- EdgeMoE: Fast On-Device Inference of MoE-based Large Language Models
- . Yonggan Fu, Yongan Zhang, Zhongzhi Yu, Sixu Li, Zhifan Ye, Chaojian Li, Cheng Wan, Yingyan Lin. [Paper]
- Rethinking Memory and Communication Cost for Efficient Large Language Model Training
- Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models
- FlashDecoding++: Faster Large Language Model Inference on GPUs
- LLM in a flash: Efficient Large Language Model Inference with Limited Memory
- FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGA
- Efficient LLM inference solution on Intel GPU - extension-for-pytorch/tree/v2.1.10%2Bxpu/examples/gpu/inference/python/llm)
- ![Star
- ![Star - MII) DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, Yuxiong He. [Paper][Github]
- . Taesu Kim, Jongho Lee, Daehyun Ahn, Sarang Kim, Jiwoong Choi, Minkyu Kim, Hyungjun Kim. [Paper][Github]
- BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences
- ![Star - project/sglang/tree/main) Efficiently Programming Large Language Models using SGLang. Lianmin Zheng*, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng*. [Paper] [Github]
- MELTing point: Mobile Evaluation of Language Transformers
- DeFT: Flash Tree-attention with IO-Awareness for Efficient Tree-search-based LLM Inference
- Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs
- LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism
- ![Star - griggs/melange-release) Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity. Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, Ion Stoica. [Paper][Github]
- Expert Router: Orchestrating Efficient Language Model Inference through Prompt Classification
- Efficient and Economic Large Language Model Inference with Attention Offloading
Inference Acceleration
-  Junyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, Yiming Qian | |Paper|
-  Ziqian Zeng, Yihuai Hong, Hongliang Dai, Huiping Zhuang, Cen Chen | |Paper|
- ![Publish
- ![Star - ICML'23%20Oral-blue)]()Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time Zichang Liu, Jue WANG, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, Beidi Chen | |Github Paper|
- ![Star - Yew Lin, Yuqing Yang, Lili Qiu | |Github Paper|
- ![Star - EMNLP'23-blue)]()Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding Sangmin Bae, Jongwoo Ko, Hwanjun Song, Se-Young Yun | |Github Paper|
- ![Star - EMNLP'23-blue)]()Compressing Context to Enhance Inference Efficiency of Large Language Models Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin | |Github Paper|
- Inference with Reference: Lossless Acceleration of Large Language Models
-  Murong Yue, Jie Zhao, Min Zhang, Liang Du, Ziyu Yao | |Github Paper|
- CacheGen: Fast Context Loading for Language Model Applications
- ![Star - EMNLP'23-blue)]()Context Compression for Auto-regressive Transformers with Sentinel Tokens Siyu Ren, Qi Jia, Kenny Q. Zhu | |Github Paper|
- ![Star - rankers)A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, Guido Zuccon | |Github Paper|
- SPEED: Speculative Pipelined Execution for Efficient Decoding
- Accelerating LLM Inference by Enabling Intermediate Layer Decoding
- Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster
- ![Star - mllab/context-memory)Compressed Context Memory For Online Language Model Interaction Jang-Hyun Kim, Junyoung Yeom, Sangdoo Yun, Hyun Oh Song | |Github Paper|
- SparQ Attention: Bandwidth-Efficient LLM Inference - Galley, Charlie Blake, Carlo Luschi, Douglas Orr | |Paper|
- Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy
- Cascade Speculative Drafting for Even Faster LLM Inference - Chuan Chang | |Paper|
- ![Star - offloading)Fast Inference of Mixture-of-Experts Language Models with Offloading Artyom Eliseev, Denis Mazur | |Github Paper|
- ![Star - NeurIPS'23-blue)]()H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen | |Github Paper|
-  Feng Lin, Hanling Yi, Hongbin Li, Yifan Yang, Xiaotian Yu, Guangming Lu, Rong Xiao | |Github Paper|
-  Jie Ou, Yueming Chen, Wenhong Tian | |Github Paper|
- Self-Selected Attention Span for Accelerating Large Language Model Inference
- Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration
- XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference - André Noël, Valentina Zantedeschi, David Vázquez, Nicolas Chapados, Christopher Pal, Perouz Taslakian | |Paper|
-  Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, Yunhe Wang | |Github Paper|
- Accelerating Speculative Decoding using Dynamic Speculation Length
- Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge
- 
Knowledge Distillation
- ![Star - CoT-Specialization)Specializing Smaller Language Models towards Multi-Step Reasoning Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, Tushar Khot | |Github Paper|
- ![Star - ACL'23%20Outstanding-blue)]()Distilling Script Knowledge from Large Language Models for Constrained Language Planning Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, Soham Shah, Charles Robert Jankowski, Yanghua Xiao, Deqing Yang | |Github Paper|
-  Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, Xiang Ren | |Paper|
- ![Star - ACL'23-blue)]()DISCO: Distilling Counterfactuals with Large Language Models Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, Kyle Richardson | |Github Paper|
- ![Star - ACL'23-blue)]()I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Lianhui Qin, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, Yejin Choi | |Github Paper Project |
- ![Star - ACL'23-blue)]()Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, Yejin Choi | |Github Paper|
- ![Star - NeurIPS'23-blue)]() Can Language Models Teach? Teacher Explanations Improve Student Performance via Theory of Mind Swarnadeep Saha, Peter Hase, and Mohit Bansal | |Github Paper|
- ![Star - EMNLP-2023)PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation Gaurav Sahu, Olga Vechtomova, Dzmitry Bahdanau, Issam H. Laradji | |Github Paper|
- ![Star - AAAI'24-blue)]()Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Bin Sun, Xinglin Wang, Heda Wang, Kan Li | |Github Paper|
- ![Star - ACL'23%20Industry%20Track-blue)]() GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model Shicheng Tan, Weng Lam Tam, Yuanchun Wang, Wenwen Gong, Yang Yang, Hongyin Tang, Keqing He, Jiahao Liu, Jingang Wang, Shu Zhao, Peng Zhang, Jie Tang | |Github Paper|
- ![Star - research/distilling-step-by-step) Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister | | Github Paper |
- ![Star - EMNLP'23%20Findings-blue)]()Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models Ilias Stogiannidis, Stavros Vassos, Prodromos Malakasiotis, Ion Androutsopoulos | |Github Paper|
- ![Star - nlp/LaMini-LM) LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, Alham Fikri Aji | | Github paper |
- Teaching Small Language Models to Reason
- ![Star - distill) Large Language Model Distillation Doesn't Need a Teacher Ananya Harsh Jha, Dirk Groeneveld, Emma Strubell, Iz Beltagy | | Github paper |
- The False Promise of Imitating Proprietary LLMs
- ![Star - distillation) Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, Yejin Choi | |Github paper |
- PaD: Program-aided Distillation Specializes Large Models in Reasoning
- RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment
- Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA - COT.png"> |Paper|
- ![Star - ner/universal-ner)UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, Hoifung Poon | |Github Paper Project |
- ![Star - Loup Tastet | |Github Paper | Model |
- DistillSpec: Improving Speculative Decoding via Knowledge Distillation - François Kagy, Rishabh Agarwal | |Paper|
- ![Star - handbook)Zephyr: Direct Distillation of LM Alignment Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, Thomas Wolf | |Github Paper|
- ![Star
- Unlock the Power: Competitive Distillation for Multi-Modal Large Language Models
- Mixed Distillation Helps Smaller Language Model Better Reasoning
- Distilling Event Sequence Knowledge From Large Language Models
- Knowledge Distillation for Closed-Source Language Models
- Improving Small Language Models' Mathematical Reasoning via Equation-of-Thought Distillation
- Scavenging Hyena: Distilling Transformers into Long Convolution Models - Transfer-HD.png"> |Paper|
- ![Star - Young Yun | |Github Paper|
- Large Language Model Meets Graph Neural Network in Knowledge Distillation
- ![Star - river/LLM_unlearning)Unmemorization in Large Language Models via Self-Distillation and Deliberate Imagination Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, Ivan Vulić | |Github Paper|
- ![Star - BZRD/llm-recipes)Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs Nicolas Boizard, Kevin El-Haddad, Céline Hudelot, Pierre Colombo | |Github Github Paper Model|
- Revisiting Knowledge Distillation for Autoregressive Language Models
- PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning
- Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
- Wisdom of Committee: Distilling from Foundation Model to Specialized Application Model
- Divide-or-Conquer? Which Part Should You Distill Your LLM?
- ![Star - cd)Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation Phuc Phan, Hieu Tran, Long Phan | |Github Paper|
- ![Star - to-Reason)Democratizing Reasoning Ability: Tailored Learning from Large Language Model Zhaoyang Wang, Shaohan Huang, Yuxuan Liu, Jiahai Wang, Minghui Song, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang | |Github Paper|
- Leveraging Zero-Shot Prompting for Efficient Language Model Distillation
- ![Publish
- ![Star - large-metaie)|
- Gecko: Versatile Text Embeddings Distilled from Large Language Models
- Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models - AKL.png"> |Paper Blog-Eng Blog-中|
- Post-Semantic-Thinking: A Robust Strategy to Distill Reasoning Capacity from Large Language Models
- Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation
- Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning
- ![Star - ACL'24-blue)]()RDRec: Rationale Distillation for LLM-based Recommendation Xinfeng Wang, Jin Cui, Yoshimi Suzuki, Fumiyo Fukumoto | |Github Paper|
KV Cache Compression
- WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More
- DB-LLM: Accurate Dual-Binarization for Efficient LLMs - LLM.png"> |Paper|
- No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
- ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition
- Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference
- ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
-  Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Yipeng Ma, Tao Wang, Ji-Rong Wen | |Github Paper|
- ![Star - ACL'24-blue)]()PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, Hai Zhao | |Github Paper|
- Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
- ![Star - ACL'24-blue)]()Layer-Condensed KV Cache for Efficient Inference of Large Language Models Haoyi Wu, Kewei Tu | |Github Paper|
Leaderboard
Low-Rank Decomposition
- ![Star - ICML'23-blue)]() LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, Tuo Zhao | |Github Paper|
- ![Star - compressor)Matrix Compression via Randomized Low Rank and Low Precision Factorization Rajarshi Saha, Varun Srivastava, Mert Pilanci | |Github Paper|
- TensorGPT: Efficient Compression of the Embedding Layer in LLMs based on the Tensor-Train Decomposition - SVD.png"> |Paper|
- LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression
- ![Star - rom)Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models Arnav Chavan, Nahush Lele, Deepak Gupta | |Github Paper|
- Data-free Weight Compress and Denoise for Large Language Models
- ![Star - MLSys-Lab/SVD-LLM)SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression Xin Wang, Yu Zheng, Zhongwei Wan, Mi Zhang | |Github Paper|
- ![Star - ACL'24%20Findings-blue)]()Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization Yixin Ji, Yang Xiang, Juntao Li, Wei Chen, Zhongyi Liu, Kehai Chen, Min Zhang | |Github Paper|
- 
Network Pruning
- ![Star - DASLab/sparsegpt) SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot Elias Frantar, Dan Alistarh| |Github paper |
- ![Star - Pruner) LLM-Pruner: On the Structural Pruning of Large Language Models Xinyin Ma, Gongfan Fang, Xinchao Wang | | Github paper|
- ![Star - Group/essential_sparsity) The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter Ajay Jaiswal, Shiwei Liu, Tianlong Chen, Zhangyang Wang | |Github Paper|
- ![Star - llm) Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, Shuaiwen Leon Song | |Github Paper|
- ![Star - Pruning-Official) NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models Jongwoo Ko, Seungjoon Park, Yujin Kim, Sumyeong Ahn, Du-Seong Chang, Euijai Ahn, Se-Young Yun | |Github Paper|
- ![Star - C2A4A6)]()A Simple and Effective Pruning Approach for Large Language Models Mingjie Sun, Zhuang Liu, Anna Bair, J. Zico Kolter | |Github Paper|
- ![Type
- ![Type - KICK.png"> |Paper|
- ![Star - Group/Junk_DNA_Hypothesis)Junk DNA Hypothesis: A Task-Centric Angle of LLM Pre-trained Weights through Sparsity Lu Yin, Shiwei Liu, Ajay Jaiswal, Souvik Kundu, Zhangyang Wang | |Github Paper|
- ![Star - C2A4A6)]() Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, Shiwei Liu | |Github Paper|
- 
