Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
Awesome-LLM-Inference
📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.
https://github.com/DefTruth/Awesome-LLM-Inference
Last synced: 3 days ago
JSON representation
-
📖Contents
-
📖KV Cache Scheduling/Quantize/Dropping ([©️back👆🏻](#paperlist))
- KV Cache Compress with LoRA - Context-Memory]](https://github.com/snu-mllab/Context-Memory)  |⭐️⭐️ |
- FASTDECODE
- Sparsity-Aware KV Caching
- **AdaKV**
- ThinK
- Palu
- ZipCache
- Chunked Prefills
- **Adaptive KV Cache Compress**
- CompressKV
- **DistKV-LLM**
- Prompt Caching
- **AlignedKV**
- **Shared Prefixes**
- SqueezeAttention
- Zero-Delay QKV Compression
- Less
- MiKV
- QAQ - KVCacheQuantization]](https://github.com/ClubieDong/QAQ-KVCacheQuantization)  |⭐️⭐️ |
- DMC
- Keyformer - matrix-ai/keyformer-llm) |⭐️⭐️ |
- GEAR - project/GEAR) |⭐️ |
- **ChunkAttention** - attention]](https://github.com/microsoft/chunk-attention)  |⭐️⭐️ |
- SnapKV
- KVCache-1Bit
- LTP
- **GQA**
- KV Cache Compress
- H2O
- **Adaptive KV Cache Compress**
- CacheGen
- Less
- MiKV
- FASTDECODE
- CacheGen
- Prompt Caching
- **GQA**
- MQA
- LTP
- KV Cache Compress
- H2O
- **LayerKV**
- QK-Sparse/Dropping Attention - sparse-flash-attention]](https://github.com/epfml/dynamic-sparse-flash-attention) |⭐️ |
- MQA
- **PagedAttention** - project/vllm) |⭐️⭐️ |
- DMC
- Keyformer - matrix-ai/keyformer-llm) |⭐️⭐️ |
- QAQ - KVCacheQuantization]](https://github.com/ClubieDong/QAQ-KVCacheQuantization)  |⭐️⭐️ |
- **RadixAttention** - project/sglang)  |⭐️⭐️ |
- KV-Runahead
- Chunked Prefills
- KV Cache FP8 + WINT4
- SqueezeAttention
- **Shared Prefixes**
- **TensorRT-LLM KV Cache FP8** - LLM]](https://github.com/NVIDIA/TensorRT-LLM)  |⭐️⭐️ |
- KV Cache Compress with LoRA - Context-Memory]](https://github.com/snu-mllab/Context-Memory)  |⭐️⭐️ |
- Sparsity-Aware KV Caching
- MiniCache
- CacheBlend
- MemServe
- QK-Sparse/Dropping Attention - sparse-flash-attention]](https://github.com/epfml/dynamic-sparse-flash-attention) |⭐️ |
- **PagedAttention** - project/vllm) |⭐️⭐️ |
- MLKV - mlkv]](https://github.com/zaydzuhri/pythia-mlkv) |⭐️ |
- vAttention
-
📖CPU/Single GPU/FPGA/Mobile Inference ([©️back👆🏻](#paperlist))
- FlightLLM
- LLM CPU Inference - extension-for-transformers]](https://github.com/intel/intel-extension-for-transformers)  |⭐️ |
- LinguaLinked
- FlexGen
- OpenVINO
-
📖LLM Train/Inference Framework ([©️back👆🏻](#paperlist))
- **Megatron-LM** - LM]](https://github.com/NVIDIA/Megatron-LM) |⭐️⭐️ |
- SpecInfer
- FastServe
- StreamingLLM - llm]](https://github.com/mit-han-lab/streaming-llm) |⭐️ |
- **PETALS** - workshop/petals) |⭐️⭐️ |
-
📖IO/FLOPs-Aware/Sparse Attention ([©️back👆🏻](#paperlist))
- FLOP, I/O
- **FlashAttention-2** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
- Flash-Decoding++
- SparseGPT - DASLab/sparsegpt)  |⭐️ |
- **CHESS**
- MoA - nics/MoA)  | ⭐️ |
- INT-FLASHATTENTION - FlashAttention]](https://github.com/INT-FlashAttention2024/INT-FlashAttention)  | ⭐️ |
- SCCA
- **FlashLLM**
- CHAI
- **FlashAttention** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
- Flash-Decoding++
- Online Softmax
- Hash Attention
- **FlashAttention** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
- Online Softmax
- **FlashLLM**
- **FlashAttention-2** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
- Online Softmax
- FlashAttention
- Online Softmax
- Hash Attention
- **GLA**
- CHAI
- **Flash-Decoding** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
- Flash Tree Attention
- SparseGPT - DASLab/sparsegpt)  |⭐️ |
- **GLA**
- SCCA
- DeFT
- **FlashAttention-3** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
- Shared Attention
-
📖Long Context Attention/KV Cache Optimization ([©️back👆🏻](#paperlist))
- **HyperAttention** - attn](https://github.com/insuhan/hyper-attn) |⭐️⭐️ |
- **Streaming Attention**
- **CLA**
- **Blockwise Attention**
- **StripedAttention** - forall/striped_attention/)  |⭐️⭐️ |
- **RingAttention**
- **LightningAttention-1**
- RAGCache
- LOOK-M - M]](https://github.com/SUSTechBruce/LOOK-M)  |⭐️⭐️ |
- **SentenceVAE**
- **YOCO** - YOCO]](https://github.com/microsoft/unilm/tree/master/YOCO)  |⭐️⭐️ |
- **Blockwise Attention**
- RAGCache
- **KCache**
- **Prompt Cache**
- **Streaming Attention**
- **Prompt Cache**
- **LightningAttention-1**
- **LightningAttention-2** - attention](https://github.com/OpenNLPLab/lightning-attention) |⭐️⭐️ |
- **RelayAttention**
- **RingAttention**
- **StripedAttention** - forall/striped_attention/)  |⭐️⭐️ |
- **RelayAttention**
- Landmark Attention - attention](https://github.com/epfml/landmark-attention/) |⭐️⭐️ |
- **RetrievalAttention**
- **InstInfer**
- **KVQuant**
- Landmark Attention - attention](https://github.com/epfml/landmark-attention/) |⭐️⭐️ |
- **LightningAttention-2** - attention](https://github.com/OpenNLPLab/lightning-attention) |⭐️⭐️ |
- **HyperAttention** - attn](https://github.com/insuhan/hyper-attn) |⭐️⭐️ |
- Infini-attention
- SKVQ
- **ShadowKV**
- **MInference**
- **InfiniGen**
- **Quest** - han-lab/Quest)  |⭐️⭐️ |
- PQCache
-
📖GEMM/Tensor Cores/WMMA/Parallel ([©️back👆🏻](#paperlist))
-
📖Parallel Decoding/Sampling ([©️back👆🏻](#paperlist))
- **OSD**
- **Cascade Speculative**
- Token Recycling
- **Hidden Transfer**
- **Medusa**
- LookaheadDecoding - ai-lab/LookaheadDecoding)  |⭐️⭐️ |
- **Speculative Decoding** - mad-dash/decoding-speculative-decoding)  |⭐️|
- **Hybrid Inference**
- **TriForce** - AI-Lab/TriForce) |⭐️⭐️ |
- **Parallel Decoding**
- **Speculative Sampling**
- **Speculative Sampling**
- **OSD**
- **Cascade Speculative**
- LookaheadDecoding - ai-lab/LookaheadDecoding)  |⭐️⭐️ |
- **Speculative Decoding**
- **Speculative Decoding**
- **Speculative Decoding** - lty/ParallelSpeculativeDecoding)  |⭐️⭐️ |
- **FocusLLM**
- **MagicDec** - AI-Lab/MagicDec/) |⭐️ |
- Multi-Token Speculative Decoding
- **Parallel Decoding**
- **Speculative Sampling**
- **Speculative Sampling**
- **Hidden Transfer**
- **PARALLELSPEC**
- **Fast Best-of-N**
- Instructive Decoding - Decoding]](https://github.com/joonkeekim/Instructive-Decoding) |⭐️ |
- S3D
-
📖Structured Prune/KD/Weight Sparse ([©️back👆🏻](#paperlist))
- **FLAP** - IVA-Lab/FLAP) |⭐️⭐️ |
- **LASER**
- **Admm Pruning** - pruning]](https://github.com/fmfi-compbio/admm-pruning) |⭐️ |
- FFSplit
- **LASER**
- PowerInfer - IPADS/PowerInfer) |⭐️ |
- **FLAP** - IVA-Lab/FLAP) |⭐️⭐️ |
-
📖Mixture-of-Experts(MoE) LLM Inference ([©️back👆🏻](#paperlist))
- **Mixtral Offloading** - offloading]](https://github.com/dvmazur/mixtral-offloading) |⭐️⭐️ |
- **WINT8/4**
- **Mixtral Offloading** - offloading]](https://github.com/dvmazur/mixtral-offloading) |⭐️⭐️ |
- MoE-Mamba
- MoE-Mamba
- **WINT8/4**
- MoE Inference
- DeepSeek-V2 - V2]](https://github.com/deepseek-ai/DeepSeek-V2) | ⭐️⭐️ |
- MoE
-
📖Non Transformer Architecture ([©️back👆🏻](#paperlist))
- **RWKV** - LM]](https://github.com/BlinkDL/RWKV-LM) |⭐️⭐️ |
- Kraken
- **RWKV-CLIP** - CLIP]](https://github.com/deepglint/RWKV-CLIP) |⭐️⭐️ |
- **RWKV** - LM]](https://github.com/BlinkDL/RWKV-LM) |⭐️⭐️ |
- **FLA** - linear-attention]](https://github.com/sustcsonglin/flash-linear-attention) |⭐️⭐️ |
- **Mamba** - spaces/mamba) |⭐️⭐️ |
-
📖Early-Exit/Intermediate Layer Decoding ([©️back👆🏻](#paperlist))
- FastBERT
- **SkipDecode**
- **EE-Tuning** - Tuning]](https://github.com/pan-x-c/EE-LLM)  |⭐️⭐️ |
- **KOALA**
- DeeBERT
- BERxiT
- **LITE**
- DeeBERT
- **LITE**
- **EE-LLM** - LLM]](https://github.com/pan-x-c/EE-LLM)  |⭐️⭐️ |
- **FREE**
- Skip Attention
- **EE-LLM** - LLM]](https://github.com/pan-x-c/EE-LLM)  |⭐️⭐️ |
- **FREE**
-
📖LLM Train/Inference Framework/Design ([©️back👆🏻](#paperlist))
- **LightLLM**
- **llama.cpp**
- **DeepSpeed-FastGen 2x vLLM?** - fastgen]](https://github.com/microsoft/DeepSpeed)  |⭐️⭐️ |
- inferflow
- **TensorRT-LLM** - LLM]](https://github.com/NVIDIA/TensorRT-LLM)  |⭐️⭐️ |
- DynamoLLM
- Medusa
- SpecInfer
- FastServe
- StreamingLLM - llm]](https://github.com/mit-han-lab/streaming-llm) |⭐️ |
- **PETALS** - workshop/petals) |⭐️⭐️ |
- **Decentralized LLM**
- NanoFlow
- **Megatron-LM** - LM]](https://github.com/NVIDIA/Megatron-LM) |⭐️⭐️ |
- **flashinfer** - ai/flashinfer) |⭐️⭐️ |
- **Mooncake** - ai/Mooncake) |⭐️⭐️ |
- **LMDeploy**
- **MLC-LLM** - llm]](https://github.com/mlc-ai/mlc-llm) |⭐️⭐️ |
-
📖GEMM/Tensor Cores/MMA/Parallel ([©️back👆🏻](#paperlist))
- **LUT TENSOR CORE**
- **SpMM**
- **TEE**
- QUICK
- Tensor Parallel
- Microbenchmark
- **HiFloat8**
- **Tensor Cores**
- Tensor Core
- Intra-SM Parallelism
- FP8
- Tensor Cores
- **MARLIN** - DASLab/marlin) |⭐️⭐️ |
- **cutlass/cute**
- **Tensor Product**
- **flute**
-
📖Prompt/Context/KV Compression ([©️back👆🏻](#paperlist))
- **Prompt Compression**
- **Context Distillation**
- **LLMLingua-2**
- **Eigen Attention**
- **AutoCompressor** - nlp/AutoCompressors) |⭐️ |
- **500xCompressor**
- **LongLLMLingua**
- **Selective-Context** - Context](https://github.com/liyucheng09/Selective_Context) |⭐️⭐️ |
- **LLMLingua**
- **KV-COMPRESS** - kvcompress](https://github.com/IsaacRe/vllm-kvcompress) |⭐️⭐️ |
- **LORC**
- **CRITIPREFILL**
-
📖Continuous/In-flight Batching ([©️back👆🏻](#paperlist))
- **DeepSpeed-FastGen 2x vLLM?** - fastgen]](https://github.com/microsoft/DeepSpeed)  |⭐️⭐️ |
- Automatic Inference Engine Tuning
- Splitwise
- SpotServe
- LightSeq
- **SJF Scheduling**
- **Continuous Batching**
- **In-flight Batching** - LLM]](https://github.com/NVIDIA/TensorRT-LLM)  |⭐️⭐️ |
- Splitwise
- SpotServe
- LightSeq
- **vTensor** - machine-learning/glake/tree/master/GLakeServe) |⭐️⭐️ |
-
📖Weight/Activation Quantize/Compress ([©️back👆🏻](#paperlist))
- I-LLM
- ABQ-LLM - LLM]](https://github.com/bytedance/ABQ-LLM) |⭐️ |
- FP6-LLM
- VPTQ
- **ZeroQuant**
- LLM.int8()
- **GPTQ** - DASLab/gptq) |⭐️⭐️ |
- 2-bit LLM
- SqueezeLLM
- ZeroQuant-FP
- **ZeroQuant**
- FP8-Quantization - quantization]](https://github.com/Qualcomm-AI-research/FP8-quantization)  |⭐️ |
- LLM.int8()
- **GPTQ** - DASLab/gptq) |⭐️⭐️ |
- **SmoothQuant** - han-lab/smoothquant) |⭐️⭐️ |
- ZeroQuant-V2
- **SmoothQuant+**
- OdysseyLLM W4A8
- CBQ
- QLLM
- **SmoothQuant** - han-lab/smoothquant) |⭐️⭐️ |
- ZeroQuant-V2
- Agile-Quant
- CBQ
- QLLM
- ACTIVATION SPARSITY
- 1-bit LLMs
- BitNet
- **SmoothQuant+**
- OdysseyLLM W4A8
- **SparQ**
- **AWQ** - awq]](https://github.com/mit-han-lab/llm-awq) |⭐️⭐️ |
- SpQR
- SqueezeLLM
- ZeroQuant-FP
- FP8-LM - AMP]](https://github.com/Azure/MS-AMP)  |⭐️ |
- LLM-Shearing - Shearing]](https://github.com/princeton-nlp/LLM-Shearing)  |⭐️ |
- LLM-FP4 - FP4]](https://github.com/nbasyl/LLM-FP4)  |⭐️ |
- 2-bit LLM
- **SparQ**
- Agile-Quant
- **W4A8KV4** - han-lab/qserve)  |⭐️⭐️ |
- FP8-Quantization - quantization]](https://github.com/Qualcomm-AI-research/FP8-quantization)  |⭐️ |
- SpinQuant
- OutlierTune
- FP8-LM - AMP]](https://github.com/Azure/MS-AMP)  |⭐️ |
- LLM-Shearing - Shearing]](https://github.com/princeton-nlp/LLM-Shearing)  |⭐️ |
- LLM-FP4 - FP4]](https://github.com/nbasyl/LLM-FP4)  |⭐️ |
- GPTQT
-
📖LLM Algorithmic/Eval Survey ([©️back👆🏻](#paperlist))
- Understanding LLMs
- LLM-Viewer - Viewer]](https://github.com/hahnyuan/LLM-Viewer)  |⭐️⭐️ |
- **Low-bit**
- Evaluating - LLMs-Evaluation]](https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers)  |⭐️ |
- **Runtime Performance**
- ChatGPT Anniversary
- Algorithmic Survey
- Security and Privacy
- **LLMCompass**
- Evaluating - LLMs-Evaluation]](https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers)  |⭐️ |
- **Runtime Performance**
- ChatGPT Anniversary
- Algorithmic Survey
- Security and Privacy
- **LLMCompass**
- **Efficient LLMs** - LLMs-Survey]](https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey)  |⭐️⭐️ |
- **Serving Survey**
- **Efficient LLMs** - LLMs-Survey]](https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey)  |⭐️⭐️ |
- **Serving Survey**
- LLM-Viewer - Viewer]](https://github.com/hahnyuan/LLM-Viewer)  |⭐️⭐️ |
- **LLM Inference**
- **Internal Consistency & Self-Feedback** - Survey]](https://github.com/IAAR-Shanghai/ICSFSurvey)  | ⭐️⭐️ |
-
📖CPU/Single GPU/FPGA/NPU/Mobile Inference ([©️back👆🏻](#paperlist))
- Summary
- LLM CPU Inference - extension-for-transformers]](https://github.com/intel/intel-extension-for-transformers)  |⭐️ |
- LinguaLinked
- FlightLLM
- OpenVINO
- FlexGen
- Transformer-Lite
- **FastAttention**
- **xFasterTransformer**
-
📖Prompt/Context Compression ([©️back👆🏻](#paperlist))
- **Selective-Context** - Context](https://github.com/liyucheng09/Selective_Context) |⭐️⭐️ |
- **AutoCompressor** - nlp/AutoCompressors) |⭐️ |
- **LLMLingua**
- **LLMLingua-2**
-
📖Trending LLM/VLM Topics ([©️back👆🏻](#paperlist))
- Open-Sora Plan - Sora-Plan]](https://github.com/PKU-YuanGroup/Open-Sora-Plan) | ⭐️⭐️ |
- Open-Sora - Sora]](https://github.com/hpcaitech/Open-Sora) | ⭐️⭐️ |
-
📖VLM/Position Embed/Others ([©️back👆🏻](#paperlist))
-
📖Position Embed/Others ([©️back👆🏻](#paperlist))
-
📖Data/Model/Pipeline/Tensor/Sequence/Context Parallelism ([©️back👆🏻](#paperlist))
- **Model Parallel**
- **Sequence Parallel** - LM]](https://github.com/NVIDIA/Megatron-LM) |⭐️⭐️ |
- **Sequence Parallel**
- **Context Parallel** - LM]](https://github.com/NVIDIA/Megatron-LM) |⭐️⭐️ |
- **Context Parallel**
-
-
📙Awesome LLM Inference Papers with Codes
-
📖Non Transformer Architecture ([©️back👆🏻](#paperlist))
-
📖Parallel Decoding/Sampling ([©️back👆🏻](#paperlist))
-
📖Structured Prune/KD/Weight Sparse ([©️back👆🏻](#paperlist))
- FFSplit
- **Admm Pruning** - pruning]](https://github.com/fmfi-compbio/admm-pruning) |⭐️ |
-
📖Long Context Attention/KV Cache Optimization ([©️back👆🏻](#paperlist))
-
📖LLM Train/Inference Framework ([©️back👆🏻](#paperlist))
- **DeepSpeed-FastGen 2x vLLM?** - fastgen]](https://github.com/microsoft/DeepSpeed)  |⭐️⭐️ |
- inferflow
-
📖KV Cache Scheduling/Quantize/Dropping ([©️back👆🏻](#paperlist))
-
📖Weight/Activation Quantize/Compress ([©️back👆🏻](#paperlist))
-
📖LLM Algorithmic/Eval Survey ([©️back👆🏻](#paperlist))
-
Sub Categories
📖KV Cache Scheduling/Quantize/Dropping ([©️back👆🏻](#paperlist))
65
📖Weight/Activation Quantize/Compress ([©️back👆🏻](#paperlist))
50
📖Long Context Attention/KV Cache Optimization ([©️back👆🏻](#paperlist))
38
📖IO/FLOPs-Aware/Sparse Attention ([©️back👆🏻](#paperlist))
32
📖Parallel Decoding/Sampling ([©️back👆🏻](#paperlist))
30
📖LLM Algorithmic/Eval Survey ([©️back👆🏻](#paperlist))
23
📖LLM Train/Inference Framework/Design ([©️back👆🏻](#paperlist))
18
📖GEMM/Tensor Cores/MMA/Parallel ([©️back👆🏻](#paperlist))
16
📖Early-Exit/Intermediate Layer Decoding ([©️back👆🏻](#paperlist))
14
📖Prompt/Context/KV Compression ([©️back👆🏻](#paperlist))
12
📖Continuous/In-flight Batching ([©️back👆🏻](#paperlist))
12
📖CPU/Single GPU/FPGA/NPU/Mobile Inference ([©️back👆🏻](#paperlist))
10
📖Structured Prune/KD/Weight Sparse ([©️back👆🏻](#paperlist))
9
📖Mixture-of-Experts(MoE) LLM Inference ([©️back👆🏻](#paperlist))
9
📖Non Transformer Architecture ([©️back👆🏻](#paperlist))
8
📖LLM Train/Inference Framework ([©️back👆🏻](#paperlist))
7
📖GEMM/Tensor Cores/WMMA/Parallel ([©️back👆🏻](#paperlist))
6
📖CPU/Single GPU/FPGA/Mobile Inference ([©️back👆🏻](#paperlist))
5
📖Data/Model/Pipeline/Tensor/Sequence/Context Parallelism ([©️back👆🏻](#paperlist))
5
📖VLM/Position Embed/Others ([©️back👆🏻](#paperlist))
4
📖Prompt/Context Compression ([©️back👆🏻](#paperlist))
4
📖Position Embed/Others ([©️back👆🏻](#paperlist))
2
📖Trending LLM/VLM Topics ([©️back👆🏻](#paperlist))
2