Awesome-LLM-Inference
📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉
https://github.com/xlite-dev/Awesome-LLM-Inference
Last synced: 3 days ago
JSON representation
-
📖Contents
-
📖KV Cache Scheduling/Quantize/Dropping ([©️back👆🏻](#paperlist))
- KV Cache Compress with LoRA - Context-Memory]](https://github.com/snu-mllab/Context-Memory)  |⭐️⭐️ |
- FASTDECODE
- Sparsity-Aware KV Caching
- H2O
- **AdaKV**
- **KV Cache Recomputation**
- **DynamicKV**
- ThinK
- Palu
- ZipCache
- **Adaptive KV Cache Compress**
- CompressKV
- **DistKV-LLM**
- Prompt Caching
- **AlignedKV**
- **Shared Prefixes**
- Zero-Delay QKV Compression
- SqueezeAttention
- Less
- MiKV
- QAQ - KVCacheQuantization]](https://github.com/ClubieDong/QAQ-KVCacheQuantization)  |⭐️⭐️ |
- DMC
- SnapKV
- **ChunkAttention** - attention]](https://github.com/microsoft/chunk-attention)  |⭐️⭐️ |
- GEAR - project/GEAR) |⭐️ |
- KVCache-1Bit
- LTP
- KV Cache Compress
- **Adaptive KV Cache Compress**
- CacheGen
- Less
- MiKV
- FASTDECODE
- CacheGen
- Prompt Caching
- KV Cache Compress
- H2O
- LTP
- MQA
- **GQA**
- **LayerKV**
- DMC
- Keyformer - matrix-ai/keyformer-llm) |⭐️⭐️ |
- QAQ - KVCacheQuantization]](https://github.com/ClubieDong/QAQ-KVCacheQuantization)  |⭐️⭐️ |
- KV-Runahead
- Chunked Prefills
- **ClusterKV**
- Chunked Prefills
- **Shared Prefixes**
- SqueezeAttention
- **TensorRT-LLM KV Cache FP8** - LLM]](https://github.com/NVIDIA/TensorRT-LLM)  |⭐️⭐️ |
- Sparsity-Aware KV Caching
- MiniCache
- CacheBlend
- MemServe
- QK-Sparse/Dropping Attention - sparse-flash-attention]](https://github.com/epfml/dynamic-sparse-flash-attention) |⭐️ |
- **PagedAttention** - project/vllm) |⭐️⭐️ |
- MLKV - mlkv]](https://github.com/zaydzuhri/pythia-mlkv) |⭐️ |
- **KV Cache Prefetch**
- **CacheCraft**
- Keyformer - matrix-ai/keyformer-llm) |⭐️⭐️ |
- **KVzip** - mllab/KVzip) |⭐️⭐️|
- MQA
- **PagedAttention** - project/vllm) |⭐️⭐️ |
- KV Cache FP8 + WINT4
- **RadixAttention** - project/sglang)  |⭐️⭐️ |
- KV Cache Compress with LoRA - Context-Memory]](https://github.com/snu-mllab/Context-Memory)  |⭐️⭐️ |
- **Inference-Time Hyper-Scaling**
- **GQA**
- QK-Sparse/Dropping Attention - sparse-flash-attention]](https://github.com/epfml/dynamic-sparse-flash-attention) |⭐️ |
- vAttention
-
📖CPU/Single GPU/FPGA/Mobile Inference ([©️back👆🏻](#paperlist))
- FlightLLM
- LLM CPU Inference - extension-for-transformers]](https://github.com/intel/intel-extension-for-transformers)  |⭐️ |
- LinguaLinked
- FlexGen
- OpenVINO
-
📖LLM Train/Inference Framework ([©️back👆🏻](#paperlist))
- SpecInfer
- FastServe
- StreamingLLM - llm]](https://github.com/mit-han-lab/streaming-llm) |⭐️ |
- **PETALS** - workshop/petals) |⭐️⭐️ |
-
📖Long Context Attention/KV Cache Optimization ([©️back👆🏻](#paperlist))
- **HyperAttention** - attn](https://github.com/insuhan/hyper-attn) |⭐️⭐️ |
- **Streaming Attention**
- **CLA**
- **RingAttention**
- **StripedAttention** - forall/striped_attention/)  |⭐️⭐️ |
- **LightningAttention-1**
- RAGCache
- LOOK-M - M]](https://github.com/SUSTechBruce/LOOK-M)  |⭐️⭐️ |
- **SentenceVAE**
- **Blockwise Attention**
- RAGCache
- **KCache**
- **Prompt Cache**
- **Streaming Attention**
- **Prompt Cache**
- **LightningAttention-1**
- **LightningAttention-2** - attention](https://github.com/OpenNLPLab/lightning-attention) |⭐️⭐️ |
- **RelayAttention**
- Landmark Attention - attention](https://github.com/epfml/landmark-attention/) |⭐️⭐️ |
- **InstInfer**
- **RetrievalAttention**
- **KVQuant**
- Landmark Attention - attention](https://github.com/epfml/landmark-attention/) |⭐️⭐️ |
- **HyperAttention** - attn](https://github.com/insuhan/hyper-attn) |⭐️⭐️ |
- Infini-attention
- SKVQ
- **ShadowKV**
- **InfiniGen**
- **Quest** - han-lab/Quest)  |⭐️⭐️ |
- PQCache
- **HOMER**
- **REFORM**
- **LightningAttention-2** - attention](https://github.com/OpenNLPLab/lightning-attention) |⭐️⭐️ |
- **RingAttention**
- **StripedAttention** - forall/striped_attention/)  |⭐️⭐️ |
- **YOCO** - YOCO]](https://github.com/microsoft/unilm/tree/master/YOCO)  |⭐️⭐️ |
- **MInference**
- **Lightning Attention** - 01]](https://github.com/MiniMax-AI/MiniMax-01)  | ⭐️⭐️ |
- **RelayAttention**
-
📖IO/FLOPs-Aware/Sparse Attention ([©️back👆🏻](#paperlist))
- FLOP, I/O
- **FlashAttention-2** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
- Flash-Decoding++
- SparseGPT - DASLab/sparsegpt)  |⭐️ |
- **Sparse Frontier** - frontier)  | ⭐️⭐️ |
- **TurboAttention**
- **CHESS**
- MoA - nics/MoA)  | ⭐️ |
- INT-FLASHATTENTION - FlashAttention]](https://github.com/INT-FlashAttention2024/INT-FlashAttention)  | ⭐️ |
- SCCA
- **FlashLLM**
- CHAI
- **FlashAttention** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
- Online Softmax
- **FlashAttention** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
- Hash Attention
- Online Softmax
- **FlashAttention-2** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
- Flash-Decoding++
- **FlashLLM**
- Online Softmax
- Online Softmax
- Hash Attention
- **GLA**
- CHAI
- **Flash-Decoding** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
- Flash Tree Attention
- **GLA**
- SCCA
- **Flex Attention** - gym]](https://github.com/pytorch-labs/attention-gym)  | ⭐️⭐️ |
- DeFT
- **FFPA** - attn-mma]](https://github.com/DefTruth/ffpa-attn-mma) |⭐️⭐️ |
- **Slim attention** - ai/transformer-tricks)  | ⭐️⭐️⭐️ |
- **SeerAttention**
- Shared Attention
- **Squeezed Attention**
- **SpargeAttention** - ml/SpargeAttn)  | ⭐️⭐️ |
- **FFPA** - attn-mma]](https://github.com/xlite-dev/ffpa-attn-mma) |⭐️⭐️ |
- **FFPA** - attn]](https://github.com/xlite-dev/ffpa-attn) |⭐️⭐️ |
- **SageAttention-3** - ml/SageAttention)  | ⭐️⭐️ |
- **Parallel Encoding** - AI-Lab/APE)  | ⭐️⭐️ |
- **Parallel Encoding** - attention]](https://github.com/TemporaryLoRA/Block-attention)  | ⭐️⭐️ |
- **MMInference**
- FlashAttention
- SparseGPT - DASLab/sparsegpt)  |⭐️ |
- **SageAttention** - ml/SageAttention)  | ⭐️⭐️ |
- **SageAttention-2** - ml/SageAttention)  | ⭐️⭐️ |
- **FlashAttention-3** - attention]](https://github.com/Dao-AILab/flash-attention) |⭐️⭐️ |
-
📖GEMM/Tensor Cores/WMMA/Parallel ([©️back👆🏻](#paperlist))
-
📖Structured Prune/KD/Weight Sparse ([©️back👆🏻](#paperlist))
- **FLAP** - IVA-Lab/FLAP) |⭐️⭐️ |
- **LASER**
- SDMPrune
- FFSplit
- **LASER**
- **FLAP** - IVA-Lab/FLAP) |⭐️⭐️ |
- PowerInfer - IPADS/PowerInfer) |⭐️ |
- **Simba**
- **Admm Pruning** - pruning]](https://github.com/fmfi-compbio/admm-pruning) |⭐️ |
-
📖Parallel Decoding/Sampling ([©️back👆🏻](#paperlist))
- **OSD**
- **Cascade Speculative**
- Token Recycling
- **Medusa**
- **TriForce** - AI-Lab/TriForce) |⭐️⭐️ |
- **Cascade Speculative**
- LookaheadDecoding - ai-lab/LookaheadDecoding)  |⭐️⭐️ |
- **Speculative Decoding**
- **Speculative Decoding**
- **Speculative Decoding** - lty/ParallelSpeculativeDecoding)  |⭐️⭐️ |
- **FocusLLM**
- **MagicDec** - AI-Lab/MagicDec/) |⭐️ |
- **Parallel Decoding**
- **Speculative Sampling**
- **Speculative Sampling**
- **Hidden Transfer**
- **Hidden Transfer**
- **PARALLELSPEC**
- **Fast Best-of-N**
- Instructive Decoding - Decoding]](https://github.com/joonkeekim/Instructive-Decoding) |⭐️ |
- **STAND**
- **Mamba Drafters**
- **Parallel Decoding**
- **Speculative Sampling**
- **Speculative Sampling**
- **OSD**
- LookaheadDecoding - ai-lab/LookaheadDecoding)  |⭐️⭐️ |
- **Speculative Decoding** - mad-dash/decoding-speculative-decoding)  |⭐️|
- S3D
- **Hybrid Inference**
- Multi-Token Speculative Decoding
-
📖Mixture-of-Experts(MoE) LLM Inference ([©️back👆🏻](#paperlist))
- **Mixtral Offloading** - offloading]](https://github.com/dvmazur/mixtral-offloading) |⭐️⭐️ |
- **Mixtral Offloading** - offloading]](https://github.com/dvmazur/mixtral-offloading) |⭐️⭐️ |
- MoE-Mamba
- **WINT8/4**
- MoE Inference
- **WINT8/4**
- MoE
- MoE-Mamba
- DeepSeek-V2 - V2]](https://github.com/deepseek-ai/DeepSeek-V2) | ⭐️⭐️ |
-
📖Non Transformer Architecture ([©️back👆🏻](#paperlist))
- **RWKV** - LM]](https://github.com/BlinkDL/RWKV-LM) |⭐️⭐️ |
- Kraken
- **RWKV-CLIP** - CLIP]](https://github.com/deepglint/RWKV-CLIP) |⭐️⭐️ |
- **RWKV** - LM]](https://github.com/BlinkDL/RWKV-LM) |⭐️⭐️ |
- **Mamba** - spaces/mamba) |⭐️⭐️ |
- **FLA** - linear-attention]](https://github.com/sustcsonglin/flash-linear-attention) |⭐️⭐️ |
-
📖Early-Exit/Intermediate Layer Decoding ([©️back👆🏻](#paperlist))
- FastBERT
- **SkipDecode**
- **EE-Tuning** - Tuning]](https://github.com/pan-x-c/EE-LLM)  |⭐️⭐️ |
- **KOALA**
- BERxiT
- DeeBERT
- **LITE**
- DeeBERT
- **EE-LLM** - LLM]](https://github.com/pan-x-c/EE-LLM)  |⭐️⭐️ |
- **FREE**
- **LITE**
- **EE-LLM** - LLM]](https://github.com/pan-x-c/EE-LLM)  |⭐️⭐️ |
- **FREE**
- Skip Attention
-
📖Weight/Activation Quantize/Compress ([©️back👆🏻](#paperlist))
- **BitNet v2**
- I-LLM
- ABQ-LLM - LLM]](https://github.com/bytedance/ABQ-LLM) |⭐️ |
- FP6-LLM
- VPTQ
- **ZeroQuant**
- LLM.int8()
- **GPTQ** - DASLab/gptq) |⭐️⭐️ |
- 2-bit LLM
- SqueezeLLM
- ZeroQuant-FP
- **ZeroQuant**
- LLM.int8()
- **SmoothQuant** - han-lab/smoothquant) |⭐️⭐️ |
- ZeroQuant-V2
- OdysseyLLM W4A8
- **SmoothQuant+**
- CBQ
- **SmoothQuant** - han-lab/smoothquant) |⭐️⭐️ |
- ZeroQuant-V2
- Agile-Quant
- CBQ
- QLLM
- ACTIVATION SPARSITY
- 1-bit LLMs
- BitNet
- **SmoothQuant+**
- OdysseyLLM W4A8
- **SparQ**
- **AWQ** - awq]](https://github.com/mit-han-lab/llm-awq) |⭐️⭐️ |
- SpQR
- SqueezeLLM
- ZeroQuant-FP
- FP8-LM - AMP]](https://github.com/Azure/MS-AMP)  |⭐️ |
- LLM-Shearing - Shearing]](https://github.com/princeton-nlp/LLM-Shearing)  |⭐️ |
- LLM-FP4 - FP4]](https://github.com/nbasyl/LLM-FP4)  |⭐️ |
- **SparQ**
- Agile-Quant
- **W4A8KV4** - han-lab/qserve)  |⭐️⭐️ |
- 2-bit LLM
- FP8-Quantization - quantization]](https://github.com/Qualcomm-AI-research/FP8-quantization)  |⭐️ |
- SpinQuant
- OutlierTune
- FP8-LM - AMP]](https://github.com/Azure/MS-AMP)  |⭐️ |
- LLM-Shearing - Shearing]](https://github.com/princeton-nlp/LLM-Shearing)  |⭐️ |
- LLM-FP4 - FP4]](https://github.com/nbasyl/LLM-FP4)  |⭐️ |
- GPTQT
- **GuidedQuant** - mllab/GuidedQuant) |⭐️⭐️ |
- FP8-Quantization - quantization]](https://github.com/Qualcomm-AI-research/FP8-quantization)  |⭐️ |
- **GPTQ** - DASLab/gptq) |⭐️⭐️ |
- QLLM
-
📖CPU/Single GPU/FPGA/NPU/Mobile Inference ([©️back👆🏻](#paperlist))
- **NITRO** - lab/nitro) |⭐️ |
- Summary
- FlightLLM
- Transformer-Lite
- **FastAttention**
- **xFasterTransformer**
- FlexGen
- LLM CPU Inference - extension-for-transformers]](https://github.com/intel/intel-extension-for-transformers)  |⭐️ |
- LinguaLinked
- OpenVINO
-
📖GEMM/Tensor Cores/MMA/Parallel ([©️back👆🏻](#paperlist))
- **FLASH-ATTENTION RNG**
- **SpMM**
- **TEE**
- Microbenchmark
- Tensor Parallel
- **TRITONBENCH**
- **HiFloat8**
- **Tensor Cores**
- Tensor Core
- FP8
- Tensor Cores
- **MARLIN** - DASLab/marlin) |⭐️⭐️ |
- **cutlass/cute**
- QUICK
- **flute**
- **Triton-distributed** - distributed]](https://github.com/ByteDance-Seed/Triton-distributed) |⭐️⭐️ |
- Intra-SM Parallelism
- **LUT TENSOR CORE**
- **HADACORE** - labs/applied-ai/tree/main/kernels/cuda/inference/hadamard_transform) |⭐️ |
-
📖DeepSeek/Multi-head Latent Attention(MLA) ([©️back👆🏻](#paperlist))
- **TransMLA**
- **X-EcoMLA**
- **DeepSeek-NSA**
- **FlashMLA** - ai/FlashMLA.svg?style=social) |⭐️⭐️ |
- **DualPipe** - ai/DualPipe.svg?style=social) |⭐️⭐️ |
- **DeepGEMM** - ai/DeepGEMM.svg?style=social) |⭐️⭐️ |
- **DeepEP** - ai/DeepEP.svg?style=social) |⭐️⭐️ |
- **EPLB** - ai/EPLB.svg?style=social) |⭐️⭐️ |
- **3FS** - ai/3FS.svg?style=social) |⭐️⭐️ |
- **推理系统**
- **MHA2MLA** - Ushio/MHA2MLA)  |⭐️⭐️ |
- **DeepSeek-R1** - R1]](https://github.com/deepseek-ai/DeepSeek-R1)  | ⭐️⭐️ |
-
📖LLM Train/Inference Framework/Design ([©️back👆🏻](#paperlist))
- **LightLLM**
- **siiRL** - research/siiRL)<br> |⭐️⭐️ |
- **DeepSpeed-FastGen 2x vLLM?** - fastgen]](https://github.com/microsoft/DeepSpeed)  |⭐️⭐️ |
- **TensorRT-LLM** - LLM]](https://github.com/NVIDIA/TensorRT-LLM)  |⭐️⭐️ |
- DynamoLLM
- Medusa
- FastServe
- SpecInfer
- **PETALS** - workshop/petals) |⭐️⭐️ |
- StreamingLLM - llm]](https://github.com/mit-han-lab/streaming-llm) |⭐️ |
- **Decentralized LLM**
- NanoFlow
- **flashinfer** - ai/flashinfer) |⭐️⭐️ |
- **MLC-LLM** - llm]](https://github.com/mlc-ai/mlc-llm) |⭐️⭐️ |
- **LMDeploy**
- **SparseInfer**
- prima.cpp
- **Megatron-LM** - LM]](https://github.com/NVIDIA/Megatron-LM) |⭐️⭐️ |
- inferflow
- **llama.cpp**
-
📖Prompt/Context/KV Compression ([©️back👆🏻](#paperlist))
- **Prompt Compression**
- **Context Distillation**
- **Eigen Attention**
- **AutoCompressor** - nlp/AutoCompressors) |⭐️ |
- **500xCompressor**
- **LongLLMLingua**
- **LLMLingua**
- **KV-COMPRESS** - kvcompress](https://github.com/IsaacRe/vllm-kvcompress) |⭐️⭐️ |
- **LLMLingua-2**
- **LORC**
- **CRITIPREFILL**
- **Selective-Context** - Context](https://github.com/liyucheng09/Selective_Context) |⭐️⭐️ |
-
📖Continuous/In-flight Batching ([©️back👆🏻](#paperlist))
- **DeepSpeed-FastGen 2x vLLM?** - fastgen]](https://github.com/microsoft/DeepSpeed)  |⭐️⭐️ |
- Automatic Inference Engine Tuning
- Splitwise
- SpotServe
- **SJF Scheduling**
- Splitwise
- SpotServe
- **Continuous Batching**
- **In-flight Batching** - LLM]](https://github.com/NVIDIA/TensorRT-LLM)  |⭐️⭐️ |
- **BatchLLM**
- LightSeq
- LightSeq
- **vTensor** - machine-learning/glake/tree/master/GLakeServe) |⭐️⭐️ |
- **DeepSpeed-FastGen 2x vLLM?** - fastgen]](https://github.com/microsoft/DeepSpeed)  |⭐️⭐️ |
-
📖Multi-GPUs/Multi-Nodes Parallelism ([©️back👆🏻](#paperlist))
- **SP: BPT**
- **SP: DEEPSPEED ULYSSES**
- **CP: Megatron-LM** - LM]](https://github.com/NVIDIA/Megatron-LM) |⭐️⭐️ |
- **CP: Meta**
- **SP: Megatron-LM** - LM]](https://github.com/NVIDIA/Megatron-LM) |⭐️⭐️ |
- **TP: Comm Compression**
- **SP: TokenRing** - ring]](https://github.com/ACA-Lab-SJTU/token-ring) |⭐️⭐️ |
- **MP: ZeRO**
- **FSDP 1/2**
- **SP: Star-Attention, 11x~ speedup** - Attention]](https://github.com/NVIDIA/Star-Attention) |⭐️⭐️ |
-
📖LLM Algorithmic/Eval Survey ([©️back👆🏻](#paperlist))
- Understanding LLMs
- LLM-Viewer - Viewer]](https://github.com/hahnyuan/LLM-Viewer)  |⭐️⭐️ |
- **Low-bit**
- Evaluating - LLMs-Evaluation]](https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers)  |⭐️ |
- **Runtime Performance**
- ChatGPT Anniversary
- Algorithmic Survey
- Security and Privacy
- **LLMCompass**
- Evaluating - LLMs-Evaluation]](https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers)  |⭐️ |
- **Runtime Performance**
- ChatGPT Anniversary
- Algorithmic Survey
- **LLMCompass**
- Security and Privacy
- **Efficient LLMs** - LLMs-Survey]](https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey)  |⭐️⭐️ |
- **Serving Survey**
- **Efficient LLMs** - LLMs-Survey]](https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey)  |⭐️⭐️ |
- **Serving Survey**
- LLM-Viewer - Viewer]](https://github.com/hahnyuan/LLM-Viewer)  |⭐️⭐️ |
- **LLM Inference**
- **Internal Consistency & Self-Feedback** - Survey]](https://github.com/IAAR-Shanghai/ICSFSurvey)  | ⭐️⭐️ |
-
📖Trending LLM/VLM Topics ([©️back👆🏻](#paperlist))
- **Mooncake** - ai/Mooncake) |⭐️⭐️ |
- Open-Sora Plan - Sora-Plan]](https://github.com/PKU-YuanGroup/Open-Sora-Plan) | ⭐️⭐️ |
- Open-Sora - Sora]](https://github.com/hpcaitech/Open-Sora) | ⭐️⭐️ |
-
📖Disaggregating Prefill and Decoding ([©️back👆🏻](#paperlist))
-
📖Prompt/Context Compression ([©️back👆🏻](#paperlist))
- **LLMLingua**
- **Selective-Context** - Context](https://github.com/liyucheng09/Selective_Context) |⭐️⭐️ |
- **AutoCompressor** - nlp/AutoCompressors) |⭐️ |
- **LLMLingua-2**
-
📖VLM/Position Embed/Others ([©️back👆🏻](#paperlist))
-
📖Position Embed/Others ([©️back👆🏻](#paperlist))
-
-
📙Awesome LLM Inference Papers with Codes
-
📖Parallel Decoding/Sampling ([©️back👆🏻](#paperlist))
-
📖Structured Prune/KD/Weight Sparse ([©️back👆🏻](#paperlist))
- FFSplit
- **Admm Pruning** - pruning]](https://github.com/fmfi-compbio/admm-pruning) |⭐️ |
-
📖Non Transformer Architecture ([©️back👆🏻](#paperlist))
- **Mamba** - spaces/mamba) |⭐️⭐️ |
-
📖Long Context Attention/KV Cache Optimization ([©️back👆🏻](#paperlist))
-
📖LLM Train/Inference Framework ([©️back👆🏻](#paperlist))
- **DeepSpeed-FastGen 2x vLLM?** - fastgen]](https://github.com/microsoft/DeepSpeed)  |⭐️⭐️ |
- inferflow
-
📖KV Cache Scheduling/Quantize/Dropping ([©️back👆🏻](#paperlist))
-
📖Weight/Activation Quantize/Compress ([©️back👆🏻](#paperlist))
-
📖LLM Algorithmic/Eval Survey ([©️back👆🏻](#paperlist))
-
-
🎉Awesome LLM Inference Papers with Codes
-
📖 News 🔥🔥
- 2025-08-18 - free Cache Acceleration Toolbox for DiTs: Cache Acceleration with One-line Code ~ ♥️. Feel free to take a try!
- 2025-07-13 - fast](https://github.com/huggingface/flux-fast) that **makes flux-fast even faster** with **[cache-dit](https://github.com/vipshop/cache-dit)**, **3.3x** speedup on NVIDIA L20 while still maintaining **high precision**.
Categories
Sub Categories
📖KV Cache Scheduling/Quantize/Dropping ([©️back👆🏻](#paperlist))
72
📖Weight/Activation Quantize/Compress ([©️back👆🏻](#paperlist))
52
📖IO/FLOPs-Aware/Sparse Attention ([©️back👆🏻](#paperlist))
48
📖Long Context Attention/KV Cache Optimization ([©️back👆🏻](#paperlist))
40
📖Parallel Decoding/Sampling ([©️back👆🏻](#paperlist))
32
📖LLM Algorithmic/Eval Survey ([©️back👆🏻](#paperlist))
23
📖LLM Train/Inference Framework/Design ([©️back👆🏻](#paperlist))
20
📖GEMM/Tensor Cores/MMA/Parallel ([©️back👆🏻](#paperlist))
19
📖Continuous/In-flight Batching ([©️back👆🏻](#paperlist))
14
📖Early-Exit/Intermediate Layer Decoding ([©️back👆🏻](#paperlist))
14
📖Prompt/Context/KV Compression ([©️back👆🏻](#paperlist))
12
📖DeepSeek/Multi-head Latent Attention(MLA) ([©️back👆🏻](#paperlist))
12
📖Structured Prune/KD/Weight Sparse ([©️back👆🏻](#paperlist))
11
📖CPU/Single GPU/FPGA/NPU/Mobile Inference ([©️back👆🏻](#paperlist))
11
📖Multi-GPUs/Multi-Nodes Parallelism ([©️back👆🏻](#paperlist))
10
📖Mixture-of-Experts(MoE) LLM Inference ([©️back👆🏻](#paperlist))
9
📖Non Transformer Architecture ([©️back👆🏻](#paperlist))
7
📖GEMM/Tensor Cores/WMMA/Parallel ([©️back👆🏻](#paperlist))
6
📖LLM Train/Inference Framework ([©️back👆🏻](#paperlist))
6
📖CPU/Single GPU/FPGA/Mobile Inference ([©️back👆🏻](#paperlist))
5
📖VLM/Position Embed/Others ([©️back👆🏻](#paperlist))
5
📖Disaggregating Prefill and Decoding ([©️back👆🏻](#paperlist))
4
📖Prompt/Context Compression ([©️back👆🏻](#paperlist))
4
📖Trending LLM/VLM Topics ([©️back👆🏻](#paperlist))
3
📖Position Embed/Others ([©️back👆🏻](#paperlist))
2
Keywords
mlsys
3
sdpa
3
tensor-cores
3
flash-attention
3
cuda
3
attention
3
deepseek
2
deepseek-r1
2
deepseek-v3
2
flash-mla
2
fused-mla
2
mla
2
openai-triton
1
nlp
1
model-serving
1
llm
1
llama
1
gpt
1
deep-learning
1
large-language-models
1
machine-learning-systems
1
natural-language-processing
1
acceleration
1
cogvideox
1
diffusion
1
dit
1
flux
1
transformers
1
wan
1