Paper-Code

Attention

• ![Star - flash-attention.git) | - | |

Communication

• ARK: GPU-driven Code Execution for Distributed Deep Learning NSDI'23
• ARK: GPU-driven Code Execution for Distributed Deep Learning NSDI'23
• TopoOpt: Optimizing the Network Topology for Distributed DNN Training NSDI'23
• TopoOpt: Optimizing the Network Topology for Distributed DNN Training NSDI'23
• Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads ASPLOS'22
• Efficient Sparse Collective Communication and its application to Accelerate Distributed Deep Learning SIGCOMM'21
• Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads ASPLOS'22
• Efficient Sparse Collective Communication and its application to Accelerate Distributed Deep Learning SIGCOMM'21

Energy

• Zeus: Understanding and Optimizing {GPU} Energy Consumption of {DNN} Training NSDI'23
• EnvPipe: Performance-preserving DNN Training Framework for Saving Energy ATC'23
• Zeus: Understanding and Optimizing {GPU} Energy Consumption of {DNN} Training NSDI'23
• EnvPipe: Performance-preserving DNN Training Framework for Saving Energy ATC'23

Fancy LLM

• LLMCompiler: An LLM Compiler for Parallel Function Calling 23arxiv
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
• EAGLE: Lossless Acceleration of LLM Decoding by Feature Extrapolation
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Fine-Tune

• Fine-tuning giant neural networks on commodity hardware with automatic pipeline model parallelism ATC'21
• Fine-tuning giant neural networks on commodity hardware with automatic pipeline model parallelism ATC'21

Framework

• Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning OSDI'22
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM SC21
• A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters OSDI'20
• HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework VLDB'22
• Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning OSDI'22
• A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters OSDI'20
• Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training ICPP'23
• HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework VLDB'22
• Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training ICPP'23

GNN

• Legion: Automatically Pushing the Envelope of Multi-GPU System for Billion-Scale GNN Training ATC'23
• TC-GNN: Accelerating Sparse Graph Neural Network Computation Via Dense Tensor Core on GPUs ATC'23
• Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platforms OSDI'23
• GNNAdvisor: An Efficient Runtime System for GNN Acceleration on GPUs OSDI'21
• Marius: Learning Massive Graph Embeddings on a Single Machine OSDI'21
• Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads OSDI'21
• BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling MLSYS'22
• Accelerating Large Scale Real-Time GNN Inference Using Channel Pruning VLDB'21
• gSampler: Efficient GPU-Based Graph Sampling for Graph Learning SOSP'23
• Legion: Automatically Pushing the Envelope of Multi-GPU System for Billion-Scale GNN Training ATC'23
• TC-GNN: Accelerating Sparse Graph Neural Network Computation Via Dense Tensor Core on GPUs ATC'23
• Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platforms OSDI'23
• CoGNN: Efficient Scheduling for Concurrent GNN Training on GPUs SC'22
• GNNAdvisor: An Efficient Runtime System for GNN Acceleration on GPUs OSDI'21
• Marius: Learning Massive Graph Embeddings on a Single Machine OSDI'21
• Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads OSDI'21
• BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling MLSYS'22
• Accelerating Large Scale Real-Time GNN Inference Using Channel Pruning VLDB'21
• Reducing Communication in Graph Neural Network Training SC'20
• gSampler: Efficient GPU-Based Graph Sampling for Graph Learning SOSP'23
• CoGNN: Efficient Scheduling for Concurrent GNN Training on GPUs SC'22
• BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling MLSYS'22
• Accelerating Large Scale Real-Time GNN Inference Using Channel Pruning VLDB'21
• Reducing Communication in Graph Neural Network Training SC'20
• awesome GNN

GPU Cluster Management

• Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning NSDI'23
• Synergy : Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters OSDI'22
• Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning OSDI'21
• Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads OSDI'20
• Tiresias -- A GPU Cluster Manager for Distributed Deep Learning Training without complete job information NSDI'19
• Chronus: A Novel Deadline-aware Scheduler for Deep Learning Training Jobs SOCC'21
• Lucid: A Non-Intrusive, Scalable and Interpretable Scheduler for Deep Learning Training Jobs ASPLOS'23
• Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning NSDI'23
• Synergy : Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters OSDI'22
• Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning OSDI'21
• Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads OSDI'20
• Tiresias -- A GPU Cluster Manager for Distributed Deep Learning Training without complete job information NSDI'19
• Chronus: A Novel Deadline-aware Scheduler for Deep Learning Training Jobs SOCC'21
• awesome DL scheduler
• Lucid: A Non-Intrusive, Scalable and Interpretable Scheduler for Deep Learning Training Jobs ASPLOS'23
• Synergy : Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters OSDI'22

LLM Evaluation Platform

• ![Star - sys/FastChat.git)| |

LLM FineTune

• S-LoRA: Serving Thousands of Concurrent LoRA Adapters
• Punica: Serving multiple LoRA finetuned LLM as one

LLM Inference (System Side)

• ![Star - Liger) |-| PPOPP'24

LLM Robustness and Debugging

• ![DOI

LLM Serving

• Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity vldb'24
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
• Efficient Memory Management for Large Language Model Serving with PagedAttention SOSP'23
• SpecInfer: Accelerating Generative Large Language Model Serving with Speculative Inference and Token Tree Verification 23arxiv
• SpecInfer: Accelerating Generative Large Language Model Serving with Speculative Inference and Token Tree Verification 23arxiv

LLM Serving Framework

• ![Star - ai/mlc-llm/) |
• ![Star - LLM.git) |
• ![Star
• ![Star
• ![Star

LoRA

•  | - | Oct,2023 |

Misc

• AI-Enabling Workloads on Large-Scale GPU-Accelerated System: Characterization, Opportunities, and Implications HPCA'22
• Characterizing Variability in Large-Scale, Accelerator-Rich Systems SC'22
• Prediction of the Resource Consumption of Distributed Deep Learning Systems SIGMETRICS'22
• AI-Enabling Workloads on Large-Scale GPU-Accelerated System: Characterization, Opportunities, and Implications HPCA'22
• Characterizing Variability in Large-Scale, Accelerator-Rich Systems SC'22
• Prediction of the Resource Consumption of Distributed Deep Learning Systems SIGMETRICS'22

MoE

• SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Static and Dynamic Parallelization ATC'23
• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts MLSYS'23
• Tutel: Adaptive Mixture-of-Experts at Scale MLSYS'23
• AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers ICLR'23
• SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Static and Dynamic Parallelization ATC'23
• Tutel: Adaptive Mixture-of-Experts at Scale MLSYS'23
• FastMoE: A Fast Mixture-of-Expert Training System PPOPP'22
• AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers ICLR'23
• awesome MoE
• MoE Paper
• Tutel: Adaptive Mixture-of-Experts at Scale MLSYS'23
• FastMoE: A Fast Mixture-of-Expert Training System PPOPP'22
• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts MLSYS'23
• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts MLSYS'23

Optimization

• MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant GPU Clusters SOCC'22
• Accpar: Tensor partitioning for heterogeneous deep learning accelerators HPCA'20
• iGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud TPDS'22
• CheckFreq: Frequent, Fine-Grained DNN Checkpointing FAST'22
• Harmony: Overcoming the hurdles of GPU memory capacity to train massive DNN models on commodity servers VLDB'22
• PetS: A Unified Framework for Parameter-Efficient Transformers Serving ATC'22
• PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections OSDI'21
• Fluid: Resource-Aware Hyperparameter Tuning Engine MLSYS'21
• Baechi: Fast Device Placement on Machine Learning Graphs SOCC'20
• Dynamic Parameter Allocation in Parameter Servers VLDB'20
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• Spada: Accelerating Sparse Matrix Multiplication with Adaptive Dataflow ASPLOS'23
• Hidet: Task Mapping Programming Paradigm for Deep Learning Tensor Programs ASPLOS'23
• GLake: optimizing GPU memory management and IO transmission ASPLOS'24
• Spada: Accelerating Sparse Matrix Multiplication with Adaptive Dataflow ASPLOS'23
• MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant GPU Clusters SOCC'22
• Accpar: Tensor partitioning for heterogeneous deep learning accelerators HPCA'20
• iGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud TPDS'22
• CheckFreq: Frequent, Fine-Grained DNN Checkpointing FAST'22
• Efficient Quantized Sparse Matrix Operations on Tensor Cores SC'22
• Harmony: Overcoming the hurdles of GPU memory capacity to train massive DNN models on commodity servers VLDB'22
• PetS: A Unified Framework for Parameter-Efficient Transformers Serving ATC'22
• PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections OSDI'21
• APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Core SC'21
• iGUARD: In-GPU Advanced Race Detection SOSP'21
• Fluid: Resource-Aware Hyperparameter Tuning Engine MLSYS'21
• Baechi: Fast Device Placement on Machine Learning Graphs SOCC'20
• Dynamic Parameter Allocation in Parameter Servers VLDB'20
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Core SC'21
• MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant GPU Clusters SOCC'22
• Accpar: Tensor partitioning for heterogeneous deep learning accelerators HPCA'20
• Hidet: Task Mapping Programming Paradigm for Deep Learning Tensor Programs ASPLOS'23
• iGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud TPDS'22
• Efficient Quantized Sparse Matrix Operations on Tensor Cores SC'22
• iGUARD: In-GPU Advanced Race Detection SOSP'21

Parallellism Training

• Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization OSDI'22
• DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines Eurosys'24
• DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines Eurosys'24
• HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis Eurosys'24
• PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices MLSYS'23
• Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs NSDI'23
• MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism HPCA'23
• MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism HPCA'23
• Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression ASPLOS'23
• Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning OSDI'22
• AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness NeurIPS '22
• Piper: Multidimensional Planner for DNN Parallelization NeurIPS'21
• PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models ICML'21
• TeraPipe:Large-Scale Language Modeling with Pipeline Parallelism ICML'21
• PipeDream: Pipeline Parallelism for DNN Training SOSP'19
• SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient
• Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
• MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism HPCA'23
• Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
• MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism HPCA'23
• Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
• MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism HPCA'23
• Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
• NEAR ZERO BUBBLE PIPELINE PARALLELISM ICLR'24
• zero-bubble-pipeline-parallelism
• Aceso: Efficient Parallel DNN Training through Iterative Bottleneck Alleviation Eurosys'24
• HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis Eurosys'24
• Calculon: A Methodology and Tool for High-Level Co-Design of Systems and Large Language Models SC'23
• Calculon: A Methodology and Tool for High-Level Co-Design of Systems and Large Language Models SC'23
• PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices MLSYS'23
• Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs NSDI'23
• Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression ASPLOS'23
• AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness NeurIPS '22
• NASPipe: High Performance and Reproducible Pipeline Parallel Supernet Training via Causal Synchronous Parallelism
• Varuna: Scalable, Low-cost Training of Massive Deep Learning Models
• Chimera: efficiently training large-scale neural networks with bidirectional pipelines SC'21
• Piper: Multidimensional Planner for DNN Parallelization NeurIPS'21
• PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models ICML'21
• DAPPLE: An Efficient Pipelined Data Parallel Approach for Large Models Training PPOPP'21
• TeraPipe:Large-Scale Language Modeling with Pipeline Parallelism ICML'21
• PipeDream: Pipeline Parallelism for DNN Training SOSP'19
• SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient
• Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
• awesome distributed deep learning
• awsome parallelism
• MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism HPCA'23
• Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
• MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism HPCA'23
• Varuna: Scalable, Low-cost Training of Massive Deep Learning Models
• Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
• NASPipe: High Performance and Reproducible Pipeline Parallel Supernet Training via Causal Synchronous Parallelism
• Chimera: efficiently training large-scale neural networks with bidirectional pipelines SC'21
• Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization OSDI'22
• PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices MLSYS'23
• Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression ASPLOS'23
• AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness NeurIPS '22
• PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models ICML'21
• DAPPLE: An Efficient Pipelined Data Parallel Approach for Large Models Training PPOPP'21