{"id":27160840,"url":"https://github.com/HuaizhengZhang/Awesome-System-for-Machine-Learning","last_synced_at":"2025-04-09T00:01:40.386Z","repository":{"id":38433135,"uuid":"164433070","full_name":"HuaizhengZhang/AI-Infra-from-Zero-to-Hero","owner":"HuaizhengZhang","description":"🚀 Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice. ⚡️ System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI). 🍻 OSDI, NSDI, SIGCOMM, SoCC, MLSys, etc. 🗃️ Llama3, Mistral, etc. 🧑‍💻 Video Tutorials. ","archived":false,"fork":false,"pushed_at":"2024-08-14T05:12:47.000Z","size":910,"stargazers_count":2838,"open_issues_count":12,"forks_count":321,"subscribers_count":127,"default_branch":"master","last_synced_at":"2025-03-30T19:01:43.487Z","etag":null,"topics":["ai-infra","genai","large-language-models","llmsys","mlsys","model-serving","model-training"],"latest_commit_sha":null,"homepage":"https://huaizheng.xyz/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HuaizhengZhang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-07T12:59:19.000Z","updated_at":"2025-03-30T11:06:44.000Z","dependencies_parsed_at":"2023-02-19T04:16:04.453Z","dependency_job_id":"79da80ab-f845-4ec3-9120-4bd27b8919a7","html_url":"https://github.com/HuaizhengZhang/AI-Infra-from-Zero-to-Hero","commit_stats":null,"previous_names":["huaizhengzhang/ai-system-school","huaizhengzhang/awesome-system-for-machine-learning","huaizhengzhang/ai-infra-from-zero-to-hero"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuaizhengZhang%2FAI-Infra-from-Zero-to-Hero","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuaizhengZhang%2FAI-Infra-from-Zero-to-Hero/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuaizhengZhang%2FAI-Infra-from-Zero-to-Hero/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuaizhengZhang%2FAI-Infra-from-Zero-to-Hero/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HuaizhengZhang","download_url":"https://codeload.github.com/HuaizhengZhang/AI-Infra-from-Zero-to-Hero/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247947842,"owners_count":21023065,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-infra","genai","large-language-models","llmsys","mlsys","model-serving","model-training"],"created_at":"2025-04-09T00:01:32.694Z","updated_at":"2025-04-09T00:01:40.379Z","avatar_url":"https://github.com/HuaizhengZhang.png","language":null,"funding_links":[],"categories":["Uncategorized","Technical","Others","Table of Contents","Other Lists","Lists","System \u0026 Production","Famous Toolkit","General ML","Related Awesome Lists"],"sub_categories":["Uncategorized","awesome-*","Julia Lists","Survey","Workshop","Surveys"],"readme":"[![Maintenance](https://img.shields.io/badge/Maintained%3F-YES-green.svg)](https://github.com/HuaizhengZhang/Awesome-System-for-Machine-Learning/graphs/commit-activity)\n[![Commit Activity](https://img.shields.io/github/commit-activity/m/HuaizhengZhang/Awesome-System-for-Machine-Learning.svg?color=red)](https://github.com/HuaizhengZhang/Awesome-System-for-Machine-Learning/graphs/commit-activity)\n[![Last Commit](https://img.shields.io/github/last-commit/HuaizhengZhang/Awesome-System-for-Machine-Learning.svg)](https://github.com/HuaizhengZhang/Awesome-System-for-Machine-Learning/commits/master)\n[![Ask Me Anything !](https://img.shields.io/badge/Ask%20me-anything-1abc9c.svg)](https://GitHub.com/Naereen/ama)\n[![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n[![GitHub license](https://img.shields.io/github/license/HuaizhengZhang/Awesome-System-for-Machine-Learning.svg?color=blue)](https://github.com/HuaizhengZhang/Awesome-System-for-Machine-Learning/blob/master/LICENSE)\n[![GitHub stars](https://img.shields.io/github/stars/HuaizhengZhang/Awesome-System-for-Machine-Learning.svg?style=social)](https://GitHub.com/HuaizhengZhang/Awesome-System-for-Machine-Learning/stargazers/)\n\n# AI System School \n\n### 💫💫💫 System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI)\n\n### Updates: \n\n- Video Tutorials [[YouTube]](https://youtu.be/ChD1_aVZJ0g?si=Kg-yB3F4Iea0Xp9J) [[bilibili]](https://www.bilibili.com/video/BV1ZwYUerEtL/) [[小红书]](http://xhslink.com/MmrjcT)\n- We are preparing a new website [[Lets Go AI]](https://letsgoai.pro/) for this repo!!!\n\n### *Path to System for AI* [[Whitepaper You Must Read]](./paper/mlsys-whitepaper.pdf)\n\nA curated list of research in machine learning systems. Link to the code if available is also present. Now we have a [team](#maintainer) to maintain this project. *You are very welcome to pull request by using our template*.\n\n![AI system](https://github.com/HuaizhengZhang/Awesome-System-for-Machine-Learning/blob/master/imgs/AI_system.png)\n\n## System for AI (Ordered by Category)\n\n### ML / DL Infra\n\n- [Data Processing](data_processing.md#data-processing)\n- [Training System](training.md#training-system)\n- [Inference System](inference.md#inference-system)\n- [Machine Learning Infrastructure](infra.md#machine-learning-infrastructure)\n\n### LLM Infra\n\n- [LLM Training](llm_training.md#llm_training)\n- [LLM Serving](llm_serving.md#llm_serving)\n\n### Domain-Specific Infra\n\n- [Video System](video_system.md#video-system)\n- [AutoML System](AutoML_system.md#automl-system)\n- [Edge AI](edge_system.md#edge-or-mobile-papers)\n- [GNN System](GNN_system.md#system-for-gnn-traininginference)\n- [Federated Learning System](federated_learning_system.md#federated-learning-system)\n- [Deep Reinforcement Learning System](drl_system.md#deep-reinforcement-learning-system)\n\n## System for ML/LLM Conference\n\n### Conference\n\n- OSDI\n- SOSP\n- SIGCOMM\n- NSDI\n- MLSys\n- ATC\n- Eurosys \n- Middleware\n- SoCC\n- TinyML\n\n## General Resources\n\n- [Survey](#survey)\n- [Book](#book)\n- [Video](#video)\n- [Course](#course)\n- [Blog](#blog)\n\n## Survey\n\n- Toward Highly Available, Intelligent Cloud and ML Systems [[Slide]](http://sysnetome.com/Talks/cguo_netai_2018.pdf)\n- A curated list of awesome System Designing articles, videos and resources for distributed computing, AKA Big Data. [[GitHub]](https://github.com/madd86/awesome-system-design)\n- awesome-production-machine-learning: A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning [[GitHub]](https://github.com/EthicalML/awesome-production-machine-learning)\n- Opportunities and Challenges Of Machine Learning Accelerators In Production [[Paper]](https://www.usenix.org/system/files/opml19papers-ananthanarayanan.pdf)\n  - Ananthanarayanan, Rajagopal, et al. \"\n  - 2019 {USENIX} Conference on Operational Machine Learning (OpML 19). 2019.\n- How (and How Not) to Write a Good Systems Paper [[Advice]](https://www.usenix.org/legacy/events/samples/submit/advice_old.html)\n- Applied machine learning at Facebook: a datacenter infrastructure perspective [[Paper]](https://research.fb.com/wp-content/uploads/2017/12/hpca-2018-facebook.pdf)\n  - Hazelwood, Kim, et al. (*HPCA 2018*)\n- Infrastructure for Usable Machine Learning: The Stanford DAWN Project\n  - Bailis, Peter, Kunle Olukotun, Christopher Ré, and Matei Zaharia. (*preprint 2017*)\n- Hidden technical debt in machine learning systems [[Paper]](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf)\n  - Sculley, David, et al. (*NIPS 2015*)\n- End-to-end arguments in system design [[Paper]](http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf)\n  - Saltzer, Jerome H., David P. Reed, and David D. Clark. \n- System Design for Large Scale Machine Learning [[Thesis]](http://shivaram.org/publications/shivaram-dissertation.pdf)\n- Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications [[Paper]](https://arxiv.org/pdf/1811.09886.pdf)\n  - Park, Jongsoo, Maxim Naumov, Protonu Basu et al. *arXiv 2018*\n  - Summary: This paper presents a characterizations of DL models and then shows the new design principle of DL hardware.\n- A Berkeley View of Systems Challenges for AI [[Paper]](https://arxiv.org/pdf/1712.05855.pdf)\n\n\n## Book\n\n- Computer Architecture: A Quantitative Approach [[Must read]](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.115.1881\u0026rep=rep1\u0026type=pdf)\n- Distributed Machine Learning Patterns [[Website]](https://www.manning.com/books/distributed-machine-learning-patterns)\n- Streaming Systems [[Book]](https://www.oreilly.com/library/view/streaming-systems/9781491983867/)\n- Kubernetes in Action (start to read) [[Book]](https://www.oreilly.com/library/view/kubernetes-in-action/9781617293726/)\n- Machine Learning Systems: Designs that scale [[Website]](https://www.manning.com/books/machine-learning-systems)\n- Trust in Machine Learning [[Website]](https://www.manning.com/books/trust-in-machine-learning)\n- Automated Machine Learning in Action [[Website]](https://www.manning.com/books/automated-machine-learning-in-action)\n\n## Video\n\n- ScalaDML2020: Learn from the best minds in the machine learning community. [[Video]](https://info.matroid.com/scaledml-media-archive-preview)\n- Jeff Dean: \"Achieving Rapid Response Times in Large Online Services\" Keynote - Velocity 2014 [[YouTube]](https://www.youtube.com/watch?v=1-3Ahy7Fxsc)\n- From Research to Production with PyTorch [[Video]](https://www.infoq.com/presentations/pytorch-torchscript-botorch/#downloadPdf/)\n- Introduction to Microservices, Docker, and Kubernetes [[YouTube]](https://www.youtube.com/watch?v=1xo-0gCVhTU)\n- ICML Keynote: Lessons Learned from Helping 200,000 non-ML experts use ML [[Video]](https://slideslive.com/38916584/keynote-lessons-learned-from-helping-200000-nonml-experts-use-ml)\n- Adaptive \u0026 Multitask Learning Systems [[Website]](https://www.amtl-workshop.org/schedule)\n- System thinking. A TED talk. [[YouTube]](https://www.youtube.com/watch?v=_vS_b7cJn2A)\n- Flexible systems are the next frontier of machine learning. Jeff Dean [[YouTube]](https://www.youtube.com/watch?v=Jnunp-EymJQ\u0026list=WL\u0026index=12)\n- Is It Time to Rewrite the Operating System in Rust? [[YouTube]](https://www.youtube.com/watch?v=HgtRAbE1nBM\u0026list=WL\u0026index=17\u0026t=0s)\n- InfoQ: AI, ML and Data Engineering [[YouTube]](https://www.youtube.com/playlist?list=PLndbWGuLoHeYsZk6VpCEj_SSd9IFgjJ-2)\n  - Start to watch.\n- Netflix: Human-centric Machine Learning Infrastructure [[InfoQ]](https://www.infoq.com/presentations/netflix-ml-infrastructure?utm_source=youtube\u0026utm_medium=link\u0026utm_campaign=qcontalks)\n- SysML 2019: [[YouTube]](https://www.youtube.com/channel/UChutDKIa-AYyAmbT45s991g/videos)\n- ScaledML 2019: David Patterson, Ion Stoica, Dawn Song and so on [[YouTube]](https://www.youtube.com/playlist?list=PLRM2gQVaW_wWXoUnSfZTxpgDmNaAS1RtG)\n- ScaledML 2018: Jeff Dean, Ion Stoica, Yangqing Jia and so on [[YouTube]](https://www.youtube.com/playlist?list=PLRM2gQVaW_wW9KAxcibxdqY_TDyvmEjzm) [[Slides]](https://www.matroid.com/blog/post/slides-and-videos-from-scaledml-2018)\n- A New Golden Age for Computer Architecture History, Challenges, and Opportunities. David Patterson [[YouTube]](https://www.youtube.com/watch?v=uyc_pDBJotI\u0026t=767s)\n- How to Have a Bad Career. David Patterson (I am a big fan) [[YouTube]](https://www.youtube.com/watch?v=Rn1w4MRHIhc)\n- SysML 18: Perspectives and Challenges. Michael Jordan [[YouTube]](https://www.youtube.com/watch?v=4inIBmY8dQI\u0026t=26s)\n- SysML 18: Systems and Machine Learning Symbiosis. Jeff Dean [[YouTube]](https://www.youtube.com/watch?v=Nj6uxDki6-0)\n- AutoML Basics: Automated Machine Learning in Action. Qingquan Song, Haifeng Jin, Xia Hu [[YouTube]](https://www.youtube.com/watch?v=9KpieG0B7VM)\n\n## Course\n\n- CS692 Seminar: Systems for Machine Learning, Machine Learning for Systems [[GitHub]](https://github.com/guanh01/CS692-mlsys)\n- Topics in Networks: Machine Learning for Networking and Systems, Autumn 2019 [[Course Website]](https://people.cs.uchicago.edu/~junchenj/34702-fall19/syllabus.html)\n- CS6465: Emerging Cloud Technologies and Systems Challenges [[Cornell]](http://www.cs.cornell.edu/courses/cs6465/2019fa/)\n- CS294: AI For Systems and Systems For AI. [[UC Berkeley Spring]](https://github.com/ucbrise/cs294-ai-sys-sp19) (*Strong Recommendation*) [[Machine Learning Systems (Fall 2019)]](https://ucbrise.github.io/cs294-ai-sys-fa19/)\n- CSE 599W: System for ML.  [[Chen Tianqi]](https://github.com/tqchen) [[University of Washington]](http://dlsys.cs.washington.edu/)\n- EECS 598: Systems for AI (W'21). [[Mosharaf Chowdhury]](https://www.mosharaf.com/) [[Systems for AI (W'21)]](https://github.com/mosharaf/eecs598/tree/w21-ai)\n- Tutorial code on how to build your own Deep Learning System in 2k Lines [[GitHub]](https://github.com/tqchen/tinyflow)\n- CSE 291F: Advanced Data Analytics and ML Systems. [[UCSD]](http://cseweb.ucsd.edu/classes/wi19/cse291-f/)\n- CSci 8980: Machine Learning in Computer Systems [[University of Minnesota, Twin Cities]](http://www-users.cselabs.umn.edu/classes/Spring-2019/csci8980/)\n- Mu Li (MxNet, Parameter Server): Introduction to Deep Learning [[Best DL Course I think]](https://courses.d2l.ai/berkeley-stat-157/index.html)  [[Book]](https://www.d2l.ai/)\n- 10-605: Machine Learning with Large Datasets. [[CMU]](https://10605.github.io/fall2020/index.html)\n- CS 329S: Machine Learning Systems Design. [[Stanford]](https://stanford-cs329s.github.io/index.html)\n\n## Blog\n\n- Parallelizing across multiple CPU/GPUs to speed up deep learning inference at the edge [[Amazon Blog]](https://aws.amazon.com/blogs/machine-learning/parallelizing-across-multiple-cpu-gpus-to-speed-up-deep-learning-inference-at-the-edge/)\n- Building Robust Production-Ready Deep Learning Vision Models in Minutes [[Blog]](https://medium.com/google-developer-experts/building-robust-production-ready-deep-learning-vision-models-in-minutes-acd716f6450a)\n- Deploy Machine Learning Models with Keras, FastAPI, Redis and Docker [[Blog]](https://medium.com/@shane.soh/deploy-machine-learning-models-with-keras-fastapi-redis-and-docker-4940df614ece)\n- How to Deploy a Machine Learning Model -- Creating a production-ready API using FastAPI + Uvicorn [[Blog]](https://towardsdatascience.com/how-to-deploy-a-machine-learning-model-dc51200fe8cf) [[GitHub]](https://github.com/MaartenGr/ML-API)\n- Deploying a Machine Learning Model as a REST API [[Blog]](https://towardsdatascience.com/deploying-a-machine-learning-model-as-a-rest-api-4a03b865c166)\n- Continuous Delivery for Machine Learning [[Blog]](https://martinfowler.com/articles/cd4ml.html)\n- Kubernetes CheatSheets In A4 [[GitHub]](https://github.com/HuaizhengZhang/cheatsheet-kubernetes-A4)\n- A Gentle Introduction to Kubernetes [[Blog]](https://medium.com/faun/a-gentle-introduction-to-kubernetes-4961e443ba26)\n- Train and Deploy Machine Learning Model With Web Interface - Docker, PyTorch \u0026 Flask [[GitHub]](https://github.com/imadelh/ML-web-app)\n- Learning Kubernetes, The Chinese Taoist Way [[GitHub]](https://github.com/caicloud/kube-ladder)\n- Data pipelines, Luigi, Airflow: everything you need to know [[Blog]](https://towardsdatascience.com/data-pipelines-luigi-airflow-everything-you-need-to-know-18dc741449b7)\n- The Deep Learning Toolset — An Overview [[Blog]](https://medium.com/luminovo/the-deep-learning-toolset-an-overview-b71756016c06)\n- Summary of CSE 599W: Systems for ML [[Chinese Blog]](http://jcf94.com/2018/10/04/2018-10-04-cse559w/)\n- Polyaxon, Argo and Seldon for Model Training, Package and Deployment in Kubernetes [[Blog]](https://medium.com/analytics-vidhya/polyaxon-argo-and-seldon-for-model-training-package-and-deployment-in-kubernetes-fa089ba7d60b)\n- Overview of the different approaches to putting Machine Learning (ML) models in production [[Blog]](https://medium.com/analytics-and-data/overview-of-the-different-approaches-to-putting-machinelearning-ml-models-in-production-c699b34abf86)\n- Being a Data Scientist does not make you a Software Engineer [[Part1]](https://towardsdatascience.com/being-a-data-scientist-does-not-make-you-a-software-engineer-c64081526372)\n  Architecting a Machine Learning Pipeline [[Part2]](https://towardsdatascience.com/architecting-a-machine-learning-pipeline-a847f094d1c7)\n- Model Serving in PyTorch [[Blog]](https://pytorch.org/blog/model-serving-in-pyorch/)\n- Machine learning in Netflix [[Medium]](https://medium.com/@NetflixTechBlog)\n- SciPy Conference Materials (slides, repo) [[GitHub]](https://github.com/deniederhut/Slides-SciPyConf-2018)\n- 继Spark之后，UC Berkeley 推出新一代AI计算引擎——Ray [[Blog]](http://www.qtmuniao.com/2019/04/06/ray/)\n- 了解/从事机器学习/深度学习系统相关的研究需要什么样的知识结构？ [[Zhihu]](https://www.zhihu.com/question/315611053/answer/623529977)\n- Learn Kubernetes in Under 3 Hours: A Detailed Guide to Orchestrating Containers [[Blog]](https://www.freecodecamp.org/news/learn-kubernetes-in-under-3-hours-a-detailed-guide-to-orchestrating-containers-114ff420e882/) [[GitHub]](https://github.com/rinormaloku/k8s-mastery)\n- data-engineer-roadmap: Learning from multiple companies in Silicon Valley. Netflix, Facebook, Google, Startups [[GitHub]](https://github.com/hasbrain/data-engineer-roadmap)\n- TensorFlow Serving + Docker + Tornado机器学习模型生产级快速部署 [[Blog]](https://zhuanlan.zhihu.com/p/52096200?utm_source=wechat_session\u0026utm_medium=social\u0026utm_oi=38612796178432)\n- Deploying a Machine Learning Model as a REST API [[Blog]](https://towardsdatascience.com/deploying-a-machine-learning-model-as-a-rest-api-4a03b865c166)\n- Colossal-AI: A Unified Deep Learning System for Big Model Era [[Blog]](https://medium.com/@hpcaitech/train-18-billion-parameter-gpt-models-with-a-single-gpu-on-your-personal-computer-8793d08332dc) [[GitHub]](https://github.com/hpcaitech/ColossalAI)\n- Data Engineer Roadmap [[Scaler Blogs]](https://www.scaler.com/blog/data-engineer-roadmap/)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHuaizhengZhang%2FAwesome-System-for-Machine-Learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FHuaizhengZhang%2FAwesome-System-for-Machine-Learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHuaizhengZhang%2FAwesome-System-for-Machine-Learning/lists"}