Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mryab/efficient-dl-systems

Efficient Deep Learning Systems course materials (HSE, YSDA)
https://github.com/mryab/efficient-dl-systems

cuda deep-learning distributed-training efficient-deep-learning machine-learning ml-infrastructure mlops pytorch

Last synced: about 9 hours ago
JSON representation

Efficient Deep Learning Systems course materials (HSE, YSDA)

Host: GitHub
URL: https://github.com/mryab/efficient-dl-systems
Owner: mryab
License: mit
Created: 2021-12-06T00:30:40.000Z (about 3 years ago)
Default Branch: main
Last Pushed: 2025-01-24T10:11:05.000Z (10 days ago)
Last Synced: 2025-01-27T10:03:08.687Z (7 days ago)
Topics: cuda, deep-learning, distributed-training, efficient-deep-learning, machine-learning, ml-infrastructure, mlops, pytorch
Language: Jupyter Notebook
Homepage:
Size: 48.3 MB
Stars: 733
Watchers: 14
Forks: 118
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

awesome-oneapi - efficient-dl-systems - This repository contains materials for the Efficient Deep Learning Systems course taught at the HSE University and Yandex School of Data Analysis. (Tutorials / Tools and Development)

README

        # Efficient Deep Learning Systems

This repository contains materials for the Efficient Deep Learning Systems course taught at the [Faculty of Computer Science](https://cs.hse.ru/en/) of [HSE University](https://www.hse.ru/en/) and [Yandex School of Data Analysis](https://academy.yandex.com/dataschool/).

__This branch corresponds to the ongoing 2025 course. If you want to see full materials of past years, see the ["Past versions"](#past-versions) section.__

# Syllabus

- [__Week 1:__](./week01_intro) __Introduction__

  - Lecture: Course overview and organizational details. Core concepts of the GPU architecture and CUDA API.

  - Seminar: CUDA operations in PyTorch. Introduction to benchmarking.

- [__Week 2:__](./week02_management_and_testing) __Experiment tracking, model and data versioning, testing DL code in Python__

  - Lecture: Experiment management basics and pipeline versioning. Configuring Python applications. Intro to regular and property-based testing.

  - Seminar: Example DVC+Weights & Biases project walkthrough. Intro to testing with pytest.

- [__Week 3:__ ](./week03_fast_pipelines) __Training optimizations, FP16/BF16/FP8 formats, profiling deep learning code__

  - Lecture: Measuring performance of GPU-accelerated software. Mixed-precision training. Data storage and loading optimizations. Tools for profiling deep learning workloads. 

  - Seminar: Automatic Mixed Precision in PyTorch. Dynamic padding for sequence data and JPEG decoding benchmarks. Basics of profiling with py-spy, PyTorch Profiler, Memory Snapshot and Nsight Systems.

- __Week 4:__ __Data-parallel training and All-Reduce__

- __Week 5:__ __Sharded data-parallel training, distributed training optimizations__

- __Week 6:__ __Training large models__

- __Week 7:__ __Python web application deployment__

- __Week 8:__ __LLM inference optimizations and software__

- __Week 9:__ __Efficient model inference__

- __Week 10:__ Guest lecture

## Grading

There will be several home assignments (spread over multiple weeks) on the following topics:

- Training pipelines and code profiling

- Distributed and memory-efficient training

- Deploying and optimizing models for production

The final grade is a weighted sum of per-assignment grades.

Please refer to the course page of your institution for details.

# Staff

- [Max Ryabinin](https://github.com/mryab)

- [Just Heuristic](https://github.com/justheuristic)

- [Yaroslav Zolotarev](https://github.com/Q-c7)

- [Maksim Abraham](https://github.com/fdrose)

- [Gregory Leleytner](https://github.com/RunFMe)

- [Antony Frolov](https://github.com/antony-frolov)

- [Anton Chigin](https://github.com/achigin)

- [Alexander Markovich](https://github.com/markovka17)

- [Roman Gorb](https://github.com/rvg77)

# Past versions

- [2024](https://github.com/mryab/efficient-dl-systems/tree/2024)

- [2023](https://github.com/mryab/efficient-dl-systems/tree/2023)

- [2022](https://github.com/mryab/efficient-dl-systems/tree/2022)

- [2021](https://github.com/yandexdataschool/dlatscale_draft)