https://github.com/hundredblocks/large-model-parallelism

Functional local implementations of main model parallelism approaches
https://github.com/hundredblocks/large-model-parallelism

Last synced: 3 months ago
JSON representation

Functional local implementations of main model parallelism approaches

• Host: GitHub
• URL: https://github.com/hundredblocks/large-model-parallelism
• Owner: hundredblocks
• License: gpl-3.0
• Created: 2023-02-21T18:15:17.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2023-02-21T18:44:13.000Z (over 1 year ago)
• Last Synced: 2024-06-15T11:33:02.317Z (5 months ago)
• Language: Jupyter Notebook
• Size: 823 KB
• Stars: 91
• Watchers: 12
• Forks: 5
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-ChatGPT-repositories - large-model-parallelism - Functional local implementations of main model parallelism approaches (Reimplementations)

README

        
![large-model-parallelism-illustration](https://user-images.githubusercontent.com/7229234/220430946-4b4ab497-7f86-40af-ac43-3e2793900bce.jpg)

# Model parallelism 101

Learn how model parallelism enables training models like stable diffusion and Chat GPT in less than 300 lines of code. This [notebook](https://github.com/hundredblocks/large-model-parallelism/blob/main/large-model-parallelism.ipynb) provides practical local implementations of the main model parallelism methods. It explores three approaches: data parallelism, tensor parallelism, and pipeline parallelism with a 2-layer MLP example that can be naturally extended to more complex models.

Reading this notebook will give you a solid overview of model parallelism techniques and an intuition for how to implement them.

Pull requests welcome. Illustration above generated with Lexica's Aperture model.