Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/NousResearch/DisTrO

Distributed Training Over-The-Internet
https://github.com/NousResearch/DisTrO

Last synced: 12 days ago
JSON representation

Distributed Training Over-The-Internet

Host: GitHub
URL: https://github.com/NousResearch/DisTrO
Owner: NousResearch
Created: 2024-08-26T15:30:54.000Z (5 months ago)
Default Branch: main
Last Pushed: 2024-12-02T16:42:45.000Z (about 1 month ago)
Last Synced: 2024-12-02T17:25:39.819Z (about 1 month ago)
Homepage:
Size: 987 KB
Stars: 692
Watchers: 78
Forks: 25
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

StarryDivineSky - NousResearch/DisTrO

README

        # DisTrO

This is the repository for DisTrO (Distributed Training Over-The-Internet), a family of low latency distributed optimizers that reduce inter-GPU communication requirements by three to four orders of magnitude.

- [x] Aug. 26th, 2024: DisTrO [(Preliminary Report)](https://github.com/NousResearch/DisTrO/raw/main/A_Preliminary_Report_on_DisTrO.pdf)

- [x] Dec. 2nd, 2024: DeMo Optimization [(Paper)](https://arxiv.org/abs/2411.19870) [(Code)](https://github.com/bloc97/DeMo), original seed research/idea for DisTrO

- [x] Dec. 2nd, 2024: [Nous trains a 15b model using DisTrO](https://distro.nousresearch.com/)

- [ ] Coming Soon: DisTrO Paper and Code

- [ ] In The Near Future: 👀

[Join us on Discord](https://discord.com/invite/jqVphNsB4H) if you're interested in helping research and build the future of distributed training.