https://github.com/NousResearch/DisTrO
Distributed Training Over-The-Internet
https://github.com/NousResearch/DisTrO
Last synced: 8 months ago
JSON representation
Distributed Training Over-The-Internet
- Host: GitHub
- URL: https://github.com/NousResearch/DisTrO
- Owner: NousResearch
- Created: 2024-08-26T15:30:54.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-05-15T14:43:24.000Z (12 months ago)
- Last Synced: 2025-08-29T02:36:27.600Z (8 months ago)
- Homepage:
- Size: 990 KB
- Stars: 955
- Watchers: 99
- Forks: 41
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- StarryDivineSky - NousResearch/DisTrO
README
# DisTrO
This is the repository for DisTrO (Distributed Training Over-The-Internet), a family of low latency distributed optimizers that reduce inter-GPU communication requirements by three to four orders of magnitude.
- [x] Aug. 26th, 2024: DisTrO [(Preliminary Report)](https://github.com/NousResearch/DisTrO/raw/main/A_Preliminary_Report_on_DisTrO.pdf)
- [x] Dec. 2nd, 2024: DeMo Optimization [(Paper)](https://arxiv.org/abs/2411.19870) [(Code)](https://github.com/bloc97/DeMo), original seed research/idea for DisTrO
- [x] Dec. 2nd, 2024: [Nous trains a 15b model using DisTrO](https://distro.nousresearch.com/)
- [x] May. 14th, 2025: [Psyche Network](https://nousresearch.com/nous-psyche/)
- [x] May. 14th, 2025: [Nous Consilience 40b LLM](https://psyche.network/runs/consilience-40b-1/0), [Huggingface](https://huggingface.co/PsycheFoundation/consilience-40b-7Y9v38s5)
- [ ] Coming Soon: DisTrO Paper and Code
[Join us on Discord](https://discord.com/invite/jqVphNsB4H) if you're interested in helping research and build the future of distributed training.