https://github.com/cloneofsimo/scaling-guide

WIP
https://github.com/cloneofsimo/scaling-guide

Last synced: 9 months ago
JSON representation

WIP

• Host: GitHub
• URL: https://github.com/cloneofsimo/scaling-guide
• Owner: cloneofsimo
• Created: 2024-07-30T07:28:05.000Z (about 2 years ago)
• Default Branch: master
• Last Pushed: 2024-08-13T19:10:09.000Z (almost 2 years ago)
• Last Synced: 2025-02-07T00:09:28.204Z (over 1 year ago)
• Language: Python
• Homepage:
• Size: 266 KB
• Stars: 93
• Watchers: 9
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Almost Comprehensive Scaling Guide for Transformer Training

WIP

# Feature

Activation Tracking:

![alt text](contents/activation.png)

Weight Update Tracking:

![alt text](contents/weight_shift.png)

# Scope:

* Kaplan vs Chinchilla

* SP vs muP vs layerwise SP

* adamW weight decay

* infinite lr scheduler

* batch_size vs lr (sqrt BS law)

* agd-muP (spectral initialization vs classic muP)

* adam-atan2

* data dependent lr tranfer

* embedding lr transfer

* u-muP

# Shoutout

Thanks to [Fal.ai](https://fal.ai/) for providing compute to run these experiments.