https://github.com/ahxt/mini-llm-pretraining
https://github.com/ahxt/mini-llm-pretraining
Last synced: 9 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/ahxt/mini-llm-pretraining
- Owner: ahxt
- Created: 2025-01-29T06:28:53.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-17T04:33:15.000Z (over 1 year ago)
- Last Synced: 2025-04-05T13:51:23.566Z (over 1 year ago)
- Language: Python
- Size: 125 KB
- Stars: 5
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# How to speed up LLaMA 3 training?
A clean and efficient implementation for training LLaMA 3 models, focused on simplicity and performance. This codebase provides a streamlined version of LLaMA 3 pretraining based on Hugging Face's implementation and uses FSDP for distributed training.
## Files
- `modeling_llama.py`: minimal LLaMA 3 model implementation
- `train_fsdp.py`: FSDP distributed training script
- `preprocess_findweb.py`: data preprocessing and cache
## How to run
1. Recommend to use Docker to run the code
```
docker pull xiaotian99/llm_pt25:v0.3
```
or use the [sglang official image](https://hub.docker.com/r/lmsysorg/sglang/tags).
2. Prepare the dataset:
```bash
./run_prepare_datasets.sh
```
3. Start training:
```bash
./run_pretraining_fsdp.sh
```
## Key Features
- Minimal codebase with clear structure
- Optimized for modern GPU architectures
- Support for multiple attention implementations
- Clean and easy to use code