awesome-llm-pretraining

Awesome LLM pre-training resources, including data, frameworks, and methods.
https://github.com/RUCAIBox/awesome-llm-pretraining

Last synced: 22 days ago
JSON representation

III. Open-source Datasets

3.1 Web Pages

[paper - baseline-1.0)]
[paper - edu)]
[paper - CC/index.html)]
[resource
[resource

3.2 Mathematics

3.3 Code

3.4 General-purpose (Books, Encyclopedias, Instructions, Long Contexts, etc.)

II. Training Strategies

2.1 Training Frameworks

2.2 Training Strategies

2.3 Interpretability

2.4 Model Architecture Improvements

2.5 Learning Rate Annealing

I. Technical Reports

1.1 Dense Models

[paper

Baichuan Series

DeepSeek Series

Falcon Series

Gemini Series

Gemma Series

GLM Series

Hunyuan Series

[paper

InternLM Series

Ling Series

[paper

LLaMA Series

LLM360 Project

[paper

MAP-Neo Series

[paper

MiniCPM

[paper

Minimax Series

[paper

Mistral Series

Nemotron Series

OLMo Series

Phi Series

Qwen Series

Reka Series

[paper

Skywork Series

Yi Series

YuLan Series

[code - team/yulan-mini-676d214b24376739b00d95f3)] [paper]
[resource - GSAI/YuLan-Chat)] [paper]

IV. Data Methods

4.1 Tokenizers

[code
[code - with-vocab-demo)] [paper]
[code

4.2 Data Mixing and Curriculum

[paper
[paper
[homepage
[demo - gram.io/)] [paper]

4.3 Data Synthesis

Star History

4.3 Data Synthesis

![Star History Chart - history.com/#RUCAIBox/awesome-llm-pretraining&Date)
![Star History Chart - history.com/#RUCAIBox/awesome-llm-pretraining&Date)
![Star History Chart - history.com/#RUCAIBox/awesome-llm-pretraining&Date)