AMD OLMo (Oct. 2024)

• 1B model
• SFT
• SFT-DPO
• Blog post

Cerebras

• Models Collection
• Paper
• Training Code

Databricks / formerly Mosaic ML

• Streaming Datasets
• Eval Gauntlet
• Pretraining Code
• Megablocks (MoE Training)
• DBRX Collection

DataComp (Jun. 2024)

• 1B Model
• Instruct
• 7B Model
• Instruct
• 8K Extension
• Pretrain Dataset
• Training + Eval Code
• Paper
• Evalchemy (Post-training Eval Code)

EleutherAI

• Models Collection
• Central Repo
• Paper
• Model
• Paper
• Central Repo
• Paper
• Dataset
• Blogpost
• Training Code
• Eval Code
• Paper
• The Pile Dataset
• Paper

HuggingFace SmolLM (v2 Oct. 2024)

• SmolLM 2 collection
• SmolLM instruction mix
• SmolLM collection
• SmolLM pretraining data
• Synthetic pretrain corpus
• Fineweb pretrain corpus
• Edu Subset
• Fineweb 2 (multilingual)
• SmolLM repo
• Training Code
• Eval Code
• Data Processing Code
• Fineweb
• Fineweb v2
• SmolLM

LLäMmlein (German LLaMa)

• Paper
• 120M - UniWue/LLaMmlein_1B), 7B, with intermediate checkpoints
• Dataset
• Pretraining codebase

LLM 360

• Analysis360
• Collection
• Data
• Data Preparation Code
• Training Code
• Data
• Data Preparation Code
• Training Code
• Collection
• Data
• Data Preparation Code
• Training Code
• Paper

M.A.P.

• Models Collection
• Datasets
• Paper
• Code

NVIDIA

• Megatron-LM
• NeMo
• Post-training Code
• Data Processing

OLMo 2 (Nov. 2024)

• Collection
• 7B base model
• 13B base model
• 7B instruct
• 13B instruct
• Annealing dataset
• Training Code (1st gen.)
• Training Code (2nd gen.)
• Post-train Code
• Eval Code
• Data Processing Toolkit
• Demo

Open-sci, OpenEuroLLM, LAION AI (Sept. 2025)

• Paper
• Blog
• HF collections

PyTorch / Meta

• Torch Titan Pretraining Code
• Meta Lingua Pretraining Code

RWKV

• RWKV-5 Models Collection
• RWKV-6 Models Collection
• Training Code
• RWKV-4
• RWKV-5 and RWKV-6

Together.AI

• RedPajama v1 Dataset
• RedPajama v2 Dataset
• Data Code

Zyphra

• 7B
• 7B Instruct
• Blogpost
• 2.7B
• Blogpost
• 1.2B
• Blogpost
• Paper
• Zyda 2 Dataset
• Blogpost
• Paper