{"id":51417097,"url":"https://github.com/bob798/ohmygpt","last_synced_at":"2026-07-04T20:30:37.007Z","repository":{"id":364988298,"uuid":"1270037153","full_name":"bob798/ohmygpt","owner":"bob798","description":"从0到1手写的中文小型大语言模型 — minimind 风格教学项目：RMSNorm·RoPE·GQA·SwiGLU，完整 tokenizer→pretrain→SFT 管线，单张 RTX 3060 可训。A from-scratch Chinese LLM for learning.","archived":false,"fork":false,"pushed_at":"2026-06-15T10:50:51.000Z","size":41,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-15T12:22:23.296Z","etag":null,"topics":["chinese-nlp","education","from-scratch","gqa","llm","minimind","pretraining","pytorch","rope","sft","transformer"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bob798.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-15T10:20:34.000Z","updated_at":"2026-06-15T10:50:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/bob798/ohmygpt","commit_stats":null,"previous_names":["bob798/ohmygpt"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/bob798/ohmygpt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bob798%2Fohmygpt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bob798%2Fohmygpt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bob798%2Fohmygpt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bob798%2Fohmygpt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bob798","download_url":"https://codeload.github.com/bob798/ohmygpt/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bob798%2Fohmygpt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35135341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-04T02:00:05.987Z","response_time":113,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chinese-nlp","education","from-scratch","gqa","llm","minimind","pretraining","pytorch","rope","sft","transformer"],"created_at":"2026-07-04T20:30:36.526Z","updated_at":"2026-07-04T20:30:36.998Z","avatar_url":"https://github.com/bob798.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/banner.svg\" alt=\"ohmygpt — 从0到1手写的中文小型大语言模型\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cb\u003e中文\u003c/b\u003e · \u003ca href=\"README_EN.md\"\u003eEnglish\u003c/a\u003e\u003c/p\u003e\n\n# ohmygpt\n\n\u003e 从 0 到 1 手写一个中文小型大语言模型 —— minimind 风格的教学项目。\n\u003e A small Chinese LLM built **from scratch**, for learning how modern LLMs actually work.\n\n`tokenizer → pretrain → SFT`，每个核心机制都手写实现、可读可改，只在「无聊的管道」（数据加载、BPE 训练）上依赖现成库。\n\n## About\n\nohmygpt 的目标是**学习 LLM 内部原理**，而不是追求 SOTA 性能。它复刻了现代解码器（Llama / Qwen 同款组件）的关键设计，并提供完整但精简的训练管线，可在一张 **RTX 3060 (12GB)** 上从零训练出一个能补全中文、能简单对话的小模型。\n\n代码刻意保持轻量与透明：\n\n- **无重型框架**：不用 accelerate / deepspeed，训练循环自己写，每一步都看得见。\n- **现代架构，全部手写**：RMSNorm、RoPE 旋转位置编码、分组查询注意力（GQA）、SwiGLU 前馈、权重共享（tied embeddings）。\n- **完整管线**：训练分词器 → 预训练基座模型 → 指令微调（SFT，带 prompt 损失掩码）→ 推理（top-p 采样 + 多轮对话）。\n- **正确性优先**：含 overfit-one-batch 等单元测试，作为「架构是否接对」的关键信号。\n\n设计与实现计划见 [`docs/superpowers/`](docs/superpowers/)。灵感来自 [jingyaogong/minimind](https://github.com/jingyaogong/minimind)。\n\n## Features\n\n| 模块 | 文件 | 说明 |\n|------|------|------|\n| 模型 | `model/config.py`, `model/model.py` | 解码器：RMSNorm · RoPE · GQA · SwiGLU · 权重共享 |\n| 分词器 | `train/train_tokenizer.py` | byte-level BPE，词表 6400，特殊符 `\u003cunk\u003e/\u003cs\u003e/\u003c/s\u003e` + 对话模板 |\n| 数据集 | `dataset.py` | `PretrainDataset`（打包定长窗口）+ `SFTDataset`（对话模板 + 仅对答案计损失） |\n| 训练 | `train/pretrain.py`, `train/sft.py` | AdamW、cosine 调度 + warmup、bf16/fp16 混合精度、梯度累积、梯度裁剪 |\n| 推理 | `inference.py` | top-p（nucleus）采样，KV-cache 加速，支持补全（complete）与对话（chat）两种模式 |\n| 测试 | `tests/` | RMSNorm/RoPE/注意力/前馈/整模型单测 + overfit 校验 + 分词器往返 + 损失掩码 + 缓存生成一致性测试 |\n\n## Model presets\n\n两套配置，先用 `small` 快速跑通管线，再用 `base` 正式训练。\n\n| 参数 | `small`（≈6M） | `base`（≈26M） |\n|------|---------------|----------------|\n| dim（隐藏维度） | 256 | 512 |\n| n_layers | 4 | 8 |\n| n_heads | 8 | 16 |\n| n_kv_heads（GQA） | 4 | 8 |\n| max_seq_len | 512 | 512 |\n| vocab_size | 6400 | 6400 |\n\n## Project layout\n\n```\nohmygpt/\n├── model/\n│   ├── config.py          # ModelConfig 数据类 + small/base 预设\n│   ├── model.py           # Transformer：RMSNorm, RoPE, GQA, SwiGLU\n│   └── tokenizer/         # 训练好的 BPE 分词器文件\n├── dataset.py             # PretrainDataset / SFTDataset + 加载器\n├── train/\n│   ├── train_tokenizer.py # 训练 BPE 分词器\n│   ├── pretrain.py        # 预训练循环\n│   └── sft.py             # 指令微调循环\n├── inference.py           # 加载 checkpoint，采样 / 对话\n├── tests/                 # 单元与正确性测试\n├── data/                  # minimind 数据集（已 gitignore）\n└── docs/superpowers/      # 设计文档与实现计划\n```\n\n## Setup\n\n```bash\npython -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -r requirements.txt\n```\n\n\u003e 训练需要 NVIDIA GPU（推荐 RTX 3060 12GB 或更高）。CPU/MPS 可运行单元测试，但不适合实际训练。\n\n## Pipeline\n\n1. 下载 minimind 数据集到 `data/`（来自 HuggingFace `jingyaogong/minimind_dataset`）：\n   ```bash\n   python scripts/download_data.py            # 国内慢/被墙可加：--endpoint https://hf-mirror.com\n   ```\n2. 训练分词器：`python train/train_tokenizer.py`\n3. 预训练：`python train/pretrain.py --preset base`\n4. 指令微调：`python train/sft.py --preset base`\n5. 对话：`python inference.py --ckpt checkpoints/sft.pt --mode chat --prompt \"你好\"`\n\n**建议先用 `--preset small` 端到端跑通一遍**，确认管线无误后再上 `base`：\n\n```bash\npython train/pretrain.py --preset small --limit 2000 --batch_size 4 --accum_steps 2\n```\n\n显存不够时，调小 `--batch_size` 并调大 `--accum_steps`，保持有效 batch 不变。\n\n## Sanity checks\n\n训练前务必先跑这两个检查，它们能在浪费 GPU 时间前抓出接线错误：\n\n- `pytest tests/test_model.py::test_overfit_single_batch` —— 单批过拟合，loss 必须降到 0.1 以下。\n- `pytest tests/test_generate.py` —— KV-cache 生成结果须与朴素全量重算一致。\n- `pytest tests/test_tokenizer.py` —— 分词器编解码往返一致。\n\n完整测试：`pytest -v`。\n\n## Acknowledgements\n\n- 灵感与数据集来自 [jingyaogong/minimind](https://github.com/jingyaogong/minimind)。\n- 架构参考 Llama / Qwen 系列的现代解码器设计。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbob798%2Fohmygpt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbob798%2Fohmygpt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbob798%2Fohmygpt/lists"}