{"id":30702409,"url":"https://github.com/mlnlp-world/minimind-notes","last_synced_at":"2025-09-08T00:02:15.757Z","repository":{"id":312866698,"uuid":"1048794651","full_name":"MLNLP-World/minimind-notes","owner":"MLNLP-World","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-02T12:39:46.000Z","size":9596,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-02T14:38:39.383Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MLNLP-World.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-02T03:24:07.000Z","updated_at":"2025-09-02T12:39:50.000Z","dependencies_parsed_at":"2025-09-02T14:48:57.527Z","dependency_job_id":null,"html_url":"https://github.com/MLNLP-World/minimind-notes","commit_stats":null,"previous_names":["mlnlp-world/minimind-notes"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/MLNLP-World/minimind-notes","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MLNLP-World%2Fminimind-notes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MLNLP-World%2Fminimind-notes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MLNLP-World%2Fminimind-notes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MLNLP-World%2Fminimind-notes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MLNLP-World","download_url":"https://codeload.github.com/MLNLP-World/minimind-notes/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MLNLP-World%2Fminimind-notes/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273303048,"owners_count":25081354,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-02T15:20:28.927Z","updated_at":"2025-09-02T15:20:32.166Z","avatar_url":"https://github.com/MLNLP-World.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# MiniMind Notes 📝\n\n\u003cdiv align=\"center\"\u003e\n\n![logo](./images/logo.png)\n\n[![GitHub Repo stars](https://img.shields.io/github/stars/llm-chaser/minimind-notes?style=social)](https://github.com/MLNLP-World/minimind-notes/stargazers)  \n[![GitHub pull request](https://img.shields.io/badge/PRs-welcome-blue)](https://github.com/MLNLP-World/minimind-notes/pulls)  \n[![License](https://img.shields.io/github/license/MLNLP-World/minimind-notes)](LICENSE)\n\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n[项目动机](#项目动机) / [项目简介](#项目简介) / [项目资源](#项目资源) / [注释风格与示例](#注释风格与示例) / [原书readme](#原书readme) / [贡献指南](#贡献指南) / [致谢](#致谢)\n\n\u003c/div\u003e\n\n---\n\n\u003ch2 id=\"项目动机\"\u003e🎯 项目动机\u003c/h2\u003e\n\n原项目与地址：[MiniMind](https://github.com/jingyaogong/minimind)  \n\n本项目是对 GitHub 项目 **《MiniMind》** 的 **注释增强版**。在原始代码的基础上，我们补充了逐文件、逐模块的详细中文注释，帮助读者更好地理解大模型的训练流程。  \n\n在整理过程中，我们尽可能保持原始实现的完整性和准确性，同时在注释上进行了适度优化，以更贴合中文学习者的阅读习惯。需要特别说明的是，**原作者才是该项目的主要贡献者**，本仓库仅作为学习辅助资料，不对原内容进行修改或扩展。  \n\n由于个人能力有限，注释中难免存在不完善之处，欢迎大家提出宝贵意见。希望通过这一项目，更多学习者能够快速从“能跑起来”过渡到“能理解、能修改”，也希望为国内社区的大模型学习与研究贡献一份力量。    \n\n---\n\n\n\u003ch2 id=\"项目简介\"\u003e📚 项目简介\u003c/h2\u003e\n\n大语言模型（LLMs）已经成为当前人工智能发展的核心驱动力，但对大多数学习者来说，**从零开始理解并实现一个 LLM 的训练流程仍然充满挑战**。  \n\n原始的 [MiniMind](https://github.com/jingyaogong/minimind) 项目提供了一个极轻量的全流程实现：  \n- 仅需低成本 GPU，就能在 2 小时内训练一个 26M 参数的 GPT 模型；  \n- 覆盖了从 *Tokenizer → Pretrain → SFT → LoRA → DPO → 蒸馏* 的完整链路。  \n\n在此基础上，**MiniMind-Notes** 的定位是：在原项目的实现基础上，补充逐文件、逐模块的详尽注释，帮助学习者降低理解门槛，从而更快掌握训练流程与实现原理。  \n\n### ✨ 本项目的特色\n- 每个源码文件包含用途说明、核心原理与特点总结；  \n- 函数配备清晰的 docstring 与行内注释；  \n- 注释均为中文编写，适合初学者快速上手；  \n- 与原仓库保持同步更新，保证可复现性。  \n\n### 💡 通过本项目，你将能够\n- 理解从零实现一个 GPT 风格 LLM 的完整流程；  \n- 掌握预训练 (Pretrain)、监督微调 (SFT)、LoRA、DPO 等核心环节；  \n- 学会如何准备数据集、运行训练、评估模型；  \n- 提升源码阅读与理解能力，为后续研究与改进打下基础。  \n\n---\n\n\u003ch2 id=\"项目资源\"\u003e📖 项目资源\u003c/h2\u003e\n\n- 原版项目地址：[MiniMind](https://github.com/jingyaogong/minimind)  \n- 注释增强版：[MiniMind-Notes](https://github.com/MLNLP-World/minimind-notes)  \n- 数据集下载：\n  - [MiniMind Dataset @ HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind_dataset/tree/main)  \n  - [MiniMind Dataset @ ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files)  \n- 模型下载：[MiniMind2](https://huggingface.co/jingyaogong/MiniMind2)  \n\n快速体验：\n```bash\ngit clone https://github.com/MLNLP-World/minimind-notes.git\n\n````\n\n\n---\n\n\u003ch2 id=\"注释风格与示例\"\u003e📖 注释风格与示例\u003c/h2\u003e\n\n\n### 🖊️ 注释风格\n代码采用 **文件级三段式说明 + 函数 docstring + 关键行内注释**，三者结合，保证既能宏观理解模块作用，又能在细节处快速上手。\n\n### 📝 示例\n以下为完整示例，分别展示三种注释形式：\n```python\n# ===== 1) 文件级三段式说明 =====\n文件用途：\n  使用 Direct Preference Optimization (DPO) 对 MiniMind 进行偏好对齐训练。\n\n核心原理：\n  - 无需 RL 中的奖励模型（RM）和 PPO 复杂流程，直接通过成对偏好数据（chosen/rejected）优化模型。\n  - 核心思想：让模型对“人类偏好的回答（chosen）”输出概率高于“不偏好的回答（rejected）”，以参考模型（ref_model）为基准。\n\n特点：\n  - 参考模型 ref_model 固定不训练；被训练模型与其对比 log 概率比\n  - 支持 AMP 混合精度、DDP 分布式、余弦退火学习率\n  - 数据集为成对样本（chosen / rejected），由 DPODataset 提供\n  - 仅对 assistant 回复区域计入损失（由 mask 控制）\n```\n```python\n# ===== 2) 函数 docstring 示例 =====\nclass MOEFeedForward(nn.Module):\n    \"\"\"\n    MoE 版本的前馈网络 (FFN)：\n      - 训练：按 top-k 将 token 分配给专家并加权合并\n      - 推理：按专家分组批量前向（moe_infer，无梯度）\n      - 可选 shared_experts：对残差进行额外的共享专家修正\n    \"\"\"\n```\n\n```python\n# ===== 3) 关键行内注释示例 =====\n    def forward(self, x):\n        identity = x\n        orig_shape = x.shape\n        bsz, seq_len, _ = x.shape\n        # 1) 门控选择专家\n        topk_idx, topk_weight, aux_loss = self.gate(x)\n        # 2) 训练/推理两条路径\n        x = x.view(-1, x.shape[-1])          # [bs*seq, hidden]\n        flat_topk_idx = topk_idx.view(-1)    # [bs*seq*topk]\n        if self.training:\n            # 训练：复制 token，按专家掩码分别前向，再按权重聚合\n            x = x.repeat_interleave(self.config.num_experts_per_tok, dim=0)\n            y = torch.empty_like(x, dtype=torch.float16)  # 缓存各专家输出（节省显存）\n            for i, expert in enumerate(self.experts):\n                y[flat_topk_idx == i] = expert(x[flat_topk_idx == i]).to(y.dtype)\n            # 按 topk 权重加权融合\n            y = (y.view(*topk_weight.shape, -1) * topk_weight.unsqueeze(-1)).sum(dim=1)\n            y = y.view(*orig_shape)\n        else:\n            # 推理：无梯度 + 分组批量前向，减少小批次 kernel 启动开销\n            y = self.moe_infer(x, flat_topk_idx, topk_weight.view(-1, 1)).view(*orig_shape)\n        # 3) 共享专家（残差修正）\n        if self.config.n_shared_experts \u003e 0:\n            for expert in self.shared_experts:\n                y = y + expert(identity)\n        # 暴露 aux_loss，供上层汇总\n        self.aux_loss = aux_loss\n        return y\n```\n\n---\n\n\u003ch2 id=\"原书readme\"\u003e📜 原书readme\u003c/h2\u003e\n\n以下为 [MiniMind 官方项目](https://github.com/jingyaogong/minimind) 的核心内容：\n\n\u003e 此开源项目旨在完全从0开始，仅用 **3块钱成本 + 2小时**，即可训练出仅 25.8M 的超小语言模型 MiniMind。\n\u003e MiniMind 系列极其轻量，最小版本体积是 GPT-3 的 1/7000，力求做到最普通的个人 GPU 也可快速训练。\n\u003e 项目同时开源了从 **Tokenizer → Pretrain → SFT → LoRA → DPO → 蒸馏** 的全过程代码。\n\n### 🔧 快速开始\n\n```bash\ngit clone https://github.com/jingyaogong/minimind.git\ncd minimind\npip install -r requirements.txt\n```\n\n下载模型：\n\n```bash\ngit clone https://huggingface.co/jingyaogong/MiniMind2\n```\n\n预训练：\n\n```bash\npython trainer/train_pretrain.py\n```\n\n监督微调：\n\n```bash\npython trainer/train_full_sft.py\n```\n\n评估：\n\n```bash\npython eval_model.py --model_mode 1\n```\n\n### 📂 数据集示例\n\n推荐最小组合：\n\n* `pretrain_hq.jsonl`\n* `sft_mini_512.jsonl`\n\n更多数据集详见：[MiniMind Dataset](https://huggingface.co/datasets/jingyaogong/minimind_dataset/tree/main)。\n\n---\n\n\u003ch2 id=\"贡献指南\"\u003e🤝 贡献指南\u003c/h2\u003e\n\n* 欢迎提交 **更详细的注释**、**学习笔记** 或 **改进建议**。\n* 我们的目标是让这个项目成为 **LLM 学习入门的参考手册**。\n\n---\n\n\u003ch2 id=\"致谢\"\u003e📝 致谢\u003c/h2\u003e\n\n* 原项目作者：[jingyaogong/minimind](https://github.com/jingyaogong/minimind)\n* 已参与的贡献者 🙌\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/jingyaogong\"\u003e\n        \u003cimg src=\"./images/contributors/jingyaogong.jpeg\" width=\"80\" height=\"80\" alt=\"jingyaogong\"/\u003e\u003cbr/\u003e\n        \u003csub\u003e\u003cb\u003ejingyaogong\u003c/b\u003e\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/llm-chaser\"\u003e\n        \u003cimg src=\"./images/contributors/llm-chaser.jpg\" width=\"80\" height=\"80\" alt=\"llm-chaser\"/\u003e\u003cbr/\u003e\n        \u003csub\u003e\u003cb\u003ellm-chaser\u003c/b\u003e\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/jiusheng58\"\u003e\n        \u003cimg src=\"./images/contributors/jiusheng58.png\" width=\"80\" height=\"80\" alt=\"jiusheng58\"/\u003e\u003cbr/\u003e\n        \u003csub\u003e\u003cb\u003ejiusheng58\u003c/b\u003e\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/lul0308\"\u003e\n        \u003cimg src=\"./images/contributors/lul0308.png\" width=\"80\" height=\"80\" alt=\"jiusheng58\"/\u003e\u003cbr/\u003e\n        \u003csub\u003e\u003cb\u003elul0308\u003c/b\u003e\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/qinzheng-li\"\u003e\n        \u003cimg src=\"./images/contributors/qinzheng-li.png\" width=\"80\" height=\"80\" alt=\"jiusheng58\"/\u003e\u003cbr/\u003e\n        \u003csub\u003e\u003cb\u003eqinzheng-li\u003c/b\u003e\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n---\n\n\u003ch2 id=\"license\"\u003e📜 License\u003c/h2\u003e\n\n本项目采用 [Apache-2.0 License](LICENSE)。\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlnlp-world%2Fminimind-notes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlnlp-world%2Fminimind-notes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlnlp-world%2Fminimind-notes/lists"}