{"id":13641398,"url":"https://github.com/charent/Phi2-mini-Chinese","last_synced_at":"2025-04-20T07:33:19.469Z","repository":{"id":214006382,"uuid":"734557915","full_name":"charent/Phi2-mini-Chinese","owner":"charent","description":"Phi2-Chinese-0.2B 从0开始训练自己的Phi2中文小模型，支持接入langchain加载本地知识库做检索增强生成RAG。Training your own Phi2 small chat model from scratch.","archived":false,"fork":false,"pushed_at":"2024-07-11T15:12:53.000Z","size":183,"stargazers_count":545,"open_issues_count":1,"forks_count":61,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-04-04T19:11:11.196Z","etag":null,"topics":["language-model","phi-2","retrieval-augmented-generation","text-generation"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/charent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-22T02:03:27.000Z","updated_at":"2025-04-01T14:01:39.000Z","dependencies_parsed_at":"2024-11-09T11:32:23.429Z","dependency_job_id":"0f2b10a4-f01a-49b2-92b9-1992f7acb5c9","html_url":"https://github.com/charent/Phi2-mini-Chinese","commit_stats":null,"previous_names":["charent/phi2-mini-chinese"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/charent%2FPhi2-mini-Chinese","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/charent%2FPhi2-mini-Chinese/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/charent%2FPhi2-mini-Chinese/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/charent%2FPhi2-mini-Chinese/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/charent","download_url":"https://codeload.github.com/charent/Phi2-mini-Chinese/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249864350,"owners_count":21336727,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["language-model","phi-2","retrieval-augmented-generation","text-generation"],"created_at":"2024-08-02T01:01:20.411Z","updated_at":"2025-04-20T07:33:19.161Z","avatar_url":"https://github.com/charent.png","language":"Jupyter Notebook","funding_links":[],"categories":["Summary"],"sub_categories":[],"readme":"# Phi2-Chinese-0.2B 从0开始训练自己的Phi2中文小模型\n\n**本项目为实验项目，开源代码及模型权重，预训练数据较少，如果需要效果更好的中文小模型，可以参考项目[ChatLM-mini-Chinese](https://github.com/charent/ChatLM-mini-Chinese)**\n\n\u003e [!CAUTION]\n\u003e 本项目为实验性项目，随时会大改，包括训练数据、模型结构、文件目录结构等。\n\u003e 第一版模型及请查看`tag v1.0`\n\n- 支持flash attention 2 加速\n\n# 1. ⚗️数据清洗\n比如句末添加句号、繁体转简体、删除重复的标点符号（比如有些对话语料非常多`\"。。。。。\"`）、NFKC Unicode标准化（主要是全角转半角及网页数据的\\u3000 \\xa0问题）等等。   \n具体的数据清洗过程请参考项目[ChatLM-mini-Chinese](https://github.com/charent/ChatLM-mini-Chinese)。  \n\n# 2. 🗨️tokenizer训练 \n本项目使用`byte level`的`BPE`分词器。共提供的两种分词器`char level` 和`byte level`的训练代码。  \n\n训练完的tokenizer记得检查词表中是否有常见的特殊符号，如`\\t`、`\\n`等，可以尝试编一句包含特殊字符的文本`encode`、`decode`看看能不能还原。如果不包含这些特殊字符，通过`add_tokens`函数添加。使用`len(tokenizer)`获取词表大小，`tokenizer.vocab_size`不统计自己通过`add_tokens`函数添加的字符。     \n\ntokenizer训练非常吃内存：  \n\n- `byte level`训练1亿个字符至少需要`32G`内存（其实`32G`还是不太够，会频繁触发swap），`13600k`训练时长大概1个小时。   \n\n- `char level`训练6.5亿个字符（刚好是中文wiki百科的数据量）至少需要32G内存，因为多次触发了swap，实际使用量远不止32G，`13600K`训练时长约半个小时。   \n\n所以大数据集时（GB级别），建议训练`tokenizer`时从数据集中进行采样。  \n\n\n# 3. ⛏️CLM因果模型预训练 \n\n用大量文本进行无监督预训练，主要使用`bell open source`的数据集[BELLE](https://github.com/LianjiaTech/BELLE)。  \n\n数据集格式：一个样本一句话，太长的可以截断分为多个样本。  \n\nCLM预训练过程中，模型输入和输出是一样的，计算交叉熵损失的时候，要错开一位（`shift`）。  \n\n处理百科语料时，建议在每个词条结束后加上`'[EOS]'`标记。其他语料处理也类似，一个`doc`的结束（可以时一篇文章结束或者段落结束）都要加上`'[EOS]'`标记。开始标记`'[BOS]'`可加可不加。\n\n\n# 4. ⚒️SFT指令微调 \n\n主要使用`bell open source`的数据集。感谢大佬[BELLE](https://github.com/LianjiaTech/BELLE)。  \n\nSFT训练的数据格式如下：  \n```python\ntext = f\"##提问:\\n{example['instruction']}\\n##回答:\\n{example['output'][EOS]\"\n```\n模型计算损失时会忽略标记`\"##回答:\"`之前的部分（`\"##回答:\"`也会被忽略），从`\"##回答:\"`后面开始。\n\n记得添加`EOS`句子结束特殊标记，否则模型`decode`的时候不知道要什么时候停下来。`BOS`句子开始标记可填可不填。\n\n\n# 5. 📝RLHF优化\n\n采用更简单、更节省显存的dpo偏好优化方法。  \n\n根据个人喜好对SFT模型微调，数据集要构造三列`prompt`、`chosen`和 `rejected`，`rejected`这一列有部分数据我是从sft阶段初级模型（比如sft训练4个`epoch`，取0.5个`epoch`检查点的模型）生成，如果生成的`rejected`和`chosen`相似度在0.9以上，则不要这条数据。  \n\nDPO过程中要有两个模型，一个是要训练的模型，一个是参考的模型，在加载的时候其实是同一个模型，只不过参考模型不参与参数更新。  \n\n\n\n# 6. 📑本项目模型使用方法\n## 6.1 普通对话能力\n模型权重`huggingface`仓库：[Phi2-Chinese-0.2B](https://huggingface.co/charent/Phi2-Chinese-0.2B)  \n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig\nimport torch\n\ndevice = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n\ntokenizer = AutoTokenizer.from_pretrained('charent/Phi2-Chinese-0.2B')\nmodel = AutoModelForCausalLM.from_pretrained('charent/Phi2-Chinese-0.2B').to(device)\n\ntxt = '感冒了要怎么办？'\nprompt = f\"##提问:\\n{txt}\\n##回答:\\n\"\n\n# greedy search\ngen_conf = GenerationConfig(\n    num_beams=1,\n    do_sample=False,\n    max_length=320,\n    max_new_tokens=256,\n    no_repeat_ngram_size=4,\n    eos_token_id=tokenizer.eos_token_id,\n    pad_token_id=tokenizer.pad_token_id,\n)\n\ntokend = tokenizer.encode_plus(text=prompt)\ninput_ids, attention_mask = torch.LongTensor([tokend.input_ids]).to(device), \\\n    torch.LongTensor([tokend.attention_mask]).to(device)\n\noutputs = model.generate(\n    inputs=input_ids,\n    attention_mask=attention_mask,\n    generation_config=gen_conf,\n)\n\nouts = tokenizer.decode(outputs[0].cpu().numpy(), clean_up_tokenization_spaces=True, skip_special_tokens=True,)\nprint(outs)\n\n```\n```txt\n##提问:\n感冒了要怎么办？\n##回答:\n感冒是由病毒引起的，感冒一般由病毒引起，以下是一些常见感冒的方法：\n- 洗手，特别是在接触其他人或物品后。\n- 咳嗽或打喷嚏时用纸巾或手肘遮住口鼻。\n- 用手触摸口鼻，特别是喉咙和鼻子。\n- 如果咳嗽或打喷嚏，可以用纸巾或手绢来遮住口鼻，但要远离其他人。\n- 如果你感冒了，最好不要触摸自己的眼睛、鼻子和嘴巴。\n- 在感冒期间，最好保持充足的水分和休息，以缓解身体的疲劳。\n- 如果您已经感冒了，可以喝一些温水或盐水来补充体液。\n- 另外，如果感冒了，建议及时就医。\n```\n\n## 6.2 检索式生成（RAG）\n具体代码见`rag_with_langchain.ipynb`\n\n![rag](./imgs/rag.png)\n\n\n# 7、🎓引用\n如果你觉得本项目对你有所帮助，欢迎引用。  \n```conf\n@misc{Charent2023,\n    author={Charent Chen},\n    title={A small Chinese causal language model with 0.2B parameters base on Phi2},\n    year={2023},\n    publisher = {GitHub},\n    journal = {GitHub repository},\n    howpublished = {\\url{https://github.com/charent/Phi2-mini-Chinese}},\n}\n```\n\n# 8、🤔其他事项\n本项目不承担开源模型和代码导致的数据安全、舆情风险或发生任何模型被误导、滥用、传播、不当利用而产生的风险和责任。\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcharent%2FPhi2-mini-Chinese","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcharent%2FPhi2-mini-Chinese","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcharent%2FPhi2-mini-Chinese/lists"}