{"id":13617948,"url":"https://github.com/yuanzhoulvpi2017/zero_nlp","last_synced_at":"2025-05-14T01:06:12.644Z","repository":{"id":65956912,"uuid":"597644883","full_name":"yuanzhoulvpi2017/zero_nlp","owner":"yuanzhoulvpi2017","description":"中文nlp解决方案(大模型、数据、模型、训练、推理) ","archived":false,"fork":false,"pushed_at":"2025-02-12T13:56:56.000Z","size":53308,"stargazers_count":3434,"open_issues_count":100,"forks_count":403,"subscribers_count":31,"default_branch":"main","last_synced_at":"2025-05-11T01:01:45.680Z","etag":null,"topics":["bert","chatglm-6b","clip","gpt","gpt2","huggingface-transformers","llama","llama2","llava","nlp","pytorch","text-generation","transformers"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yuanzhoulvpi2017.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-02-05T07:11:18.000Z","updated_at":"2025-05-10T11:46:26.000Z","dependencies_parsed_at":"2024-07-18T10:04:07.577Z","dependency_job_id":"558387dc-535e-42c6-ba55-8e01d513ad92","html_url":"https://github.com/yuanzhoulvpi2017/zero_nlp","commit_stats":{"total_commits":232,"total_committers":3,"mean_commits":77.33333333333333,"dds":0.008620689655172376,"last_synced_commit":"3c8d6cfa8eaf5e9115f8855e893b88e0fe81232f"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanzhoulvpi2017%2Fzero_nlp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanzhoulvpi2017%2Fzero_nlp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanzhoulvpi2017%2Fzero_nlp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanzhoulvpi2017%2Fzero_nlp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yuanzhoulvpi2017","download_url":"https://codeload.github.com/yuanzhoulvpi2017/zero_nlp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253660822,"owners_count":21943822,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","chatglm-6b","clip","gpt","gpt2","huggingface-transformers","llama","llama2","llava","nlp","pytorch","text-generation","transformers"],"created_at":"2024-08-01T20:01:51.313Z","updated_at":"2025-05-14T01:06:07.635Z","avatar_url":"https://github.com/yuanzhoulvpi2017.png","language":"Jupyter Notebook","funding_links":[],"categories":["其他_NLP自然语言处理","AI应用","Jupyter Notebook"],"sub_categories":["其他_文本生成、文本对话"],"readme":"# zero to nlp\n\n## 特点\n\n1. 🎯`目标`：基于`pytorch`、`transformers`做中文领域的nlp开箱即用的训练框架，提供全套的训练、微调模型（包括大模型、文本转向量、文本生成、多模态等模型）的解决方案；\n2. 💽`数据`：\n    - 从开源社区，整理了海量的训练数据，帮助用户可以快速上手；\n    - 同时也开放训练数据模版，可以快速处理垂直领域数据；\n    - 结合多线程、内存映射等更高效的数据处理方式，即使需要处理`百GB`规模的数据，也是轻而易举；\n3. 💻`流程`：每一个项目有完整的模型训练步骤，如：数据清洗、数据处理、模型构建、模型训练、模型部署、模型图解；\n4. 🔥`模型`：当前已经支持`gpt2`、`clip`、`gpt-neox`、`dolly`、`llama`、`chatglm-6b`、`VisionEncoderDecoderModel`等多模态大模型；\n5. 🚀`多卡串联`\n   ：当前，多数的大模型的尺寸已经远远大于单个消费级显卡的显存，需要将多个显卡串联，才能训练大模型、才能部署大模型。因此对部分模型结构进行修改，实现了`训练时`、`推理时`\n   的多卡串联功能。\n6. ⚙️`模型工具`：添加了大模型的`词表裁切`和`词表扩充`\n   教程[model_modify](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/model_modify)\n\n## 目录\n\n[//]: # (### 源码解读)\n\n[//]: # ()\n\n[//]: # (当前`transformers`包，确实好用，包括训练等，但是我们不能停留于表面，不能浅尝辄止。要深入源码底部，挖掘出每一个细节。因此，在这个模块中，我将把)\n\n[//]: # (`transfrmers`包中用到的python高级用法、优秀的数据处理思路和方法，尽可能的讲解清楚。)\n\n[//]: # ()\n\n[//]: # (⚠️将逐步完善，敬请期待)\n\n[//]: # (| 模块         | 文件名称 | 作用  | 实现细节 |)\n\n[//]: # (|------------|------|-----|------|)\n\n[//]: # (| Tokenizer  | ☑️   | ☑️  | ☑️   |)\n\n[//]: # (| Datasets   | ☑️   | ☑️  | ☑️   |)\n\n[//]: # (| Model      | ☑️   | ☑️  | ☑️   |)\n\n[//]: # (| Trainer    | ☑️   | ☑️  | ☑️   |)\n\n[//]: # (| AutoClass  | ☑️   | ☑️  | ☑️   |)\n\n[//]: # (| AutoConfig | ☑️   | ☑️  | ☑️   |)\n\n### 模型训练\n\n| 中文名称                              | 文件夹名称                                                                                                                 | 数据 | 数据清洗 | 大模型 | 模型部署 | 图解 |\n|-----------------------------------|-----------------------------------------------------------------------------------------------------------------------|----|------|-----|------|----|\n| 中文文本分类                            | [chinese_classifier](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_classifier)                       | ✅  | ✅    | ✅   | ❌    | ✅  |\n| 中文`gpt2`                          | [chinese_gpt2](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_gpt2)                                   | ✅  | ✅    | ✅   | ✅    | ❌  |\n| 中文`clip`                          | [chinese_clip](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_clip_ddp)                               | ✅  | ✅    | ✅   | ❌    | ✅  |\n| 图像生成中文文本                          | [VisionEncoderDecoderModel](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/vit-gpt2-image-chinese-captioning) | ✅  | ✅    | ✅   | ❌    | ✅  |\n| vit核心源码介绍                         | [vit model](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/vit)                                               | ❌  | ❌    | ❌   | ❌    | ✅  |\n| `Thu-ChatGlm-6b`(`v1`版本 作废)       | [simple_thu_chatglm6b](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/simple_thu_chatglm6b)                   | ✅  | ✅    | ✅   | ✅    | ❌  |\n| 🌟chatglm-`v2`-6b🎉               | [chatglm_v2_6b_lora](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chatglm_v2_6b_lora)                       | ✅  | ✅    | ✅   | ❌    | ❌  |\n| 中文`dolly_v2_3b`                   | [dolly_v2_3b](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_dolly_v2_3b)                             | ✅  | ✅    | ✅   | ❌    | ❌  |\n| 中文`llama`(作废)                     | [chinese_llama](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_llama)                                 | ✅  | ✅    | ✅   | ❌    | ❌  |\n| 中文`bloom`                         | [chinese_bloom](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_bloom)                                 | ✅  | ✅    | ✅   | ❌    | ❌  |\n| 中文`falcon`(注意：falcon模型和bloom结构类似) | [chinese_bloom](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/chinese_bloom)                                 | ✅  | ✅    | ✅   | ❌    | ❌  |\n| 中文**预训练**代码                       | [model_clm](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/model_clm)                                         | ✅  | ✅    | ✅   | ❌    | ❌  |\n| 百川大模型                             | [model_baichuan](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/model_baichuan)                               | ✅  | ✅    | ✅   | ✅    | ❌  |\n| 模型修剪✂️                            | [model_modify](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/model_modify)                                   | ✅  | ✅    | ✅   |      |    |\n| llama2 流水线并行                      | [pipeline](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/pipeline)                                           | ✅  | ✅    | ✅   | ❌    | ❌  |\n| 百川2-7b-chat的`dpo`                 | [DPO baichuan2-7b-chat ](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/baichuan2_dpo)                        | ✅  | ✅    | ✅   | ❌    | ❌  |\n| 训练时候，数据占比发生变化                     | [train_data_sample ](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/train_data_sample)                        | ✅  | ✅    | ✅   | ❌    | ❌  |\n| internlm-base sft                 | [internlm-sft ](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/internlm-sft)                                  | ✅  | ✅    | ✅   | ❌    | ❌  |\n| train qwen2                       | [train_qwen2 ](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/train_qwen)                                     | ✅  | ✅    | ✅   | ✅    | ❌  |\n| train llava                       | [train_llava ](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/train_llava)                                    | ✅  | ✅    | ✅   | ✅    | ✅  |\n\n### 工程介绍 debug vllm\n\n1. 介绍如何debug\n   vllm，对vllm工程上了解的更加透彻 [debug vllm](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/debug_vllm)\n\n\u003cdetails\u003e\u003csummary\u003e\u003cb\u003e数据流程图解\u003c/b\u003e\u003c/summary\u003e\n\n\n我一直觉得，数据流程通过图解的形式表达出来，其实是最清楚的，因此我都会尽可能的把每一个任务的都图解出来。\n\n### 文本分类数据图解\n\n![](images/文本分类.003.png)\n\n### 中文gpt2\n\n![](images/chinesegpt2_bot.png)\n\n### 中文clip\n\n![model](images/clip001.png)\n\n### 图像生成中文文本\n\n![model](images/vision-encoder-decoder.png)\n\n### vit 源码\n\n![](images/vit_architecture.jpg)\n\u003c/details\u003e\n\n# 分享transformers源码解读\n\n一直在做transformers的源码解读，可以去B站查看视频👉[良睦路程序员](https://space.bilibili.com/45156039)\n\n[//]: # (# 分享数据)\n\n[//]: # ()\n\n[//]: # (一直在整理开源数据，如果有需要，可以关注公众号`统计学人`，回复`nlp数据`即可。目前还在整理数据中)\n\n[//]: # ()\n\n[//]: # (![统计学人]\u0026#40;images/gzh.jpg\u0026#41;)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuanzhoulvpi2017%2Fzero_nlp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuanzhoulvpi2017%2Fzero_nlp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuanzhoulvpi2017%2Fzero_nlp/lists"}