{"id":15293026,"url":"https://github.com/920232796/bert_seq2seq_ddp","last_synced_at":"2025-04-13T12:20:30.026Z","repository":{"id":53890092,"uuid":"484710371","full_name":"920232796/bert_seq2seq_DDP","owner":"920232796","description":"bert_seq2seq的DDP版本，支持bert、roberta、nezha、t5、gpt2等模型，支持seq2seq、ner、关系抽取等任务，无需添加额外代码，轻松启动DDP多卡训练。","archived":false,"fork":false,"pushed_at":"2022-08-29T09:55:00.000Z","size":319,"stargazers_count":51,"open_issues_count":2,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-27T03:22:29.119Z","etag":null,"topics":["bert","data-parallel","ddp","gpt2","nezha","pytorch","roberta","t5","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/920232796.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-23T10:03:02.000Z","updated_at":"2025-01-22T10:32:59.000Z","dependencies_parsed_at":"2022-08-13T03:20:41.284Z","dependency_job_id":null,"html_url":"https://github.com/920232796/bert_seq2seq_DDP","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/920232796%2Fbert_seq2seq_DDP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/920232796%2Fbert_seq2seq_DDP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/920232796%2Fbert_seq2seq_DDP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/920232796%2Fbert_seq2seq_DDP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/920232796","download_url":"https://codeload.github.com/920232796/bert_seq2seq_DDP/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248711037,"owners_count":21149282,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","data-parallel","ddp","gpt2","nezha","pytorch","roberta","t5","transformer"],"created_at":"2024-09-30T16:37:45.965Z","updated_at":"2025-04-13T12:20:29.996Z","avatar_url":"https://github.com/920232796.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bert_seq2seq_DDP\nbert_seq2seq的DDP(分布式训练)版本。\n此项目是对bert_seq2seq项目的重构并且很好的支持pytorch的DDP多卡训练。examples里面是各种训练例子，data中是样例数据。\n\n本项目可以轻松调用不同种类transformer结构的模型（Bert、Roberta、T5、Nezha、Bart等）针对不同的任务（生成、序列标注、文本分类、关系抽取、命名实体识别等）进行快速的训练、预测，并且无缝进行分布式（DDP）训练。\n\n**一个不同的数据集，只需要花5-10分钟修改好构建输入输出的函数，即可快速开始训练！**\n#### 欢迎加入交流群～ 可以提问题，提建议，互相交流，还会提供部分数据与模型的下载 QQ群: 975907202 微信群: w11267191 加好友拉入群～\n\n\n更多关于bert_seq2seq相关的内容请看：https://github.com/920232796/bert_seq2seq\n\n### 项目特点一：\n单卡训练与多卡训练方式相同，无需添加额外代码和使用额外命令运行。\n\n单卡与多卡的运行方式均为：\n```shell\npython \"./train.py\" ## train.py为example中以train开头的训练脚本文件\n```\n切换多卡训练只需要修改 ```train.py``` 文件中的环境设置即可：\n\n```python\nnum_gpus = 4 # gpu个数\nnum_nodes = 1 ## 机器个数 目前只支持1 ，多机待测试。\ntrainer = Trainer(env_type=\"DDP\",## DDP为pytorch的分布式数据并行训练\n                  epoches=5, model_save_dir=model_save_dir,\n                  val_every_step=500, device=device,\n                  batch_size=16, num_gpus=num_gpus, num_nodes=num_nodes,\n                  training_script=__file__,\n                  )\n```\n具体例子代码可以参考:\n\n[train_roberta_auto_title_multi_gpu.py](https://github.com/920232796/bert_seq2seq_DDP/blob/master/examples/seq2seq/bert/train_roberta_auto_title_multi_gpu.py) 自动标题任务，多gpu训练。\n\n### 项目特点二:\n虽然使用Trainer类进行了封装，也能做到比较灵活的evaluation.\n\n#### 自定义Evaluator类，可以自由进行验证\n\n[train_roberta_auto_title.py](https://github.com/920232796/bert_seq2seq_DDP/blob/master/examples/seq2seq/bert/train_roberta_auto_title.py) 自动标题任务，在训练过程中很方便打印下生成内容。\n\n[train_roberta_semantic_matching.py](https://github.com/920232796/bert_seq2seq_DDP/blob/master/examples/text_classification/train_roberta_semantic_matching.py) 语义匹配任务，在训练过程中轻松插入验证过程。\n\n### 项目特点三:\n提供样例数据在data目录中，帮助理解代码运行过程，供参考（qq群文件里提供部分任务的全部数据）。\n### 环境配置\n#### 安装pytorch，不是太旧的版本即可。\nhttps://pytorch.org/\n#### 安装额外的包\n```commandline\npip install bert_seq2seq_DDP \npip install tqdm\npip install scikit-learn //可选\n```\n网络不好请切换国内源进行安装\n\n#### 模型预训练参数、字典下载\n1. roberta模型(支持base、large)，模型和字典文件下载地址：https://drive.google.com/file/d/1iNeYFhCBJWeUsIlnW_2K6SMwXkM4gLb_/view 这里下载。 参考github仓库：https://github.com/ymcui/Chinese-BERT-wwm ，roberta-large模型也是在里面进行下载即可。\n2. bert模型(支持base、large)，下载bert中文预训练权重 \"bert-base-chinese\": \"https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-pytorch_model.bin\", 下载bert中文字典 \"bert-base-chinese\": \"https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-vocab.txt\".\n3. nezha模型，字典权重位置（支持base、large）：nezha-base模型下载：链接: https://pan.baidu.com/s/1Z0SJbISsKzAgs0lT9hFyZQ 提取码: 4awe\n4. gpt2中文模型：gpt2中文通用模型和字典下载地址：https://pan.baidu.com/s/1vTYc8fJUmlQrre5p0JRelw  密码: f5un，下载好即可在 [examples/seq2seq/gpt2_text_writting.py](https://github.com/920232796/bert_seq2seq_DDP/blob/master/examples/seq2seq/gpt2_text_writting.py) 中进行续写测试。\n6. t5中文模型(支持base、small)，预训练参数下载：https://github.com/renmada/t5-pegasus-pytorch\n7. SimBert模型，相似句的生成，预训练模型使用bert、roberta、nezha均可。\n8. bart中文模型下载地址：https://huggingface.co/fnlp/bart-base-chinese\n\n#### 参数说明，以文本分类任务为例\n```python\nimport torch \nfrom bert_seq2seq import Tokenizer, Predictor, load_model, Trainer\ntarget = [\"财经\", \"彩票\", \"房产\", \"股票\", \"家居\", \"教育\", \"科技\", \"社会\", \"时尚\", \"时政\", \"体育\", \"星座\", \"游戏\", \"娱乐\"] # 所有labels\ntrain_path = \"../data/新闻标题文本分类/Train.txt\" # 数据存放位置\nmodel_name = \"roberta\" # 选择模型名字\ntask_name = \"cls\" # 任务名字\nvocab_path = \"../state_dict/roberta-large/vocab.txt\" # roberta模型字典的位置\nmodel_path = \"../state_dict/roberta-large/pytorch_model.bin\" # roberta模型位置\nmodel_save_path = \"./bert_news_title_classification.bin\" ## 训练好的模型保存位置\nbatch_size = 16\nlr = 1e-5\n# 加载字典\ntokenizer = Tokenizer(vocab_path)\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n# 加载roberta large 模型，做利用cls向量分类的任务。\nbert_model = load_model(tokenizer.vocab,\n                        model_name=model_name,\n                        size=\"large\", ## load large model\n                        task_name=task_name,\n                        target_size=len(target))\n## 加载预训练的模型参数～\nbert_model.load_pretrain_params(model_path)\n# trainer设置\ntrainer = Trainer(epoches=3, val_every_step=500,  # 每500步进行验证\n                  batch_size=batch_size, \n                  env_type=\"pytorch\", # 单卡训练方式\n                  device=device, \n                  )\n```\n#### 运行\n确定要做哪个任务，找到examples中对应的train_*.py文件，下载好模型与字典后，理解数据构建过程，运行即可（样例数据在data目录中，帮助理解代码过程，供参考）。\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F920232796%2Fbert_seq2seq_ddp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F920232796%2Fbert_seq2seq_ddp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F920232796%2Fbert_seq2seq_ddp/lists"}