{"id":19465238,"url":"https://github.com/luozhouyang/deepseg","last_synced_at":"2025-04-25T09:31:53.301Z","repository":{"id":39738415,"uuid":"155837694","full_name":"luozhouyang/deepseg","owner":"luozhouyang","description":"Chinese word segmentation in tensorflow 2.x","archived":false,"fork":false,"pushed_at":"2023-03-25T00:52:11.000Z","size":83,"stargazers_count":23,"open_issues_count":1,"forks_count":9,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-03T19:40:15.844Z","etag":null,"topics":["albert-crf","bert-bilstm-crf","bert-crf","bilstm-crf","crf","deepseg","keras","segmentation","sequence-labeling","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luozhouyang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-02T08:40:19.000Z","updated_at":"2025-01-17T13:12:16.000Z","dependencies_parsed_at":"2022-08-29T00:21:32.695Z","dependency_job_id":null,"html_url":"https://github.com/luozhouyang/deepseg","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luozhouyang%2Fdeepseg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luozhouyang%2Fdeepseg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luozhouyang%2Fdeepseg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luozhouyang%2Fdeepseg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luozhouyang","download_url":"https://codeload.github.com/luozhouyang/deepseg/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250790131,"owners_count":21487754,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["albert-crf","bert-bilstm-crf","bert-crf","bilstm-crf","crf","deepseg","keras","segmentation","sequence-labeling","tensorflow"],"created_at":"2024-11-10T18:19:18.085Z","updated_at":"2025-04-25T09:31:53.015Z","avatar_url":"https://github.com/luozhouyang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# deepseg\n\nTensorflow 2.x 实现的神经网络分词模型！一键训练\u0026一键部署！\n\n\u003e tensorflow 1.x的实现请切换到`tf1`分支\n\n推荐本项目使用到的两个库：\n\n* [luozhouyang/transformers-keras](https://github.com/luozhouyang/transformers-keras) 用于加载各种预训练`Bert`和`Albert`模型。\n* [luozhouyang/keras-crf](https://github.com/luozhouyang/keras-crf) 基于[tensorflow/addons](https://github.com/tensorflow/addons)之上的CRF层实现。\n\n\n## 开发环境\n\n```bash\nconda create -n deepseg python=3.6\nconda activate deepseg \npip install -r requirements.txt\n```\n\n## 数据集下载\n\n* [SIGHAN](https://lzy-oss-files.oss-cn-hangzhou.aliyuncs.com/segmentation/sighan-icwb2-data.zip)\n\n## 训练模型\n\n可以使用`deepseg/run_deepseg.py`脚本来训练你的模型。需要提供以下参数：\n\n* `--model`，模型，可选择 `bisltm-crf`, `bigru-crf`, `bert-crf`, `albert-crf`, `bert-bilstm-crf`, `albert-bilstm-crf`\n    - 如果是`bert-based`或者`albert-based`模型，请提供预训练模型路径，使用`--pretrained_model_dir`参数制定。\n* `--model_dir`，模型保存路径\n* `--vocab_file`，词典文件路径，注意是**字符**级别的词典，参考`testdata/vocab_small.txt`\n* `--train_input_files`，训练文件，分好词的文本文件，参考`testdata/train_small.txt`\n\n对于`bilstm-crf`和`bigru-crf`模型，还需要指定以下参数：\n\n* `--vocab_size`，词典大小\n* `--embedding_size`，潜入层的维度\n\n一个使用`bert-crf`模型的例子如下：\n\n```bash\npython -m deepseg.run_deepseg \\\n    --model=bert-crf \\\n    --model_dir=models/bert-crf-model \\\n    --pretrained_model_dir=/home/zhouyang.lzy/pretrain-models/chinese_roberta_wwm_ext_L-12_H-768_A-12 \\\n    --train_input_files=testdata/train_small.txt \\\n    --vocab_file=/home/zhouyang.lzy/pretrain-models/chinese_roberta_wwm_ext_L-12_H-768_A-12/vocab.txt \\\n    --epochs=2 \n```\n\n什么？你觉得我的训练脚本写得太烂了，想自己写训练过程？\n\n完全OK啊！\n\n### 自己写训练脚本\n\n```python\nfrom deepseg.dataset import DatasetBuilder, LabelMapper, TokenMapper\nfrom deepseg.models import BiGRUCRFModel, BiLSTMCRFModel\nfrom deepseg.models import AlbertBiLSTMCRFModel, AlbertCRFModel\nfrom deepseg.models import BertBiLSTMCRFModel, BertCRFModel\n\ntoken_mapper = TokenMapper(vocab_file='testdata/vocab_small.txt')\nlabel_mapper = LabelMapper()\n\nbuilder = DatasetBuilder(token_mapper, label_mapper)\ntrain_dataset = builder.build_train_dataset('testdata/train_small.txt', batch_size=20, buffer_size=100)\nvalid_dataset = None\n\nmodel_dir = 'model/bilstm-crf'\ntensorboard_logdir = os.path.join(model_dir, 'logs')\nsaved_model_dir = os.path.join(model_dir, 'export', '{epoch}')\n\n# 更改成你自己想要的模型，或者干脆自己构建任何你想要的模型！\nmodel = BiLSTMCRFModel(100, 128, 3)\nmodel.fit(\n    train_dataset,\n    validation_data=valid_dataset,\n    epochs=10,\n    callbacks=[\n        tf.keras.callbacks.EarlyStopping(monitor='val_loss' if valid_dataset is not None else 'loss'),\n        tf.keras.callbacks.TensorBoard(tensorboard_logdir),\n        tf.keras.callbacks.ModelCheckpoint(\n            saved_model_dir,\n            save_best_only=False,\n            save_weights_only=False)\n    ]\n)\n```\n\n## 部署模型\n\n上面训练过程中，每个epoch都会保存一个SavedModel格式的模型，可以直接使用`tensorflow-serving`部署。\n\n* TODO：增加部署文档和客户端调用文档\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluozhouyang%2Fdeepseg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluozhouyang%2Fdeepseg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluozhouyang%2Fdeepseg/lists"}