{"id":15293012,"url":"https://github.com/shibing624/nerpy","last_synced_at":"2025-04-13T08:21:50.176Z","repository":{"id":50319600,"uuid":"489253513","full_name":"shibing624/nerpy","owner":"shibing624","description":"🌈  NERpy: Implementation of Named Entity Recognition using Python. 命名实体识别工具，支持BertSoftmax、BertSpan等模型，开箱即用。","archived":false,"fork":false,"pushed_at":"2024-02-19T09:43:36.000Z","size":6423,"stargazers_count":113,"open_issues_count":2,"forks_count":15,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-11T20:39:41.327Z","etag":null,"topics":["bert","bert-softmax","bert-span","named-entity-recognition","ner","nlp","pytorch","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shibing624.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-05-06T07:11:29.000Z","updated_at":"2025-04-11T18:44:38.000Z","dependencies_parsed_at":"2024-02-19T10:54:33.804Z","dependency_job_id":null,"html_url":"https://github.com/shibing624/nerpy","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibing624%2Fnerpy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibing624%2Fnerpy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibing624%2Fnerpy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibing624%2Fnerpy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shibing624","download_url":"https://codeload.github.com/shibing624/nerpy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248681644,"owners_count":21144728,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","bert-softmax","bert-span","named-entity-recognition","ner","nlp","pytorch","transformers"],"created_at":"2024-09-30T16:37:42.965Z","updated_at":"2025-04-13T08:21:50.154Z","avatar_url":"https://github.com/shibing624.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![PyPI version](https://badge.fury.io/py/nerpy.svg)](https://badge.fury.io/py/nerpy)\n[![Downloads](https://pepy.tech/badge/nerpy)](https://pepy.tech/project/nerpy)\n[![Contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg)](CONTRIBUTING.md)\n[![License Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)\n[![python_version](https://img.shields.io/badge/Python-3.8%2B-green.svg)](requirements.txt)\n[![GitHub issues](https://img.shields.io/github/issues/shibing624/nerpy.svg)](https://github.com/shibing624/nerpy/issues)\n[![Wechat Group](https://img.shields.io/badge/wechat-group-green.svg?logo=wechat)](#Contact)\n\n# NERpy\n🌈 Implementation of Named Entity Recognition using Python. \n\n**nerpy**实现了BertSoftmax、BertCrf、BertSpan等多种命名实体识别模型，并在标准数据集上比较了各模型的效果。\n\n\n**Guide**\n- [Feature](#Feature)\n- [Evaluation](#Evaluation)\n- [Install](#install)\n- [Usage](#usage)\n- [Contact](#Contact)\n- [Reference](#reference)\n\n\n# Feature\n### 命名实体识别模型\n- [BertSoftmax](nerpy/ner_model.py)：BertSoftmax基于BERT预训练模型实现实体识别，本项目基于PyTorch实现了BertSoftmax模型的训练和预测\n- [BertSpan](nerpy/bertspan.py)：BertSpan基于BERT训练span边界的表示，模型结构更适配实体边界识别，本项目基于PyTorch实现了BertSpan模型的训练和预测\n\n# Evaluation\n\n### 实体识别\n\n- 英文实体识别数据集的评测结果：\n\n| Arch | Backbone | Model Name | CoNLL-2003 | QPS |\n| :-- | :--- | :--- | :-: | :--: |\n| BertSoftmax | bert-base-uncased | bert4ner-base-uncased | 90.43 | 235 |\n| BertSoftmax | bert-base-cased | bert4ner-base-cased | 91.17 | 235 |\n| BertSpan | bert-base-uncased | bertspan4ner-base-uncased | 90.61 | 210 |\n| BertSpan | bert-base-cased | bertspan4ner-base-cased | 91.90 | 224 |\n\n- 中文实体识别数据集的评测结果：\n\n| Arch | Backbone | Model Name | CNER | PEOPLE | MSRA-NER | QPS |\n| :-- | :--- | :--- | :-: | :-: | :-: | :-: |\n| BertSoftmax | bert-base-chinese | bert4ner-base-chinese | 94.98 | 95.25 | 94.65 | 222 |\n| BertSpan | bert-base-chinese | bertspan4ner-base-chinese | 96.03 | 96.06 | 95.03 | 254 |\n\n- 本项目release模型的实体识别评测结果：\n\n| Arch | Backbone | Model Name | CNER(zh) | PEOPLE(zh) | CoNLL-2003(en) | QPS |\n| :-- | :--- | :---- | :-: | :-: | :-: | :-: |\n| BertSpan | bert-base-chinese | shibing624/bertspan4ner-base-chinese | 96.03 | 96.06 | - | 254 |\n| BertSoftmax | bert-base-chinese | shibing624/bert4ner-base-chinese | 94.98 | 95.25 | - | 222 |\n| BertSoftmax | bert-base-uncased | shibing624/bert4ner-base-uncased | - | - | 90.43 | 243 |\n\n说明：\n- 结果值均使用F1\n- 结果均只用该数据集的train训练，在test上评估得到的表现，没用外部数据\n- `shibing624/bertspan4ner-base-chinese`模型达到base级别里SOTA效果，是用BertSpan方法训练的，\n 运行[examples/training_bertspan_zh_demo.py](examples/training_bertspan_zh_demo.py)代码可在各中文数据集复现结果\n- `shibing624/bert4ner-base-chinese`模型达到base级别里较好效果，是用BertSoftmax方法训练的，\n 运行[examples/training_ner_model_zh_demo.py](examples/training_ner_model_zh_demo.py)代码可在各中文数据集复现结果\n- `shibing624/bert4ner-base-uncased`模型是用BertSoftmax方法训练的，\n 运行[examples/training_ner_model_en_demo.py](examples/training_ner_model_en_demo.py)代码可在CoNLL-2003英文数据集复现结果\n- 各预训练模型均可以通过transformers调用，如中文BERT模型：`--model_name bert-base-chinese`\n- 中文实体识别数据集下载[链接见下方](#数据集)\n- QPS的GPU测试环境是Tesla V100，显存32GB\n\n# Demo\n\nDemo: https://huggingface.co/spaces/shibing624/nerpy\n\n![](docs/hf.png)\n\nrun example: [examples/gradio_demo.py](examples/gradio_demo.py) to see the demo:\n```shell\npython examples/gradio_demo.py\n```\n\n \n# Install\npython 3.8+\n\n```shell\npip install torch # conda install pytorch\npip install -U nerpy\n```\n\nor\n\n```shell\npip install torch # conda install pytorch\npip install -r requirements.txt\n\ngit clone https://github.com/shibing624/nerpy.git\ncd nerpy\npip install --no-deps .\n```\n\n# Usage\n\n## 命名实体识别\n\n#### 英文实体识别：\n\n```shell\n\u003e\u003e\u003e from nerpy import NERModel\n\u003e\u003e\u003e model = NERModel(\"bert\", \"shibing624/bert4ner-base-uncased\")\n\u003e\u003e\u003e predictions, raw_outputs, entities = model.predict([\"AL-AIN, United Arab Emirates 1996-12-06\"], split_on_space=True)\nentities:  [('AL-AIN,', 'LOC'), ('United Arab Emirates', 'LOC')]\n```\n\n#### 中文实体识别：\n\n```shell\n\u003e\u003e\u003e from nerpy import NERModel\n\u003e\u003e\u003e model = NERModel(\"bert\", \"shibing624/bert4ner-base-chinese\")\n\u003e\u003e\u003e predictions, raw_outputs, entities = model.predict([\"常建良，男，1963年出生，工科学士，高级工程师\"], split_on_space=False)\nentities: [('常建良', 'PER'), ('1963年', 'TIME')]\n```\n\nexample: [examples/base_zh_demo.py](examples/base_zh_demo.py)\n\n```python\nimport sys\n\nsys.path.append('..')\nfrom nerpy import NERModel\n\nif __name__ == '__main__':\n    # BertSoftmax中文实体识别模型: NERModel(\"bert\", \"shibing624/bert4ner-base-chinese\")\n    # BertSpan中文实体识别模型: NERModel(\"bertspan\", \"shibing624/bertspan4ner-base-chinese\")\n    model = NERModel(\"bert\", \"shibing624/bert4ner-base-chinese\")\n    sentences = [\n        \"常建良，男，1963年出生，工科学士，高级工程师，北京物资学院客座副教授\",\n        \"1985年8月-1993年在国家物资局、物资部、国内贸易部金属材料流通司从事国家统配钢材中特种钢材品种的调拨分配工作，先后任科员、主任科员。\"\n    ]\n    predictions, raw_outputs, entities = model.predict(sentences)\n    print(entities)\n```\n\noutput:\n```\n[('常建良', 'PER'), ('1963年', 'TIME'), ('北京物资学院', 'ORG')]\n[('1985年', 'TIME'), ('8月', 'TIME'), ('1993年', 'TIME'), ('国家物资局', 'ORG'), ('物资部', 'ORG'), ('国内贸易部金属材料流通司', 'ORG')]\n```\n\n- `shibing624/bert4ner-base-chinese`模型是BertSoftmax方法在中文PEOPLE(人民日报)数据集训练得到的，模型已经上传到huggingface的\n模型库[shibing624/bert4ner-base-chinese](https://huggingface.co/shibing624/bert4ner-base-chinese)，\n是`nerpy.NERModel`指定的默认模型，可以通过上面示例调用，或者如下所示用[transformers库](https://github.com/huggingface/transformers)调用，\n模型自动下载到本机路径：`~/.cache/huggingface/transformers`\n- `shibing624/bertspan4ner-base-chinese`模型是BertSpan方法在中文PEOPLE(人民日报)数据集训练得到的，模型已经上传到huggingface的\n模型库[shibing624/bertspan4ner-base-chinese](https://huggingface.co/shibing624/bertspan4ner-base-chinese)\n\n\n#### Usage (HuggingFace Transformers)\nWithout [nerpy](https://github.com/shibing624/nerpy), you can use the model like this: \n\nFirst, you pass your input through the transformer model, then you have to apply the bio tag to get the entity words.\n\nexample: [examples/predict_use_origin_transformers_zh_demo.py](examples/predict_use_origin_transformers_zh_demo.py)\n\n```python\nimport os\nimport torch\nfrom transformers import AutoTokenizer, AutoModelForTokenClassification\nfrom seqeval.metrics.sequence_labeling import get_entities\n\nos.environ[\"KMP_DUPLICATE_LIB_OK\"] = \"TRUE\"\n\n# Load model from HuggingFace Hub\ntokenizer = AutoTokenizer.from_pretrained(\"shibing624/bert4ner-base-chinese\")\nmodel = AutoModelForTokenClassification.from_pretrained(\"shibing624/bert4ner-base-chinese\")\nlabel_list = ['I-ORG', 'B-LOC', 'O', 'B-ORG', 'I-LOC', 'I-PER', 'B-TIME', 'I-TIME', 'B-PER']\n\nsentence = \"王宏伟来自北京，是个警察，喜欢去王府井游玩儿。\"\n\n\ndef get_entity(sentence):\n    tokens = tokenizer.tokenize(sentence)\n    inputs = tokenizer.encode(sentence, return_tensors=\"pt\")\n    with torch.no_grad():\n        outputs = model(inputs).logits\n    predictions = torch.argmax(outputs, dim=2)\n    char_tags = [(token, label_list[prediction]) for token, prediction in zip(tokens, predictions[0].numpy())][1:-1]\n    print(sentence)\n    print(char_tags)\n\n    pred_labels = [i[1] for i in char_tags]\n    entities = []\n    line_entities = get_entities(pred_labels)\n    for i in line_entities:\n        word = sentence[i[1]: i[2] + 1]\n        entity_type = i[0]\n        entities.append((word, entity_type))\n\n    print(\"Sentence entity:\")\n    print(entities)\n\n\nget_entity(sentence)\n```\noutput:\n```shell\n王宏伟来自北京，是个警察，喜欢去王府井游玩儿。\n[('王', 'B-PER'), ('宏', 'I-PER'), ('伟', 'I-PER'), ('来', 'O'), ('自', 'O'), ('北', 'B-LOC'), ('京', 'I-LOC'), ('，', 'O'), ('是', 'O'), ('个', 'O'), ('警', 'O'), ('察', 'O'), ('，', 'O'), ('喜', 'O'), ('欢', 'O'), ('去', 'O'), ('王', 'B-LOC'), ('府', 'I-LOC'), ('井', 'I-LOC'), ('游', 'O'), ('玩', 'O'), ('儿', 'O'), ('。', 'O')]\nSentence entity:\n[('王宏伟', 'PER'), ('北京', 'LOC'), ('王府井', 'LOC')]\n```\n\n### 数据集\n\n#### 实体识别数据集\n\n\n| 数据集 | 语料 | 下载链接 | 文件大小 |\n| :------- | :--------- | :---------: | :---------: |\n| **`CNER中文实体识别数据集`** | CNER(12万字) | [CNER github](https://github.com/shibing624/nerpy/tree/main/examples/data/cner)| 1.1MB |\n| **`PEOPLE中文实体识别数据集`** | 人民日报数据集（200万字） | [PEOPLE github](https://github.com/shibing624/nerpy/tree/main/examples/data/people)| 12.8MB |\n| **`MSRA-NER中文实体识别数据集`** | MSRA-NER数据集（4.6万条，221.6万字） | [MSRA-NER github](https://github.com/shibing624/nerpy/releases/download/0.1.0/msra_ner.tar.gz)| 3.6MB |\n| **`CoNLL03英文实体识别数据集`** | CoNLL-2003数据集（22万字） | [CoNLL03 github](https://github.com/shibing624/nerpy/tree/main/examples/data/conll03)| 1.7MB |\n\n\nCNER中文实体识别数据集，数据格式：\n\n```text\n美\tB-LOC\n国\tI-LOC\n的\tO\n华\tB-PER\n莱\tI-PER\n士\tI-PER\n\n我\tO\n跟\tO\n他\tO\n```\n\n\n## BertSoftmax 模型\n\nBertSoftmax实体识别模型，基于BERT的标准序列标注方法：\n\nNetwork structure:\n\n\n\u003cimg src=\"docs/bert.png\" width=\"500\" /\u003e\n\n\n模型文件组成：\n```\nshibing624/bert4ner-base-chinese\n    ├── config.json\n    ├── model_args.json\n    ├── pytorch_model.bin\n    ├── special_tokens_map.json\n    ├── tokenizer_config.json\n    └── vocab.txt\n```\n\n#### BertSoftmax 模型训练和预测\n\ntraining example: [examples/training_ner_model_toy_demo.py](examples/bert_softmax_demo.py)\n\n\n```python\nimport sys\nimport pandas as pd\n\nsys.path.append('..')\nfrom nerpy.ner_model import NERModel\n\n\n# Creating samples\ntrain_samples = [\n    [0, \"HuggingFace\", \"B-MISC\"],\n    [0, \"Transformers\", \"I-MISC\"],\n    [0, \"started\", \"O\"],\n    [0, \"with\", \"O\"],\n    [0, \"text\", \"O\"],\n    [0, \"classification\", \"B-MISC\"],\n    [1, \"Nerpy\", \"B-MISC\"],\n    [1, \"Model\", \"I-MISC\"],\n    [1, \"can\", \"O\"],\n    [1, \"now\", \"O\"],\n    [1, \"perform\", \"O\"],\n    [1, \"NER\", \"B-MISC\"],\n]\ntrain_data = pd.DataFrame(train_samples, columns=[\"sentence_id\", \"words\", \"labels\"])\n\ntest_samples = [\n    [0, \"HuggingFace\", \"B-MISC\"],\n    [0, \"Transformers\", \"I-MISC\"],\n    [0, \"was\", \"O\"],\n    [0, \"built\", \"O\"],\n    [0, \"for\", \"O\"],\n    [0, \"text\", \"O\"],\n    [0, \"classification\", \"B-MISC\"],\n    [1, \"Nerpy\", \"B-MISC\"],\n    [1, \"Model\", \"I-MISC\"],\n    [1, \"then\", \"O\"],\n    [1, \"expanded\", \"O\"],\n    [1, \"to\", \"O\"],\n    [1, \"perform\", \"O\"],\n    [1, \"NER\", \"B-MISC\"],\n]\ntest_data = pd.DataFrame(test_samples, columns=[\"sentence_id\", \"words\", \"labels\"])\n\n# Create a NERModel\nmodel = NERModel(\n    \"bert\",\n    \"bert-base-uncased\",\n    args={\"overwrite_output_dir\": True, \"reprocess_input_data\": True, \"num_train_epochs\": 1},\n    use_cuda=False,\n)\n\n# Train the model\nmodel.train_model(train_data)\n\n# Evaluate the model\nresult, model_outputs, predictions = model.eval_model(test_data)\nprint(result, model_outputs, predictions)\n\n# Predictions on text strings\nsentences = [\"Nerpy Model perform sentence NER\", \"HuggingFace Transformers build for text\"]\npredictions, raw_outputs, entities = model.predict(sentences, split_on_space=True)\nprint(predictions, entities)\n```\n\n- 在中文CNER数据集训练和评估`BertSoftmax`模型\n\nexample: [examples/training_ner_model_zh_demo.py](examples/training_ner_model_zh_demo.py)\n\n```shell\ncd examples\npython training_ner_model_zh_demo.py --do_train --do_predict --num_epochs 5 --task_name cner\n```\n- 在英文CoNLL-2003数据集训练和评估`BertSoftmax`模型\n\nexample: [examples/training_ner_model_en_demo.py](examples/training_ner_model_en_demo.py)\n\n```shell\ncd examples\npython training_ner_model_en_demo.py --do_train --do_predict --num_epochs 5\n```\n\n\n#### BertSpan 模型训练和预测\n\n- 在中文CNER数据集训练和评估`BertSpan`模型\n\nexample: [examples/training_bertspan_zh_demo.py](examples/training_bertspan_zh_demo.py)\n\n```shell\ncd examples\npython training_bertspan_zh_demo.py --do_train --do_predict --num_epochs 5 --task_name cner\n```\n\n# Contact\n\n- Issue(建议)：[![GitHub issues](https://img.shields.io/github/issues/shibing624/nerpy.svg)](https://github.com/shibing624/nerpy/issues)\n- 邮件我：xuming: xuming624@qq.com\n- 微信我：\n加我*微信号：xuming624, 备注：姓名-公司-NLP* 进NLP交流群。\n\n\u003cimg src=\"docs/wechat.jpeg\" width=\"200\" /\u003e\n\n\n# Citation\n\n如果你在研究中使用了nerpy，请按如下格式引用：\n\nAPA:\n```latex\nXu, M. nerpy: Named Entity Recognition Toolkit (Version 0.0.2) [Computer software]. https://github.com/shibing624/nerpy\n```\n\nBibTeX:\n```latex\n@software{Xu_nerpy_Text_to,\nauthor = {Xu, Ming},\ntitle = {{nerpy: Named Entity Recognition Toolkit}},\nurl = {https://github.com/shibing624/nerpy},\nversion = {0.0.2}\n}\n```\n\n# License\n\n\n授权协议为 [The Apache License 2.0](LICENSE)，可免费用做商业用途。请在产品说明中附加nerpy的链接和授权协议。\n\n\n# Contribute\n项目代码还很粗糙，如果大家对代码有所改进，欢迎提交回本项目，在提交之前，注意以下两点：\n\n - 在`tests`添加相应的单元测试\n - 使用`python -m pytest`来运行所有单元测试，确保所有单测都是通过的\n\n之后即可提交PR。\n\n# Reference\n- [huggingface/transformers](https://github.com/huggingface/transformers)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshibing624%2Fnerpy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshibing624%2Fnerpy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshibing624%2Fnerpy/lists"}