{"id":19287783,"url":"https://github.com/deeppavlov/slavic-bert-ner","last_synced_at":"2025-04-22T04:32:21.433Z","repository":{"id":83451121,"uuid":"186630771","full_name":"deeppavlov/Slavic-BERT-NER","owner":"deeppavlov","description":"Shared BERT model for 4 languages of Bulgarian, Czech, Polish and Russian. Slavic NER model.","archived":false,"fork":false,"pushed_at":"2022-01-28T14:29:51.000Z","size":817,"stargazers_count":72,"open_issues_count":1,"forks_count":11,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-10-07T15:02:03.608Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deeppavlov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-05-14T13:41:48.000Z","updated_at":"2024-03-27T08:27:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"2bba082e-c69f-47bc-9cfb-2549434c928a","html_url":"https://github.com/deeppavlov/Slavic-BERT-NER","commit_stats":null,"previous_names":["deepmipt/slavic-bert-ner"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2FSlavic-BERT-NER","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2FSlavic-BERT-NER/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2FSlavic-BERT-NER/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeppavlov%2FSlavic-BERT-NER/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deeppavlov","download_url":"https://codeload.github.com/deeppavlov/Slavic-BERT-NER/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223888420,"owners_count":17220083,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T22:07:12.102Z","updated_at":"2024-11-09T22:07:12.705Z","avatar_url":"https://github.com/deeppavlov.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Slavic BERT NER\n\n***Notice:** The repo is left as-is, the Slavic BERT model is now as part of [DeepPavlov repo](https://github.com/deepmipt/DeepPavlov).*\n\n**BERT** is a method of pre-training language representations, meaning that we train a general-purpose \"language understanding\" model on a large text corpus (like Wikipedia), and then use that model for downstream NLP tasks that we care about (like question answering). For details see original [BERT github](https://github.com/google-research/bert).\n\nThe repository contains **Bulgarian**+**Czech**+**Polish**+**Russian** specific:\n- [shared BERT model](#slavic-bert)\n- [NER model (`PER`, `LOC`, `ORG`, `PRO`, `EVT`)](#slavic-bert)\n\nOur academic paper which describes tuning Transformers for NER task in detail can be found here: https://www.aclweb.org/anthology/W19-3712/.\n\n## Slavic BERT\n\nThe Slavic model is the result of transfer from `2018_11_23/multi_cased_L-12_H-768_A-12` Multilingual BERT model to languages of Bulgarian (`bg`), Czech (`cs`), Polish (`pl`) and Russian (`ru`). The fine-tuning was performed with a stratified dataset of `bg`, `cs` and `pl` Wikipedias and `ru` news.\n\nThe model format is the same as in the original repository.\n\n*   **[`BERT, Slavic Cased`](http://files.deeppavlov.ai/deeppavlov_data/bg_cs_pl_ru_cased_L-12_H-768_A-12.tar.gz)**:\n    4 languages, 12-layer, 768-hidden, 12-heads, 110M parameters, 600Mb\n\n## Slavic NER\n\nNamed Entity Recognition (further, **NER**) is a task of recognizing named entities in text, as well as detecting their type.\n\nWe used Slavic BERT model as a base to build NER system. First, we feed each input word into WordPiece case-sensitive tokenizer and extract the final hidden representation corresponding to the first subtoken in each word. These representations are fed into a classification dense layer over the NER label set. A token-level CRF layer is also added on top.\n\n![BERT NER diagram](bert_ner_diagram.png)\n\nThe model was trained on [BSNLP-2019 dataset](http://bsnlp.cs.helsinki.fi/shared_task.html). The pre-trained model can recognize such entities as:\n\n- Persons (PER)\n- Locations (LOC)\n- Organizations (ORG)\n- Products (PRO)\n- Events (EVT)\n\nThe metrics for all languages and entities on test set are:\n\n| Language     | Tag  | Precision     | Recall       | RPM (Relaxed Partial Matching)    |\n|--------------|:----:|:-------------:|:------------:|:---------------------------------:|\n| cs           |      | 94.3          | 93.4         | 93.9                              |\n| ru           |      | 88.1          | 86.6         | 87.3                              |\n| bg           |      | 90.3          | 84.3         | 87.2                              |\n| pl           |      | 93.3          | 93.0         | 93.2                              |\n|              | PER  | 94.2          | 95.6         | 94.9                              |\n|              | LOC  | 96.6          | 96.4         | 96.5                              |\n|              | ORG  | 84.3          | 92.1         | 88.0                              |\n|              | PRO  | 87.6          | 51.3         | 64.7                              |\n|              | EVT  | 39.4          | 27.7         | 93.9                              |\n|              |      | **89.8**      | **91.8**     | **90.8**                          |\n\nFor detailed description of evaluation method see [BSNLP-2019 Shared Task page](http://bsnlp.cs.helsinki.fi/shared_task.html).\n\n*   **[`NER, Slavic Cased`](http://files.deeppavlov.ai/deeppavlov_data/ner_bert_slav.tar.gz)**:\n    4 languages, 13-layer + CRF, 768-hidden, 2.0Gb\n    \n# Usage\n\n#### Install\n\nThe toolkit support Python 3.6 and Python3.7. To install required packages use:\n\n```bash\n$ pip3 install -r requirements.txt\n```\n\nCAUTION: Python\u003c=3.5 and Python\u003e=3.8 are not supported, see [DeepPavlov rep](https://github.com/deepmipt/deeppavlov) for details.\n\n#### Ner usage\n\n```python\nfrom deeppavlov import build_model\n\n# Download and load model (set download=False to skip download phase)\nner = build_model(\"./ner_bert_slav.json\", download=True)\n\n# Get predictions\nner([\"To Bert z ulicy Sezamkowej\"])\n# [[['To', 'Bert', 'z', 'ulicy', 'Sezamkowej']], [['O', 'B-PER', 'O', 'B-LOC', 'I-LOC']]]\nner([\"Это\", \"Берт\", \"из\", \"России\"])\n# [[['Это'], ['Берт'], ['из'], ['России']], [['O'], ['B-PER'], ['O'], ['B-LOC']]]\n```\n \n#### Bert usage\n\nThe Slavic Bert model can be used in any way proposed by the BERT developers.\n\nOne approach may be:\n\n```python\n\nimport tensorflow as tf\n  \nfrom bert_dp.modeling import BertConfig, BertModel\nfrom deeppavlov.models.preprocessors.bert_preprocessor import BertPreprocessor\n\n\nbert_config = BertConfig.from_json_file('./bg_cs_pl_ru_cased_L-12_H-768_A-12/bert_config.json')\n\ninput_ids = tf.placeholder(shape=(None, None), dtype=tf.int32)\ninput_mask = tf.placeholder(shape=(None, None), dtype=tf.int32)\ntoken_type_ids = tf.placeholder(shape=(None, None), dtype=tf.int32)\n\nbert = BertModel(config=bert_config,\n                 is_training=False,\n                 input_ids=input_ids,\n                 input_mask=input_mask,\n                 token_type_ids=token_type_ids,\n                 use_one_hot_embeddings=False)\n\npreprocessor = BertPreprocessor(vocab_file='./bg_cs_pl_ru_cased_L-12_H-768_A-12/vocab.txt',\n                                do_lower_case=False,\n                                max_seq_length=512)\n\nwith tf.Session() as sess:\n\n    # Load model\n    tf.train.Saver().restore(sess, './bg_cs_pl_ru_cased_L-12_H-768_A-12/bert_model.ckpt')\n\n    # Get predictions\n    features = preprocessor([\"Bert z ulicy Sezamkowej\"])[0]\n\n    print(sess.run(bert.sequence_output, feed_dict={input_ids: [features.input_ids],\n                                                    input_mask: [features.input_mask],\n                                                    token_type_ids: [features.input_type_ids]}))\n\n    features = preprocessor([\"Берт\", \"с\", \"Улицы\", \"Сезам\"])[0]\n\n    print(sess.run(bert.sequence_output, feed_dict={input_ids: [features.input_ids],\n                                                    input_mask: [features.input_mask],\n                                                    token_type_ids: [features.input_type_ids]}))\n```\n\n## Changes\n\n[Jan 2021] fixed 'Model bert_ner is not registered' issue, updated to\n  `deeppavlov==0.17.2`, `tensorflow==1.15.5`\n\n## Citation\n```\n@inproceedings{arkhipov-etal-2019-tuning,\n    title = \"Tuning Multilingual Transformers for Language-Specific Named Entity Recognition\",\n    author = \"Arkhipov, Mikhail  and\n      Trofimova, Maria  and\n      Kuratov, Yuri  and\n      Sorokin, Alexey\",\n    booktitle = \"Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing\",\n    month = aug,\n    year = \"2019\",\n    address = \"Florence, Italy\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://www.aclweb.org/anthology/W19-3712\",\n    doi = \"10.18653/v1/W19-3712\",\n    pages = \"89--93\"\n}\n```\n\n## References\n\n[1] - [Jacob Devlin et all: *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*, 2018](https://arxiv.org/abs/1810.04805)\n\n[2] - [Mozharova V., Loukachevitch N.: *Two-stage approach in Russian named entity recognition*, 2016](https://ieeexplore.ieee.org/document/7584769)\n\n[3] - [BSNLP-2019 Shared Task](http://bsnlp.cs.helsinki.fi/shared_task.html)\n\n[4] - [DeepPavlov: open-source library for dialog systems](https://github.com/deepmipt/deeppavlov)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeppavlov%2Fslavic-bert-ner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeeppavlov%2Fslavic-bert-ner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeppavlov%2Fslavic-bert-ner/lists"}