{"id":18305300,"url":"https://github.com/guillaumegenthial/tf_ner","last_synced_at":"2025-04-13T02:15:43.692Z","repository":{"id":113442382,"uuid":"149046413","full_name":"guillaumegenthial/tf_ner","owner":"guillaumegenthial","description":"Simple and Efficient Tensorflow implementations of NER models with tf.estimator and tf.data","archived":false,"fork":false,"pushed_at":"2018-12-18T22:16:50.000Z","size":147,"stargazers_count":923,"open_issues_count":37,"forks_count":275,"subscribers_count":42,"default_branch":"master","last_synced_at":"2025-04-13T02:14:42.446Z","etag":null,"topics":["bi-lstm-crf","character-embeddings","conll-2003","exponential-moving-average","glove","lstm-crf","named-entity-recognition","ner","state-of-the-art","tensorflow","tf-data","tf-estimator"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guillaumegenthial.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-09-16T23:58:35.000Z","updated_at":"2025-04-06T12:34:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"43f29968-83fa-4043-8a3e-5c6386077e33","html_url":"https://github.com/guillaumegenthial/tf_ner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guillaumegenthial%2Ftf_ner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guillaumegenthial%2Ftf_ner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guillaumegenthial%2Ftf_ner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guillaumegenthial%2Ftf_ner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guillaumegenthial","download_url":"https://codeload.github.com/guillaumegenthial/tf_ner/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248654095,"owners_count":21140236,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bi-lstm-crf","character-embeddings","conll-2003","exponential-moving-average","glove","lstm-crf","named-entity-recognition","ner","state-of-the-art","tensorflow","tf-data","tf-estimator"],"created_at":"2024-11-05T15:33:47.906Z","updated_at":"2025-04-13T02:15:43.642Z","avatar_url":"https://github.com/guillaumegenthial.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tensorflow - Named Entity Recognition\n\nEach folder contains a __standalone__, __short (~100 lines of Tensorflow)__, `main.py` that implements a neural-network based model for Named Entity Recognition (NER) using [`tf.estimator`](https://www.tensorflow.org/guide/custom_estimators) and [`tf.data`](https://www.tensorflow.org/guide/datasets).\n\n![Named Entity Recognition](images/ner.png)\n\n\nThese implementations are __simple, efficient, and state-of-the-art__, in the sense that they do __as least as well as the results reported in the papers__. The best model achieves in *average* an __f1 score of 91.21__. To my knowledge, *existing implementations available on the web are convoluted, outdated and not always accurate* (including my [previous work](https://github.com/guillaumegenthial/sequence_tagging)). This repo is an attempt to fix this, in the hope that it will enable people to test and validate new ideas quickly.\n\nThe script [`lstm_crf/main.py`](https://github.com/guillaumegenthial/tf_ner/blob/master/models/lstm_crf/main.py) can also be seen as a __simple introduction to Tensorflow high-level APIs [`tf.estimator`](https://www.tensorflow.org/guide/custom_estimators) and [`tf.data`](https://www.tensorflow.org/guide/datasets) applied to Natural Language Processing__. [Here is a longer discussion about this implementation along with an introduction to tf.estimator and tf.data](https://guillaumegenthial.github.io/introduction-tensorflow-estimator.html)\n\n\n## Install\n\nYou need __python3__ -- If you haven't switched yet, do it.\n\nYou need to install [`tf_metrics` ](https://github.com/guillaumegenthial/tf_metrics) (multi-class precision, recall and f1 metrics for Tensorflow).\n```\npip install git+https://github.com/guillaumegenthial/tf_metrics.git\n```\nOR\n```\ngit clone https://github.com/guillaumegenthial/tf_metrics.git\ncd tf_metrics\npip install .\n```\n\n## Data Format\n\nFollow the [`data/example`](https://github.com/guillaumegenthial/tf_ner/tree/master/data/example).\n\n1. For `name` in `{train, testa, testb}`, create files `{name}.words.txt` and `{name}.tags.txt` that contain one sentence per line, each\nword / tag separated by space. I recommend using the `IOBES` tagging scheme.\n2. Create files `vocab.words.txt`, `vocab.tags.txt` and `vocab.chars.txt` that contain one token per line.\n3. Create a `glove.npz` file containing one array `embeddings` of shape `(size_vocab_words, 300)` using [GloVe 840B vectors](https://nlp.stanford.edu/projects/glove/) and [`np.savez_compressed`](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.savez_compressed.html).\n\nAn example of scripts to build the `vocab` and the `glove.npz` files from the  `{name}.words.txt` and `{name}.tags.txt` files is provided in [`data/example`](https://github.com/guillaumegenthial/tf_ner/tree/master/data/example). See\n\n1. [`build_vocab.py`](https://github.com/guillaumegenthial/tf_ner/blob/master/data/example/build_vocab.py)\n2. [`build_glove.py`'](https://github.com/guillaumegenthial/tf_ner/blob/master/data/example/build_glove.py)\n\n![Data Format](images/data.png)\n\nIf you just want to get started, once you have created your `{name}.words.txt` and `{name}.tags.txt` files, simply do\n\n```\ncd data/example\nmake download-glove\nmake build\n```\n\n(These commands will build the __example__ dataset)\n\n*Note that the example dataset is here for debugging purposes only and won't be of much use to train an actual model*\n\n## Get Started\n\nOnce you've produced all the required data files, simply pick one of the `main.py` scripts. Then, modify the `DATADIR` variable at the top of `main.py`.\n\nTo train, evaluate and write predictions to file, run\n\n```\ncd models/lstm_crf\npython main.py\n```\n\n(These commands will train a bi-LSTM + CRF on the __example__ dataset if you haven't changed `DATADIR` in the `main.py`.)\n\n__Each model subdirectory contains a breakdown of the instructions__.\n\n## Models\n\nTook inspiration from these papers\n\n- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://arxiv.org/abs/1508.01991) by Huang, Xu and Yu\n- [Neural Architectures for Named Entity Recognition](https://arxiv.org/abs/1603.01360) by Lample et al.\n- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://arxiv.org/abs/1603.01354) by Ma et Hovy\n\nYou can also read [this blog post](https://guillaumegenthial.github.io/sequence-tagging-with-tensorflow.html).\n\nWord-vectors are __not retrained__ to avoid any undesirable shift (explanation in [these CS224N notes](https://github.com/stanfordnlp/cs224n-winter17-notes/blob/master/notes2.pdf)).\n\nThe models are tested on the [CoNLL2003 shared task](https://www.clips.uantwerpen.be/conll2003/ner/).\n\nTraining times are provided for indicative purposes only. Obtained on a 2016 13-inch MBPro 3.3 GHz Intel Core i7.\n\nFor each model, we run 5 experiments\n\n- Train on `train` only\n- __Early stopping__ on `testa`\n- Select best of 5 on the perfomance on `testa` (token-level F1)\n- Report __F1 score mean and standard deviation__ (entity-level F1 from the official `conlleval` script)\n- Select best on `testb` for reference (but shouldn't be used for comparison as this is just overfitting on the final test set)\n\nIn addition, we run 5 other experiments, keeping an __Exponential Moving Average (EMA)__ of the weights (used for evaluation) and report the best F1, mean / std.\n\nAs you can see, there's no clear statistical evidence of which of the 2 character-based models is the best. EMA seems to help most of the time. Also, considering the complexity of the models and the relatively small gap in performance (0.6 F1), using the `lstm_crf` model is probably a safe bet for most of the concrete applications.\n\n---\n\n### `lstm_crf`\n\n__Architecture__\n\n1. [GloVe 840B vectors](https://nlp.stanford.edu/projects/glove/)\n2. Bi-LSTM\n3. CRF\n\n__Related Paper__ [Bidirectional LSTM-CRF Models for Sequence Tagging](https://arxiv.org/abs/1508.01991) by Huang, Xu and Yu\n\n__Training time__ ~ 20 min\n\n|| `train` | `testa` | `testb` | Paper, `testb` |\n|---|:---:|:---:|:---:|:---:|\n|best | 98.45 |93.81 | __90.61__ |  90.10 |\n|best (EMA)| 98.82 | 94.06 | 90.43 | |\n|mean ± std| 98.85 ± 0.22| 93.68 ± 0.12| 90.42 ± 0.10|  |\n|mean ± std (EMA)| 98.71 ± 0.47 | 93.81 ± 0.24 | __90.50__ ± 0.21| |\n|abs. best |   | | 90.61 |  |\n|abs. best (EMA) | |  | 90.75 |  |\n\n\n---\n\n### `chars_lstm_lstm_crf`\n\n__Architecture__\n\n1. [GloVe 840B vectors](https://nlp.stanford.edu/projects/glove/)\n2. Chars embeddings\n3. Chars bi-LSTM\n4. Bi-LSTM\n5. CRF\n\n__Related Paper__ [Neural Architectures for Named Entity Recognition](https://arxiv.org/abs/1603.01360) by Lample et al.\n\n__Training time__ ~ 35 min\n\n|| `train` | `testa` | `testb` | Paper, `testb` |\n|---|:---:|:---:|:---:|:---:|\n|best| 98.81 | 94.36 | 91.02 | 90.94 |\n|best (EMA) |98.73 | 94.50 | __91.14__ | |\n|mean ± std | 98.83 ± 0.27| 94.02 ± 0.26| 91.01 ± 0.16 |  |\n|mean ± std (EMA) | 98.51 ± 0.25| 94.20 ± 0.28| __91.21__ ± 0.05 |  |\n|abs. best |   | |91.22 | |\n|abs. best (EMA) | |   | 91.28 |  |\n\n---\n\n### `chars_conv_lstm_crf`\n\n__Architecture__\n\n1. [GloVe 840B vectors](https://nlp.stanford.edu/projects/glove/)\n2. Chars embeddings\n3. Chars 1d convolution and max-pooling\n4. Bi-LSTM\n5. CRF\n\n__Related Paper__ [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://arxiv.org/abs/1603.01354) by Ma et Hovy\n\n__Training time__ ~ 35 min\n\n|| `train` | `testa` | `testb` | Paper, `testb` |\n|---|:---:|:---:|:---:|:---:|\n|best| 99.16 | 94.53 | __91.18__ | 91.21 |\n|best (EMA) |99.44 | 94.50 | 91.17 | |\n|mean ± std | 98.86 ± 0.30| 94.10 ± 0.26| __91.20__ ± 0.15 |  |\n|mean ± std (EMA) | 98.67 ± 0.39| 94.29 ± 0.17| 91.13 ± 0.11 |  |\n|abs. best |  | | 91.42 |  |\n|abs. best (EMA) |   | | 91.22 |  |\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguillaumegenthial%2Ftf_ner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguillaumegenthial%2Ftf_ner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguillaumegenthial%2Ftf_ner/lists"}