{"id":13673792,"url":"https://github.com/MantisAI/nervaluate","last_synced_at":"2025-04-28T11:30:43.395Z","repository":{"id":43318809,"uuid":"172034914","full_name":"MantisAI/nervaluate","owner":"MantisAI","description":"Full named-entity (i.e., not tag/token) evaluation metrics based on SemEval’13","archived":false,"fork":false,"pushed_at":"2025-03-25T09:43:57.000Z","size":290,"stargazers_count":171,"open_issues_count":19,"forks_count":23,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-12T01:26:49.856Z","etag":null,"topics":["evaluation-metrics","machine-learning","named-entity-recognition","natural-language-processing","sequence-models"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MantisAI.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-02-22T09:19:55.000Z","updated_at":"2025-03-25T09:44:02.000Z","dependencies_parsed_at":"2023-01-21T04:36:25.875Z","dependency_job_id":"74c0c7d0-33e0-4c9f-abbb-7c99ffce1645","html_url":"https://github.com/MantisAI/nervaluate","commit_stats":null,"previous_names":["ivyleavedtoadflax/nervaluate"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MantisAI%2Fnervaluate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MantisAI%2Fnervaluate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MantisAI%2Fnervaluate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MantisAI%2Fnervaluate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MantisAI","download_url":"https://codeload.github.com/MantisAI/nervaluate/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251304679,"owners_count":21567920,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation-metrics","machine-learning","named-entity-recognition","natural-language-processing","sequence-models"],"created_at":"2024-08-02T11:00:22.580Z","updated_at":"2025-04-28T11:30:43.380Z","avatar_url":"https://github.com/MantisAI.png","language":"Python","funding_links":[],"categories":["Evaluation and analysis"],"sub_categories":["Other"],"readme":"[![python](https://img.shields.io/badge/Python-3.9-3776AB.svg?style=flat\u0026logo=python\u0026logoColor=white)](https://www.python.org)\n\u0026nbsp;\n![example event parameter](https://github.com/davidsbatista/BREDS/actions/workflows/code_checks.yml/badge.svg?event=pull_request)\n\u0026nbsp;\n![code coverage](https://raw.githubusercontent.com/MantisAI/nervaluate/coverage-badge/coverage.svg?raw=true)\n\u0026nbsp;\n[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)\n\u0026nbsp;\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\u0026nbsp;\n![GitHub](https://img.shields.io/github/license/ivyleavedtoadflax/nervaluate)\n\u0026nbsp;\n![Pull Requests Welcome](https://img.shields.io/badge/pull%20requests-welcome-brightgreen.svg)\n\u0026nbsp;\n![PyPI](https://img.shields.io/pypi/v/nervaluate)\n\n# nervaluate\n\nnervaluate is a python module for evaluating Named Entity Recognition (NER) models as defined in the SemEval 2013 - 9.1 task.\n\nThe evaluation metrics output by nervaluate go beyond a simple token/tag based schema, and consider different scenarios \nbased on whether all the tokens that belong to a named entity were classified or not, and also whether the correct \nentity type was assigned.\n\nThis full problem is described in detail in the [original blog](http://www.davidsbatista.net/blog/2018/05/09/Named_Entity_Evaluation/) \npost by [David Batista](https://github.com/davidsbatista), and extends the code in the \n[original repository](https://github.com/davidsbatista/NER-Evaluation) which accompanied the blog post.\n\nThe code draws heavily on:\n\n* Segura-bedmar, I., \u0026 Mart, P. (2013). 2013 SemEval-2013 Task 9 Extraction of Drug-Drug Interactions from. Semeval, 2(DDIExtraction), 341–350. [link](https://www.aclweb.org/anthology/S13-2056)\n* [PDF link](https://aclanthology.org/S13-2056.pdf)\n\n## The problem\n\n### Token level evaluation for NER is too simplistic\n\nWhen running machine learning models for NER, it is common to report metrics at the individual token level. This may \nnot be the best approach, as a named entity can be made up of multiple tokens, so a full-entity accuracy would be \ndesirable.\n\nWhen comparing the golden standard annotations with the output of a NER system different scenarios might occur:\n\n__I. Surface string and entity type match__\n\n|Token|Gold|Prediction|\n|---|---|---|\n|in|O|O|\n|New|B-LOC|B-LOC|\n|York|I-LOC|I-LOC|\n|.|O|O|\n\n__II. System hypothesized an incorrect entity__\n\n|Token|Gold|Prediction|\n|---|---|---|\n|an|O|O|\n|Awful|O|B-ORG|\n|Headache|O|I-ORG|\n|in|O|O|\n\n__III. System misses an entity__\n\n|Token|Gold|Prediction|\n|---|---|---|\n|in|O|O|\n|Palo|B-LOC|O|\n|Alto|I-LOC|O|\n|,|O|O|\n\nBased on these three scenarios we have a simple classification evaluation that can be measured in terms of false \npositives, true positives, false negatives and false positives, and subsequently compute precision, recall and \nF1-score for each named-entity type.\n\nHowever, this simple schema ignores the possibility of partial matches or other scenarios when the NER system gets\nthe named-entity surface string correct but the type wrong, and we might also want to evaluate these scenarios \nagain at a full-entity level.\n\nFor example:\n\n__IV. System assigns the wrong entity type__\n\n|Token|Gold|Prediction|\n|---|---|---|\n|I|O|O|\n|live|O|O|\n|in|O|O|\n|Palo|B-LOC|B-ORG|\n|Alto|I-LOC|I-ORG|\n|,|O|O|\n\n__V. System gets the boundaries of the surface string wrong__\n\n|Token|Gold|Prediction|\n|---|---|---|\n|Unless|O|B-PER|\n|Karl|B-PER|I-PER|\n|Smith|I-PER|I-PER|\n|resigns|O|O|\n\n__VI. System gets the boundaries and entity type wrong__\n\n|Token|Gold|Prediction|\n|---|---|---|\n|Unless|O|B-ORG|\n|Karl|B-PER|I-ORG|\n|Smith|I-PER|I-ORG|\n|resigns|O|O|\n\nHow can we incorporate these described scenarios into evaluation metrics? See the [original blog](http://www.davidsbatista.net/blog/2018/05/09/Named_Entity_Evaluation/) \nfor a great explanation, a summary is included here:\n\nWe can use the following five metrics to consider difference categories of errors:\n\n|Error type|Explanation|\n|---|---|\n|Correct (COR)|both are the same|\n|Incorrect (INC)|the output of a system and the golden annotation don’t match|\n|Partial (PAR)|system and the golden annotation are somewhat “similar” but not the same|\n|Missing (MIS)|a golden annotation is not captured by a system|\n|Spurious (SPU)|system produces a response which doesn’t exist in the golden annotation|\n\nThese five metrics can be measured in four different ways:\n\n|Evaluation schema|Explanation|\n|---|---|\n|Strict|exact boundary surface string match and entity type|\n|Exact|exact boundary match over the surface string, regardless of the type|\n|Partial|partial boundary match over the surface string, regardless of the type|\n|Type|some overlap between the system tagged entity and the gold annotation is required|\n\nThese five errors and four evaluation schema interact in the following ways:\n\n|Scenario|Gold entity|Gold string|Pred entity|Pred string|Type|Partial|Exact|Strict|\n|---|---|---|---|---|---|---|---|---|\n|III|BRAND|tikosyn| | |MIS|MIS|MIS|MIS|\n|II| | |BRAND|healthy|SPU|SPU|SPU|SPU|\n|V|DRUG|warfarin|DRUG|of warfarin|COR|PAR|INC|INC|\n|IV|DRUG|propranolol|BRAND|propranolol|INC|COR|COR|INC|\n|I|DRUG|phenytoin|DRUG|phenytoin|COR|COR|COR|COR|\n|VI|GROUP|contraceptives|DRUG|oral contraceptives|INC|PAR|INC|INC|\n\nThen precision/recall/f1-score are calculated for each different evaluation schema. In order to achieve data, two more \nquantities need to be calculated:\n\n```\nPOSSIBLE (POS) = COR + INC + PAR + MIS = TP + FN\nACTUAL (ACT) = COR + INC + PAR + SPU = TP + FP\n```\n\nThen we can compute precision/recall/f1-score, where roughly describing precision is the percentage of correct \nnamed-entities found by the NER system, and recall is the percentage of the named-entities in the golden annotations \nthat are retrieved by the NER system. This is computed in two different ways depending on whether we want an exact \nmatch (i.e., strict and exact ) or a partial match (i.e., partial and type) scenario:\n\n__Exact Match (i.e., strict and exact )__\n```\nPrecision = (COR / ACT) = TP / (TP + FP)\nRecall = (COR / POS) = TP / (TP+FN)\n```\n__Partial Match (i.e., partial and type)__\n```\nPrecision = (COR + 0.5 × PAR) / ACT = TP / (TP + FP)\nRecall = (COR + 0.5 × PAR)/POS = COR / ACT = TP / (TP + FN)\n```\n\n__Putting all together:__\n\n|Measure|Type|Partial|Exact|Strict|\n|---|---|---|---|---|\n|Correct|3|3|3|2|\n|Incorrect|2|0|2|3|\n|Partial|0|2|0|0|\n|Missed|1|1|1|1|\n|Spurious|1|1|1|1|\n|Precision|0.5|0.66|0.5|0.33|\n|Recall|0.5|0.66|0.5|0.33|\n|F1|0.5|0.66|0.5|0.33|\n\n\n## Notes:\n\nIn scenarios IV and VI the entity type of the `true` and `pred` does not match, in both cases we only scored against \nthe true entity, not the predicted one. You can argue that the predicted entity could also be scored as spurious, \nbut according to the definition of `spurious`:\n\n* Spurious (SPU) : system produces a response which does not exist in the golden annotation;\n\nIn this case there exists an annotation, but with a different entity type, so we assume it's only incorrect.\n\n## Installation\n\n```\npip install nervaluate\n```\n\n## Example:\n\nThe main `Evaluator` class will accept a number of formats:\n\n* [prodi.gy](https://prodi.gy) style lists of spans.\n* Nested lists containing NER labels.\n* CoNLL style tab delimited strings.\n\n### Prodigy spans\n\n```\ntrue = [\n    [{\"label\": \"PER\", \"start\": 2, \"end\": 4}],\n    [{\"label\": \"LOC\", \"start\": 1, \"end\": 2},\n     {\"label\": \"LOC\", \"start\": 3, \"end\": 4}]\n]\n\npred = [\n    [{\"label\": \"PER\", \"start\": 2, \"end\": 4}],\n    [{\"label\": \"LOC\", \"start\": 1, \"end\": 2},\n     {\"label\": \"LOC\", \"start\": 3, \"end\": 4}]\n]\n\nfrom nervaluate import Evaluator\n\nevaluator = Evaluator(true, pred, tags=['LOC', 'PER'])\n\n# Returns overall metrics and metrics for each tag\n\nresults, results_per_tag, result_indices, result_indices_by_tag = evaluator.evaluate()\n\nprint(results)\n```\n\n```\n{\n    'ent_type':{\n        'correct':3,\n        'incorrect':0,\n        'partial':0,\n        'missed':0,\n        'spurious':0,\n        'possible':3,\n        'actual':3,\n        'precision':1.0,\n        'recall':1.0\n    },\n    'partial':{\n        'correct':3,\n        'incorrect':0,\n        'partial':0,\n        'missed':0,\n        'spurious':0,\n        'possible':3,\n        'actual':3,\n        'precision':1.0,\n        'recall':1.0\n    },\n    'strict':{\n        'correct':3,\n        'incorrect':0,\n        'partial':0,\n        'missed':0,\n        'spurious':0,\n        'possible':3,\n        'actual':3,\n        'precision':1.0,\n        'recall':1.0\n    },\n    'exact':{\n        'correct':3,\n        'incorrect':0,\n        'partial':0,\n        'missed':0,\n        'spurious':0,\n        'possible':3,\n        'actual':3,\n        'precision':1.0,\n        'recall':1.0\n    }\n}\n```\n\n```\nprint(results_by_tag)\n```\n\n```\n{\n    'LOC':{\n        'ent_type':{\n            'correct':2,\n            'incorrect':0,\n            'partial':0,\n            'missed':0,\n            'spurious':0,\n            'possible':2,\n            'actual':2,\n            'precision':1.0,\n            'recall':1.0\n        },\n        'partial':{\n            'correct':2,\n            'incorrect':0,\n            'partial':0,\n            'missed':0,\n            'spurious':0,\n            'possible':2,\n            'actual':2,\n            'precision':1.0,\n            'recall':1.0\n        },\n        'strict':{\n            'correct':2,\n            'incorrect':0,\n            'partial':0,\n            'missed':0,\n            'spurious':0,\n            'possible':2,\n            'actual':2,\n            'precision':1.0,\n            'recall':1.0\n        },\n        'exact':{\n            'correct':2,\n            'incorrect':0,\n            'partial':0,\n            'missed':0,\n            'spurious':0,\n            'possible':2,\n            'actual':2,\n            'precision':1.0,\n            'recall':1.0\n        }\n    },\n    'PER':{\n        'ent_type':{\n            'correct':1,\n            'incorrect':0,\n            'partial':0,\n            'missed':0,\n            'spurious':0,\n            'possible':1,\n            'actual':1,\n            'precision':1.0,\n            'recall':1.0\n        },\n        'partial':{\n            'correct':1,\n            'incorrect':0,\n            'partial':0,\n            'missed':0,\n            'spurious':0,\n            'possible':1,\n            'actual':1,\n            'precision':1.0,\n            'recall':1.0\n        },\n        'strict':{\n            'correct':1,\n            'incorrect':0,\n            'partial':0,\n            'missed':0,\n            'spurious':0,\n            'possible':1,\n            'actual':1,\n            'precision':1.0,\n            'recall':1.0\n        },\n        'exact':{\n            'correct':1,\n            'incorrect':0,\n            'partial':0,\n            'missed':0,\n            'spurious':0,\n            'possible':1,\n            'actual':1,\n            'precision':1.0,\n            'recall':1.0\n        }\n    }\n}\n```\n\n```\nfrom nervaluate import summary_report_overall_indices\n\nprint(summary_report_overall_indices(evaluation_indices=result_indices, error_schema='partial', preds=pred))\n```\n\n```\nIndices for error schema 'partial':\n\nCorrect indices:\n  - Instance 0, Entity 0: Label=PER, Start=2, End=4\n  - Instance 1, Entity 0: Label=LOC, Start=1, End=2\n  - Instance 1, Entity 1: Label=LOC, Start=3, End=4\n\nIncorrect indices:\n  - None\n\nPartial indices:\n  - None\n\nMissed indices:\n  - None\n\nSpurious indices:\n  - None\n```\n\n### Nested lists\n\n```\ntrue = [\n    ['O', 'O', 'B-PER', 'I-PER', 'O'],\n    ['O', 'B-LOC', 'I-LOC', 'B-LOC', 'I-LOC', 'O'],\n]\n\npred = [\n    ['O', 'O', 'B-PER', 'I-PER', 'O'],\n    ['O', 'B-LOC', 'I-LOC', 'B-LOC', 'I-LOC', 'O'],\n]\n\nevaluator = Evaluator(true, pred, tags=['LOC', 'PER'], loader=\"list\")\n\nresults, results_by_tag, result_indices, result_indices_by_tag = evaluator.evaluate()\n```\n\n### CoNLL style tab delimited\n\n```\n\ntrue = \"word\\tO\\nword\\tO\\B-PER\\nword\\tI-PER\\n\"\n\npred = \"word\\tO\\nword\\tO\\B-PER\\nword\\tI-PER\\n\"\n\nevaluator = Evaluator(true, pred, tags=['PER'], loader=\"conll\")\n\nresults, results_by_tag, result_indices, result_indices_by_tag = evaluator.evaluate()\n\n```\n\n## Extending the package to accept more formats\n\nAdditional formats can easily be added to the module by creating a conversion function in `nervaluate/utils.py`, \nfor example `conll_to_spans()`. This function must return the spans in the prodigy style dicts shown in the prodigy \nexample above.\n\nThe new function can then be added to the list of loaders in `nervaluate/nervaluate.py`, and can then be selection \nwith the `loader` argument when instantiating the `Evaluator` class.\n\nA list of formats we intend to include is included in https://github.com/ivyleavedtoadflax/nervaluate/issues/3.\n\n\n## Contributing to the nervaluate package\n\nImprovements, adding new features and bug fixes are welcome. If you wish to participate in the development of nervaluate \nplease read the following guidelines.\n\n## The contribution process at a glance\n\n1. Preparing the development environment\n2. Code away!\n3. Continuous Integration\n4. Submit your changes by opening a pull request\n\nSmall fixes and additions can be submitted directly as pull requests, but larger changes should be discussed in \nan issue first. You can expect a reply within a few days, but please be patient if it takes a bit longer. \n\n\n## Preparing the development environment\n\nMake sure you have Python3.8 installed on your system\n\nmacOs\n```sh\nbrew install python@3.8\npython3.8 -m pip install --user --upgrade pip\npython3.8 -m pip install virtualenv\n```\n\nClone the repository and prepare the development environment:\n\n```sh\ngit clone git@github.com:MantisAI/nervaluate.git\ncd nervaluate\npython3.8 -m virtualenv venv\nsource venv/bin/activate\npip install -r requirements_dev.txt\npip install -e .\n```\n\n\n## Continuous Integration\n\nnervaluate runs a continuous integration (CI) on all pull requests. This means that if you open a pull request (PR), \na full  test suite is run on your PR: \n\n- The code is formatted using `black`\n- Linting is done using `pyling` and `flake8`\n- Type checking is done using `mypy`\n- Tests are run using `pytest`\n\nNevertheless, if you prefer to run the tests \u0026 formatting locally, it's possible too. \n\n```sh\nmake all\n```\n\n## Opening a Pull Request\n\nEvery PR should be accompanied by short description of the changes, including:\n- Impact and  motivation for the changes\n- Any open issues that are closed by this PR\n\n---\n\nGive a ⭐️ if this project helped you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMantisAI%2Fnervaluate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMantisAI%2Fnervaluate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMantisAI%2Fnervaluate/lists"}