{"id":13935592,"url":"https://github.com/davidsbatista/NER-Evaluation","last_synced_at":"2025-07-19T20:33:22.113Z","repository":{"id":41176238,"uuid":"133837987","full_name":"davidsbatista/NER-Evaluation","owner":"davidsbatista","description":"An implementation of a full named-entity evaluation metrics based on SemEval'13 Task 9 - not at tag/token level but considering all the tokens that are part of the named-entity","archived":false,"fork":false,"pushed_at":"2024-07-02T08:58:45.000Z","size":88,"stargazers_count":213,"open_issues_count":6,"forks_count":49,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-08-08T23:21:17.912Z","etag":null,"topics":["crfsuite","evaluation-metrics","named-entity-recognition","ner","ner-evaluation","notebook-jupyter","semeval","semeval-2013"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidsbatista.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-17T16:04:33.000Z","updated_at":"2024-08-02T02:10:47.000Z","dependencies_parsed_at":"2022-08-25T15:10:27.523Z","dependency_job_id":null,"html_url":"https://github.com/davidsbatista/NER-Evaluation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidsbatista%2FNER-Evaluation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidsbatista%2FNER-Evaluation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidsbatista%2FNER-Evaluation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidsbatista%2FNER-Evaluation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidsbatista","download_url":"https://codeload.github.com/davidsbatista/NER-Evaluation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226677121,"owners_count":17666010,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crfsuite","evaluation-metrics","named-entity-recognition","ner","ner-evaluation","notebook-jupyter","semeval","semeval-2013"],"created_at":"2024-08-07T23:01:54.802Z","updated_at":"2024-11-27T03:30:51.406Z","avatar_url":"https://github.com/davidsbatista.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Named Entity Evaluation as in SemEval 2013 task 9.1\n\nMy own implementation, with lots of input from [Matt Upson](https://github.com/ivyleavedtoadflax), of the Named-Entity Recognition evaluation metrics as defined by the SemEval 2013 - 9.1 task.\n\nThis evaluation metrics go belong a simple token/tag based schema, and consider diferent scenarios based on wether all the tokens that belong to a named entity were classified or not, and also wether the correct entity type was assigned.\n\nYou can find a more detailed explanation in the following blog post:\n\n* http://www.davidsbatista.net/blog/2018/05/09/Named_Entity_Evaluation/\n\n\n## Notes:\n\nIn scenarios IV and VI the entity type of the `true` and `pred` does not match, in both cases we only scored against the true entity, not the predicted one. You can argue that the predicted entity could also be scored as spurious, but according to the definition of `spurious`:\n\n* Spurius (SPU) : system produces a response which doesn’t exist in the golden annotation;\n\nIn this case it exists an annotation, but only with a different entity type, so we assume it's only incorrect\n\n\n## Example:\n\nYou can see a working example on the following notebook:\n\n- [example-full-named-entity-evaluation.ipynb](example-full-named-entity-evaluation.ipynb)\n\nNote that in order to run that example you need to have installed:\n\n- sklearn\n- nltk\n- sklearn_crfsuite\n\nFor testing you will need:\n\n- pytest\n- coverage\n\nThese dependencies can be installed by running `pip3 install -r requirements.txt`\n\n## Code tests and tests coverage:\n\nTo run tests:\n\n`coverage run --rcfile=setup.cfg -m pytest`\n\nTo produce a coverage report:\n\n`coverage report`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidsbatista%2FNER-Evaluation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidsbatista%2FNER-Evaluation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidsbatista%2FNER-Evaluation/lists"}