{"id":34032615,"url":"https://github.com/cyk1337/eval4ner","last_synced_at":"2026-04-01T23:02:35.455Z","repository":{"id":45921967,"uuid":"158794107","full_name":"cyk1337/eval4ner","owner":"cyk1337","description":"[PyPI] NER MUC evaluation toolkit in Python","archived":false,"fork":false,"pushed_at":"2025-08-16T06:00:07.000Z","size":176,"stargazers_count":16,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-03-28T00:58:08.042Z","etag":null,"topics":["named-entity-recognition","ner","ner-evaluation","nlp","python-implementaion"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cyk1337.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-11-23T07:08:33.000Z","updated_at":"2025-08-16T06:00:11.000Z","dependencies_parsed_at":"2022-09-05T00:51:27.955Z","dependency_job_id":null,"html_url":"https://github.com/cyk1337/eval4ner","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/cyk1337/eval4ner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyk1337%2Feval4ner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyk1337%2Feval4ner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyk1337%2Feval4ner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyk1337%2Feval4ner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cyk1337","download_url":"https://codeload.github.com/cyk1337/eval4ner/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cyk1337%2Feval4ner/sbom","scorecard":{"id":314392,"data":{"date":"2025-08-11","repo":{"name":"github.com/cyk1337/eval4ner","commit":"8e4140d1b5f987c98e824e01f45027617451338c"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.6,"checks":[{"name":"Code-Review","score":0,"reason":"Found 1/20 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":9,"reason":"binaries present in source code","details":["Warn: binary detected: dist/eval4ner-0.1.0-py3-none-any.whl:1"],"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENCE:0","Info: FSF or OSI recognized license: MIT License: LICENCE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v0.1.0 not signed: https://api.github.com/repos/cyk1337/eval4ner/releases/72289185","Warn: release artifact v0.1.0 does not have provenance: https://api.github.com/repos/cyk1337/eval4ner/releases/72289185"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 12 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-17T23:53:20.253Z","repository_id":45921967,"created_at":"2025-08-17T23:53:20.253Z","updated_at":"2025-08-17T23:53:20.253Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31292785,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T21:15:39.731Z","status":"ssl_error","status_checked_at":"2026-04-01T21:15:34.046Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["named-entity-recognition","ner","ner-evaluation","nlp","python-implementaion"],"created_at":"2025-12-13T18:54:01.356Z","updated_at":"2026-04-01T23:02:35.450Z","avatar_url":"https://github.com/cyk1337.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# eval4ner: An All-Round Evaluation for Named Entity Recognition\n![Stable version](https://img.shields.io/pypi/v/eval4ner)\n![Python3](https://img.shields.io/pypi/pyversions/eval4ner)![wheel:eval4ner](https://img.shields.io/pypi/wheel/eval4ner)\n![Download](https://img.shields.io/pypi/dm/eval4ner)\n![MIT License](https://img.shields.io/pypi/l/eval4ner)\n\n\n\nTable of Contents\n=================\n\n- [TL;DR](https://github.com/cyk1337/eval4ner/#tldr)\n- [Preliminaries for NER Evaluation](https://github.com/cyk1337/eval4ner/#preliminaries-for-ner-evaluation)\n- [User Guide](https://github.com/cyk1337/eval4ner/#user-guide)\n    - [Installation](https://github.com/cyk1337/eval4ner/#installation)\n    - [Usage](https://github.com/cyk1337/eval4ner/#usage)\n- [Citation](https://github.com/cyk1337/eval4ner/#citation)\n- [References](https://github.com/cyk1337/eval4ner/#references)\n\nThis is a Python toolkit of MUC-5 evaluation metrics for evaluating Named Entity Recognition (NER) results. \n\n\n## TL;DR\nIt considers not only the mode of strict matching, *i.e.*, extracted entities are correct w.r.t both boundaries and types, but that of partial match, summarizing as following four modes:  \n- Strict：exact match (Both entity boundary and type are correct)\n- Exact boundary matching：predicted entity boundary is correct, regardless of entity boundary\n- Partial boundary matching：entity boundaries overlap, regardless of entity boundary\n- Type matching：some overlap between the system tagged entity and the gold annotation is required;\n\n\nRefer to the blog [Evaluation Metrics of Name Entity Recognition](https://ychai.uk/notes/2018/11/21/NLP/NER/NER-Evaluation-Metrics/#SemEval%E2%80%9813) for explanations of MUC metric.\n\n## Preliminaries for NER Evaluation\nIn research and production, following scenarios of NER systems can occur frequently: \n\n\u003ctable class=\"tg\"\u003e\n  \u003ctr\u003e\n    \u003cth class=\"tg-0pky\"\u003eScenario\u003c/th\u003e\n    \u003cth class=\"tg-c3ow\" colspan=\"2\"\u003eGolden Standard\u003c/th\u003e\n    \u003cth class=\"tg-c3ow\" colspan=\"2\"\u003eNER system prediction\u003c/th\u003e\n    \u003cth class=\"tg-c3ow\" colspan=\"4\"\u003eMeasure\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-0pky\"\u003e\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003eEntity Type\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003eEntity Boundary (Surface String)\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eEntity Type\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eEntity Boundary (Surface String)\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eType\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003ePartial\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eExact\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eStrict\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-0pky\"\u003eIII\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003eMUSIC_NAME\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003e告白气球\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003e\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003e\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eMIS\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eMIS\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eMIS\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eMIS\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-0pky\"\u003eII\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003e\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003e\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eMUSIC_NAME\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003e年轮\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eSPU\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eSPU\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eSPU\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eSPU\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-0pky\"\u003eV\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003eMUSIC_NAME\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003e告白气球\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eMUSIC_NAME\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003e一首告白气球\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eCOR\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003ePAR\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eINC\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eINC\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-0pky\"\u003eIV\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003eMUSIC_NAME\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003e告白气球\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eSINGER\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003e告白气球\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eINC\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eCOR\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eCOR\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eINC\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-0pky\"\u003eI\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003eMUSIC_NAME\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003e告白气球\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eMUSIC_NAME\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003e告白气球\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eCOR\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eCOR\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eCOR\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eCOR\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-0pky\"\u003eVI\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003eMUSIC_NAME\u003c/td\u003e\n    \u003ctd class=\"tg-c3ow\"\u003e告白气球\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eSINGER\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003e一首告白气球\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eINC\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003ePAR\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eINC\u003c/td\u003e\n    \u003ctd class=\"tg-0pky\"\u003eINC\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\nThus, MUC-5 takes into account all these scenarios for all-sided evaluation. \n\nThen we can compute:\n\n**Number of golden standard**:\n\nPossible(POS) = COR + INC + PAR + MIS = TP + FN\n\n**Number of predictee**: \n\nActual(ACT) = COR + INC + PAR + SPU = TP + FP\n\nThe evaluation type of exact match and partial match are as follows:\n### Exact match(i.e. Strict, Exact)\n\n$\\text{Precision = COR / ACT = TP / (TP + FP)}$\n\n$\\text{Recall = COR / POS = TP / (TP + FN)}$\n\n### Partial match (i.e. Partial, Type)\n$\\text{Precision = (COR + 0.5 * PAR) /ACT}$\n$\\text{Recall = (COR + 0.5 * PAR)/ POS }$\n\n\n### F-Measure\n\n$F_\\alpha = ((\\alpha^2 + 1)* PR) / (\\alpha^2 P + R)$\n\n$F_1 = (2PR)/ (P +R)$\n\nTherefore, we can get the results:\n\u003ctable class=\"tg\"\u003e\n  \u003ctr\u003e\n    \u003cth class=\"tg-e6bt\"\u003eMeasure\u003c/th\u003e\n    \u003cth class=\"tg-23iq\"\u003eType\u003c/th\u003e\n    \u003cth class=\"tg-23iq\"\u003ePartial\u003c/th\u003e\n    \u003cth class=\"tg-ww3v\"\u003eExact\u003c/th\u003e\n    \u003cth class=\"tg-ww3v\"\u003eStrict\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-e6bt\"\u003eCorrect\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e2\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e2\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e2\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e1\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-e6bt\"\u003eIncorrect\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e2\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e0\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e2\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e3\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-e6bt\"\u003ePartial\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e0\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e2\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e0\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e0\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-e6bt\"\u003eMissed\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e1\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e1\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e1\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e1\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-e6bt\"\u003eSpurius\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e1\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e1\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e1\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e1\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-e6bt\"\u003ePrecision\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e0.4\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e0.6\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e0.4\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e0.2\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-e6bt\"\u003eRecall\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e0.4\u003c/td\u003e\n    \u003ctd class=\"tg-23iq\"\u003e0.6\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e0.4\u003c/td\u003e\n    \u003ctd class=\"tg-ww3v\"\u003e0.2\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-gx32\"\u003eF1 score\u003c/td\u003e\n    \u003ctd class=\"tg-t0np\"\u003e0.4\u003c/td\u003e\n    \u003ctd class=\"tg-t0np\"\u003e0.6\u003c/td\u003e\n    \u003ctd class=\"tg-8l38\"\u003e0.4\u003c/td\u003e\n    \u003ctd class=\"tg-8l38\"\u003e0.2\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n## User Guide\n### Installation\n```bash\npip install [-U] eval4ner\n```\n\n### Usage\n#### 1. Evaluate single prediction\n```python\nimport eval4ner.muc as muc\nimport pprint\ngrount_truth = [('PER', 'John Jones'), ('PER', 'Peter Peters'), ('LOC', 'York')]\nprediction = [('PER', 'John Jones and Peter Peters came to York')]\ntext = 'John Jones and Peter Peters came to York'\none_result = muc.evaluate_one(prediction, grount_truth, text)\npprint.pprint(one_result)\n```\n\nOutput:\n```bash\n{'exact': {'actual': 1,\n           'correct': 0,\n           'f1_score': 0,\n           'incorrect': 1,\n           'missed': 2,\n           'partial': 0,\n           'possible': 3,\n           'precision': 0.0,\n           'recall': 0.0,\n           'spurius': 0},\n 'partial': {'actual': 1,\n             'correct': 0,\n             'f1_score': 0.25,\n             'incorrect': 0,\n             'missed': 2,\n             'partial': 1,\n             'possible': 3,\n             'precision': 0.5,\n             'recall': 0.16666666666666666,\n             'spurius': 0},\n 'strict': {'actual': 1,\n            'correct': 0,\n            'f1_score': 0,\n            'incorrect': 1,\n            'missed': 2,\n            'partial': 0,\n            'possible': 3,\n            'precision': 0.0,\n            'recall': 0.0,\n            'spurius': 0},\n 'type': {'actual': 1,\n          'correct': 1,\n          'f1_score': 0.5,\n          'incorrect': 0,\n          'missed': 2,\n          'partial': 0,\n          'possible': 3,\n          'precision': 1.0,\n          'recall': 0.3333333333333333,\n          'spurius': 0}}\n\n```\n\n#### 2. Evaluate all predictions\n```python\nimport eval4ner.muc as muc\n# ground truth\ngrount_truths = [\n    [('PER', 'John Jones'), ('PER', 'Peter Peters'), ('LOC', 'York')],\n    [('PER', 'John Jones'), ('PER', 'Peter Peters'), ('LOC', 'York')],\n    [('PER', 'John Jones'), ('PER', 'Peter Peters'), ('LOC', 'York')]\n]\n# NER model prediction\npredictions = [\n    [('PER', 'John Jones and Peter Peters came to York')],\n    [('LOC', 'John Jones'), ('PER', 'Peters'), ('LOC', 'York')],\n    [('PER', 'John Jones'), ('PER', 'Peter Peters'), ('LOC', 'York')]\n]\n# input texts\ntexts = [\n    'John Jones and Peter Peters came to York',\n    'John Jones and Peter Peters came to York',\n    'John Jones and Peter Peters came to York'\n]\nmuc.evaluate_all(predictions, grount_truths * 1, texts, verbose=True)\n```\n\nOutput:\n```bash\n NER evaluation scores:\n  strict mode, Precision=0.4444, Recall=0.4444, F1:0.4444\n   exact mode, Precision=0.5556, Recall=0.5556, F1:0.5556\n partial mode, Precision=0.7778, Recall=0.6667, F1:0.6944\n    type mode, Precision=0.8889, Recall=0.6667, F1:0.7222\n```\n\nThis repo will be long-term supported. Welcome to contribute and PR.\n\n## Citation\nFor attribution in academic contexts, please cite this work as:\n```\n@misc{eval4ner,\n  title={Evaluation Metrics of Named Entity Recognition},\n  author={Chai, Yekun},\n  year={2018},\n  howpublished={\\url{https://cyk1337.github.io/notes/2018/11/21/NLP/NER/NER-Evaluation-Metrics/}},\n}\n\n@misc{chai2018-ner-eval,\n  author = {Chai, Yekun},\n  title = {eval4ner: An All-Round Evaluation for Named Entity Recognition},\n  year = {2019},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/cyk1337/eval4ner}}\n}\n```\n\n## References\n1. [Evaluation of the SemEval-2013 Task 9.1: Recognition and Classification of pharmacological substances](https://www.cs.york.ac.uk/semeval-2013/task9/data/uploads/semeval_2013-task-9_1-evaluation-metrics.pdf)\n2. [MUC-5 Evaluation Metrics](https://www.aclweb.org/anthology/M93-1007.pdf)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyk1337%2Feval4ner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcyk1337%2Feval4ner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyk1337%2Feval4ner/lists"}