{"id":13935589,"url":"https://github.com/jantrienes/nereval","last_synced_at":"2025-08-27T14:26:13.304Z","repository":{"id":42879170,"uuid":"111694450","full_name":"jantrienes/nereval","owner":"jantrienes","description":"Evaluation script for named entity recognition (NER) systems based on entity-level F1 score.","archived":false,"fork":false,"pushed_at":"2021-04-20T17:32:43.000Z","size":23,"stargazers_count":70,"open_issues_count":2,"forks_count":8,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-07-05T11:07:08.520Z","etag":null,"topics":["evaluation-metrics","machine-learning","named-entity-recognition","nlp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jantrienes.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-22T14:29:03.000Z","updated_at":"2025-01-17T09:51:39.000Z","dependencies_parsed_at":"2022-07-08T03:03:34.842Z","dependency_job_id":null,"html_url":"https://github.com/jantrienes/nereval","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jantrienes/nereval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jantrienes%2Fnereval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jantrienes%2Fnereval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jantrienes%2Fnereval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jantrienes%2Fnereval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jantrienes","download_url":"https://codeload.github.com/jantrienes/nereval/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jantrienes%2Fnereval/sbom","scorecard":{"id":505736,"data":{"date":"2025-08-11","repo":{"name":"github.com/jantrienes/nereval","commit":"3b943d9a2470b0884db1b78d3279fe8c07b23b6a"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.8,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 0/18 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":8,"reason":"2 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2020-92 / GHSA-hj5v-574p-mj7c","Warn: Project is vulnerable to: PYSEC-2022-42969"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 9 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-19T23:12:36.846Z","repository_id":42879170,"created_at":"2025-08-19T23:12:36.847Z","updated_at":"2025-08-19T23:12:36.847Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272340216,"owners_count":24917342,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-27T02:00:09.397Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation-metrics","machine-learning","named-entity-recognition","nlp"],"created_at":"2024-08-07T23:01:54.729Z","updated_at":"2025-08-27T14:26:13.271Z","avatar_url":"https://github.com/jantrienes.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"nereval\n=======\n.. image:: https://travis-ci.org/jantrienes/nereval.svg?branch=master\n    :target: https://travis-ci.org/jantrienes/nereval\n\nEvaluation script for named entity recognition (NER) systems based on entity-level F1 score.\n\nDefinition\n----------\nThe metric as implemented here has been described by Nadeau and Sekine (2007) and was widely used as part of the Message Understanding Conferences (Grishman and Sundheim, 1996). It evaluates an NER system according to two axes: whether it is able to assign the right type to an entity, and whether it finds the exact entity boundaries. For both axes, the number of correct predictions (COR), the number of actual predictions (ACT) and the number of possible predictions (POS) are computed. From these statistics, precision and recall can be derived:\n\n::\n\n  precision = COR/ACT\n  recall = COR/POS\n\n\nThe final score is the micro-averaged F1 measure of precision and recall of both type and boundary axes.\n\nInstallation\n------------\n.. code-block:: bash\n\n  pip install nereval\n\n\nUsage\n-----\nThe script can either be used from within Python or from the command line when classification results have been written to a JSON file.\n\nUsage from Command Line\n~~~~~~~~~~~~~~~~~~~~~~~\nAssume we have the following classification results in ``input.json``:\n\n.. code-block:: json\n\n  [\n    {\n      \"text\": \"CILINDRISCHE PLUG\",\n      \"true\": [\n        {\n          \"text\": \"CILINDRISCHE PLUG\",\n          \"type\": \"Productname\",\n          \"start\": 0\n        }\n      ],\n      \"predicted\": [\n        {\n          \"text\": \"CILINDRISCHE\",\n          \"type\": \"Productname\",\n          \"start\": 0\n        },\n        {\n          \"text\": \"PLUG\",\n          \"type\": \"Productname\",\n          \"start\": 13\n        }\n      ]\n    }\n  ]\n\n\nThen the script can be executed as follows:\n\n.. code-block:: bash\n\n  python nereval.py input.json\n  F1-score: 0.33\n\n\nUsage from Python\n~~~~~~~~~~~~~~~~~\nAlternatively, the evaluation metric can be directly invoked from within python. Example:\n\n.. code-block:: python\n\n  import nereval\n  from nereval import Entity\n\n  # Ground-truth:\n  # CILINDRISCHE PLUG\n  # B_PROD       I_PROD\n  y_true = [\n      Entity('CILINDRISCHE PLUG', 'Productname', 0)\n  ]\n\n  # Prediction:\n  # CILINDRISCHE PLUG\n  # B_PROD       B_PROD\n  y_pred = [\n      # correct type, wrong text\n      Entity('CILINDRISCHE', 'Productname', 0),\n      # correct type, wrong text\n      Entity('PLUG', 'Productname', 13)\n  ]\n\n  score = nereval.evaluate([y_true], [y_pred])\n  print('F1-score: %.2f' % score)\n  F1-score: 0.33\n\n\nNote on Symmetry\n----------------\nThe metric itself is not symmetric due to the inherent problem of word overlaps in NER. So ``evaluate(y_true, y_pred) != evaluate(y_pred, y_true)``. This comes apparent if we consider the following example (tagger uses an BIO scheme):\n\n.. code-block:: bash\n\n  # Example 1:\n  Input:     CILINDRISCHE PLUG     DIN908  M10X1   Foo\n  Truth:     B_PROD       I_PROD   B_PROD  B_DIM   O\n  Predicted: B_PROD       B_PROD   B_PROD  B_PROD  B_PROD\n\n  Correct Text: 2\n  Correct Type: 2\n\n  # Example 2 (inversed):\n  Input:     CILINDRISCHE PLUG     DIN908  M10X1   Foo\n  Truth:     B_PROD       B_PROD   B_PROD  B_PROD  B_PROD\n  Predicted: B_PROD       I_PROD   B_PROD  B_DIM   O\n\n  Correct Text: 2\n  Correct Type: 3\n\n\nNotes and References\n--------------------\nUsed in a student research project on natural language processing at `University of Twente, Netherlands \u003chttps://www.utwente.nl\u003e`_.\n\n**References**\n\n* Grishman, R., \u0026 Sundheim, B. (1996). `Message understanding conference-6: A brief history \u003chttp://www.aclweb.org/anthology/C96-1079\u003e`_. *In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics* (Vol. 1).\n* Nadeau, D., \u0026 Sekine, S. (2007). `A survey of named entity recognition and classification \u003chttp://www.jbe-platform.com/content/journals/10.1075/li.30.1.03nad\u003e`_. *Lingvisticae Investigationes*, 30(1), 3-26.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjantrienes%2Fnereval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjantrienes%2Fnereval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjantrienes%2Fnereval/lists"}