{"id":32175163,"url":"https://github.com/maridda/nerval","last_synced_at":"2026-02-22T14:44:09.586Z","repository":{"id":41093483,"uuid":"478206797","full_name":"maridda/nerval","owner":"maridda","description":"Python framework to evaluate Named Entity Recognition (NER) models. Creates entity-level confusion matrix and classification report.","archived":false,"fork":false,"pushed_at":"2024-06-27T16:29:22.000Z","size":396,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-12-13T03:10:55.984Z","etag":null,"topics":["classification-report","confusion-matrix","deep-learning","f1-score","machine-learning","named-entity-recognition","natural-language-processing","ner","nlp","precison-recall","python","sequence-labeling","sequence-labeling-evaluation"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maridda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-05T16:12:16.000Z","updated_at":"2024-06-27T16:25:52.000Z","dependencies_parsed_at":"2024-05-31T18:29:54.154Z","dependency_job_id":"b7cbe101-2007-4382-a9b9-2589f62e1388","html_url":"https://github.com/maridda/nerval","commit_stats":{"total_commits":53,"total_committers":1,"mean_commits":53.0,"dds":0.0,"last_synced_commit":"66e49988d5797c05fb2765e8d452c4fe8b4cbccf"},"previous_names":["maridda/nerval","mdadda/nerval"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/maridda/nerval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maridda%2Fnerval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maridda%2Fnerval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maridda%2Fnerval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maridda%2Fnerval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maridda","download_url":"https://codeload.github.com/maridda/nerval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maridda%2Fnerval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29716358,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-22T13:30:57.152Z","status":"ssl_error","status_checked_at":"2026-02-22T13:30:28.561Z","response_time":110,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["classification-report","confusion-matrix","deep-learning","f1-score","machine-learning","named-entity-recognition","natural-language-processing","ner","nlp","precison-recall","python","sequence-labeling","sequence-labeling-evaluation"],"created_at":"2025-10-21T19:26:20.712Z","updated_at":"2026-02-22T14:44:09.578Z","avatar_url":"https://github.com/maridda.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nerval\r\nEntity-level confusion matrix and classification report to evaluate Named Entity Recognition (NER) models.\r\n\r\n\r\n## Labelling schemes supported:\r\n- IO\r\n- BIO1 (IOB1)\r\n- BIO2 (IOB2)\r\n- IOE1\r\n- IOE2\r\n- IOBES\r\n- BILOU\r\n- BMEWO\r\n\r\n\r\n## Options for the 'scheme' argument:\r\n- **BIO** for the following schemes: IO / BIO1 (IOB1) / BIO2 (IOB2) / IOBES / BILOU / BMEWO\r\n- **IOE** for the following schemes: IOE1 / IOE2\r\n- **BIO** is the default scheme.\r\n\r\n\r\n## Output:\r\n- Classification report\r\n- Confusion matrix\r\n- Labels for the confusion matrix\r\n- Confusion matrix plot\r\n\r\n\r\n## How to use it:\r\n\r\n```python\r\n\u003e\u003e\u003e from nerval import crm\r\n\r\n\u003e\u003e\u003e y_true = [['O', 'B-PER', 'I-PER', 'O', 'O', 'O', 'O', 'B-LOC', 'I-LOC']]\r\n\u003e\u003e\u003e y_pred = [['O', 'B-PER', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC']]\r\n\r\n\u003e\u003e\u003e cr, cm, cm_labels = crm(y_true, y_pred, scheme='BIO')\r\nTrue Entities: 2\r\nPred Entities: 2\r\n\r\nTrue Entities with 3 or more tags: 0\r\nPred Entities with 3 or more tags: 0\r\n\r\nTrue positives:  0\r\nFalse positives (true = 'O'):  1\r\nFalse positives (true \u003c\u003e pred):  1\r\nToT False positives:  2\r\nFalse negatives:  1\r\n\r\n\u003e\u003e\u003e print(cr)\r\nprecision  recall  f1_score  true_entities  pred_entities\r\nPER                0.00    0.00      0.00           1.00           0.00\r\nLOC                0.00    0.00      0.00           1.00           1.00\r\nPER__              0.00    0.00      0.00           0.00           1.00\r\nmicro_avg          0.00    0.00      0.00           2.00           2.00\r\nmacro_avg          0.00    0.00      0.00           2.00           2.00\r\nweighted_avg       0.00    0.00      0.00           2.00           2.00\r\n\r\n\u003e\u003e\u003e print(cm)\r\n[[0 1 0 0]\r\n [1 0 0 0]\r\n [0 0 0 1]\r\n [0 0 0 0]]\r\n\r\n\u003e\u003e\u003e print(cm_labels)\r\n['LOC', 'O', 'PER', 'PER__']\r\n```\r\n\r\n```python\r\n\u003e\u003e\u003e from nerval import plot_confusion_matrix\r\n\r\n\u003e\u003e\u003e y_true = [['O', 'B-PER', 'I-PER', 'O', 'O', 'O', 'O', 'B-LOC', 'I-LOC']]\r\n\u003e\u003e\u003e y_pred = [['O', 'B-PER', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC']]\r\n\r\n\u003e\u003e\u003e plot_confusion_matrix(cm, cm_labels, show=True, save=False, img_path=None, normalize=None, decimal_places=2, figsize=(15,15), SMALL_SIZE=8, MEDIUM_SIZE=12, BIGGER_SIZE=14, cmap='OrRd', xticks_rotation='vertical', title='Confusion Matrix')\r\n\r\n\u003e\u003e\u003e plot_confusion_matrix(cm, cm_labels, show=True, save=True, img_path=None)\r\n\r\n\u003e\u003e\u003e plot_confusion_matrix(cm, cm_labels, show=True, save=True, img_path=r'C:\\Users\\...\\my_conf_matrix.png')\r\n\r\n\u003e\u003e\u003e plot_confusion_matrix(cm, cm_labels, show=False, save=True, img_path=None)\r\n\r\n\u003e\u003e\u003e plot_confusion_matrix(cm, cm_labels, show=False, save=True, img_path=r'C:\\Users\\...\\my_conf_matrix.png')\r\n```\r\n\r\n### Note 1:\r\n**y_true** and **y_pred** could be:\r\n- flat lists\r\n- lists of flat lists\r\n- lists of nested lists.\r\n\r\nFlat lists and lists of nested lists will be converted to lists of lists.\r\n\r\n\r\n### Note 2:\r\nThe __ at the end of some entities means that true and pred have the same entity name for the first token but the prediction is somewhat different from the true label.\r\nExamples:\r\n```python\r\n\u003e\u003e\u003e y_true = ['B-ORG', 'I-ORG', 'I-ORG']\r\n\u003e\u003e\u003e y_pred = ['B-ORG']\r\n\r\n\u003e\u003e\u003e y_true = ['B-ORG', 'I-ORG', 'I-ORG']\r\n\u003e\u003e\u003e y_pred = ['B-ORG', 'I-ORG', 'I-ORG', 'I-ORG', 'I-ORG']\r\n\r\n\u003e\u003e\u003e y_true = ['B-ORG', 'I-ORG', 'I-ORG']\r\n\u003e\u003e\u003e y_pred = ['B-ORG', 'I-PER']\r\n\r\n\u003e\u003e\u003e y_true = ['B-ORG', 'I-ORG', 'I-ORG']\r\n\u003e\u003e\u003e y_pred = ['I-ORG', 'I-PER']\r\n```\r\n\r\n### Note 3:\r\nThe normalize argument could be:\r\n- None\r\n- 'true'\r\n- 'pred'\r\n- 'all'\r\n\r\nDefault is None.\r\n\r\n\r\n### Note 4:\r\nIn case of division by zero, the result will default to zero.\r\n\r\n### Note 5:\r\nParameters in function plot_confusion_matrix():\r\n- show: show the plot (default: True)\r\n- save: save the plot (default: False) \r\n- img_path: where to save the plot - e.g. r'C:\\Users\\User\\...\\my_conf_matrix.png' (default: None - this means save the plot in current dir)\r\n\r\n\r\n## Installation\r\n```bash\r\npip install nerval\r\nconda install -c conda-forge nerval\r\n```\r\n\r\n\r\n## License\r\n[MIT](https://github.com/maridda/nerval/blob/main/LICENCE.txt)\r\n\r\n\r\n## Citation\r\n```text\r\n@misc{nerval,\r\n  title={{nerval}: Entity-level confusion matrix and classification report to evaluate Named Entity Recognition (NER) models.},\r\n  url={https://github.com/maridda/nerval},\r\n  note={Software available from https://github.com/maridda/nerval},\r\n  author={Mariangela D'Addato},\r\n  year={2022},\r\n}\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaridda%2Fnerval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaridda%2Fnerval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaridda%2Fnerval/lists"}