{"id":13935591,"url":"https://github.com/clovaai/CLEval","last_synced_at":"2025-07-19T20:33:22.192Z","repository":{"id":37635220,"uuid":"267226708","full_name":"clovaai/CLEval","owner":"clovaai","description":"CLEval: Character-Level Evaluation for Text Detection and Recognition Tasks","archived":false,"fork":false,"pushed_at":"2023-10-17T01:48:32.000Z","size":2640,"stargazers_count":184,"open_issues_count":8,"forks_count":28,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-11-21T06:02:18.344Z","etag":null,"topics":["end-to-end-ocr","evaluation-metrics","text-detection","text-detection-recognition","text-recognition"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clovaai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-05-27T05:12:23.000Z","updated_at":"2024-10-30T07:16:25.000Z","dependencies_parsed_at":"2024-01-15T23:27:12.688Z","dependency_job_id":"aa6607cf-0f8c-43ee-b7d6-76f6e2194441","html_url":"https://github.com/clovaai/CLEval","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2FCLEval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2FCLEval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2FCLEval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2FCLEval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clovaai","download_url":"https://codeload.github.com/clovaai/CLEval/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226677121,"owners_count":17666010,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["end-to-end-ocr","evaluation-metrics","text-detection","text-detection-recognition","text-recognition"],"created_at":"2024-08-07T23:01:54.773Z","updated_at":"2024-11-27T03:30:51.191Z","avatar_url":"https://github.com/clovaai.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# CLEval: Character-Level Evaluation for Text Detection and Recognition Tasks\n\nOfficial implementation of CLEval | [paper](https://arxiv.org/abs/2006.06244)\n\n## Overview\nWe propose a Character-Level Evaluation metric (CLEval). To perform fine-grained assessment of the results, *instance matching* process handles granularity difference and *scoring process* conducts character-level evaluation. Please refer to the paper for more details. This code is based on [ICDAR15 official evaluation code](http://rrc.cvc.uab.es/).\n\n### 2023.10.16 Huge Update\n- **Much More Faster Version** of CLEval has been Uploaded!!\n- Support CLI \n- Support torchmetric\n- Support scale-wise evaluation\n\n\n### Simplified Method Description\n![Explanation](resources/screenshots/explanation.gif)\n\n## Supported annotation types\n* **LTRB**(xmin, ymin, xmax, ymax)\n* **QUAD**(x1, y1, x2, y2, x3, y3, x4, y4)\n* **POLY**(x1, y1, x2, y2, ..., x_2n, y_2n)\n\n## Supported datasets\n* ICDAR 2013 Focused Scene Text [Link](https://rrc.cvc.uab.es/?ch=2)\n* ICDAR 2015 Incidental Scene Text [Link](https://rrc.cvc.uab.es/?ch=4)\n* TotalText [Link](https://github.com/cs-chan/Total-Text-Dataset)\n* Any other datasets that have a similar format with the datasets mentioned above\n\n## Installation\n\n### Build from pip\ndownload from Clova OCR pypi\n```bash\n$ pip install cleval\n```\n\nor build with url\n```bash\n$ pip install git+https://github.com/clovaai/CLEval.git --user\n```\n\n### Build from source\n\n```bash\n$ git clone https://github.com/clovaai/CLEval.git\n$ cd cleval\n$ python setup.py install --user\n```\n\n## How to use\nYou can replace `cleval` with `PYTHONPATH=$PWD python cleval/main.py` for evaluation using source.\n```bash\n$ PYTHONPATH=$PWD python cleval/main.py -g=gt/gt_IC13.zip -s=[result.zip] --BOX_TYPE=LTRB \n```\n\n### Detection evaluation (CLI)\n```bash\n$ cleval -g=gt/gt_IC13.zip -s=[result.zip] --BOX_TYPE=LTRB          # IC13\n$ cleval -g=gt/gt_IC15.zip -s=[result.zip]                          # IC15\n$ cleval -g=gt/gt_TotalText.zip -s=[result.zip] --BOX_TYPE=POLY     # TotalText\n```\n* Notes\n  * The default value of ```BOX_TYPE``` is set to ```QUAD```. It can be explicitly set to ```--BOX_TYPE=QUAD``` when running evaluation on IC15 dataset.\n  * Add ```--TANSCRIPTION``` option if the result file contains transcription.\n  * Add ```--CONFIDENCES``` option if the result file contains confidence.\n\n### End-to-end evaluation (CLI)\n```bash\n$ cleval -g=gt/gt_IC13.zip -s=[result.zip] --E2E --BOX_TYPE=LTRB        # IC13\n$ cleval -g=gt/gt_IC15.zip -s=[result.zip] --E2E                        # IC15\n$ cleval -g=gt/gt_TotalText.zip -s=[result.zip] --E2E --BOX_TYPE=POLY   # TotalText\n```\n* Notes\n  * Adding ```--E2E``` also automatically adds ```--TANSCRIPTION``` option. Make sure that the transcriptions are included in the result file.  \n  * Add ```--CONFIDENCES``` option if the result file contains confidence.\n\n### TorchMetric\n```python\nfrom cleval import CLEvalMetric\nmetric = CLEvalMetric()\n\nfor gt, det in zip(gts, dets):\n    # your fancy algorithm\n    # ...\n    # gt_quads = ...\n    # det_quads = ...\n    # ...\n    _ = metric(det_quads, gt_quads, det_letters, gt_letters, gt_is_dcs)\n\nmetric_out = metric.compute()\nmetric.reset()\n```\n\n### Profiling\n```bash\n$ cleval -g=resources/test_data/gt/gt_eval_doc_v1_kr_single.zip -s=resources/test_data/pred/res_eval_doc_v1_kr_single.zip --E2E -v --DEBUG --PPROFILE \u003e profile.txt\n$ PYTHONPATH=$PWD python cleval/main.py -g resources/test_data/gt/dummy_dataset_val.json -s resources/test_data/pred/dummy_dataset_val.json --SCALE_WISE --DOMAIN_WISE --ORIENTATION --E2E --ORIENTATION -v --PROFILE --DEBUG \u003e profile.txt\n```\n\n### Paramters for evaluation script\n| name | type | default | description |\n| ---- | ---- | ------- | ---- |\n| -g | ```string``` | | path to ground truth zip file |\n| -s | ```string``` | | path to result zip file |\n| -o | ```string``` | | path to save per-sample result file 'results.zip' |\n\n| name | type | default | description |\n| ---- | ---- | ------- | ---- |\n| --BOX_TYPE | ```string``` | ```QUAD``` | annotation type of box (LTRB, QUAD, POLY) |\n| --TRANSCRIPTION | ```boolean``` | ```False``` | set True if result file has transcription |\n| --CONFIDENCES | ```boolean``` | ```False``` | set True if result file has confidence |\n| --E2E | ```boolean``` | ```False``` | to measure end-to-end evaluation (if not, detection evalution only) |\n| --CASE_SENSITIVE | ```boolean``` | ```True``` | set True to evaluate case-sensitively. (only used in end-to-end evaluation) |\n* Note : Please refer to ```arg_parser.py``` file for additional parameters and default settings used internally.\n\n* Note : For scalewise evaluation, we measure the ratio of the shorter length (text height) of the text-box to the longer length of the image. \nThrough this, evaluation for each ratio can be performed. To adjust the scales, please use SCALE_BINS argument.\n\n## Citation\n```\n@article{baek2020cleval,\n  title={CLEval: Character-Level Evaluation for Text Detection and Recognition Tasks},\n  author={Youngmin Baek, Daehyun Nam, Sungrae Park, Junyeop Lee, Seung Shin, Jeonghun Baek, Chae Young Lee and Hwalsuk Lee},\n  journal={arXiv preprint arXiv:2006.06244},\n  year={2020}\n}\n```\n\n## Contact us\nCLEval has been proposed to make fair evaluation in the OCR community, so we want to hear from many researchers. We welcome any feedbacks to our metric, and appreciate pull requests if you have any comments or improvements.\n\n## License\n```\nCopyright (c) 2020-present NAVER Corp.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n```\n\n### Contribute\nPlease use pre-commit which uses Black and Isort.\n```\n$ pip install pre-commit\n$ pre-commit install\n```\n\n##### Step By Step\n1. Write an issue.\n2. Match code style (black, isort)\n3. Wirte test code.\n4. Delete branch after Squash\u0026Merge.\n\nRequired Approve: 1\n\n## Code Maintainer\n- Donghyun Kim (artit.anthony@gmail.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclovaai%2FCLEval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclovaai%2FCLEval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclovaai%2FCLEval/lists"}