{"id":25595626,"url":"https://github.com/sotirismos/object-detection-metrics","last_synced_at":"2025-07-17T18:04:23.382Z","repository":{"id":164853270,"uuid":"478449651","full_name":"sotirismos/object-detection-metrics","owner":"sotirismos","description":"Various metrics for object detection written from scratch","archived":false,"fork":false,"pushed_at":"2023-05-13T15:02:30.000Z","size":7967,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-04T01:15:09.883Z","etag":null,"topics":["computer-vision","image-ai","iou","machine-vision","mmdetection","object-detection","pr-curve"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sotirismos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-06T07:26:28.000Z","updated_at":"2023-08-30T15:42:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"bb79cebf-5653-45cf-b61e-027f47a86a02","html_url":"https://github.com/sotirismos/object-detection-metrics","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sotirismos/object-detection-metrics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sotirismos%2Fobject-detection-metrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sotirismos%2Fobject-detection-metrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sotirismos%2Fobject-detection-metrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sotirismos%2Fobject-detection-metrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sotirismos","download_url":"https://codeload.github.com/sotirismos/object-detection-metrics/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sotirismos%2Fobject-detection-metrics/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260600316,"owners_count":23034689,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","image-ai","iou","machine-vision","mmdetection","object-detection","pr-curve"],"created_at":"2025-02-21T11:29:55.773Z","updated_at":"2025-07-17T18:04:23.371Z","avatar_url":"https://github.com/sotirismos.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Comparison of various object detection metrics via their precision - recall curves ##\n\n### Usage:\n\nTo generate a PR plot of all the metrics for a given threshold of 0.75, dpi scale 200, 12 inch output image size, run:\n\n```bash\npython metrics.py --thres 0.75 --dirpath \u003cpath to a dir\u003e --pkl \u003cpath to .pkl predictions\u003e --dpi 200 --size 12\n```\n\n*Arguments*\n\n```\n --dirpath \n```\nAbsolute or relative path to output directory\n\n```\n --weight \n```\nweighting factor for hybrid and fmeasure metrics\n\n```\n --thres\n```\ndecision threshold, a sample with a metric above the threshold is classified as a TP.\n\n```\n --size\n```\noutput plot size in inches\n\n```\n --dpi\n```\nsets the dpi scale of the plot file\n\n```\n --plot\n```\na switch option, show the plot in a window\n\n```\n --pkl\n```\na serialized list of dictionaries in pickle file format\n\n```\n --ratio\n```\nset the aspec ratio of the ouput image\n  \n---\nEach list element is a dictionary that contains information about\nthe detected and ground truth objects in an image,\na summary of its key/value pairs is presented below:\n\n| Key      | Value |\n| -------  | ----- |\n| det      | list of detected bboxes in imageai format|\n| filename | the filename of the related image |\n| gt       | a list of ground truth objects |\n| matches  | a list of tupled index pairs |\n\n---\n[ImageAI](https://github.com/OlafenwaMoses/ImageAI) returns the following dictionary for each detection in an image:\n\n\n| Key        |Value  |\n| -------    | ----- |\n| box_points | bbox corner coordinates in the image's coordinate space [top-left-x, top-left-y, right-bot-x, right-bot-y]|\n| name       | the name of the label associated with the detected object |\n| percentage_probability | the detected object's confidence score |\n\n---\n- Description of matches:\n\n\tLet's assume that we tested a detection and classification system on an image resulting in\n\tno detected objects, although the image contains a ground truth object. \n\tThen the set of gt is {0} while the detection set is empty {}.\n\n\t```\n\t[(None, 0)]\n\t```\n\n\tAnother case could be that of 2 ground truth objects and 1 detected object.\n\tIn this case the list of tuples are (the second tuple implies a false negative):\n\n\n\t```\n\t[(0,0), (None,1)]\n\t```\n\n\tAn example of FP would be the following:\n\t```\n\t[(0, None)]\n\t```\n\n\tFinally, an example of perfect matching, no FP or FN\n\t```\n\t[(0,1), (1,0)]\n\t```\n---\n### Generation of license plates .pkl prediction files \n\n- [mdict_list.pkl](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/mdict_list.pkl) is generated running [two_stage_lp.py](https://github.com/sotirismos/Anonymization-Pipeline/blob/master/two_stage_lp.py) from GRUBLES-Depersonalization-pipeline repository.\n\n- [mdict_list_mmdetection.pkl](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/mdict_list_mmdetection.pkl) is generated running [two_stage_lp_mmdetection.py](https://github.com/sotirismos/Anonymization-Pipeline/blob/master/two_stage_lp_mmdetection.py) from GRUBLES-Depersonalization-pipeline repository.\n\n---\n### Generation of primary features .pkl prediction files \n\n- [mdict_list_traffic_light.pkl](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/mdict_list_traffic_light.pkl) is generated running [inference.py](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/inference.py).\n\n- [mdict_list_traffic_sign.pkl](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/mdict_list_traffic_sign.pkl) is generated running [inference.py](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/inference.py).\n\n- [mdict_list_vehicle.pkl](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/mdict_list_vehicle.pkl) is generated running [inference.py](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/inference.py).\n\n---\n[inference.py](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/inference.py) has the following functionalities.\n- Loading a pretrained model on BDD100K dataset, which consists of 10 classes.\n- Making predictions on images annotated on a subset of BDD100K classes on CVAT and exported as CVAT for images 1.1 (refer to GRUBLES-Annotation-pipeline).\n- Edit predictions and return the following dictionary for each detection in an image.\n\n| Key        |Value  |\n| -------    | ----- |\n| box_points | bbox corner coordinates in the image's coordinate space [top-left-x, top-left-y, right-bot-x, right-bot-y]|\n| name       | the name of the label associated with the detected object |\n| confidence | the detected object's confidence score |\n\n---\n### Metrics\nIn order to evaluate the performance of the models for each class (license plates, traffic lights, traffic signs, vehicles) we calculated the mAP based on the\nprecision-recall curves and exploting various metrics on different threshold values [0.5, 0.7]. To get the TP, FP, FN in order to construct the precision-recall curves, we utilized the following metrics.\n- IoU = intersection_area / union_area\n- \"Precision\" = intersection_area / detection_area\n- \"Recall\" = intersection_area / gt_area\n- \"LP metric 1\" = detection_area / union_area\n- \"LP metric 2\" = gt_area / union_area\n\n---\n### Evaluation datasets\nFor the class of license plates, the evaluation dataset is a subset of the Stanford cars dataset and consists of 105 images of different types of vehicles (mainly cars) accompanied by 105 license plate annotation files. \u003cbr /\u003e\nFor the classes of vehicles (cars, buses, trucks, motorcycles), traffic lights and traffic signs the dataset collected during the GRUBLES projects was utilized. More specifically, videos from the 6 different Insta360 Pro2 cameras were randomly selected. The frames from these videos were extracted and 500 random frames were selected, where each frame was annotated at a bounding box level, resulting to 1616 annotated vehicles, 125 annotated traffic lights and 371 annotated traffic signs.\n\n---\n### License plates evaluation plots\n\nImageAI @ threshold = 0.5            |  mmdetection @ threshold = 0.5\n:-------------------------:|:-------------------------:\n![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_license_plates/pr_t50.png)  |  ![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_license_plates_mmdetection/pr_t50.png)\n\nImageAI @ threshold = 0.6            |  mmdetection @ threshold = 0.6\n:-------------------------:|:-------------------------:\n![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_license_plates/pr_t60.png)  |  ![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_license_plates_mmdetection/pr_t60.png)\n\nImageAI @ threshold = 0.7            |  mmdetection @ threshold = 0.7\n:-------------------------:|:-------------------------:\n![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_license_plates/pr_t70.png)  |  ![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_license_plates_mmdetection/pr_t70.png)\n\n---\n### Primary features evaluation plots\n\nTraffic Light @ threshold = 0.5            |  Traffic Light @ threshold = 0.7\n:-------------------------:|:-------------------------:\n![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_traffic_light/pr_t50.png)  |  ![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_traffic_light/pr_t70.png)\n\nTraffic Sign @ threshold = 0.5            |  Traffic Sign @ threshold = 0.7\n:-------------------------:|:-------------------------:\n![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_traffic_sign/pr_t50.png)  |  ![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_traffic_sign/pr_t70.png)\n\nVehicle @ threshold = 0.5            |  Vehicle @ threshold = 0.7\n:-------------------------:|:-------------------------:\n![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_vehicle/pr_t50.png)  |  ![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/plots_vehicle/pr_t70.png)\n\n---\n### Conclusion\nThe two-stage license plate detection approach utilizing [mmdetection](https://github.com/sotirismos/mmdetection) provides much better perfomance that the [ImageAI](https://github.com/OlafenwaMoses/ImageAI) approach. \u003cbr /\u003e\nFor the class of traffic lights, during the annotation process for a specific traffic light we outline 1 bounding box, while during the prediction process this can be \"broken\" into 2 or more, as shown in the figure below. Also, during prediction inverted traffic lights that have not been annotated are predicted as *True Positives*. Setting the threshold for IoU to 0.5, the mAP results to 40.69%. This is justified by the fact that there are many cases where we have 2 or more predictions and 1 annotation, where the predictions have a very small IoU and as a result are categorized as *False Positives*, and the prediction of inverted traffic lights as *True Positives*. Thus, the number of *False Positives* is enormous and the mAP is low. \u003cbr /\u003e\nFor the vehicles the model works perfectly. Setting the threshold for IoU equal to 0.5 results to mAP = 62.28%, which is due to the fact that vehicles were not exhaustively annotated, resulting in a high number of *False Positives*. The ability of the model to locate almost every vehicle without fail is illustrated below and is verified by the fact that the Recall metric for this class is equal to 1. \u003cbr /\u003e\nFor the class of traffic signs, the model works satisfactorily. Setting the threshold for the IoU equal to 0.5 results to mAP = 34.71%, where this particular low value is due to the fact that the model also detects the inverted signs that we did not annotate resulting in a high number of *False Positives*, as illustrated below.\n\n![](https://github.com/sotirismos/Object-Detection-Metrics/blob/master/conclusion.jpg)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsotirismos%2Fobject-detection-metrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsotirismos%2Fobject-detection-metrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsotirismos%2Fobject-detection-metrics/lists"}