{"id":21581423,"url":"https://github.com/dlr-eoc/ukis-metrics","last_synced_at":"2025-08-08T02:35:16.446Z","repository":{"id":62586203,"uuid":"372803763","full_name":"dlr-eoc/ukis-metrics","owner":"dlr-eoc","description":"numpy-based implementation of common performance metrics for semantic image segmentation","archived":false,"fork":false,"pushed_at":"2023-05-03T13:51:35.000Z","size":2280,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-10T19:13:22.303Z","etag":null,"topics":["deep-learning","earth-observation","metrics","object-detection","semantic-segmentation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dlr-eoc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.rst","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-06-01T11:22:16.000Z","updated_at":"2024-11-28T20:33:02.000Z","dependencies_parsed_at":"2023-02-16T06:45:23.332Z","dependency_job_id":null,"html_url":"https://github.com/dlr-eoc/ukis-metrics","commit_stats":{"total_commits":39,"total_committers":6,"mean_commits":6.5,"dds":0.6153846153846154,"last_synced_commit":"782180241559ffdde9c8e0db564a90d43207da7c"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/dlr-eoc/ukis-metrics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlr-eoc%2Fukis-metrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlr-eoc%2Fukis-metrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlr-eoc%2Fukis-metrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlr-eoc%2Fukis-metrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dlr-eoc","download_url":"https://codeload.github.com/dlr-eoc/ukis-metrics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlr-eoc%2Fukis-metrics/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269354447,"owners_count":24403256,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","earth-observation","metrics","object-detection","semantic-segmentation"],"created_at":"2024-11-24T14:12:28.156Z","updated_at":"2025-08-08T02:35:16.419Z","avatar_url":"https://github.com/dlr-eoc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [![UKIS](img/ukis-logo.png)](https://www.dlr.de/eoc/en/desktopdefault.aspx/tabid-5413/10560_read-21914/) ukis-metrics\n\n![ukis-metrics](https://github.com/dlr-eoc/ukis-metrics/workflows/ukis-metrics/badge.svg)\n[![codecov](https://codecov.io/gh/dlr-eoc/ukis-metrics/branch/main/graph/badge.svg)](https://codecov.io/gh/dlr-eoc/ukis-metrics)\n![Upload Python Package](https://github.com/dlr-eoc/ukis-metrics/workflows/Upload%20Python%20Package/badge.svg)\n[![PyPI version](https://img.shields.io/pypi/v/ukis-metrics)](https://pypi.python.org/pypi/ukis-metrics/)\n[![GitHub license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)\n[![Code Style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://black.readthedocs.io/en/stable/)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5561771.svg)](https://doi.org/10.5281/zenodo.5561771)\n\nA pure Numpy-based implementation of the most common performance metrics for semantic image segmentation. \n\n## Installation\n```shell\npip install ukis_metrics\n```\n\n```shell\n\u003e\u003e\u003e import ukis_metrics\n\u003e\u003e\u003e ukis_metrics.__version__\n'0.1.3'\n```\n\n## Why?\nSimply because we wanted a lightweight and fast alternative to scikit learn for tracking during training. \n\n[execution_time.py](https://github.com/dlr-eoc/ukis-metrics/blob/main/performance/execution_time.py)\ncompares the execution time of ukis-metrics with sklearn. Here's an example output:\n```\nShape of array: (256, 256, 2)\n\n                                        ### Metrics execution time [s] ###\n\n                ukis-metrics                            sklearn metrics         speed gain\nacc             0.001900                                0.007627                4.01\nrec             0.001716                                0.024509                14.28\npre             0.001815                                0.025021                13.79\nf1              0.001798                                0.024770                13.78\niou             0.001797                                0.024247                13.49\nkap             0.001824                                0.034577                18.96\n```\n\n## Workings and included metrics\nIn a first step the **true positives** *tp*, **true negatives** *tn*, **false positives** *fp*, **false negatives** *fn*\nand the number of valid pixels **n_valid_pixels** are computed. These values are then used to compute the following \nmetrics:\n- Accuracy [1]:   \n  ```math\n  acc = \\frac{tp + tn}{tp + fn + fp + tn}\n  ```\n- Recall [1]:\n  ```math\n  rec = \\frac{tp}{tp + fn}\n  ```\n- Precision [1]:\n  ```math\n  prec = \\frac{tp}{tp + fp}\n  ```\n- F1-score [2]:\n  ```math\n  F1 = \\frac{2 * prec * rec}{prec + rec}\n  ```\n- IoU [3]:\n  ```math\n  IoU = \\frac{tp}{tp + fp + fn}\n  ```  \n- Kappa: The computation of the Kappa-score incorporates several steps. Please refer to [4] for the full \n  documentation\n\n## How to use\nSimply pass a Numpy ndarray to get a dict containing the `tpfptnfn` and pass the dict to `segmentation_metrics(tpfptnfn)`:\n```python \nimport ukis_metrics.seg_metrics as segm\nimport numpy as np\n# ndarray containing the reference data, e.g.\nshape = (256, 256, 1)\ny_true = np.ones(shape)\n# ndarray containing the model predicions, e.g.\ny_pred = np.ones(shape)\n# get tp, fp, tn, fn an n_valid_pixel\ntpfptnfn = segm.tpfptnfn(y_true, y_pred, None)\nmetrics = segm.segmentation_metrics(tpfptnfn)\n```\nSo far these metrics were only used for binary classification, although one should be able to use them for \nmulticlass segmentation too, if the slices for a given class are provided individually.\n\nThere is a [jupyter notebook](https://github.com/dlr-eoc/ukis-metrics/tree/main/examples) showing how ukis-metrics can be used and how it could be extended for multiclass problems.\n\n## References\n- [1] [Sokolova and Lapalme 2009: A systematic analysis of performance measures for classification tasks](https://www.researchgate.net/publication/222674734_A_systematic_analysis_of_performance_measures_for_classification_tasks)\n- [2] [scikit-learn](https://scikit-learn.org/stable/modules/model_evaluation.html)\n- [3] [Berman et al. 2018: The Lovász-Softmax loss: A tractable surrogate for the optimization of \n  the intersection-over-union measure in neural networks](https://arxiv.org/pdf/1705.08790.pdf)\n- [4] [Tang et al. 2015: Kappa coefficient: a popular measure of rater agreement](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4372765/) \n\n\n## Contributors\nThe UKIS team creates and adapts libraries which simplify the usage of satellite data. Our team includes (in alphabetical order):\n* Fichtner, Florian\n* Helleis, Max\n* Martinis, Sandro\n* Wieland, Marc\n\nGerman Aerospace Center (DLR)\n\n## Licenses\nThis software is licensed under the [Apache 2.0 License](https://github.com/dlr-eoc/ukis-metrics/blob/main/LICENSE).\n\nCopyright (c) 2021 German Aerospace Center (DLR) * German Remote Sensing Data Center * Department: Geo-Risks and Civil Security\n\n## Changelog\nSee [changelog](https://github.com/dlr-eoc/ukis-metrics/blob/main/CHANGELOG.md).\n\n\n## What is UKIS?\nThe DLR project Environmental and Crisis Information System (the German abbreviation is UKIS, standing for [Umwelt- und Kriseninformationssysteme](https://www.dlr.de/eoc/en/desktopdefault.aspx/tabid-5413/10560_read-21914/) aims at harmonizing the development of information systems at the German Remote Sensing Data Center (DFD) and setting up a framework of modularized and generalized software components.\n\nUKIS is intended to ease and standardize the process of setting up specific information systems and thus bridging the gap from EO product generation and information fusion to the delivery of products and information to end users.\n\nFurthermore, the intention is to save and broaden know-how that was and is invested and earned in the development of information systems and components in several ongoing and future DFD projects.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlr-eoc%2Fukis-metrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdlr-eoc%2Fukis-metrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlr-eoc%2Fukis-metrics/lists"}