{"id":13443021,"url":"https://github.com/open-mmlab/mmeval","last_synced_at":"2025-04-04T15:06:16.606Z","repository":{"id":62361643,"uuid":"516602779","full_name":"open-mmlab/mmeval","owner":"open-mmlab","description":"A unified evaluation library for multiple machine learning libraries","archived":false,"fork":false,"pushed_at":"2024-03-29T03:56:22.000Z","size":2126,"stargazers_count":261,"open_issues_count":18,"forks_count":50,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-28T14:05:00.715Z","etag":null,"topics":["machine-learning","metrics","python","pytorch","tensorflow"],"latest_commit_sha":null,"homepage":"https://mmeval.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/open-mmlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-22T03:56:57.000Z","updated_at":"2025-03-10T09:47:13.000Z","dependencies_parsed_at":"2024-06-19T02:47:59.314Z","dependency_job_id":"a1e40b88-c973-44ba-b841-4ef306190d53","html_url":"https://github.com/open-mmlab/mmeval","commit_stats":{"total_commits":75,"total_committers":21,"mean_commits":"3.5714285714285716","dds":0.64,"last_synced_commit":"9ce87f0f45732b220ff566e3b0c9fabf3c2baf14"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-mmlab%2Fmmeval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-mmlab%2Fmmeval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-mmlab%2Fmmeval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-mmlab%2Fmmeval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/open-mmlab","download_url":"https://codeload.github.com/open-mmlab/mmeval/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247198443,"owners_count":20900079,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","metrics","python","pytorch","tensorflow"],"created_at":"2024-07-31T03:01:54.903Z","updated_at":"2025-04-04T15:06:16.585Z","avatar_url":"https://github.com/open-mmlab.png","language":"Python","funding_links":[],"categories":["Python","Object Detection Applications"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/open-mmlab/mmeval/raw/main/docs/zh_cn/_static/image/mmeval-logo.png\" height=\"100\"/\u003e\n  \u003cdiv\u003e\u0026nbsp;\u003c/div\u003e\n  \u003cdiv align=\"center\"\u003e\n    \u003cb\u003e\u003cfont size=\"5\"\u003eOpenMMLab website\u003c/font\u003e\u003c/b\u003e\n    \u003csup\u003e\n      \u003ca href=\"https://openmmlab.com\"\u003e\n        \u003ci\u003e\u003cfont size=\"4\"\u003eHOT\u003c/font\u003e\u003c/i\u003e\n      \u003c/a\u003e\n    \u003c/sup\u003e\n    \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n    \u003cb\u003e\u003cfont size=\"5\"\u003eOpenMMLab platform\u003c/font\u003e\u003c/b\u003e\n    \u003csup\u003e\n      \u003ca href=\"https://platform.openmmlab.com\"\u003e\n        \u003ci\u003e\u003cfont size=\"4\"\u003eTRY IT OUT\u003c/font\u003e\u003c/i\u003e\n      \u003c/a\u003e\n    \u003c/sup\u003e\n  \u003c/div\u003e\n  \u003cdiv\u003e\u0026nbsp;\u003c/div\u003e\n\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/mmeval)](https://pypi.org/project/mmeval/)\n[![PyPI](https://img.shields.io/pypi/v/mmeval)](https://pypi.org/project/mmeval)\n[![license](https://img.shields.io/github/license/open-mmlab/mmeval.svg)](https://github.com/open-mmlab/mmeval/blob/main/LICENSE)\n[![open issues](https://isitmaintained.com/badge/open/open-mmlab/mmeval.svg)](https://github.com/open-mmlab/mmeval/issues)\n[![issue resolution](https://isitmaintained.com/badge/resolution/open-mmlab/mmeval.svg)](https://github.com/open-mmlab/mmeval/issues)\n\n[🤔Reporting Issues](https://github.com/open-mmlab/mmeval/issues/new/choose)\n\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\nEnglish | [简体中文](README_zh-CN.md)\n\n\u003c/div\u003e\n\n## Introduction\n\nMMEval is a machine learning evaluation library that supports efficient and accurate distributed evaluation on a variety of machine learning frameworks.\n\nMajor features:\n\n- Comprehensive metrics for various computer vision tasks (NLP will be covered soon!)\n- Efficient and accurate distributed evaluation, backed by multiple distributed communication backends\n- Support multiple machine learning frameworks via dynamic input dispatching mechanism\n\n\u003cdiv  align=\"center\"\u003e\n  \u003cimg src=\"docs/zh_cn/_static/image/mmeval-arch.png\" width=\"600\"/\u003e\n\u003c/div\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e Supported distributed communication backends \u003c/summary\u003e\n\n|                                                                       MPI4Py                                                                       |                                                                                                                                                     torch.distributed                                                                                                                                                     |                                                                           Horovod                                                                           |                                                                 paddle.distributed                                                                 |                                                                     oneflow.comm                                                                      |\n| :------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: |\n| [MPI4PyDist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.MPI4PyDist.html#mmeval.core.dist_backends.MPI4PyDist) | [TorchCPUDist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.TorchCPUDist.html#mmeval.core.dist_backends.TorchCPUDist) \u003cbr\u003e [TorchCUDADist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.TorchCUDADist.html#mmeval.core.dist_backends.TorchCUDADist) | [TFHorovodDist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.TFHorovodDist.html#mmeval.core.dist_backends.TFHorovodDist) | [PaddleDist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.PaddleDist.html#mmeval.core.dist_backends.PaddleDist) | [OneFlowDist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.OneFlowDist.html#mmeval.core.dist_backends.OneFlowDist) |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e Supported metrics and ML frameworks \u003c/summary\u003e\n\n`NOTE: MMEval tested with PyTorch 1.6+, TensorFlow 2.4+, Paddle 2.2+ and OneFlow 0.8+.`\n\n|                                                                           Metric                                                                           | numpy.ndarray | torch.Tensor | tensorflow.Tensor | paddle.Tensor | oneflow.Tensor |\n| :--------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------: | :----------: | :---------------: | :-----------: | :------------: |\n|                   [Accuracy](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.Accuracy.html#mmeval.metrics.Accuracy)                   |       ✔       |      ✔       |         ✔         |       ✔       |       ✔        |\n|     [SingleLabelMetric](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.SingleLabelMetric.html#mmeval.metrics.SingleLabelMetric)      |       ✔       |      ✔       |                   |               |       ✔        |\n|       [MultiLabelMetric](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.MultiLabelMetric.html#mmeval.metrics.MultiLabelMetric)       |       ✔       |      ✔       |                   |               |       ✔        |\n|       [AveragePrecision](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.AveragePrecision.html#mmeval.metrics.AveragePrecision)       |       ✔       |      ✔       |                   |               |       ✔        |\n|                    [MeanIoU](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.MeanIoU.html#mmeval.metrics.MeanIoU)                     |       ✔       |      ✔       |         ✔         |       ✔       |       ✔        |\n|                 [VOCMeanAP](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.VOCMeanAP.html#mmeval.metrics.VOCMeanAP)                  |       ✔       |              |                   |               |                |\n|                 [OIDMeanAP](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.OIDMeanAP.html#mmeval.metrics.OIDMeanAP)                  |       ✔       |              |                   |               |                |\n|           [COCODetection](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.COCODetection.html#mmeval.metrics.COCODetection)            |       ✔       |              |                   |               |                |\n|          [ProposalRecall](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.ProposalRecall.html#mmeval.metrics.ProposalRecall)          |       ✔       |              |                   |               |                |\n|                    [F1Score](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.F1Score.html#mmeval.metrics.F1Score)                     |       ✔       |      ✔       |                   |               |       ✔        |\n|                   [HmeanIoU](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.HmeanIoU.html#mmeval.metrics.HmeanIoU)                   |       ✔       |              |                   |               |                |\n|              [PCKAccuracy](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.PCKAccuracy.html#mmeval.metrics.PCKAccuracy)               |       ✔       |              |                   |               |                |\n|        [MpiiPCKAccuracy](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.MpiiPCKAccuracy.html#mmeval.metrics.MpiiPCKAccuracy)         |       ✔       |              |                   |               |                |\n|       [JhmdbPCKAccuracy](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.JhmdbPCKAccuracy.html#mmeval.metrics.JhmdbPCKAccuracy)       |       ✔       |              |                   |               |                |\n|           [EndPointError](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.EndPointError.html#mmeval.metrics.EndPointError)            |       ✔       |      ✔       |                   |               |       ✔        |\n|                 [AVAMeanAP](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.AVAMeanAP.html#mmeval.metrics.AVAMeanAP)                  |       ✔       |              |                   |               |                |\n| [StructuralSimilarity](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.StructuralSimilarity.html#mmeval.metrics.StructuralSimilarity) |       ✔       |              |                   |               |                |\n|       [SignalNoiseRatio](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.SignalNoiseRatio.html#mmeval.metrics.SignalNoiseRatio)       |       ✔       |              |                   |               |                |\n| [PeakSignalNoiseRatio](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.PeakSignalNoiseRatio.html#mmeval.metrics.PeakSignalNoiseRatio) |       ✔       |              |                   |               |                |\n|     [MeanAbsoluteError](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.MeanAbsoluteError.html#mmeval.metrics.MeanAbsoluteError)      |       ✔       |              |                   |               |                |\n|       [MeanSquaredError](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.MeanSquaredError.html#mmeval.metrics.MeanSquaredError)       |       ✔       |              |                   |               |                |\n\n\u003c/details\u003e\n\n## Installation\n\n`MMEval` requires Python 3.6+ and can be installed via pip.\n\n```bash\npip install mmeval\n```\n\nTo install the dependencies required for all the metrics provided in `MMEval`, you can install them with the following command.\n\n```bash\npip install 'mmeval[all]'\n```\n\n## Get Started\n\nThere are two ways to use `MMEval`'s metrics, using `Accuracy` as an example:\n\n```python\nfrom mmeval import Accuracy\nimport numpy as np\n\naccuracy = Accuracy()\n```\n\nThe first way is to directly call the instantiated `Accuracy` object to calculate the metric.\n\n```python\nlabels = np.asarray([0, 1, 2, 3])\npreds = np.asarray([0, 2, 1, 3])\naccuracy(preds, labels)\n# {'top1': 0.5}\n```\n\nThe second way is to calculate the metric after accumulating data from multiple batches.\n\n```python\nfor i in range(10):\n    labels = np.random.randint(0, 4, size=(100, ))\n    predicts = np.random.randint(0, 4, size=(100, ))\n    accuracy.add(predicts, labels)\n\naccuracy.compute()\n# {'top1': ...}\n```\n\n## Learn More\n\n\u003cdetails\u003e\n\u003csummary\u003eTutorials\u003c/summary\u003e\n\n- [Implementing a Metric](https://mmeval.readthedocs.io/en/latest/tutorials/custom_metric.html)\n- [Using Distributed Evaluation](https://mmeval.readthedocs.io/en/latest/tutorials/dist_evaluation.html)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eExamples\u003c/summary\u003e\n\n- [MMCls](https://mmeval.readthedocs.io/en/latest/examples/mmclassification.html)\n- [TensorPack](https://mmeval.readthedocs.io/en/latest/examples/tensorpack.html)\n- [PaddleSeg](https://mmeval.readthedocs.io/en/latest/examples/paddleseg.html)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eDesign\u003c/summary\u003e\n\n- [BaseMetric Design](https://mmeval.readthedocs.io/en/latest/design/base_metric.html)\n- [Distributed Communication Backend](https://mmeval.readthedocs.io/en/latest/design/distributed_backend.html)\n- [Multiple Dispatch](https://mmeval.readthedocs.io/en/latest/design/multiple_dispatch.html)\n\n\u003c/details\u003e\n\n## In the works\n\n- Continue to add more metrics and expand more tasks (e.g. NLP, audio).\n- Support more ML frameworks and explore multiple ML framework support paradigms.\n\n## Contributing\n\nWe appreciate all contributions to improve MMEval. Please refer to [CONTRIBUTING.md](CONTRIBUTING.md) for the contributing guideline.\n\n## License\n\nThis project is released under the [Apache 2.0 license](LICENSE).\n\n## Projects in OpenMMLab\n\n- [MMEngine](https://github.com/open-mmlab/mmengine): OpenMMLab foundational library for training deep learning models.\n- [MIM](https://github.com/open-mmlab/mim): MIM installs OpenMMLab packages.\n- [MMCV](https://github.com/open-mmlab/mmcv): OpenMMLab foundational library for computer vision.\n- [MMClassification](https://github.com/open-mmlab/mmclassification): OpenMMLab image classification toolbox and benchmark.\n- [MMDetection](https://github.com/open-mmlab/mmdetection): OpenMMLab detection toolbox and benchmark.\n- [MMDetection3D](https://github.com/open-mmlab/mmdetection3d): OpenMMLab's next-generation platform for general 3D object detection.\n- [MMRotate](https://github.com/open-mmlab/mmrotate): OpenMMLab rotated object detection toolbox and benchmark.\n- [MMYOLO](https://github.com/open-mmlab/mmyolo): OpenMMLab YOLO series toolbox and benchmark.\n- [MMSegmentation](https://github.com/open-mmlab/mmsegmentation): OpenMMLab semantic segmentation toolbox and benchmark.\n- [MMOCR](https://github.com/open-mmlab/mmocr): OpenMMLab text detection, recognition, and understanding toolbox.\n- [MMPose](https://github.com/open-mmlab/mmpose): OpenMMLab pose estimation toolbox and benchmark.\n- [MMHuman3D](https://github.com/open-mmlab/mmhuman3d): OpenMMLab 3D human parametric model toolbox and benchmark.\n- [MMSelfSup](https://github.com/open-mmlab/mmselfsup): OpenMMLab self-supervised learning toolbox and benchmark.\n- [MMRazor](https://github.com/open-mmlab/mmrazor): OpenMMLab model compression toolbox and benchmark.\n- [MMFewShot](https://github.com/open-mmlab/mmfewshot): OpenMMLab fewshot learning toolbox and benchmark.\n- [MMAction2](https://github.com/open-mmlab/mmaction2): OpenMMLab's next-generation action understanding toolbox and benchmark.\n- [MMTracking](https://github.com/open-mmlab/mmtracking): OpenMMLab video perception toolbox and benchmark.\n- [MMFlow](https://github.com/open-mmlab/mmflow): OpenMMLab optical flow toolbox and benchmark.\n- [MMEditing](https://github.com/open-mmlab/mmediting): OpenMMLab image and video editing toolbox.\n- [MMGeneration](https://github.com/open-mmlab/mmgeneration): OpenMMLab image and video generative models toolbox.\n- [MMDeploy](https://github.com/open-mmlab/mmdeploy): OpenMMLab model deployment framework.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-mmlab%2Fmmeval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopen-mmlab%2Fmmeval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-mmlab%2Fmmeval/lists"}