Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/open-mmlab/mmeval
A unified evaluation library for multiple machine learning libraries
https://github.com/open-mmlab/mmeval
machine-learning metrics python pytorch tensorflow
Last synced: 12 days ago
JSON representation
A unified evaluation library for multiple machine learning libraries
- Host: GitHub
- URL: https://github.com/open-mmlab/mmeval
- Owner: open-mmlab
- License: apache-2.0
- Created: 2022-07-22T03:56:57.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-03-29T03:56:22.000Z (3 months ago)
- Last Synced: 2024-06-14T02:43:16.861Z (14 days ago)
- Topics: machine-learning, metrics, python, pytorch, tensorflow
- Language: Python
- Homepage: https://mmeval.readthedocs.io/en/latest/
- Size: 2.03 MB
- Stars: 248
- Watchers: 9
- Forks: 47
- Open Issues: 18
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
- Code of conduct: .github/CODE_OF_CONDUCT.md
- Codeowners: CODEOWNERS
Lists
- awesome-stars - open-mmlab/mmeval - `★248` A unified evaluation library for multiple machine learning libraries (Python)
- awesome-yolo-object-detection - open-mmlab/mmeval - mmlab/mmeval?style=social"/> : MMEval is a machine learning evaluation library that supports efficient and accurate distributed evaluation on a variety of machine learning frameworks. (Applications)
- awesome-stars - open-mmlab/mmeval - A unified evaluation library for multiple machine learning libraries (Python)
- awesome-stars - open-mmlab/mmeval - `★238` A unified evaluation library for multiple machine learning libraries (Python)
README
[](https://pypi.org/project/mmeval/)
[](https://pypi.org/project/mmeval)
[](https://github.com/open-mmlab/mmeval/blob/main/LICENSE)
[](https://github.com/open-mmlab/mmeval/issues)
[](https://github.com/open-mmlab/mmeval/issues)
[🤔Reporting Issues](https://github.com/open-mmlab/mmeval/issues/new/choose)
English | [简体中文](README_zh-CN.md)
## Introduction
MMEval is a machine learning evaluation library that supports efficient and accurate distributed evaluation on a variety of machine learning frameworks.
Major features:
- Comprehensive metrics for various computer vision tasks (NLP will be covered soon!)
- Efficient and accurate distributed evaluation, backed by multiple distributed communication backends
- Support multiple machine learning frameworks via dynamic input dispatching mechanism
Supported distributed communication backends
| MPI4Py | torch.distributed | Horovod | paddle.distributed | oneflow.comm |
| :------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: |
| [MPI4PyDist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.MPI4PyDist.html#mmeval.core.dist_backends.MPI4PyDist) | [TorchCPUDist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.TorchCPUDist.html#mmeval.core.dist_backends.TorchCPUDist)
[TorchCUDADist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.TorchCUDADist.html#mmeval.core.dist_backends.TorchCUDADist) | [TFHorovodDist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.TFHorovodDist.html#mmeval.core.dist_backends.TFHorovodDist) | [PaddleDist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.PaddleDist.html#mmeval.core.dist_backends.PaddleDist) | [OneFlowDist](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.core.dist_backends.OneFlowDist.html#mmeval.core.dist_backends.OneFlowDist) |
Supported metrics and ML frameworks
`NOTE: MMEval tested with PyTorch 1.6+, TensorFlow 2.4+, Paddle 2.2+ and OneFlow 0.8+.`
| Metric | numpy.ndarray | torch.Tensor | tensorflow.Tensor | paddle.Tensor | oneflow.Tensor |
| :--------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------: | :----------: | :---------------: | :-----------: | :------------: |
| [Accuracy](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.Accuracy.html#mmeval.metrics.Accuracy) | ✔ | ✔ | ✔ | ✔ | ✔ |
| [SingleLabelMetric](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.SingleLabelMetric.html#mmeval.metrics.SingleLabelMetric) | ✔ | ✔ | | | ✔ |
| [MultiLabelMetric](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.MultiLabelMetric.html#mmeval.metrics.MultiLabelMetric) | ✔ | ✔ | | | ✔ |
| [AveragePrecision](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.AveragePrecision.html#mmeval.metrics.AveragePrecision) | ✔ | ✔ | | | ✔ |
| [MeanIoU](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.MeanIoU.html#mmeval.metrics.MeanIoU) | ✔ | ✔ | ✔ | ✔ | ✔ |
| [VOCMeanAP](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.VOCMeanAP.html#mmeval.metrics.VOCMeanAP) | ✔ | | | | |
| [OIDMeanAP](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.OIDMeanAP.html#mmeval.metrics.OIDMeanAP) | ✔ | | | | |
| [COCODetection](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.COCODetection.html#mmeval.metrics.COCODetection) | ✔ | | | | |
| [ProposalRecall](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.ProposalRecall.html#mmeval.metrics.ProposalRecall) | ✔ | | | | |
| [F1Score](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.F1Score.html#mmeval.metrics.F1Score) | ✔ | ✔ | | | ✔ |
| [HmeanIoU](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.HmeanIoU.html#mmeval.metrics.HmeanIoU) | ✔ | | | | |
| [PCKAccuracy](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.PCKAccuracy.html#mmeval.metrics.PCKAccuracy) | ✔ | | | | |
| [MpiiPCKAccuracy](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.MpiiPCKAccuracy.html#mmeval.metrics.MpiiPCKAccuracy) | ✔ | | | | |
| [JhmdbPCKAccuracy](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.JhmdbPCKAccuracy.html#mmeval.metrics.JhmdbPCKAccuracy) | ✔ | | | | |
| [EndPointError](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.EndPointError.html#mmeval.metrics.EndPointError) | ✔ | ✔ | | | ✔ |
| [AVAMeanAP](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.AVAMeanAP.html#mmeval.metrics.AVAMeanAP) | ✔ | | | | |
| [StructuralSimilarity](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.StructuralSimilarity.html#mmeval.metrics.StructuralSimilarity) | ✔ | | | | |
| [SignalNoiseRatio](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.SignalNoiseRatio.html#mmeval.metrics.SignalNoiseRatio) | ✔ | | | | |
| [PeakSignalNoiseRatio](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.PeakSignalNoiseRatio.html#mmeval.metrics.PeakSignalNoiseRatio) | ✔ | | | | |
| [MeanAbsoluteError](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.MeanAbsoluteError.html#mmeval.metrics.MeanAbsoluteError) | ✔ | | | | |
| [MeanSquaredError](https://mmeval.readthedocs.io/en/latest/api/generated/mmeval.metrics.MeanSquaredError.html#mmeval.metrics.MeanSquaredError) | ✔ | | | | |
## Installation
`MMEval` requires Python 3.6+ and can be installed via pip.
```bash
pip install mmeval
```
To install the dependencies required for all the metrics provided in `MMEval`, you can install them with the following command.
```bash
pip install 'mmeval[all]'
```
## Get Started
There are two ways to use `MMEval`'s metrics, using `Accuracy` as an example:
```python
from mmeval import Accuracy
import numpy as np
accuracy = Accuracy()
```
The first way is to directly call the instantiated `Accuracy` object to calculate the metric.
```python
labels = np.asarray([0, 1, 2, 3])
preds = np.asarray([0, 2, 1, 3])
accuracy(preds, labels)
# {'top1': 0.5}
```
The second way is to calculate the metric after accumulating data from multiple batches.
```python
for i in range(10):
labels = np.random.randint(0, 4, size=(100, ))
predicts = np.random.randint(0, 4, size=(100, ))
accuracy.add(predicts, labels)
accuracy.compute()
# {'top1': ...}
```
## Learn More
Tutorials
- [Implementing a Metric](https://mmeval.readthedocs.io/en/latest/tutorials/custom_metric.html)
- [Using Distributed Evaluation](https://mmeval.readthedocs.io/en/latest/tutorials/dist_evaluation.html)
Examples
- [MMCls](https://mmeval.readthedocs.io/en/latest/examples/mmclassification.html)
- [TensorPack](https://mmeval.readthedocs.io/en/latest/examples/tensorpack.html)
- [PaddleSeg](https://mmeval.readthedocs.io/en/latest/examples/paddleseg.html)
Design
- [BaseMetric Design](https://mmeval.readthedocs.io/en/latest/design/base_metric.html)
- [Distributed Communication Backend](https://mmeval.readthedocs.io/en/latest/design/distributed_backend.html)
- [Multiple Dispatch](https://mmeval.readthedocs.io/en/latest/design/multiple_dispatch.html)
## In the works
- Continue to add more metrics and expand more tasks (e.g. NLP, audio).
- Support more ML frameworks and explore multiple ML framework support paradigms.
## Contributing
We appreciate all contributions to improve MMEval. Please refer to [CONTRIBUTING.md](CONTRIBUTING.md) for the contributing guideline.
## License
This project is released under the [Apache 2.0 license](LICENSE).
## Projects in OpenMMLab
- [MMEngine](https://github.com/open-mmlab/mmengine): OpenMMLab foundational library for training deep learning models.
- [MIM](https://github.com/open-mmlab/mim): MIM installs OpenMMLab packages.
- [MMCV](https://github.com/open-mmlab/mmcv): OpenMMLab foundational library for computer vision.
- [MMClassification](https://github.com/open-mmlab/mmclassification): OpenMMLab image classification toolbox and benchmark.
- [MMDetection](https://github.com/open-mmlab/mmdetection): OpenMMLab detection toolbox and benchmark.
- [MMDetection3D](https://github.com/open-mmlab/mmdetection3d): OpenMMLab's next-generation platform for general 3D object detection.
- [MMRotate](https://github.com/open-mmlab/mmrotate): OpenMMLab rotated object detection toolbox and benchmark.
- [MMYOLO](https://github.com/open-mmlab/mmyolo): OpenMMLab YOLO series toolbox and benchmark.
- [MMSegmentation](https://github.com/open-mmlab/mmsegmentation): OpenMMLab semantic segmentation toolbox and benchmark.
- [MMOCR](https://github.com/open-mmlab/mmocr): OpenMMLab text detection, recognition, and understanding toolbox.
- [MMPose](https://github.com/open-mmlab/mmpose): OpenMMLab pose estimation toolbox and benchmark.
- [MMHuman3D](https://github.com/open-mmlab/mmhuman3d): OpenMMLab 3D human parametric model toolbox and benchmark.
- [MMSelfSup](https://github.com/open-mmlab/mmselfsup): OpenMMLab self-supervised learning toolbox and benchmark.
- [MMRazor](https://github.com/open-mmlab/mmrazor): OpenMMLab model compression toolbox and benchmark.
- [MMFewShot](https://github.com/open-mmlab/mmfewshot): OpenMMLab fewshot learning toolbox and benchmark.
- [MMAction2](https://github.com/open-mmlab/mmaction2): OpenMMLab's next-generation action understanding toolbox and benchmark.
- [MMTracking](https://github.com/open-mmlab/mmtracking): OpenMMLab video perception toolbox and benchmark.
- [MMFlow](https://github.com/open-mmlab/mmflow): OpenMMLab optical flow toolbox and benchmark.
- [MMEditing](https://github.com/open-mmlab/mmediting): OpenMMLab image and video editing toolbox.
- [MMGeneration](https://github.com/open-mmlab/mmgeneration): OpenMMLab image and video generative models toolbox.
- [MMDeploy](https://github.com/open-mmlab/mmdeploy): OpenMMLab model deployment framework.