{"id":13472678,"url":"https://github.com/Lightning-AI/torchmetrics","last_synced_at":"2025-03-26T17:31:01.125Z","repository":{"id":36955885,"uuid":"323721661","full_name":"Lightning-AI/torchmetrics","owner":"Lightning-AI","description":"Machine learning metrics for distributed, scalable PyTorch applications.","archived":false,"fork":false,"pushed_at":"2024-10-28T23:30:34.000Z","size":12130,"stargazers_count":2126,"open_issues_count":93,"forks_count":404,"subscribers_count":32,"default_branch":"master","last_synced_at":"2024-10-29T09:17:11.804Z","etag":null,"topics":["analyses","data-science","deep-learning","machine-learning","metrics","python","pytorch"],"latest_commit_sha":null,"homepage":"https://lightning.ai/docs/torchmetrics/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Lightning-AI.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-22T20:02:42.000Z","updated_at":"2024-10-29T05:03:13.000Z","dependencies_parsed_at":"2023-09-22T06:23:24.037Z","dependency_job_id":"826a35b1-ad56-4c7e-bc57-7a167f3e81fc","html_url":"https://github.com/Lightning-AI/torchmetrics","commit_stats":{"total_commits":1949,"total_committers":261,"mean_commits":7.467432950191571,"dds":0.7660338635197537,"last_synced_commit":"6377aa5b6fe2863761839e6b8b5a857ef1b8acfa"},"previous_names":["Lightning-AI/metrics","lightning-ai/metrics","pytorchlightning/metrics"],"tags_count":55,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lightning-AI%2Ftorchmetrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lightning-AI%2Ftorchmetrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lightning-AI%2Ftorchmetrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lightning-AI%2Ftorchmetrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Lightning-AI","download_url":"https://codeload.github.com/Lightning-AI/torchmetrics/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245486357,"owners_count":20623243,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analyses","data-science","deep-learning","machine-learning","metrics","python","pytorch"],"created_at":"2024-07-31T16:00:56.993Z","updated_at":"2025-03-26T17:31:01.104Z","avatar_url":"https://github.com/Lightning-AI.png","language":"Python","funding_links":[],"categories":["Python","General","1. Core Frameworks \u0026 Libraries"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"docs/source/_static/images/logo.png\" width=\"400px\"\u003e\n\n**Machine learning metrics for distributed, scalable PyTorch applications.**\n\n______________________________________________________________________\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#what-is-torchmetrics\"\u003eWhat is Torchmetrics\u003c/a\u003e •\n  \u003ca href=\"#implementing-your-own-module-metric\"\u003eImplementing a metric\u003c/a\u003e •\n  \u003ca href=\"#build-in-metrics\"\u003eBuilt-in metrics\u003c/a\u003e •\n  \u003ca href=\"https://lightning.ai/docs/torchmetrics/stable/\"\u003eDocs\u003c/a\u003e •\n  \u003ca href=\"#community\"\u003eCommunity\u003c/a\u003e •\n  \u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\n\u003c/p\u003e\n\n______________________________________________________________________\n\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/torchmetrics)](https://pypi.org/project/torchmetrics/)\n[![PyPI Status](https://badge.fury.io/py/torchmetrics.svg)](https://badge.fury.io/py/torchmetrics)\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/torchmetrics)\n](https://pepy.tech/project/torchmetrics)\n[![Conda](https://img.shields.io/conda/v/conda-forge/torchmetrics?label=conda\u0026color=success)](https://anaconda.org/conda-forge/torchmetrics)\n[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/Lightning-AI/torchmetrics/blob/master/LICENSE)\n\n[![CI testing | CPU](https://github.com/Lightning-AI/torchmetrics/actions/workflows/ci-tests.yml/badge.svg?event=push)](https://github.com/Lightning-AI/torchmetrics/actions/workflows/ci-tests.yml)\n[![Build Status](https://dev.azure.com/Lightning-AI/Metrics/_apis/build/status%2FTM.unittests?branchName=master)](https://dev.azure.com/Lightning-AI/Metrics/_build/latest?definitionId=54\u0026branchName=master)\n[![codecov](https://codecov.io/gh/Lightning-AI/torchmetrics/branch/master/graph/badge.svg?token=NER6LPI3HS)](https://codecov.io/gh/Lightning-AI/torchmetrics)\n[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/Lightning-AI/torchmetrics/master.svg)](https://results.pre-commit.ci/latest/github/Lightning-AI/torchmetrics/master)\n\n[![Documentation Status](https://readthedocs.org/projects/torchmetrics/badge/?version=latest)](https://torchmetrics.readthedocs.io/en/latest/?badge=latest)\n[![Discord](https://img.shields.io/discord/1077906959069626439?style=plastic)](https://discord.gg/VptPCZkGNa)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5844769.svg)](https://doi.org/10.5281/zenodo.5844769)\n[![JOSS status](https://joss.theoj.org/papers/561d9bb59b400158bc8204e2639dca43/status.svg)](https://joss.theoj.org/papers/561d9bb59b400158bc8204e2639dca43)\n\n______________________________________________________________________\n\n\u003c/div\u003e\n\n## Installation\n\nSimple installation from PyPI\n\n```bash\npip install torchmetrics\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eOther installations\u003c/summary\u003e\n\nInstall using conda\n\n```bash\nconda install -c conda-forge torchmetrics\n```\n\nPip from source\n\n```bash\n# with git\npip install git+https://github.com/Lightning-AI/torchmetrics.git@release/stable\n```\n\nPip from archive\n\n```bash\npip install https://github.com/Lightning-AI/torchmetrics/archive/refs/heads/release/stable.zip\n```\n\nExtra dependencies for specialized metrics:\n\n```bash\npip install torchmetrics[audio]\npip install torchmetrics[image]\npip install torchmetrics[text]\npip install torchmetrics[all]  # install all of the above\n```\n\nInstall latest developer version\n\n```bash\npip install https://github.com/Lightning-AI/torchmetrics/archive/master.zip\n```\n\n\u003c/details\u003e\n\n______________________________________________________________________\n\n## What is TorchMetrics\n\nTorchMetrics is a collection of 100+ PyTorch metrics implementations and an easy-to-use API to create custom metrics. It offers:\n\n- A standardized interface to increase reproducibility\n- Reduces boilerplate\n- Automatic accumulation over batches\n- Metrics optimized for distributed-training\n- Automatic synchronization between multiple devices\n\nYou can use TorchMetrics with any PyTorch model or with [PyTorch Lightning](https://lightning.ai/docs/pytorch/stable/) to enjoy additional features such as:\n\n- Module metrics are automatically placed on the correct device.\n- Native support for logging metrics in Lightning to reduce even more boilerplate.\n\n## Using TorchMetrics\n\n### Module metrics\n\nThe [module-based metrics](https://lightning.ai/docs/torchmetrics/stable/references/metric.html) contain internal metric states (similar to the parameters of the PyTorch module) that automate accumulation and synchronization across devices!\n\n- Automatic accumulation over multiple batches\n- Automatic synchronization between multiple devices\n- Metric arithmetic\n\n**This can be run on CPU, single GPU or multi-GPUs!**\n\nFor the single GPU/CPU case:\n\n```python\nimport torch\n\n# import our library\nimport torchmetrics\n\n# initialize metric\nmetric = torchmetrics.classification.Accuracy(task=\"multiclass\", num_classes=5)\n\n# move the metric to device you want computations to take place\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmetric.to(device)\n\nn_batches = 10\nfor i in range(n_batches):\n    # simulate a classification problem\n    preds = torch.randn(10, 5).softmax(dim=-1).to(device)\n    target = torch.randint(5, (10,)).to(device)\n\n    # metric on current batch\n    acc = metric(preds, target)\n    print(f\"Accuracy on batch {i}: {acc}\")\n\n# metric on all batches using custom accumulation\nacc = metric.compute()\nprint(f\"Accuracy on all data: {acc}\")\n```\n\nModule metric usage remains the same when using multiple GPUs or multiple nodes.\n\n\u003cdetails\u003e\n  \u003csummary\u003eExample using DDP\u003c/summary\u003e\n\n\u003c!--phmdoctest-mark.skip--\u003e\n\n```python\nimport os\nimport torch\nimport torch.distributed as dist\nimport torch.multiprocessing as mp\nfrom torch import nn\nfrom torch.nn.parallel import DistributedDataParallel as DDP\nimport torchmetrics\n\n\ndef metric_ddp(rank, world_size):\n    os.environ[\"MASTER_ADDR\"] = \"localhost\"\n    os.environ[\"MASTER_PORT\"] = \"12355\"\n\n    # create default process group\n    dist.init_process_group(\"gloo\", rank=rank, world_size=world_size)\n\n    # initialize model\n    metric = torchmetrics.classification.Accuracy(task=\"multiclass\", num_classes=5)\n\n    # define a model and append your metric to it\n    # this allows metric states to be placed on correct accelerators when\n    # .to(device) is called on the model\n    model = nn.Linear(10, 10)\n    model.metric = metric\n    model = model.to(rank)\n\n    # initialize DDP\n    model = DDP(model, device_ids=[rank])\n\n    n_epochs = 5\n    # this shows iteration over multiple training epochs\n    for n in range(n_epochs):\n        # this will be replaced by a DataLoader with a DistributedSampler\n        n_batches = 10\n        for i in range(n_batches):\n            # simulate a classification problem\n            preds = torch.randn(10, 5).softmax(dim=-1)\n            target = torch.randint(5, (10,))\n\n            # metric on current batch\n            acc = metric(preds, target)\n            if rank == 0:  # print only for rank 0\n                print(f\"Accuracy on batch {i}: {acc}\")\n\n        # metric on all batches and all accelerators using custom accumulation\n        # accuracy is same across both accelerators\n        acc = metric.compute()\n        print(f\"Accuracy on all data: {acc}, accelerator rank: {rank}\")\n\n        # Resetting internal state such that metric ready for new data\n        metric.reset()\n\n    # cleanup\n    dist.destroy_process_group()\n\n\nif __name__ == \"__main__\":\n    world_size = 2  # number of gpus to parallelize over\n    mp.spawn(metric_ddp, args=(world_size,), nprocs=world_size, join=True)\n```\n\n\u003c/details\u003e\n\n### Implementing your own Module metric\n\nImplementing your own metric is as easy as subclassing an [`torch.nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html). Simply, subclass `torchmetrics.Metric`\nand just implement the `update` and `compute` methods:\n\n```python\nimport torch\nfrom torchmetrics import Metric\n\n\nclass MyAccuracy(Metric):\n    def __init__(self):\n        # remember to call super\n        super().__init__()\n        # call `self.add_state`for every internal state that is needed for the metrics computations\n        # dist_reduce_fx indicates the function that should be used to reduce\n        # state from multiple processes\n        self.add_state(\"correct\", default=torch.tensor(0), dist_reduce_fx=\"sum\")\n        self.add_state(\"total\", default=torch.tensor(0), dist_reduce_fx=\"sum\")\n\n    def update(self, preds: torch.Tensor, target: torch.Tensor) -\u003e None:\n        # extract predicted class index for computing accuracy\n        preds = preds.argmax(dim=-1)\n        assert preds.shape == target.shape\n        # update metric states\n        self.correct += torch.sum(preds == target)\n        self.total += target.numel()\n\n    def compute(self) -\u003e torch.Tensor:\n        # compute final result\n        return self.correct.float() / self.total\n\n\nmy_metric = MyAccuracy()\npreds = torch.randn(10, 5).softmax(dim=-1)\ntarget = torch.randint(5, (10,))\n\nprint(my_metric(preds, target))\n```\n\n### Functional metrics\n\nSimilar to [`torch.nn`](https://pytorch.org/docs/stable/nn.html), most metrics have both a [module-based](https://lightning.ai/docs/torchmetrics/stable/references/metric.html) and functional version.\nThe functional versions are simple python functions that as input take [torch.tensors](https://pytorch.org/docs/stable/tensors.html) and return the corresponding metric as a [torch.tensor](https://pytorch.org/docs/stable/tensors.html).\n\n```python\nimport torch\n\n# import our library\nimport torchmetrics\n\n# simulate a classification problem\npreds = torch.randn(10, 5).softmax(dim=-1)\ntarget = torch.randint(5, (10,))\n\nacc = torchmetrics.functional.classification.multiclass_accuracy(\n    preds, target, num_classes=5\n)\n```\n\n### Covered domains and example metrics\n\nIn total TorchMetrics contains [100+ metrics](https://lightning.ai/docs/torchmetrics/stable/all-metrics.html), which\ncovers the following domains:\n\n- Audio\n- Classification\n- Detection\n- Information Retrieval\n- Image\n- Multimodal (Image-Text)\n- Nominal\n- Regression\n- Segmentation\n- Text\n\nEach domain may require some additional dependencies which can be installed with `pip install torchmetrics[audio]`,\n`pip install torchmetrics['image']` etc.\n\n### Additional features\n\n#### Plotting\n\nVisualization of metrics can be important to help understand what is going on with your machine learning algorithms.\nTorchmetrics have built-in plotting support (install dependencies with `pip install torchmetrics[visual]`) for nearly\nall modular metrics through the `.plot` method. Simply call the method to get a simple visualization of any metric!\n\n```python\nimport torch\nfrom torchmetrics.classification import MulticlassAccuracy, MulticlassConfusionMatrix\n\nnum_classes = 3\n\n# this will generate two distributions that comes more similar as iterations increase\nw = torch.randn(num_classes)\ntarget = lambda it: torch.multinomial((it * w).softmax(dim=-1), 100, replacement=True)\npreds = lambda it: torch.multinomial((it * w).softmax(dim=-1), 100, replacement=True)\n\nacc = MulticlassAccuracy(num_classes=num_classes, average=\"micro\")\nacc_per_class = MulticlassAccuracy(num_classes=num_classes, average=None)\nconfmat = MulticlassConfusionMatrix(num_classes=num_classes)\n\n# plot single value\nfor i in range(5):\n    acc_per_class.update(preds(i), target(i))\n    confmat.update(preds(i), target(i))\nfig1, ax1 = acc_per_class.plot()\nfig2, ax2 = confmat.plot()\n\n# plot multiple values\nvalues = []\nfor i in range(10):\n    values.append(acc(preds(i), target(i)))\nfig3, ax3 = acc.plot(values)\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/source/_static/images/plot_example.png\" width=\"1000\"\u003e\n\u003c/p\u003e\n\nFor examples of plotting different metrics try running [this example file](_samples/plotting.py).\n\n## Contribute!\n\nThe lightning + TorchMetrics team is hard at work adding even more metrics.\nBut we're looking for incredible contributors like you to submit new metrics\nand improve existing ones!\n\nJoin our [Discord](https://discord.com/invite/tfXFetEZxv) to get help with becoming a contributor!\n\n## Community\n\nFor help or questions, join our huge community on [Discord](https://discord.com/invite/tfXFetEZxv)!\n\n## Citation\n\nWe’re excited to continue the strong legacy of open source software and have been inspired\nover the years by Caffe, Theano, Keras, PyTorch, torchbearer, ignite, sklearn and fast.ai.\n\nIf you want to cite this framework feel free to use GitHub's built-in citation option to generate a bibtex or APA-Style citation based on [this file](https://github.com/Lightning-AI/torchmetrics/blob/master/CITATION.cff) (but only if you loved it 😊).\n\n## License\n\nPlease observe the Apache 2.0 license that is listed in this repository.\nIn addition, the Lightning framework is Patent Pending.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLightning-AI%2Ftorchmetrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLightning-AI%2Ftorchmetrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLightning-AI%2Ftorchmetrics/lists"}