{"id":13688891,"url":"https://github.com/pytorch/torcheval","last_synced_at":"2025-05-14T22:09:36.855Z","repository":{"id":56775882,"uuid":"495989743","full_name":"pytorch/torcheval","owner":"pytorch","description":"A library that contains a rich collection of performant PyTorch model metrics, a simple interface to create new metrics, a toolkit to facilitate metric computation in distributed training and tools for PyTorch model evaluations.","archived":false,"fork":false,"pushed_at":"2025-01-17T14:51:10.000Z","size":3789,"stargazers_count":228,"open_issues_count":29,"forks_count":54,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-05-08T00:08:25.910Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://pytorch.org/torcheval","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pytorch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-24T21:12:51.000Z","updated_at":"2025-05-04T06:25:59.000Z","dependencies_parsed_at":"2023-09-24T07:17:18.169Z","dependency_job_id":"70d4844e-2df6-496d-9789-81a1f12e0d18","html_url":"https://github.com/pytorch/torcheval","commit_stats":{"total_commits":258,"total_committers":58,"mean_commits":4.448275862068965,"dds":0.7829457364341086,"last_synced_commit":"2c7dfb3768335ad7438bfb9bbe5c050e3f0780dc"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorcheval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorcheval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorcheval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytorch%2Ftorcheval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pytorch","download_url":"https://codeload.github.com/pytorch/torcheval/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254235701,"owners_count":22036964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T15:01:26.669Z","updated_at":"2025-05-14T22:09:31.839Z","avatar_url":"https://github.com/pytorch.png","language":"Python","funding_links":[],"categories":["Python","General"],"sub_categories":[],"readme":"# TorchEval\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://github.com/pytorch/torcheval/actions?query=branch%3Amain\"\u003e\u003cimg src=\"https://img.shields.io/github/actions/workflow/status/pytorch/torcheval/.github/workflows/unit_test.yaml?branch=main\" alt=\"build status\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/torcheval\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/torcheval\" alt=\"pypi version\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/torcheval-nightly\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/torcheval-nightly?label=nightly\" alt=\"pypi nightly version\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/pytorch/torcheval/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/pypi/l/torcheval\" alt=\"bsd license\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\u003ca href=\"https://pytorch.github.io/torcheval\"\u003e\u003cimg src=\"https://img.shields.io/badge/docs-main-brightgreen\" alt=\"docs\"\u003e\u003c/a\u003e\n\u003cp\u003e\n\n**This library is currently in Alpha and currently does not have a stable release. The API may change and may not be backward compatible. If you have suggestions for improvements, please open a GitHub issue. We'd love to hear your feedback.**\n\nA library that contains a rich collection of performant PyTorch model metrics, a simple interface to create new metrics, a toolkit to facilitate metric computation in distributed training and tools for PyTorch model evaluations.\n\n## Installing TorchEval\nRequires Python \u003e= 3.8 and PyTorch \u003e= 1.11\n\nFrom pip:\n\n```bash\npip install torcheval\n```\n\nFor nighly build version\n```bash\npip install --pre torcheval-nightly\n```\n\nFrom source:\n\n```bash\ngit clone https://github.com/pytorch/torcheval\ncd torcheval\npip install -r requirements.txt\npython setup.py install\n```\n\n## Quick Start\n\nTake a look at the [quickstart notebook](https://github.com/pytorch/torcheval/blob/main/examples/Introducing_TorchEval.ipynb), or fork it on [Colab](https://colab.research.google.com/github/pytorch/torcheval/blob/main/examples/Introducing_TorchEval.ipynb).\n\nThere are more examples in the [examples](https://github.com/pytorch/torcheval/blob/main/examples) directory:\n\n```bash\ncd torcheval\npython examples/simple_example.py\n```\n\n## Documentation\n\nDocumentation can be found at at [pytorch.org/torcheval](https://pytorch.org/torcheval)\n\n## Using TorchEval\n\nTorchEval can be run on CPU, GPU, and in a multi-process or multi-GPU setting. Metrics are provided in two interfaces, functional and class based. The functional interfaces can be found in `torcheval.metrics.functional` and are useful when your program runs in a single process setting. To use multi-process or multi-gpu configurations, the class-based interfaces, found in `torcheval.metrics` provide a much simpler experience. The class based interfaces also allow you to defer some of the computation of the metric by calling `update()` multiple times before `compute()`. This can be advantageous even in a single process setting due to saved computation overhead.\n\n### Single Process\nFor use in a single process program, the simplest use case utilizes a functional metric. We simply import the metric function and feed in our outputs and targets. The example below shows a minimal PyTorch training loop that evaluates the multiclass accuracy of every fourth batch of data.\n\n#### Functional Version (immediate computation of metric)\n```python\nimport torch\nfrom torcheval.metrics.functional import multiclass_accuracy\n\nNUM_BATCHES = 16\nBATCH_SIZE = 8\nINPUT_SIZE = 10\nNUM_CLASSES = 6\neval_frequency = 4\n\nmodel = torch.nn.Sequential(torch.nn.Linear(INPUT_SIZE, NUM_CLASSES), torch.nn.ReLU())\noptim = torch.optim.Adagrad(model.parameters(), lr=0.001)\nloss_fn = torch.nn.CrossEntropyLoss()\n\nmetric_history = []\nfor batch in range(NUM_BATCHES):\n    input = torch.rand(size=(BATCH_SIZE, INPUT_SIZE))\n    target = torch.randint(size=(BATCH_SIZE,), high=NUM_CLASSES)\n    outputs = model(input)\n\n    loss = loss_fn(outputs, target)\n    optim.zero_grad()\n    loss.backward()\n    optim.step()\n\n    # metric only computed every 4 batches,\n    # data from previous three batches is lost\n    if (batch + 1) % eval_frequency == 0:\n        metric_history.append(multiclass_accuracy(outputs, target))\n```\n### Single Process with Deferred Computation\n\n#### Class Version (enables deferred computation of metric)\n```python\nimport torch\nfrom torcheval.metrics import MulticlassAccuracy\n\nNUM_BATCHES = 16\nBATCH_SIZE = 8\nINPUT_SIZE = 10\nNUM_CLASSES = 6\neval_frequency = 4\n\nmodel = torch.nn.Sequential(torch.nn.Linear(INPUT_SIZE, NUM_CLASSES), torch.nn.ReLU())\noptim = torch.optim.Adagrad(model.parameters(), lr=0.001)\nloss_fn = torch.nn.CrossEntropyLoss()\nmetric = MulticlassAccuracy()\n\nmetric_history = []\nfor batch in range(NUM_BATCHES):\n    input = torch.rand(size=(BATCH_SIZE, INPUT_SIZE))\n    target = torch.randint(size=(BATCH_SIZE,), high=NUM_CLASSES)\n    outputs = model(input)\n\n    loss = loss_fn(outputs, target)\n    optim.zero_grad()\n    loss.backward()\n    optim.step()\n\n    # metric only computed every 4 batches,\n    # data from previous three batches is included\n    metric.update(input, target)\n    if (batch + 1) % eval_frequency == 0:\n        metric_history.append(metric.compute())\n        # remove old data so that the next call\n        # to compute is only based off next 4 batches\n        metric.reset()\n```\n\n### Multi-Process or Multi-GPU\nFor usage on multiple devices a minimal example is given below. In the normal `torch.distributed` paradigm, each device is allocated its own process gets a unique numerical ID called a \"global rank\", counting up from 0.\n\n#### Class Version (enables deferred computation and multi-processing)\n```python\nimport torch\nfrom torcheval.metrics.toolkit import sync_and_compute\nfrom torcheval.metrics import MulticlassAccuracy\n\n# Using torch.distributed\nlocal_rank = int(os.environ[\"LOCAL_RANK\"]) #rank on local machine, i.e. unique ID within a machine\nglobal_rank = int(os.environ[\"RANK\"]) #rank in global pool, i.e. unique ID within the entire process group\nworld_size  = int(os.environ[\"WORLD_SIZE\"]) #total number of processes or \"ranks\" in the entire process group\n\ndevice = torch.device(\n    f\"cuda:{local_rank}\"\n    if torch.cuda.is_available() and torch.cuda.device_count() \u003e= world_size\n    else \"cpu\"\n)\n\nmetric = MulticlassAccuracy(device=device)\nnum_epochs, num_batches = 4, 8\n\nfor epoch in range(num_epochs):\n    for i in range(num_batches):\n        input = torch.randint(high=5, size=(10,), device=device)\n        target = torch.randint(high=5, size=(10,), device=device)\n\n        # Add data to metric locally\n        metric.update(input, target)\n\n        # metric.compute() will returns metric value from\n        # all seen data on the local process since last reset()\n        local_compute_result = metric.compute()\n\n        # sync_and_compute(metric) syncs metric data across all ranks and computes the metric value\n        global_compute_result = sync_and_compute(metric)\n        if global_rank == 0:\n            print(global_compute_result)\n\n    # metric.reset() clears the data on each process so that subsequent\n    # calls to compute() only act on new data\n    metric.reset()\n```\nSee the [example directory](https://github.com/pytorch/torcheval/tree/main/examples) for more examples.\n\n## Contributing\nWe welcome PRs! See the [CONTRIBUTING](CONTRIBUTING.md) file.\n\n## License\nTorchEval is BSD licensed, as found in the [LICENSE](LICENSE) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytorch%2Ftorcheval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpytorch%2Ftorcheval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytorch%2Ftorcheval/lists"}