{"id":13738339,"url":"https://github.com/toshas/torch-fidelity","last_synced_at":"2025-05-14T19:07:09.089Z","repository":{"id":40332899,"uuid":"258172048","full_name":"toshas/torch-fidelity","owner":"toshas","description":"High-fidelity performance metrics for generative models in PyTorch","archived":false,"fork":false,"pushed_at":"2024-01-25T18:58:06.000Z","size":2349,"stargazers_count":1056,"open_issues_count":3,"forks_count":74,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-06T18:09:30.246Z","etag":null,"topics":["evaluation","frechet-inception-distance","gan","generative-model","inception-score","kernel-inception-distance","metrics","perceptual-path-length","precision","pytorch","reproducibility","reproducible-research"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/toshas.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-04-23T10:44:20.000Z","updated_at":"2025-04-04T15:26:41.000Z","dependencies_parsed_at":"2024-01-20T23:27:29.751Z","dependency_job_id":"2d71c715-2e51-4ee2-b346-5d0e3b9d5838","html_url":"https://github.com/toshas/torch-fidelity","commit_stats":{"total_commits":187,"total_committers":11,"mean_commits":17.0,"dds":0.5026737967914439,"last_synced_commit":"a61422fb9bafcf94af51440ddad4bb11b091f7aa"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toshas%2Ftorch-fidelity","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toshas%2Ftorch-fidelity/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toshas%2Ftorch-fidelity/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toshas%2Ftorch-fidelity/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/toshas","download_url":"https://codeload.github.com/toshas/torch-fidelity/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248775765,"owners_count":21159662,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation","frechet-inception-distance","gan","generative-model","inception-score","kernel-inception-distance","metrics","perceptual-path-length","precision","pytorch","reproducibility","reproducible-research"],"created_at":"2024-08-03T03:02:19.251Z","updated_at":"2025-04-13T20:20:12.153Z","avatar_url":"https://github.com/toshas.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"![High-fidelity performance metrics for generative models in PyTorch](doc/img/header.png)\n\n[![Documentation Status](https://readthedocs.org/projects/torch-fidelity/badge/?version=latest)](https://torch-fidelity.readthedocs.io/en/latest/?badge=latest)\n[![TestStatus](https://circleci.com/gh/toshas/torch-fidelity.svg?style=shield)](https://circleci.com/gh/toshas/torch-fidelity)\n[![PyPiVersion](https://badge.fury.io/py/torch-fidelity.svg)](https://pypi.org/project/torch-fidelity/)\n[![PyPiDownloads](https://pepy.tech/badge/torch-fidelity)](https://pepy.tech/project/torch-fidelity)\n[![Twitter Follow](https://img.shields.io/twitter/follow/AntonObukhov1?style=social\u0026label=Subscribe!)](https://twitter.com/antonobukhov1)\n\nThis repository provides **precise**, **efficient**, and **extensible** implementations of the popular metrics for \ngenerative model evaluation, including:\n- Inception Score ([ISC](https://arxiv.org/pdf/1606.03498.pdf))\n- Fréchet Inception Distance ([FID](https://arxiv.org/pdf/1706.08500.pdf))\n- Kernel Inception Distance ([KID](https://arxiv.org/pdf/1801.01401.pdf))\n- Precision and Recall ([PRC](https://arxiv.org/pdf/1904.06991.pdf))\n- Perceptual Path Length ([PPL](https://arxiv.org/pdf/1812.04948.pdf))\n\n**Numerical Precision**: Unlike many other reimplementations, the values produced by torch-fidelity match reference \nimplementations up to floating point's machine precision. This allows using torch-fidelity for reporting metrics in papers instead of \nscattered and slow reference implementations. [Read more about numerical precision](https://torch-fidelity.readthedocs.io/en/latest/precision.html) \n\n**Efficiency**: Feature sharing between different metrics saves recomputation time, and an additional caching \nlevel avoids recomputing features and statistics whenever possible. High efficiency allows using torch-fidelity in the \ntraining loop, for example at the end of every epoch. [Read more about efficiency](https://torch-fidelity.readthedocs.io/en/latest/miscellaneous.html)\n\n**Extensibility**: Going beyond 2D image generation is easy due to high modularity and abstraction of the metrics from\ninput data, models, and feature extractors. For example, one can swap out InceptionV3 feature extractor for a one\naccepting 3D scan volumes, such as used in MRI. [Read more about extensibility](https://torch-fidelity.readthedocs.io/en/latest/extensibility.html)\n\n**TLDR; fast and reliable GAN evaluation in PyTorch**\n\n## Installation\n\n```shell script\npip install torch-fidelity\n```\n\nSee also: [Installing the latest GitHub code](https://torch-fidelity.readthedocs.io/en/latest/installation.html#nightly-version) \n\n## Usage Examples with Command Line\n\nBelow are three examples of using torch-fidelity to evaluate metrics from the command line. See more examples in the \ndocumentation.\n\n### Simple \n\nInception Score of CIFAR-10 training split:\n```shell script\n\u003e fidelity --gpu 0 --isc --input1 cifar10-train\n\ninception_score_mean: 11.23678\ninception_score_std: 0.09514061\n```\n\n### Medium \n\nInception Score of a directory of images stored in `~/images/`:\n```shell script\n\u003e fidelity --gpu 0 --isc --input1 ~/images/\n```\n\n### Pro\n\nEfficient computation of ISC and PPL for `input1`, and FID, KID, PRC between a generative model stored in `~/generator.onnx` and CIFAR-10 training split:\n```shell script\n\u003e fidelity \\\n  --gpu 0 \\\n  --isc \\\n  --fid \\\n  --kid \\\n  --ppl \\\n  --prc \\\n  --input1 ~/generator.onnx \\ \n  --input1-model-z-type normal \\\n  --input1-model-z-size 128 \\\n  --input1-model-num-samples 50000 \\ \n  --input2 cifar10-train \n```\n\nSee also: [Other usage examples](https://torch-fidelity.readthedocs.io/en/latest/usage_cmd.html)\n\n## Quick Start with Python API\n\nWhen it comes to tracking the performance of generative models as they train, evaluating metrics after every epoch \nbecomes prohibitively expensive due to long computation times. \n`torch_fidelity` tackles this problem by making full use \nof caching to avoid recomputing common features and per-metric statistics whenever possible. \nComputing all metrics for 50000 32x32 generated images and `cifar10-train` takes only 2 min 26 seconds on NVIDIA P100 \nGPU, compared to \u003e10 min if using original codebases. \nThus, computing metrics 20 times over the whole training cycle makes overall training time just one hour longer.\n\nIn the following example, assume unconditional image generation setting with CIFAR-10, and the generative model \n`generator`, which takes a 128-dimensional standard normal noise vector.\n\nFirst, import the module:\n\n```python\nimport torch_fidelity\n```\n\nAdd the following lines at the end of epoch evaluation:\n```python\nwrapped_generator = torch_fidelity.GenerativeModelModuleWrapper(generator, 128, 'normal', 0)\n\nmetrics_dict = torch_fidelity.calculate_metrics(\n    input1=wrapped_generator, \n    input2='cifar10-train', \n    cuda=True, \n    isc=True, \n    fid=True, \n    kid=True, \n    prc=True, \n    verbose=False,\n)\n```\n\nThe resulting dictionary with computed metrics can logged directly to tensorboard, wandb, or console: \n\n```python\nprint(metrics_dict)\n```\n\nOutput:\n\n```python\n{\n    'inception_score_mean': 11.23678, \n    'inception_score_std': 0.09514061, \n    'frechet_inception_distance': 18.12198,\n    'kernel_inception_distance_mean': 0.01369556, \n    'kernel_inception_distance_std': 0.001310059\n    'precision': 0.51369556, \n    'recall': 0.501310059\n}\n```\n\nSee also: [Full API reference](https://torch-fidelity.readthedocs.io/en/latest/api.html)\n\n### Example of Integration with the Training Loop\n\nRefer to [sngan_cifar10.py](examples/sngan_cifar10.py) for a complete training example.\n\nEvolution of fixed generator latents in the example:\n\n![Evolution of fixed generator latents](doc/img/sngan-cifar10.gif)\n\nA generator checkpoint resulting from training the example can be downloaded \n[here](https://github.com/toshas/torch-fidelity/releases/download/v0.2.0/example-sngan-cifar10-generator.pth). \n\n## Troubleshooting\n\n###   WARNING: The script fidelity is installed in '\u003cSOMEPATH\u003e' which is not on PATH.\n\nSuggests that the standalone `fidelity` tool will not be available unless the above path is added to the PATH \nenvironment variable. If modifying it is undesirable, the tool can still be called by its full path: `\u003cSOMEPATH\u003e/fidelity`.\n\n## Citation\n\nCitation is recommended to reinforce the evaluation protocol in works relying on torch-fidelity. \nTo ensure reproducibility when citing this repository, use the following BibTeX:\n\n```\n@misc{obukhov2020torchfidelity,\n  author={Anton Obukhov and Maximilian Seitzer and Po-Wei Wu and Semen Zhydenko and Jonathan Kyl and Elvis Yu-Jing Lin},\n  year=2020,\n  title={High-fidelity performance metrics for generative models in PyTorch},\n  url={https://github.com/toshas/torch-fidelity},\n  publisher={Zenodo},\n  version={v0.3.0},\n  doi={10.5281/zenodo.4957738},\n  note={Version: 0.3.0, DOI: 10.5281/zenodo.4957738}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoshas%2Ftorch-fidelity","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftoshas%2Ftorch-fidelity","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoshas%2Ftorch-fidelity/lists"}