{"id":19731632,"url":"https://github.com/clovaai/generative-evaluation-prdc","last_synced_at":"2025-04-09T23:20:10.207Z","repository":{"id":55067809,"uuid":"242021861","full_name":"clovaai/generative-evaluation-prdc","owner":"clovaai","description":"Code base for the precision, recall, density, and coverage metrics for generative models. ICML 2020.","archived":false,"fork":false,"pushed_at":"2023-01-09T15:19:33.000Z","size":297,"stargazers_count":254,"open_issues_count":8,"forks_count":28,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-02T21:11:14.184Z","etag":null,"topics":["deep-learning","diversity","evaluation","evaluation-metrics","fidelity","generative-adversarial-network","generative-model","icml","icml-2020","icml2020","machine-learning","precision","recall"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clovaai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-21T00:38:27.000Z","updated_at":"2025-03-13T09:51:16.000Z","dependencies_parsed_at":"2023-02-08T12:25:34.533Z","dependency_job_id":null,"html_url":"https://github.com/clovaai/generative-evaluation-prdc","commit_stats":null,"previous_names":["clovaai/prdc"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Fgenerative-evaluation-prdc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Fgenerative-evaluation-prdc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Fgenerative-evaluation-prdc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Fgenerative-evaluation-prdc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clovaai","download_url":"https://codeload.github.com/clovaai/generative-evaluation-prdc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248126075,"owners_count":21051864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","diversity","evaluation","evaluation-metrics","fidelity","generative-adversarial-network","generative-model","icml","icml-2020","icml2020","machine-learning","precision","recall"],"created_at":"2024-11-12T00:22:14.552Z","updated_at":"2025-04-09T23:20:10.126Z","avatar_url":"https://github.com/clovaai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![PyPI version](https://badge.fury.io/py/prdc.svg)](https://badge.fury.io/py/prdc)\n[![PyPI download month](https://img.shields.io/pypi/dm/prdc.svg)](https://pypi.python.org/pypi/prdc/)\n[![PyPI license](https://img.shields.io/pypi/l/prdc.svg)](https://pypi.python.org/pypi/prdc/)\n\n## Reliable Fidelity and Diversity Metrics for Generative Models (ICML 2020)\n\n[Paper: Reliable Fidelity and Diversity Metrics for Generative Models](https://arxiv.org/abs/2002.09797)\n\nMuhammad Ferjad Naeem \u003csup\u003e1,3*\u003c/sup\u003e, Seong Joon Oh\u003csup\u003e2*\u003c/sup\u003e, Yunjey Choi\u003csup\u003e1\u003c/sup\u003e, \nYoungjung Uh\u003csup\u003e1\u003c/sup\u003e, Jaejun Yoo\u003csup\u003e1,4\u003c/sup\u003e  \n\n\u003csub\u003e**Work done at Clova AI Research**\u003c/sub\u003e\n\n\u003csub\u003e\\* Equal contribution\u003c/sub\u003e\n\u003csup\u003e1\u003c/sup\u003e \u003csub\u003eClova AI Research, NAVER Corp.\u003c/sub\u003e\n\u003csup\u003e2\u003c/sup\u003e \u003csub\u003eClova AI Research, LINE Plus Corp.\u003c/sub\u003e\n\u003csup\u003e3\u003c/sup\u003e \u003csub\u003eTechnische Universit\u0026auml;t M\u0026uuml;nchen\u003c/sub\u003e\n\u003csup\u003e4\u003c/sup\u003e \u003csub\u003eEPFL\u003c/sub\u003e\n\nDevising indicative evaluation metrics for the image generation task remains an open problem.\nThe most widely used metric for measuring the similarity between real and generated images has been the Fr\u0026eacute;chet Inception Distance (FID) score. \nBecause it does not differentiate the _fidelity_ and _diversity_ aspects of the generated images, recent papers have introduced variants of precision and recall metrics to diagnose those properties separately.\nIn this paper, we show that even the latest version of the precision and recall (Kynk\u0026auml;\u0026auml;nniemi et al., 2019) metrics are not reliable yet. For example, they fail to detect the match between two identical distributions, they are not robust against outliers, and the evaluation hyperparameters are selected arbitrarily. We propose **density and coverage** metrics that solve the above issues. We analytically and experimentally show that density and coverage provide more interpretable and reliable signals for practitioners than the existing metrics.\n\n\u003ca href=\"http://www.youtube.com/watch?feature=player_embedded\u0026v=_XwsGkryVpk\n\" target=\"_blank\"\u003e\u003cimg src=\"http://img.youtube.com/vi/_XwsGkryVpk/0.jpg\" \nalt=\"VIDEO\" width=\"700\" border=\"10\" /\u003e\u003c/a\u003e\n\n## Updates\n\n* **1 June 2020**: Paper accepted at ICML 2020.\n\n## 1. Background\n\n### Precision and recall metrics\n\nPrecision and recall are defined below:\n\n\u003ca href=\"https://www.codecogs.com/eqnedit.php?latex=\\fn_cm\u0026space;\\text{precision}:=\\frac{1}{M}\\sum_{j=1}^{M}1_{Y_j\\in\\text{manifold}(X_1,\\cdots,X_N)}\" target=\"_blank\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.latex?\\fn_cm\u0026space;\\text{precision}:=\\frac{1}{M}\\sum_{j=1}^{M}1_{Y_j\\in\\text{manifold}(X_1,\\cdots,X_N)}\" title=\"\\text{precision}:=\\frac{1}{M}\\sum_{j=1}^{M}1_{Y_j\\in\\text{manifold}(X_1,\\cdots,X_N)}\" /\u003e\u003c/a\u003e\n\n\u003ca href=\"https://www.codecogs.com/eqnedit.php?latex=\\fn_cm\u0026space;\\text{recall}:=\\frac{1}{N}\\sum_{i=1}^{N}1_{X_i\\in\\text{manifold}(Y_1,\\cdots,Y_M)}\" target=\"_blank\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.latex?\\fn_cm\u0026space;\\text{recall}:=\\frac{1}{N}\\sum_{i=1}^{N}1_{X_i\\in\\text{manifold}(Y_1,\\cdots,Y_M)}\" title=\"\\text{recall}:=\\frac{1}{N}\\sum_{i=1}^{N}1_{X_i\\in\\text{manifold}(Y_1,\\cdots,Y_M)}\" /\u003e\u003c/a\u003e\n\nwhere the manifold is the defined as\n\n\u003ca href=\"https://www.codecogs.com/eqnedit.php?latex=\\fn_cm\u0026space;\\text{manifold}(X_1,\\cdots,X_N):=\u0026space;\\bigcup_{i=1}^{N}\u0026space;B(X_i,\\text{NND}_k(X_i))\" target=\"_blank\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.latex?\\fn_cm\u0026space;\\text{manifold}(X_1,\\cdots,X_N):=\u0026space;\\bigcup_{i=1}^{N}\u0026space;B(X_i,\\text{NND}_k(X_i))\" title=\"\\text{manifold}(X_1,\\cdots,X_N):= \\bigcup_{i=1}^{N} B(X_i,\\text{NND}_k(X_i))\" /\u003e\u003c/a\u003e\n\n\u003ca href=\"https://www.codecogs.com/eqnedit.php?latex=\\inline\u0026space;\\fn_cm\u0026space;B(x,r)\" target=\"_blank\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.latex?\\inline\u0026space;\\fn_cm\u0026space;B(x,r)\" title=\"B(x,r)\" /\u003e\u003c/a\u003e \nis the ball around the point `x` with radius `r`. \n\n\u003ca href=\"https://www.codecogs.com/eqnedit.php?latex=\\inline\u0026space;\\fn_cm\u0026space;\\text{NND}_k(X_i)\" target=\"_blank\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.latex?\\inline\u0026space;\\fn_cm\u0026space;\\text{NND}_k(X_i)\" title=\"\\text{NND}_k(X_i)\" /\u003e\u003c/a\u003e\nis the distance to the kth-nearest neighbour. \n\n### Density and coverage metrics\n\nDensity and coverage are defined below:\n\n\u003ca href=\"https://www.codecogs.com/eqnedit.php?latex=\\fn_cm\u0026space;\\text{density}:=\\frac{1}{kM}\\sum_{j=1}^{M}\\sum_{i=1}^{N}1_{Y_j\\in\u0026space;B(X_i,\\text{NND}_k(X_i))}\" target=\"_blank\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.latex?\\fn_cm\u0026space;\\text{density}:=\\frac{1}{kM}\\sum_{j=1}^{M}\\sum_{i=1}^{N}1_{Y_j\\in\u0026space;B(X_i,\\text{NND}_k(X_i))}\" title=\"\\text{density}:=\\frac{1}{kM}\\sum_{j=1}^{M}\\sum_{i=1}^{N}1_{Y_j\\in B(X_i,\\text{NND}_k(X_i))}\" /\u003e\u003c/a\u003e\n\n\u003ca href=\"https://www.codecogs.com/eqnedit.php?latex=\\fn_cm\u0026space;\\text{coverage}:=\\frac{1}{N}\\sum_{i=1}^{N}1_{\\exists\\text{\u0026space;}j\\text{\u0026space;s.t.\u0026space;}\u0026space;Y_j\\in\u0026space;B(X_i,\\text{NND}_k(X_i))}\" target=\"_blank\"\u003e\u003cimg src=\"https://latex.codecogs.com/svg.latex?\\fn_cm\u0026space;\\text{coverage}:=\\frac{1}{N}\\sum_{i=1}^{N}1_{\\exists\\text{\u0026space;}j\\text{\u0026space;s.t.\u0026space;}\u0026space;Y_j\\in\u0026space;B(X_i,\\text{NND}_k(X_i))}\" title=\"\\text{coverage}:=\\frac{1}{N}\\sum_{i=1}^{N}1_{\\exists\\text{ }j\\text{ s.t. } Y_j\\in B(X_i,\\text{NND}_k(X_i))}\" /\u003e\u003c/a\u003e\n\n\n### Why are DC better than PR?\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/clovaai/prdc/blob/master/figure/p_vs_d.png?raw=true\" alt=\"Precision versus density.\" width=\"500\"/\u003e\n\u003c/p\u003e\n\n**Precision versus Density.** \nBecause of the real outlier sample, the manifold is overestimated. Generating many fake samples around the real outlier is enough to increase the precision measure. \nThe problem of overestimating precision (100%) is resolved using the density estimate (60%). \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/clovaai/prdc/blob/master/figure/r_vs_c.png?raw=true\" alt=\"Recall versus coverage.\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n**Recall versus Coverage.** \nThe real and fake samples are identical across left and right.\nSince models often generate many unrealistic yet diverse samples, the fake manifold is often an overestimation of the true fake distribution. \nIn the figure above, while the fake samples are generally far from the modes in real samples, the recall measure is rewarded by the fact that real samples are contained in the overestimated fake manifold.\n\n\n## 2. Usage\n\n### Installation\n\n```bash\npip3 install prdc\n```\n\n### Example\n\nTest 10000 real and fake samples form the standard normal distribution N(0,I) in 1000-dimensional Euclidean space.\nSet the nearest neighbour `k=5`. We compute precision, recall, density, and coverage estimates below.\n\n```python\nimport numpy as np\nfrom prdc import compute_prdc\n\n\nnum_real_samples = num_fake_samples = 10000\nfeature_dim = 1000\nnearest_k = 5\nreal_features = np.random.normal(loc=0.0, scale=1.0,\n                                 size=[num_real_samples, feature_dim])\n\nfake_features = np.random.normal(loc=0.0, scale=1.0,\n                                 size=[num_fake_samples, feature_dim])\n\nmetrics = compute_prdc(real_features=real_features,\n                       fake_features=fake_features,\n                       nearest_k=nearest_k)\n\nprint(metrics)\n```\nAbove test code will result in the following estimates (may fluctuate due to randomness).\n```python\n{'precision': 0.4772,\n 'recall': 0.4705,\n 'density': 1.0555,\n 'coverage': 0.9735}\n```\n\n## 3. Miscellaneous\n\n### References\n\nKynk\u0026auml;\u0026auml;nniemi et al., 2019. Improved precision and recall metric for assessing generative models. Neurips 2019.\n\n### License\n\n```\nCopyright (c) 2020-present NAVER Corp.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n```\n\n### Cite this work\n\n```\n@article{ferjad2020icml,\n  title = {Reliable Fidelity and Diversity Metrics for Generative Models},\n  author = {Naeem, Muhammad Ferjad and Oh, Seong Joon and Uh, Youngjung and Choi, Yunjey and Yoo, Jaejun},\n  year = {2020},\n  booktitle = {International Conference on Machine Learning},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclovaai%2Fgenerative-evaluation-prdc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclovaai%2Fgenerative-evaluation-prdc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclovaai%2Fgenerative-evaluation-prdc/lists"}