{"id":18664523,"url":"https://github.com/lzhbrian/metrics","last_synced_at":"2025-04-11T21:32:57.445Z","repository":{"id":45711824,"uuid":"177364167","full_name":"lzhbrian/metrics","owner":"lzhbrian","description":"IS, FID score Pytorch and TF implementation, TF implementation is a wrapper of the official ones.","archived":false,"fork":false,"pushed_at":"2021-03-02T00:52:13.000Z","size":38931,"stargazers_count":123,"open_issues_count":4,"forks_count":23,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-25T19:16:01.185Z","etag":null,"topics":["api","commandline","fid","frechet-distance","frechet-inception-distance","gan","inception","inception-score","is","metrics","pytorch","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lzhbrian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-24T02:58:16.000Z","updated_at":"2025-03-04T05:03:37.000Z","dependencies_parsed_at":"2022-08-31T22:01:47.910Z","dependency_job_id":null,"html_url":"https://github.com/lzhbrian/metrics","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lzhbrian%2Fmetrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lzhbrian%2Fmetrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lzhbrian%2Fmetrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lzhbrian%2Fmetrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lzhbrian","download_url":"https://codeload.github.com/lzhbrian/metrics/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248483374,"owners_count":21111435,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","commandline","fid","frechet-distance","frechet-inception-distance","gan","inception","inception-score","is","metrics","pytorch","tensorflow"],"created_at":"2024-11-07T08:23:53.251Z","updated_at":"2025-04-11T21:32:52.426Z","avatar_url":"https://github.com/lzhbrian.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# metrics\n\nThis repo contains information/implementation (PyTorch, Tensorflow) about IS and FID score. This is a handy toolbox that you can easily add to your projects. TF implementations are intended to compute the exact same output as the official ones for reporting in papers. Discussion/PR/Issues are very welcomed.\n\n\n\n## Usage\n\nPut this `metrics/` folder in your projects, and __see below (Pytorch), and each .py's head comment__ for usage.\n\nWe also need to download some files in [res/](res/), see [res/README.md](res/README.md) for more details.\n\n\n\n## TF implementations (almost the same as official, just changed the interface, can be reported in papers)\n\n- [x] [inception_score_official_tf.py](inception_score_official_tf.py): inception score\n- [x] [fid_official_tf.py](fid_official_tf.py): FID score\n- [x] [precalc_stats_official_tf.py](precalc_stats_official_tf.py): calculate stats (mu, sigma)\n\n\n\n## Pytorch Implementation (CANNOT report in papers, but can get an quick view)\n\n* Requirements\n\n    * pytorch, torchvision, scipy, numpy, tqdm\n* [is_fid_pytorch.py](is_fid_pytorch.py)\n    * [x] inception score, get around `mean=9.67278, std=0.14992` for CIFAR-10 train data when n_split=10\n    * [x] FID score\n    * [x] calculate stats for custom images in a folder (mu, sigma)\n    * [x] multi-GPU support by `nn.DataParallel`\n        * e.g. `CUDA_VISIBLE_DEVICES=0,1,2,3` will use 4 GPU.\n* command line usage\n    * calculate IS, FID\n        ```bash\n        # calc IS score on CIFAR10, will download CIFAR10 data to ../data/cifar10\n        python is_fid_pytorch.py\n        \n        # calc IS score on custom images in a folder/\n        python is_fid_pytorch.py --path foldername/\n        \n        # calc IS, FID score on custom images in a folder/, compared to CIFAR10 (given precalculated stats)\n        python is_fid_pytorch.py --path foldername/ --fid res/stats_pytorch/fid_stats_cifar10_train.npz\n        \n        # calc FID on custom images in two folders/\n        python is_fid_pytorch.py --path foldername1/ --fid foldername2/\n        \n        # calc FID on two precalculated stats\n        python is_fid_pytorch.py --path res/stats_pytorch/fid_stats_cifar10_train.npz --fid res/stats_pytorch/fid_stats_cifar10_train.npz\n        ```\n\n    * precalculate stats\n        ```bash\n        # precalculate stats store as npz for CIFAR 10, will download CIFAR10 data to ../data/cifar10\n        python is_fid_pytorch.py --save-stats-path res/stats_pytorch/fid_stats_cifar10_train.npz\n        \n        # precalculate stats store as npz for images in folder/\n        python is_fid_pytorch.py --path foldername/ --save-stats-path res/stats_pytorch/fid_stats_folder.npz\n        ```\n\n        \n\n* in code usage\n\n    * `mode=1`: image tensor has already normalized by `mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]`\n    * `mode=2`: image tensor has already normalized by `mean=[0.500, 0.500, 0.500], std=[0.500, 0.500, 0.500]`\n        ```python\n        from metrics import is_fid_pytorch\n        \n        # using precalculated stats (.npz) for FID calculation\n        is_fid_model = is_fid_pytorch.ScoreModel(mode=2, stats_file='res/stats_pytorch/fid_stats_cifar10_train.npz', cuda=cuda)\n        imgs_nchw = torch.Tensor(50000, C, H, W) # torch.Tensor in -1~1, normalized by mean=[0.500, 0.500, 0.500], std=[0.500, 0.500, 0.500]\n        is_mean, is_std, fid = is_fid_model.get_score_image_tensor(imgs_nchw)\n        \n        # we can also pass in mu, sigma for get_score_image_tensor()\n        is_fid_model = is_fid_pytorch.ScoreModel(mode=2, cuda=cuda)\n        mu, sigma = is_fid_pytorch.read_stats_file('res/stats_pytorch/fid_stats_cifar10_train.npz')\n        is_mean, is_std, fid = is_fid_model.get_score_image_tensor(imgs_nchw, mu1=mu, sigma1=sigma)\n        \n        # if no need FID\n        is_fid_model = is_fid_pytorch.ScoreModel(mode=2, cuda=cuda)\n        is_mean, is_std, _ = is_fid_model.get_score_image_tensor(imgs_nchw)\n        \n        # if want stats (mu, sigma) for imgs_nchw, send in return_stats=True\n        is_mean, is_std, _, mu, sigma = is_fid_model.get_score_image_tensor(imgs_nchw, return_stats=True)\n        \n        # from pytorch dataset, use get_score_dataset(), instead of get_score_image_tensor(), other usage is the same\n        cifar = dset.CIFAR10(root='../data/cifar10', download=True,\n                             transform=transforms.Compose([\n                                 transforms.Resize(32),\n                                 transforms.ToTensor(),\n                                 transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n                             ])\n                            )\n        IgnoreLabelDataset(cifar)\n        is_mean, is_std, _ = is_fid_model.get_score_dataset(IgnoreLabelDataset(cifar))\n        ```\n\n\n## TODO\n\n- [ ] Refactor TF implementation of IS, FID Together\n- [ ] MS-SSIM score - PyTorch\n- [ ] MS-SSIM score - Tensorflow\n\n\n\n## Info\n\n### Inception Score (IS)\n\n* Assumption\n    * MEANINGFUL: The generated image should be clear, the output probability of a classifier network should be [0.9, 0.05, ...] (largely skewed to a class). $p(y|\\mathbf{x})$ is of __low entropy__.\n    * DIVERSITY: If we have 10 classes, the generated image should be averagely distributed. So that the marginal distribution $p(y) = \\frac{1}{N} \\sum_{i=1}^{N} p(y|\\mathbf{x}^{(i)})$ is of __high entropy__.\n    * Better models: KL Divergence of $p(y|\\mathbf{x})$ and $p(y)$ should be high.\n* Formulation\n    * $\\mathbf{IS} = \\exp (\\mathbb{E}_{\\mathbf{x} \\sim p_g} D_{KL} [p(y|\\mathbf{x}) || p(y)] )$\n    * where\n        * $\\mathbf{x}$ is sampled from generated data\n        * $p(y|\\mathbf{x})​$ is the output probability of Inception v3 when input is $\\mathbf{x}​$\n        * $p(y) = \\frac{1}{N} \\sum_{i=1}^{N} p(y|\\mathbf{x}^{(i)})$ is the average output probability of all generated data (from InceptionV3, 1000-dim vector)\n        * $D_{KL} (\\mathbf{p}||\\mathbf{q}) = \\sum_{j} p_{j} \\log \\frac{p_j}{q_j}$, where $j$ is the dimension of the output probability.\n\n* Explanation\n    * $p(y)$ is a evenly distributed vector\n    * larger $\\mathbf{IS}​$ score -\u003e larger KL divergence -\u003e larger diversity and clearness\n* Reference\n    * Official TF implementation is in [openai/improved-gan](https://github.com/openai/improved-gan)\n    * Pytorch Implementation: [sbarratt/inception-score-pytorch](https://github.com/sbarratt/inception-score-pytorch)\n    * TF seemed to provide a [good implementation](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/gan/python/eval/python/classifier_metrics_impl.py)\n    * [scipy.stats.entropy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html)\n    * [zhihu: Inception Score 的原理和局限性](https://zhuanlan.zhihu.com/p/54146307)\n    * [A Note on the Inception Score](https://arxiv.org/abs/1801.01973)\n\n\n\n### Fréchet Inception Distance (FID)\n\n* Formulation\n    * $\\mathbf{FID} = ||\\mu_r - \\mu_g||^2 + Tr(\\Sigma_{r} + \\Sigma_{g} - 2(\\Sigma_r \\Sigma_g)^{1/2})​$\n    * where\n        * $Tr$ is [trace of a matrix (wikipedia)](https://en.wikipedia.org/wiki/Trace_(linear_algebra))\n        * $X_r \\sim \\mathcal{N}(\\mu_r, \\Sigma_r)$ and $X_g \\sim \\mathcal{N}(\\mu_g, \\Sigma_g)$ are the 2048-dim activations  the InceptionV3 pool3 layer\n        * $\\mu_r$ is the mean of real photo's feature\n        * $\\mu_g$ is the mean of generated photo's feature\n        * $\\Sigma_r$ is the covariance matrix of real photo's feature\n        * $\\Sigma_g$ is the covariance matrix of generated photo's feature\n\n* Reference\n    * Official TF implementation: [bioinf-jku/TTUR](https://github.com/bioinf-jku/TTUR)\n    * Pytorch Implementation: [mseitzer/pytorch-fid](https://github.com/mseitzer/pytorch-fid)\n    * TF seemed to provide a [good implementation](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/gan/python/eval/python/classifier_metrics_impl.py)\n    * [zhihu: Frechet Inception Score (FID)](https://zhuanlan.zhihu.com/p/54213305)\n    * [Explanation from Neal Jean](https://nealjean.com/ml/frechet-inception-distance/)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flzhbrian%2Fmetrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flzhbrian%2Fmetrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flzhbrian%2Fmetrics/lists"}