{"id":13738211,"url":"https://github.com/EndlessSora/focal-frequency-loss","last_synced_at":"2025-05-08T16:32:57.922Z","repository":{"id":43510227,"uuid":"323530376","full_name":"EndlessSora/focal-frequency-loss","owner":"EndlessSora","description":"[ICCV 2021] Focal Frequency Loss for Image Reconstruction and Synthesis","archived":false,"fork":false,"pushed_at":"2024-08-21T03:25:56.000Z","size":5171,"stargazers_count":655,"open_issues_count":8,"forks_count":64,"subscribers_count":37,"default_branch":"master","last_synced_at":"2025-04-21T22:02:55.435Z","etag":null,"topics":["autoencoder","complementary","frequency-analysis","frequency-domain","gan","generative-adversarial-network","generative-models","generic","iccv2021","image-generation","image-reconstruction","image-synthesis","loss","loss-function","pix2pix","spade","stylegan2","variational-autoencoder"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EndlessSora.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-22T05:30:16.000Z","updated_at":"2025-04-19T23:45:43.000Z","dependencies_parsed_at":"2024-11-15T07:30:41.781Z","dependency_job_id":"72ccb899-749f-4dc3-92ad-c4017c0d7f49","html_url":"https://github.com/EndlessSora/focal-frequency-loss","commit_stats":{"total_commits":14,"total_committers":1,"mean_commits":14.0,"dds":0.0,"last_synced_commit":"c0d94cad8876d750cb9034f1b846035b0436decd"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndlessSora%2Ffocal-frequency-loss","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndlessSora%2Ffocal-frequency-loss/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndlessSora%2Ffocal-frequency-loss/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EndlessSora%2Ffocal-frequency-loss/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EndlessSora","download_url":"https://codeload.github.com/EndlessSora/focal-frequency-loss/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253105397,"owners_count":21855019,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoencoder","complementary","frequency-analysis","frequency-domain","gan","generative-adversarial-network","generative-models","generic","iccv2021","image-generation","image-reconstruction","image-synthesis","loss","loss-function","pix2pix","spade","stylegan2","variational-autoencoder"],"created_at":"2024-08-03T03:02:14.613Z","updated_at":"2025-05-08T16:32:57.250Z","avatar_url":"https://github.com/EndlessSora.png","language":"Python","funding_links":[],"categories":["Python","Image Super-Resolution \u0026 Restoration"],"sub_categories":[],"readme":"## Focal Frequency Loss - Official PyTorch Implementation\n\n![teaser](https://raw.githubusercontent.com/EndlessSora/focal-frequency-loss/master/resources/teaser.jpg)\n\nThis repository provides the official PyTorch implementation for the following paper:\n\n**Focal Frequency Loss for Image Reconstruction and Synthesis**\u003cbr\u003e\n[Liming Jiang](https://liming-jiang.com/), [Bo Dai](http://daibo.info/), [Wayne Wu](https://wywu.github.io/) and [Chen Change Loy](http://personal.ie.cuhk.edu.hk/~ccloy/)\u003cbr\u003e\nIn ICCV 2021.\u003cbr\u003e\n[**Project Page**](https://www.mmlab-ntu.com/project/ffl/index.html) | [**Paper**](https://arxiv.org/abs/2012.12821) | [**Poster**](https://liming-jiang.com/projects/FFL/resources/poster.pdf) | [**Slides**](https://liming-jiang.com/projects/FFL/resources/slides.pdf) | [**YouTube Demo**](https://www.youtube.com/watch?v=RNTnDtKvcpc)\n\u003e **Abstract:** *Image reconstruction and synthesis have witnessed remarkable progress thanks to the development of generative models. Nonetheless, gaps could still exist between the real and generated images, especially in the frequency domain. In this study, we show that narrowing gaps in the frequency domain can ameliorate image reconstruction and synthesis quality further. We propose a novel focal frequency loss, which allows a model to adaptively focus on frequency components that are hard to synthesize by down-weighting the easy ones. This objective function is complementary to existing spatial losses, offering great impedance against the loss of important frequency information due to the inherent bias of neural networks. We demonstrate the versatility and effectiveness of focal frequency loss to improve popular models, such as VAE, pix2pix, and SPADE, in both perceptual quality and quantitative performance. We further show its potential on StyleGAN2.*\n\n## Updates\n\n- [09/2021] The **code** of Focal Frequency Loss is **released**.\n\n- [07/2021] The [paper](https://arxiv.org/abs/2012.12821) of Focal Frequency Loss is accepted by **ICCV 2021**.\n\n## Quick Start\n\nRun `pip install focal-frequency-loss` for installation. Then, the following code is all you need.\n\n```python\nfrom focal_frequency_loss import FocalFrequencyLoss as FFL\nffl = FFL(loss_weight=1.0, alpha=1.0)  # initialize nn.Module class\n\nimport torch\nfake = torch.randn(4, 3, 64, 64)  # replace it with the predicted tensor of shape (N, C, H, W)\nreal = torch.randn(4, 3, 64, 64)  # replace it with the target tensor of shape (N, C, H, W)\n\nloss = ffl(fake, real)  # calculate focal frequency loss\n```\n\n**Tips:** \n\n1. Current supported PyTorch version: `torch\u003e=1.1.0`. Warnings can be ignored. Please note that experiments in the paper were conducted with `torch\u003c=1.7.1,\u003e=1.1.0`.\n2. Arguments to initialize the `FocalFrequencyLoss` class:\n\t- `loss_weight (float)`: weight for focal frequency loss. Default: 1.0\n\t- `alpha (float)`: the scaling factor alpha of the spectrum weight matrix for flexibility. Default: 1.0\n\t- `patch_factor (int)`: the factor to crop image patches for patch-based focal frequency loss. Default: 1\n\t- `ave_spectrum (bool)`: whether to use minibatch average spectrum. Default: False\n\t- `log_matrix (bool)`: whether to adjust the spectrum weight matrix by logarithm. Default: False\n\t- `batch_matrix (bool)`: whether to calculate the spectrum weight matrix using batch-based statistics. Default: False\n3. Experience shows that the main hyperparameters you need to adjust are `loss_weight` and `alpha`. The loss weight may always need to be adjusted first. Then, a larger alpha indicates that the model is more focused. We use `alpha=1.0` as default.\n\n## Exmaple: Image Reconstruction (Vanilla AE)\n\nAs a guide, we provide an example of applying the proposed focal frequency loss (FFL) for Vanilla AE image reconstruction on CelebA. Applying FFL is pretty easy. The core details can be found [here](https://github.com/EndlessSora/focal-frequency-loss/blob/master/VanillaAE/models.py).\n\n### Installation\n\nAfter installing [Anaconda](https://www.anaconda.com/), we recommend you to create a new conda environment with python 3.8.3:\n\n```bash\nconda create -n ffl python=3.8.3 -y\nconda activate ffl\n```\n\nClone this repo, install PyTorch 1.4.0 (`torch\u003e=1.1.0` may also work) and other dependencies:\n\n```bash\ngit clone https://github.com/EndlessSora/focal-frequency-loss.git\ncd focal-frequency-loss\npip install -r VanillaAE/requirements.txt\n```\n\n### Dataset Preparation\n\nIn this example, please download [img\\_align\\_celeba.zip](https://drive.google.com/file/d/0B7EVK8r0v71pZjFTYXZWM3FlRnM/view?usp=sharing\u0026resourcekey=0-dYn9z10tMJOBAkviAcfdyQ) of the CelebA dataset from its [official website](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html). Then, we highly recommend you to unzip this file and symlink the `img_align_celeba` folder to `./datasets/celeba` by:\n\n```bash\nbash scripts/datasets/prepare_celeba.sh [PATH_TO_IMG_ALIGN_CELEBA]\n```\n\nOr you can simply move the `img_align_celeba` folder to `./datasets/celeba`. The resulting directory structure should be:\n\n```\n├── datasets\n│    ├── celeba\n│    │    ├── img_align_celeba  \n│    │    │    ├── 000001.jpg\n│    │    │    ├── 000002.jpg\n│    │    │    ├── 000003.jpg\n│    │    │    ├── ...\n```\n\n### Test and Evaluation Metrics\n\nDownload the [pretrained models](https://drive.google.com/file/d/1YIH09eoDyP2JLmiYJpju4hOkVFO7M3b_/view?usp=sharing) and unzip them to `./VanillaAE/experiments`.\n\nWe have provided the example [test scripts](https://github.com/EndlessSora/focal-frequency-loss/tree/master/scripts/VanillaAE/test). If you only have a CPU environment, please specify `--no_cuda` in the script. Run:\n\n```bash\nbash scripts/VanillaAE/test/celeba_recon_wo_ffl.sh\nbash scripts/VanillaAE/test/celeba_recon_w_ffl.sh\n```\n\nThe Vanilla AE image reconstruction results will be saved at `./VanillaAE/results` by default.\n\nAfter testing, you can further calculate the evaluation metrics for this example. We have implemented a series of [evaluation metrics](https://github.com/EndlessSora/focal-frequency-loss/tree/master/metrics) we used and provided the [metric scripts](https://github.com/EndlessSora/focal-frequency-loss/tree/master/scripts/VanillaAE/metrics). Run:\n\n```bash\nbash scripts/VanillaAE/metrics/celeba_recon_wo_ffl.sh\nbash scripts/VanillaAE/metrics/celeba_recon_w_ffl.sh\n```\n\nYou will see the scores of different metrics. The metric logs will be saved in the respective experiment folders at `./VanillaAE/results`.\n\n### Training\n\nWe have provided the example [training scripts](https://github.com/EndlessSora/focal-frequency-loss/tree/master/scripts/VanillaAE/train). If you only have a CPU environment, please specify `--no_cuda` in the script. Run:\n\n```bash\nbash scripts/VanillaAE/train/celeba_recon_wo_ffl.sh\nbash scripts/VanillaAE/train/celeba_recon_w_ffl.sh \n```\n\nAfter training, inference on the newly trained models is similar to [Test and Evaluation Metrics](#test-and-evaluation-metrics). The results could be better reproduced on NVIDIA Tesla V100 GPUs with `torch\u003c=1.7.1,\u003e=1.1.0`.\n\n## More Results\n\nHere, we show other examples of applying the proposed focal frequency loss (FFL) under diverse settings.\n\n### Image Reconstruction (VAE)\n\n![reconvae](https://raw.githubusercontent.com/EndlessSora/focal-frequency-loss/master/resources/reconvae.jpg)\n\n### Image-to-Image Translation (pix2pix | SPADE)\n\n![consynI2I](https://raw.githubusercontent.com/EndlessSora/focal-frequency-loss/master/resources/consynI2I.jpg)\n\n### Unconditional Image Synthesis (StyleGAN2)\n\n256x256 results (without truncation) and the mini-batch average spectra (adjusted to better contrast):\n\n![unsynsg2res256](https://raw.githubusercontent.com/EndlessSora/focal-frequency-loss/master/resources/unsynsg2res256.jpg)\n\n1024x1024 results (without truncation) synthesized by StyleGAN2 with FFL:\n\n![unsynsg2res1024](https://raw.githubusercontent.com/EndlessSora/focal-frequency-loss/master/resources/unsynsg2res1024.jpg)\n\n## Citation\n\nIf you find this work useful for your research, please cite our paper:\n\n```\n@inproceedings{jiang2021focal,\n  title={Focal Frequency Loss for Image Reconstruction and Synthesis},\n  author={Jiang, Liming and Dai, Bo and Wu, Wayne and Loy, Chen Change},\n  booktitle={ICCV},\n  year={2021}\n}\n```\n\n## Acknowledgments\n\nThe code of Vanilla AE is inspired by [PyTorch DCGAN](https://github.com/pytorch/examples/tree/master/dcgan) and [MUNIT](https://github.com/NVlabs/MUNIT). Part of the evaluation metric code is borrowed from [MMEditing](https://github.com/open-mmlab/mmediting). We also apply [LPIPS](https://github.com/richzhang/PerceptualSimilarity) and [pytorch-fid](https://github.com/mseitzer/pytorch-fid) as evaluation metrics.\n\n## License\n\nAll rights reserved. The code is released under the [MIT License](https://github.com/EndlessSora/focal-frequency-loss/blob/master/LICENSE.md).\n\nCopyright (c) 2021\n\n## Other Implementations\n\n[[**Unofficial TensorFlow Implementation**](https://github.com/ZohebAbai/tf-focal-frequency-loss)] by [ZohebAbai](https://github.com/ZohebAbai)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEndlessSora%2Ffocal-frequency-loss","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEndlessSora%2Ffocal-frequency-loss","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEndlessSora%2Ffocal-frequency-loss/lists"}