{"id":13456888,"url":"https://github.com/sczhou/CodeFormer","last_synced_at":"2025-03-24T11:31:44.985Z","repository":{"id":37357413,"uuid":"505667511","full_name":"sczhou/CodeFormer","owner":"sczhou","description":"[NeurIPS 2022] Towards Robust Blind Face Restoration with Codebook Lookup Transformer","archived":false,"fork":false,"pushed_at":"2024-10-09T20:31:41.000Z","size":17625,"stargazers_count":15710,"open_issues_count":257,"forks_count":3306,"subscribers_count":299,"default_branch":"master","last_synced_at":"2024-10-28T17:55:20.677Z","etag":null,"topics":["codebook","codeformer","face-enhancement","face-restoration","pytorch","restoration","super-resolution","vqgan"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sczhou.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-21T02:48:14.000Z","updated_at":"2024-10-28T15:34:57.000Z","dependencies_parsed_at":"2024-06-21T19:04:14.787Z","dependency_job_id":"97935306-e83d-4bfc-b097-de1ae21c1aa8","html_url":"https://github.com/sczhou/CodeFormer","commit_stats":{"total_commits":66,"total_committers":2,"mean_commits":33.0,"dds":"0.015151515151515138","last_synced_commit":"8392d0334956108ab53d9439c4b9fc9c4af0d66d"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sczhou%2FCodeFormer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sczhou%2FCodeFormer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sczhou%2FCodeFormer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sczhou%2FCodeFormer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sczhou","download_url":"https://codeload.github.com/sczhou/CodeFormer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221946215,"owners_count":16905959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["codebook","codeformer","face-enhancement","face-restoration","pytorch","restoration","super-resolution","vqgan"],"created_at":"2024-07-31T08:01:29.607Z","updated_at":"2024-10-29T00:31:24.515Z","avatar_url":"https://github.com/sczhou.png","language":"Python","readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/CodeFormer_logo.png\" height=110\u003e\n\u003c/p\u003e\n\n## Towards Robust Blind Face Restoration with Codebook Lookup Transformer (NeurIPS 2022)\n\n[Paper](https://arxiv.org/abs/2206.11253) | [Project Page](https://shangchenzhou.com/projects/CodeFormer/) | [Video](https://youtu.be/d3VDpkXlueI)\n\n\n\u003ca href=\"https://colab.research.google.com/drive/1m52PNveE4PBhYrecj34cnpEeiHcC5LTb?usp=sharing\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"google colab logo\"\u003e\u003c/a\u003e [![Hugging Face](https://img.shields.io/badge/Demo-%F0%9F%A4%97%20Hugging%20Face-blue)](https://huggingface.co/spaces/sczhou/CodeFormer) [![Replicate](https://img.shields.io/badge/Demo-%F0%9F%9A%80%20Replicate-blue)](https://replicate.com/sczhou/codeformer) [![OpenXLab](https://img.shields.io/badge/Demo-%F0%9F%90%BC%20OpenXLab-blue)](https://openxlab.org.cn/apps/detail/ShangchenZhou/CodeFormer) ![Visitors](https://api.infinitescript.com/badgen/count?name=sczhou/CodeFormer\u0026ltext=Visitors)\n\n\n[Shangchen Zhou](https://shangchenzhou.com/), [Kelvin C.K. Chan](https://ckkelvinchan.github.io/), [Chongyi Li](https://li-chongyi.github.io/), [Chen Change Loy](https://www.mmlab-ntu.com/person/ccloy/) \n\nS-Lab, Nanyang Technological University\n\n\u003cimg src=\"assets/network.jpg\" width=\"800px\"/\u003e\n\n\n:star: If CodeFormer is helpful to your images or projects, please help star this repo. Thanks! :hugs: \n\n\n### Update\n- **2023.07.20**: Integrated to :panda_face: [OpenXLab](https://openxlab.org.cn/apps). Try out online demo! [![OpenXLab](https://img.shields.io/badge/Demo-%F0%9F%90%BC%20OpenXLab-blue)](https://openxlab.org.cn/apps/detail/ShangchenZhou/CodeFormer)\n- **2023.04.19**: :whale: Training codes and config files are public available now.\n- **2023.04.09**: Add features of inpainting and colorization for cropped and aligned face images.\n- **2023.02.10**: Include `dlib` as a new face detector option, it produces more accurate face identity.\n- **2022.10.05**: Support video input `--input_path [YOUR_VIDEO.mp4]`. Try it to enhance your videos! :clapper: \n- **2022.09.14**: Integrated to :hugs: [Hugging Face](https://huggingface.co/spaces). Try out online demo! [![Hugging Face](https://img.shields.io/badge/Demo-%F0%9F%A4%97%20Hugging%20Face-blue)](https://huggingface.co/spaces/sczhou/CodeFormer)\n- **2022.09.09**: Integrated to :rocket: [Replicate](https://replicate.com/explore). Try out online demo! [![Replicate](https://img.shields.io/badge/Demo-%F0%9F%9A%80%20Replicate-blue)](https://replicate.com/sczhou/codeformer)\n- [**More**](docs/history_changelog.md)\n\n### TODO\n- [x] Add training code and config files\n- [x] Add checkpoint and script for face inpainting\n- [x] Add checkpoint and script for face colorization\n- [x] ~~Add background image enhancement~~\n\n#### :panda_face: Try Enhancing Old Photos / Fixing AI-arts\n[\u003cimg src=\"assets/imgsli_1.jpg\" height=\"226px\"/\u003e](https://imgsli.com/MTI3NTE2) [\u003cimg src=\"assets/imgsli_2.jpg\" height=\"226px\"/\u003e](https://imgsli.com/MTI3NTE1) [\u003cimg src=\"assets/imgsli_3.jpg\" height=\"226px\"/\u003e](https://imgsli.com/MTI3NTIw) \n\n#### Face Restoration\n\n\u003cimg src=\"assets/restoration_result1.png\" width=\"400px\"/\u003e \u003cimg src=\"assets/restoration_result2.png\" width=\"400px\"/\u003e\n\u003cimg src=\"assets/restoration_result3.png\" width=\"400px\"/\u003e \u003cimg src=\"assets/restoration_result4.png\" width=\"400px\"/\u003e\n\n#### Face Color Enhancement and Restoration\n\n\u003cimg src=\"assets/color_enhancement_result1.png\" width=\"400px\"/\u003e \u003cimg src=\"assets/color_enhancement_result2.png\" width=\"400px\"/\u003e\n\n#### Face Inpainting\n\n\u003cimg src=\"assets/inpainting_result1.png\" width=\"400px\"/\u003e \u003cimg src=\"assets/inpainting_result2.png\" width=\"400px\"/\u003e\n\n\n\n### Dependencies and Installation\n\n- Pytorch \u003e= 1.7.1\n- CUDA \u003e= 10.1\n- Other required packages in `requirements.txt`\n```\n# git clone this repository\ngit clone https://github.com/sczhou/CodeFormer\ncd CodeFormer\n\n# create new anaconda env\nconda create -n codeformer python=3.8 -y\nconda activate codeformer\n\n# install python dependencies\npip3 install -r requirements.txt\npython basicsr/setup.py develop\nconda install -c conda-forge dlib (only for face detection or cropping with dlib)\n```\n\u003c!-- conda install -c conda-forge dlib --\u003e\n\n### Quick Inference\n\n#### Download Pre-trained Models:\nDownload the facelib and dlib pretrained models from [[Releases](https://github.com/sczhou/CodeFormer/releases/tag/v0.1.0) | [Google Drive](https://drive.google.com/drive/folders/1b_3qwrzY_kTQh0-SnBoGBgOrJ_PLZSKm?usp=sharing) | [OneDrive](https://entuedu-my.sharepoint.com/:f:/g/personal/s200094_e_ntu_edu_sg/EvDxR7FcAbZMp_MA9ouq7aQB8XTppMb3-T0uGZ_2anI2mg?e=DXsJFo)] to the `weights/facelib` folder. You can manually download the pretrained models OR download by running the following command:\n```\npython scripts/download_pretrained_models.py facelib\npython scripts/download_pretrained_models.py dlib (only for dlib face detector)\n```\n\nDownload the CodeFormer pretrained models from [[Releases](https://github.com/sczhou/CodeFormer/releases/tag/v0.1.0) | [Google Drive](https://drive.google.com/drive/folders/1CNNByjHDFt0b95q54yMVp6Ifo5iuU6QS?usp=sharing) | [OneDrive](https://entuedu-my.sharepoint.com/:f:/g/personal/s200094_e_ntu_edu_sg/EoKFj4wo8cdIn2-TY2IV6CYBhZ0pIG4kUOeHdPR_A5nlbg?e=AO8UN9)] to the `weights/CodeFormer` folder. You can manually download the pretrained models OR download by running the following command:\n```\npython scripts/download_pretrained_models.py CodeFormer\n```\n\n#### Prepare Testing Data:\nYou can put the testing images in the `inputs/TestWhole` folder. If you would like to test on cropped and aligned faces, you can put them in the `inputs/cropped_faces` folder. You can get the cropped and aligned faces by running the following command:\n```\n# you may need to install dlib via: conda install -c conda-forge dlib\npython scripts/crop_align_face.py -i [input folder] -o [output folder]\n```\n\n\n#### Testing:\n[Note] If you want to compare CodeFormer in your paper, please run the following command indicating `--has_aligned` (for cropped and aligned face), as the command for the whole image will involve a process of face-background fusion that may damage hair texture on the boundary, which leads to unfair comparison.\n\nFidelity weight *w* lays in [0, 1]. Generally, smaller *w* tends to produce a higher-quality result, while larger *w* yields a higher-fidelity result. The results will be saved in the `results` folder.\n\n\n🧑🏻 Face Restoration (cropped and aligned face)\n```\n# For cropped and aligned faces (512x512)\npython inference_codeformer.py -w 0.5 --has_aligned --input_path [image folder]|[image path]\n```\n\n:framed_picture: Whole Image Enhancement\n```\n# For whole image\n# Add '--bg_upsampler realesrgan' to enhance the background regions with Real-ESRGAN\n# Add '--face_upsample' to further upsample restorated face with Real-ESRGAN\npython inference_codeformer.py -w 0.7 --input_path [image folder]|[image path]\n```\n\n:clapper: Video Enhancement\n```\n# For Windows/Mac users, please install ffmpeg first\nconda install -c conda-forge ffmpeg\n```\n```\n# For video clips\n# Video path should end with '.mp4'|'.mov'|'.avi'\npython inference_codeformer.py --bg_upsampler realesrgan --face_upsample -w 1.0 --input_path [video path]\n```\n\n🌈 Face Colorization (cropped and aligned face)\n```\n# For cropped and aligned faces (512x512)\n# Colorize black and white or faded photo\npython inference_colorization.py --input_path [image folder]|[image path]\n```\n\n🎨 Face Inpainting (cropped and aligned face)\n```\n# For cropped and aligned faces (512x512)\n# Inputs could be masked by white brush using an image editing app (e.g., Photoshop) \n# (check out the examples in inputs/masked_faces)\npython inference_inpainting.py --input_path [image folder]|[image path]\n```\n### Training:\nThe training commands can be found in the documents: [English](docs/train.md) **|** [简体中文](docs/train_CN.md).\n\n### Citation\nIf our work is useful for your research, please consider citing:\n\n    @inproceedings{zhou2022codeformer,\n        author = {Zhou, Shangchen and Chan, Kelvin C.K. and Li, Chongyi and Loy, Chen Change},\n        title = {Towards Robust Blind Face Restoration with Codebook Lookup TransFormer},\n        booktitle = {NeurIPS},\n        year = {2022}\n    }\n\n### License\n\nThis project is licensed under \u003ca rel=\"license\" href=\"https://github.com/sczhou/CodeFormer/blob/master/LICENSE\"\u003eNTU S-Lab License 1.0\u003c/a\u003e. Redistribution and use should follow this license.\n\n### Acknowledgement\n\nThis project is based on [BasicSR](https://github.com/XPixelGroup/BasicSR). Some codes are brought from [Unleashing Transformers](https://github.com/samb-t/unleashing-transformers), [YOLOv5-face](https://github.com/deepcam-cn/yolov5-face), and [FaceXLib](https://github.com/xinntao/facexlib). We also adopt [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN) to support background image enhancement. Thanks for their awesome works.\n\n### Contact\nIf you have any questions, please feel free to reach me out at `shangchenzhou@gmail.com`. \n","funding_links":[],"categories":["Python","Libs With Online Books","AI","Training","图像恢复","AI应用"],"sub_categories":["Philosophy","React Components","Content Restoration","网络服务_其他"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsczhou%2FCodeFormer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsczhou%2FCodeFormer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsczhou%2FCodeFormer/lists"}