{"id":18614427,"url":"https://github.com/jgkwak95/SURF-GAN","last_synced_at":"2025-04-11T00:30:39.997Z","repository":{"id":47992138,"uuid":"516462291","full_name":"jgkwak95/SURF-GAN","owner":"jgkwak95","description":"[ECCV 2022] Official Pytorch implementation of \"Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis\"","archived":false,"fork":false,"pushed_at":"2023-02-04T07:37:30.000Z","size":522260,"stargazers_count":127,"open_issues_count":0,"forks_count":14,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-07T03:31:30.985Z","etag":null,"topics":["3d","3d-aware-image-synthesis","eccv2022","generative-adversarial-network","image-to-image-translation","neural-rendering","stylegan2","toonify"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jgkwak95.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-21T17:24:38.000Z","updated_at":"2024-11-06T10:46:30.000Z","dependencies_parsed_at":"2023-02-18T14:45:29.201Z","dependency_job_id":null,"html_url":"https://github.com/jgkwak95/SURF-GAN","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jgkwak95%2FSURF-GAN","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jgkwak95%2FSURF-GAN/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jgkwak95%2FSURF-GAN/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jgkwak95%2FSURF-GAN/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jgkwak95","download_url":"https://codeload.github.com/jgkwak95/SURF-GAN/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322216,"owners_count":21084333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","3d-aware-image-synthesis","eccv2022","generative-adversarial-network","image-to-image-translation","neural-rendering","stylegan2","toonify"],"created_at":"2024-11-07T03:25:56.463Z","updated_at":"2025-04-11T00:30:39.634Z","avatar_url":"https://github.com/jgkwak95.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF Related Tasks"],"readme":"# Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis\n\n\u003cimg src=\"https://user-images.githubusercontent.com/67986601/191766555-806a304e-25a6-4d7d-9d48-ce623febe85f.png\" width=\"90%\"\u003e\n\n## [Project page](https://jgkwak95.github.io/surfgan/) | [Paper](http://arxiv.org/abs/2207.10257)  \n\u003cbr\u003e\n  \n\u003e**\"Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis\"** \u003cbr\u003e\n\u003e[Jeong-gi Kwak](https://jgkwak95.github.io/), Yuanming Li, Dongsik Yoon, Donghyeon Kim, David Han, Hanseok Ko\u003cbr\u003e\n\u003e**ECCV 2022** \u003cbr\u003e\n\n\u003cbr\u003e\n\nThis repository includes the official Pytorch implementation of SURF-GAN. \u003cbr\u003e\n\n\n# SURF-GAN \n\nSURF-GAN, which is a NeRF-based 3D-aware GAN, can discover disentangled semantic attributes in an unsupervised manner.  \u003cbr\u003e\u003cbr\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/67986601/180636584-6ac7f46f-ff8f-4da1-a4b9-69e31e769909.gif\" width=150\u003e \u003cbr\u003e\n(Tranined on 64x64 CelebA and rendered with 256x256) \u003cbr\u003e\n\n## Get started\n\n- #### Clone the repo.\n```\ngit clone https://github.com/jgkwak95/SURF-GAN.git\ncd SURF-GAN\n```\n- #### Create virtual environment\n```\nconda create -n surfgan python=3.7.1\nconda activate surfgan\nconda install -c pytorch-lts pytorch torchvision \npip install --no-cache-dir -r requirements.txt\n```\n\n## Train SURF-GAN\nAt first, look curriculum.py and specify dataset and training options.\n```\n# CelebA\npython train_surf.py --output_dir your-exp-name \\\n--curriculum CelebA_single\n```\n### Pretrained model\nOr, you can use the [pretrained model](https://drive.google.com/file/d/19ufd6VGQ4pR2AOEMBDFwvS9seZ-C6-8U/view?usp=sharing).\n## Semantic attribute discovery\nLet's traverse each dimension with discovered semantics:\n```\npython discover_semantics.py  --experiment your-exp-name \\\n--image_size 256 \\\n--ray_step_multiplier 2 \\\n--num_id 9 \\          \n--traverse_range 3.0 \\    \n --intermediate_points 9 \\\n--curriculum CelebA_single     \n```\nThe default ckpt file to traverse is the latest file (generator.pth).\nIf you want to check specific cpkt, add this in your command line, for example,\n```\n--specific_ckpt 140000_64_generator.pth\n```\n## Control pose\nIn addition, you can control only camera paramters:\n```\npython control_pose.py --experiment your-exp-name \\\n--image_size 128 \\\n--ray_step_multiplier 2 \\\n--num_id 9 \\\n--intermediate_points 9 \\\n--mode yaw \\\n--curriculum CelebA_single \\\n```\n## Render video\n- #### Moving camera\nSet the mode: yaw, pitch, fov, etc.\nYou can also make your trajectory. \n```\npython render_video.py  --experiment your-exp-name \\\n--image_size 128 \\\n--ray_step_multiplier 2 \\\n--num_frames 100 \\\n--curriculum CelebA_single \\\n--mode yaw\n```\n\n- #### Moving camera with a specific semantic\nChoose an attribute that you want to control LiDj.\n```\npython render_video_semantic.py  --experiment your-exp-name \\\n--image_size 128 \\\n--ray_step_multiplier 2 \\\n--num_frames 100 \\ \n--traverse_range 3.0 \\\n--intermediate_points \\\n--curriculum CelebA_single \\\n--mode circle\n--L 2\n--D 4\n```\n\u003cbr\u003e\n\u003cbr\u003e\n\n# 3D-Controllable StyleGAN\n\nInjecting the prior of SURF-GAN into StyleGAN for controllable generation. \u003cbr\u003e\nAlso, it is compatible with many StyleGAN-based methods.\n\n\u003cimg src=\"https://jgkwak95.github.io/surfgan/assets/3d_stylegan.png\" width=\"40%\"\u003e \u003cbr\u003e\n\n### Video\n\n| Pose control  | + Style ([Toonify](https://github.com/justinpinkney/toonify)) | \n| ------ | ------| \n| \u003cimg src=\"https://user-images.githubusercontent.com/67986601/180636660-189a6479-084a-460f-ac69-a6491987d9b8.gif\" width=\"90%\"\u003e | \u003cimg src=\"https://user-images.githubusercontent.com/67986601/180636724-0963eb04-5ec1-441c-8305-464c4796ef02.gif\" width=\"100%\"\u003e   |\n| \u003cimg src=\"https://user-images.githubusercontent.com/67986601/180636672-4b22539a-73ee-470c-b5f8-795df3a27a5c.gif\" width=\"90%\"\u003e| \u003cimg src=\"https://user-images.githubusercontent.com/67986601/180636715-39a4438e-9176-4c3e-b8bd-b417c5b6d4c5.gif\" width=\"100%\"\u003e |\n\n\u003cbr\u003e\n\nIt is capable of editing real images directly. (with [HyperStyle](https://github.com/yuval-alaluf/hyperstyle))\n\n\n| Pose   | +Illumination (using SURF-GAN samples) | \n| ------ | ------| \n| \u003cimg src=\"https://user-images.githubusercontent.com/67986601/182040684-571dbfb7-833d-4729-84c1-e4c0544fffee.gif\" width=\"100%\"\u003e | \u003cimg src=\"https://user-images.githubusercontent.com/67986601/182040360-0b295928-0e83-4b66-b284-05fb9711425c.gif\" width=\"100%\"\u003e   |\n\n| +Hair color (using SURF-GAN samples)   | +Smile(using [InterFaceGAN](https://github.com/genforce/interfacegan)) | \n| ------ | ------| \n| \u003cimg src=\"https://user-images.githubusercontent.com/67986601/182040807-97c306a6-6a6a-4fa3-8dbc-12320f775283.gif\" width=\"100%\"\u003e | \u003cimg src=\"https://user-images.githubusercontent.com/67986601/182040808-00d04350-dc7c-4499-aa90-fa35869ff642.gif\" width=\"100%\"\u003e   | \u003cbr\u003e   \n   \n    \n     \n\u003cbr\u003e\u003cbr\u003e\n## Citation \n\n\n```\n@inproceedings{kwak2022injecting,\n  title={Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis},\n  author={Kwak, Jeong-gi and Li, Yuanming and Yoon, Dongsik and Kim, Donghyeon and Han, David and Ko, Hanseok},\n  booktitle={European Conference on Computer Vision},\n  pages={236--253},\n  year={2022},\n  organization={Springer}\n}\n```\n\n## Acknowledgments\n\n-  SURF-GAN is bulided upon the [pi-GAN](https://github.com/marcoamonteiro/pi-GAN) implementation and inspired by [EigenGAN](https://github.com/LynnHo/EigenGAN-Tensorflow) ([EigenGAN-pytorch](https://github.com/bryandlee/eigengan-pytorch)). Thanks to the authors for their excellent work!\n-  We used [pSp encoder](https://github.com/eladrich/pixel2style2pixel) and [StyleGAN2-pytorch](https://github.com/rosinality/stylegan2-pytorch) to build 3D-controllable StyleGAN. For editing  in-the-wild real images, we exploited [e4e](https://github.com/omertov/encoder4editing) and [HyperStyle](https://github.com/yuval-alaluf/hyperstyle) with our 3D-controllable StyleGAN. \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjgkwak95%2FSURF-GAN","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjgkwak95%2FSURF-GAN","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjgkwak95%2FSURF-GAN/lists"}