{"id":18614433,"url":"https://github.com/CrisHY1995/headnerf","last_synced_at":"2025-04-11T00:30:47.031Z","repository":{"id":37646014,"uuid":"470821959","full_name":"CrisHY1995/headnerf","owner":"CrisHY1995","description":"This repository contains a pytorch implementation of \"HeadNeRF: A Real-time NeRF-based Parametric Head Model (CVPR 2022)\".","archived":false,"fork":false,"pushed_at":"2022-09-15T15:29:00.000Z","size":19758,"stargazers_count":431,"open_issues_count":16,"forks_count":48,"subscribers_count":18,"default_branch":"main","last_synced_at":"2024-11-07T03:31:33.774Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CrisHY1995.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-17T02:31:36.000Z","updated_at":"2024-10-30T14:59:49.000Z","dependencies_parsed_at":"2022-07-12T16:42:13.991Z","dependency_job_id":null,"html_url":"https://github.com/CrisHY1995/headnerf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CrisHY1995%2Fheadnerf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CrisHY1995%2Fheadnerf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CrisHY1995%2Fheadnerf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CrisHY1995%2Fheadnerf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CrisHY1995","download_url":"https://codeload.github.com/CrisHY1995/headnerf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322220,"owners_count":21084333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T03:25:56.577Z","updated_at":"2025-04-11T00:30:42.013Z","avatar_url":"https://github.com/CrisHY1995.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF Related Tasks"],"readme":"# HeadNeRF: A Real-time NeRF-based Parametric Head Model\n\n\u003cspan id=\"code_demo\"\u003e\u003c/span\u003e\n\nThis repository contains a pytorch implementation of \"HeadNeRF: A Real-time NeRF-based Parametric Head Model (CVPR 2022)\".\nAuthors: [Yang Hong](https://crishy1995.github.io/), Bo Peng, Haiyao Xiao, [Ligang Liu](http://staff.ustc.edu.cn/~lgliu/) and [Juyong Zhang*](http://staff.ustc.edu.cn/~juyong/).\n\n\n![](docs/code_demo.gif)\n\n### | [Project Page](https://crishy1995.github.io/HeadNeRF-Project/) | [Paper](https://arxiv.org/abs/2112.05637) | \n\n\nThis code has been tested on ubuntu 20.04/18.04 and contains the following parts:\n\n1. **An interactive GUI** that allows users to utilize HeadNeRF to directly edit the generated images’ rendering pose and various semantic attributes.  \n2. **A fitting framework** for obtaining the latent code embedding in HeadNeRF of a single image. \n\n## Requirements\n- python3\n- torch\u003e=1.8.1\n- torchvision\n- imageio\n- kornia\n- numpy\n- opencv-python==4.3.0.36\n- pyqt5\n- tqdm\n- face-alignment\n- Pillow, plotly, matplotlib, scipy, scikit-image\n    We recommend running the following commands to create an anaconda environment called \"headnerf\" and automatically install the above requirements.\n    ```bash\n    conda env create -f environment.yaml\n    conda activate headnerf\n    ```\n\n- [Pytorch](https://pytorch.org/get-started/locally/)\n  \n  Please refer to [pytorch](https://pytorch.org/get-started/locally/) for details. \n\n\n- [Pytorch3d](https://github.com/facebookresearch/pytorch3d)\n    \n    It is recommended to install pytorch3d from a local clone. \n    ```bash\n    \n    git clone https://github.com/facebookresearch/pytorch3d.git\n    cd pytorch3d \u0026\u0026 pip install -e . \u0026\u0026 cd ..\n    ```\n\u003c!-- \n    conda install -c bottler nvidiacub\n    conda install -c fvcore -c iopath -c conda-forge fvcore iopath --\u003e\n\nNote: \n- In order to run the code smoothly, a GPU with performance higher than 1080Ti is recommended. \n- This code can also be run on Windows 10 when the mentioned above requirements are satisfied.\n\u003c!-- - For NVIDIA 30 series GPUs, installing the pytorch with the specified version may be necessary to avoid compatibility issues. Please refer to [pytorch](https://pytorch.org/get-started/locally/) for details. --\u003e\n\n\u003c!-- ``` \npip3 install torch==1.8.2+cu111 torchvision==0.9.2+cu111 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html \n``` --\u003e\n\u003c!-- By testing, it is found that the following command is working for installing pytorch on NVIDIA 30 series GPUs.  --\u003e\n\n## Getting Started\nDownload [ConfigModels.zip](https://mailustceducn-my.sharepoint.com/:u:/g/personal/hymath_mail_ustc_edu_cn/EVROtiHQqNtLoAlLweyIFV0BDrLcGn_bpHnlM6tREfcQLQ?e=n0howy), [TrainedModels.zip](https://mailustceducn-my.sharepoint.com/:u:/g/personal/hymath_mail_ustc_edu_cn/EZ78EK8mylZBoGe-jCVibjEBVtoicnFvM0jpjaE56F4ihw?e=LC0SH7), and [LatentCodeSamples.zip](https://mailustceducn-my.sharepoint.com/:u:/g/personal/hymath_mail_ustc_edu_cn/EWNBh3ZVo7VFnO9uPRuDYAABbhrx8kyQa53fHv4szdgJ-A?e=bXiAfh), then unzip them to the root dir of this project.  \n\nOther links: [Google Drive](https://drive.google.com/drive/folders/1e-DWDrCB6mUcr6MnGbMX0M9YTVl2U6KO?usp=sharing), [One Drive](https://mailustceducn-my.sharepoint.com/:f:/g/personal/hymath_mail_ustc_edu_cn/Eld8rudJhVVMrgkl-OLmKjQBBZRUg_q-1O_kWizcGYpTjQ?e=xyfLfw)\n\nThe folder structure is as follows:\n```\nheadnerf\n├── ConfigModels\n│   ├── faceparsing_model.pth\n│   ├── nl3dmm_dict.pkl\n│   └── nl3dmm_net_dict.pth\n│\n├── TrainedModels\n│   ├── model_Reso32.pth\n│   ├── model_Reso32HR.pth\n│   └── model_Reso64.pth\n│\n└── LatentCodeSamples\n    ├── model_Reso32\n    │   ├── S001_E01_I01_P02.pth\n    │   └── ...\n    ├── model_Reso32HR\n    │   ├── S001_E01_I01_P02.pth\n    │   └── ...\n    └── model_Reso64\n        ├── S001_E01_I01_P02.pth\n        └── ...\n```\nNote:\n- **faceparsing_model.pth** is from [face-parsing.PyTorch](https://github.com/zllrunning/face-parsing.PyTorch), and we utilize it to help generate the head mask.\n- **nl3dmm_dict.pkl** and **nl3dmm_net_dict.pth** are from [3D face from X](https://arxiv.org/abs/1808.05323), and they are the parameters of 3DMM.\n- **model_Reso32.pth**, **model_Reso32HR.pth** and **model_Reso64.pth** are our pre-trained models, and their properties are as follows:  \n\n    |  Pre-trained Models   | Feature Map's Reso | Result's Reso | GPU 1080Ti |  GPU 3090 |\n    |  ----  | :-:   |  :-:   | :-:   | :-:   |\n    | model_Reso32  | 32 x 32 | 256 x 256 | \\~14fps | \\~40fps |\n    | model_Reso32HR  | 32 x 32 | 512 x 512 | \\~13fps | \\~30fps |\n    | model_Reso64  | 64 x 64 | 512 x 512 | \\~ 3fps | \\~10fps |\n\n- **LatentCodeSamples.zip** contains some latent codes that correspond to some given images.\n\n\n## The Interactive GUI\n\n```bash\n#GUI, for editing the generated images’ rendering pose and various semantic attributes.\npython MainGUI.py --model_path \"TrainedModels/model_Reso64.pth\"\n```\nArgs:\n- **model_path** is the path of the specified pre-trained model.\n\nAn interactive interface like [the first figure](#code_demo) of this document will be generated after executing the above command. \n\n\u003c!-- ![](docs/Demo-Gui.png) --\u003e\n\n\u003c!-- \u003cimg src='docs/main-gui.png' align='center' style=' BORDER:#000000 1px solid'/\u003e --\u003e\n\n## The fitting framework\n\nThis part provides a framework for fitting a single image using HeadNeRF. Besides, some test images are provided in test_data/single_images dir. These images are from [FFHQ dataset](https://github.com/NVlabs/ffhq-dataset) and do not participate in building HeadNeRF's models. \n\n#### Data Preprocess\n```bash\n# generating head's mask.\npython DataProcess/Gen_HeadMask.py --img_dir \"test_data/single_images\"\n\n# generating 68-facial-landmarks by face-alignment, which is from \n# https://github.com/1adrianb/face-alignment\npython DataProcess/Gen_Landmark.py --img_dir \"test_data/single_images\"\n\n# generating the 3DMM parameters\npython Fitting3DMM/FittingNL3DMM.py --img_size 512 \\\n                                    --intermediate_size 256  \\\n                                    --batch_size 9 \\\n                                    --img_dir \"test_data/single_images\"\n```\nThe generated results will be saved to the --img_dir.\n\n#### Fitting a Single Image\n\n```Bash\n# Fitting a single image using HeadNeRF\npython FittingSingleImage.py --model_path \"TrainedModels/model_Reso32HR.pth\" \\\n                             --img \"test_data/single_images/img_000037.png\" \\\n                             --mask \"test_data/single_images/img_000037_mask.png\" \\\n                             --para_3dmm \"test_data/single_images/img_000037_nl3dmm.pkl\" \\\n                             --save_root \"test_data/fitting_res\" \\\n                             --target_embedding \"LatentCodeSamples/*/S025_E14_I01_P02.pth\"\n``` \n\nArgs:\n- **para_3dmm** is the 3DMM parameter of the input image and is provided in advance to initialize the latent codes of the corresponding image.\n- **target_embedding** is a head's latent code embedding in HeadNeRF and is an optional input. If it is provided, we will perform linear interpolation on the fitting latent code embedding and the target latent code embedding, and the corresponding head images are generated using HeadNeRF. \n- **save_root** is the directory where the following results are saved.\n\nResults: \n- The image that merges the input image and the fitting result.\n- The dynamic image generated by continuously changing the rendering pose of the fitting result.\n- The dynamic image generated by performing linear interpolation on the fitting latent code embedding and the target latent code embedding.\n- The latent codes (.pth file) of the fitting result.\n\nNote:\n- Fitting a single image based on **model_Reso32.pth** requires more than **~5 GB** GPU memory.\n- Fitting a single image based on **model_Reso32HR.pth** requires more than **~6 GB** GPU memory.\n- Fitting a single image based on **model_Reso64.pth** requires more than **~13 GB** GPU memory.\n\n## Citation\nIf you find our work useful in your research, please consider citing our paper:\n```\n@inproceedings{hong2021headnerf,\n     author     = {Yang Hong and Bo Peng and Haiyao Xiao and Ligang Liu and Juyong Zhang},\n     title      = {HeadNeRF: A Real-time NeRF-based Parametric Head Model},\n     booktitle  = {{IEEE/CVF} Conference on Computer Vision and Pattern Recognition (CVPR)},\n     year       = {2022}\n  }\n```\nIf you have questions, please contact hymath@mail.ustc.edu.cn.\n\n## Acknowledgments\n- We use [face-alignment](https://github.com/1adrianb/face-alignment) for detecting 68-facial-landmarks. \n- We use [face-parsing.PyTorch](https://github.com/zllrunning/face-parsing.PyTorch) for generating the head mask. \n- The 3DMM that we use is from [3D face from X](https://arxiv.org/abs/1808.05323) and [Noliner3DMM](https://openaccess.thecvf.com/content_cvpr_2018/papers/Tran_Nonlinear_3D_Face_CVPR_2018_paper.pdf). \n- The code of fitting a single image using 3DMM is modified from [3DMM-Fitting-Pytorch](https://github.com/ascust/3DMM-Fitting-Pytorch).\n\n\n## License\nAcademic or non-profit organization noncommercial research use only.\n\n\n\u003c!-- - This research was supported by the National Key R\u0026D Program of China (2020YFC1523102), the National Natural Science Foundation of China (No.62122071, 62025207), the Youth Innovation Promotion Association CAS (No. 2018495) and the Fundamental Research Funds for the Central Universities (No.WK3470000021). --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCrisHY1995%2Fheadnerf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCrisHY1995%2Fheadnerf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCrisHY1995%2Fheadnerf/lists"}