{"id":20984491,"url":"https://github.com/mrtornado24/next3d","last_synced_at":"2025-05-16T02:08:48.888Z","repository":{"id":63579701,"uuid":"568927674","full_name":"MrTornado24/Next3D","owner":"MrTornado24","description":"[CVPR 2023 Highlight] Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars","archived":false,"fork":false,"pushed_at":"2024-10-13T03:55:36.000Z","size":2441,"stargazers_count":482,"open_issues_count":25,"forks_count":31,"subscribers_count":27,"default_branch":"master","last_synced_at":"2025-04-08T13:05:42.106Z","etag":null,"topics":["3d-aware-image-synthesis","gan","nerf"],"latest_commit_sha":null,"homepage":"https://mrtornado24.github.io/Next3D/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MrTornado24.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-21T17:56:20.000Z","updated_at":"2025-04-05T08:04:26.000Z","dependencies_parsed_at":"2024-11-19T05:55:45.682Z","dependency_job_id":"66506124-5cdc-4c5b-a925-ebdbab0f817b","html_url":"https://github.com/MrTornado24/Next3D","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrTornado24%2FNext3D","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrTornado24%2FNext3D/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrTornado24%2FNext3D/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrTornado24%2FNext3D/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MrTornado24","download_url":"https://codeload.github.com/MrTornado24/Next3D/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254453655,"owners_count":22073617,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-aware-image-synthesis","gan","nerf"],"created_at":"2024-11-19T05:53:40.459Z","updated_at":"2025-05-16T02:08:48.870Z","avatar_url":"https://github.com/MrTornado24.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars\n\n![Teaser image](./docs/teaser_v3.jpeg)\n\n**Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars**\u003cbr\u003e\n[Jingxiang Sun](https://mrtornado24.github.io/), [Xuan Wang](https://xuanwangvc.github.io/), [Lizhen Wang](https://lizhenwangt.github.io/), [Xiaoyu Li](https://xiaoyu258.github.io/), [Yong Zhang](https://yzhang2016.github.io/yongnorriszhang.github.io/), [Hongwen Zhang](https://hongwenzhang.github.io/), [Yebin Liu](http://www.liuyebin.com/)\u003cbr\u003e\u003cbr\u003e\n\n\n[**Project**](https://mrtornado24.github.io/Next3D/) | [**Paper**](https://arxiv.org/abs/2211.11208) | [**Twitter**](https://twitter.com/JingxiangSun42/status/1630489816226988032?s=20)\n\nAbstract: *3D-aware generative adversarial networks (GANs) synthesize high-fidelity and multi-view-consistent facial images using only collections of single-view 2D imagery. Towards fine-grained control over facial attributes, recent efforts incorporate 3D Morphable Face Model (3DMM) to describe\ndeformation in generative radiance fields either explicitly or implicitly. Explicit methods provide fine-grained expression control but cannot handle topological changes caused\nby hair and accessories, while implicit ones can model varied topologies but have limited generalization caused by the\nunconstrained deformation fields. We propose a novel 3D\nGAN framework for unsupervised learning of generative,\nhigh-quality and 3D-consistent facial avatars from unstructured 2D images. To achieve both deformation accuracy\nand topological flexibility, we propose a 3D representation\ncalled Generative Texture-Rasterized Tri-planes. The proposed representation learns Generative Neural Textures on\ntop of parametric mesh templates and then projects them\ninto three orthogonal-viewed feature planes through rasterization, forming a tri-plane feature representation for volume rendering. In this way, we combine both fine-grained\nexpression control of mesh-guided explicit deformation and the flexibility of implicit volumetric representation. We further propose specific modules for modeling mouth interior\nwhich is not taken into account by 3DMM. Our method demonstrates state-of-the-art 3D-aware synthesis quality and animation ability through extensive experiments. Furthermore, serving as 3D prior, our animatable 3D repre-\nsentation boosts multiple applications including one-shot facial avatars and 3D-aware stylization.*\n\n## News\n[Oct 2024] We released the [code](https://github.com/XChenZ/invertAvatar) for our new SIGGRAPH 2024 paper: [\"InvertAvatar: Incremental GAN Inversion for Generalized Head Avatars\"](https://xchenz.github.io/invertavatar_page/)! InvertAvatar is built on an improved Next3D backbone and enables building high-quality 3D personal head avatar in 1s.\n\n## Requirements\n\n* 1\u0026ndash;8 high-end NVIDIA GPUs. We have done all testing and development using V100, RTX3090, and A100 GPUs.\n* 64-bit Python 3.9 and PyTorch 1.12.0 (or later). See https://pytorch.org for PyTorch install instructions.\n* CUDA toolkit 11.3 or later.\n* Python libraries: see [environment.yml](./environment.yml) for exact library dependencies.  You can use the following commands with Miniconda3 to create and activate your Python environment:\n  - `cd Next3D`\n  - `conda env create -f environment.yml`\n  - `conda activate next3d`\n\n## Getting started\n\nDownload our pretrained models following [the link](https://drive.google.com/drive/folders/1rbR5ZJ6LQYUSd5J5BkoVYNon_-Lb7KsZ?usp=share_link) and put it under `pretrained_models`. For training Next3D on the top of EG3D, please also download the pretrained checkpoint `ffhqrebalanced512-64.pkl` of [EG3D](https://github.com/NVlabs/eg3d/blob/main/docs/models.md).\n\n\n## Generating media\n\n```.bash\n# Generate videos for the shown cases using pre-trained model\n\npython gen_videos_next3d.py --outdir=out --trunc=0.7 --seeds=10720,12374,13393,17099 --grid=2x2 \\\n    --network=pretrained_models/next3d_ffhq_512.pkl --obj_path=data/demo/demo.obj \\\n    --lms_path=data/demo/demo_kpt2d.txt --lms_cond=True\n```\n\n```.bash\n# Generate images and shapes (as .mrc files) for the shown cases using pre-trained model\n\npython gen_samples.py --outdir=out --trunc=0.7 --shapes=true --seeds=166 \\\n    --network=pretrained_models/next3d_ffhq_512.pkl --obj_path=data/demo/demo.obj \\\n    --lms_path=data/demo/demo_kpt2d.txt --lms_cond=True\n```\n\nWe visualize our .mrc shape files with [UCSF Chimerax](https://www.cgl.ucsf.edu/chimerax/). Please refer to [EG3D](https://github.com/NVlabs/eg3d) for more detailed instructions.\n\n\n## Reenacting generative avatars\n\n### Installation\n\nEnsure the [Deep3DFaceRecon_pytorch](https://github.com/sicxu/Deep3DFaceRecon_pytorch/tree/6ba3d22f84bf508f0dde002da8fff277196fef21) submodule is properly initialized\n```.bash\ngit submodule update --init --recursive\n```\nDownload the pretrained models for FLAME estimation following [DECA](https://github.com/yfeng95/DECA) and put them into `dataset_preprocessing/ffhq/deca/data`; download the pretrained models for gaze estimation through the [link](https://drive.google.com/drive/folders/1Jgej9q5W2IYXRa-CWCldyTVXeHk-Oi-I?usp=share_link) and put them into `dataset_preprocessing/ffhq/faceverse/data`.\n\n### Preparing datasets\n\nThe video reenactment input contains three parts: camera poses `dataset.json`, FLAME meshes ('.obj') and 2D landmark files ('.txt'). For quick start, you can download the processed talking video of President Obama [here](https://drive.google.com/file/d/1ph77uSlLz-xIVlBxwXP3Et7lTR0zHXQR/view?usp=sharing) and place the downloaded folder as `data/obama`. You can also preprocess your custom datasets by running the following commands:\n\n```.bash\ncd dataset_preprocessing/ffhq\npython preprocess_in_the_wild.py --indir=INPUT_IMAGE_FOLDER\n```\n\nYou will obtain FLAME meshes and 2D landmark files for frames and a 'dataset.json'. Please put all these driving files into a same folder for reenactment later. \n\n\n### Reenacting samples\n\n```.bash\npython reenact_avatar_next3d.py --drive_root=data/obama \\\n  --network=pretrained_models/next3d_ffhq_512.pkl \\\n  --grid=2x1 --seeds=166 --outdir=out --fname=reenact.mp4 \\\n  --trunc=0.7 --lms_cond=1\n```\n\n\n## Training\n\n\nDownload and process [Flickr-Faces-HQ dataset](https://github.com/NVlabs/ffhq-dataset) using the following commands. \n```.bash\ncd dataset_preprocessing/ffhq\npython runme.py\n```\nYou can perform FLAME and landmarks estimation referring to [preprocess_in_the_wild.py](./dataset_preprocessing/ffhq/preprocess_in_the_wild.py). We will also integrate all the preprocessing steps into a script soon. \nThe dataset should be organized as below:\n```\n    ├── /path/to/dataset\n    │   ├── meshes512x512\n    │   ├── lms512x512\n    │   ├── images512x512\n    │   │   ├── 00000\n                ├──img00000000.png\n    │   │   ├── ...\n    │   │   ├── dataset.json\n```\n\nYou can train new networks using `train_next3d.py`. For example:\n\n```.bash\n# Train with FFHQ on the top of EG3D with raw neural rendering resolution=64, using 8 GPUs.\npython train_next3d.py --outdir=~/training-runs --cfg=ffhq --data=data/ffhq/images512x512 \\\n  --rdata data/ffhq/meshes512x512 --gpus=8 --batch=32 --gamma=4 --topology_path=data/demo/head_template.obj \\\n  --gen_pose_cond=True --gen_exp_cond=True --disc_c_noise=1 --load_lms=True --model_version=next3d \\\n  --resume pretrained_models/ffhqrebalanced512-64.pkl\n```\nNote that rendering-conditioned discriminator is not supported currently because obtaining rendering is still time-consuming. We are trying to accelerate this process and the training code will keep updating.\n  \n## One-shot portrait reenactment and stylization\n\nCode will come soon.\n\n## Citation\n\n```\n@inproceedings{sun2023next3d,\n  author = {Sun, Jingxiang and Wang, Xuan and Wang, Lizhen and Li, Xiaoyu and Zhang, Yong and Zhang, Hongwen and Liu, Yebin},\n  title = {Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars},\n  booktitle = {CVPR},\n  year = {2023}\n}\n```\n\n```\n@inproceedings{10.1145/3641519.3657478,\nauthor = {Zhao, Xiaochen and Sun, Jingxiang and Wang, Lizhen and Suo, Jinli and Liu, Yebin},\ntitle = {InvertAvatar: Incremental GAN Inversion for Generalized Head Avatars},\nyear = {2024},\nisbn = {9798400705250},\npublisher = {Association for Computing Machinery},\naddress = {New York, NY, USA},\nurl = {https://doi.org/10.1145/3641519.3657478},\ndoi = {10.1145/3641519.3657478},\nbooktitle = {ACM SIGGRAPH 2024 Conference Papers},\narticleno = {59},\nnumpages = {10},\nkeywords = {3D head avatar, GAN inversion, few-shot reconstruction, one-shot reconstruction, recurrent neural network},\nlocation = {Denver, CO, USA},\nseries = {SIGGRAPH '24}\n}\n\n\n```\n\n## Acknowledgements\n\nPart of the code is borrowed from [EG3D](https://github.com/NVlabs/eg3d) and [DECA](https://github.com/yfeng95/DECA).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrtornado24%2Fnext3d","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrtornado24%2Fnext3d","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrtornado24%2Fnext3d/lists"}