{"id":13958379,"url":"https://github.com/tencent-ailab/hifi3dface","last_synced_at":"2025-04-04T10:05:22.102Z","repository":{"id":41188206,"uuid":"302597513","full_name":"tencent-ailab/hifi3dface","owner":"tencent-ailab","description":"Code and data for our paper \"High-Fidelity 3D Digital Human Creation from RGB-D Selfies\". ","archived":false,"fork":false,"pushed_at":"2021-06-29T09:56:59.000Z","size":18318,"stargazers_count":779,"open_issues_count":36,"forks_count":154,"subscribers_count":36,"default_branch":"main","last_synced_at":"2024-12-20T15:12:23.963Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tencent-ailab.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-09T09:41:27.000Z","updated_at":"2024-12-20T08:59:34.000Z","dependencies_parsed_at":"2022-07-21T21:32:36.494Z","dependency_job_id":null,"html_url":"https://github.com/tencent-ailab/hifi3dface","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tencent-ailab%2Fhifi3dface","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tencent-ailab%2Fhifi3dface/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tencent-ailab%2Fhifi3dface/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tencent-ailab%2Fhifi3dface/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tencent-ailab","download_url":"https://codeload.github.com/tencent-ailab/hifi3dface/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231508406,"owners_count":18387386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-08T13:01:31.088Z","updated_at":"2024-12-27T16:08:00.647Z","avatar_url":"https://github.com/tencent-ailab.png","language":"Python","funding_links":[],"categories":["人像\\姿势\\3D人脸"],"sub_categories":["网络服务_其他"],"readme":"# High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies (ACM Transactions on Graphics 2021)\n\n\n\n\u003cdiv\u003e\n\u003cdiv align=left\u003e\u003cimg src=\"figures/teaser.png\" alt=\"teaser\" width=80%\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\nThis repository provides the python code and 3DMM of the following paper:\n\n\u003e **High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies**.\\\n\u003e Linchao Bao*, Xiangkai Lin*, Yajing Chen*, Haoxian Zhang*, Sheng Wang, Xuefei Zhe, Di Kang, Haozhi Huang, Xinwei Jiang, Jue Wang, Dong Yu, and Zhengyou Zhang. (*Equal contribution) \\\n\u003e **ACM Transactions on Graphics 2021** \\\n\u003e Project Page: https://tencent-ailab.github.io/hifi3dface_projpage/ \\\n\u003e arXiv: https://arxiv.org/abs/2010.05562\n\n\u003e **Abstract**: *We present a fully automatic system that can produce high-fidelity, photo-realistic 3D digital human heads with a consumer RGB-D selfie camera. The system only needs the user to take a short selfie RGB-D video while rotating his/her head, and can produce a high quality head reconstruction in less than 30 seconds. Our main contribution is a new facial geometry modeling and reflectance synthesis procedure that significantly improves the state-of-the-art. Specifically, given the input video a two-stage frame selection procedure is first employed to select a few high-quality frames for reconstruction. Then a differentiable renderer based 3D Morphable Model (3DMM) fitting algorithm is applied to recover facial geometries from multiview RGB-D data, which takes advantages of a powerful 3DMM basis constructed with extensive data generation and perturbation. Our 3DMM has much larger expressive capacities than conventional 3DMM, allowing us to recover more accurate facial geometry using merely linear basis. For reflectance synthesis, we present a hybrid approach that combines parametric fitting and CNNs to synthesize high-resolution albedo/normal maps with realistic hair/pore/wrinkle details. Results show that our system can produce faithful 3D digital human faces with extremely realistic details. The main code and the newly constructed 3DMM basis is publicly available.*\n\n**Note**: *Please note that the reported running time in the paper are obtained with our optimized C++ code but NOT the python code here. The python code in this repository is only for research purposes and not fully optimized.*\n\n**Note**: Please visit the **./3DMM** folder for details about our 3DMM, namely **Tencent AI-NEXT 3D Face Model (East-Asian Version)**, in short **AI-NEXT Face Model (EAs)**.\n\n## Disclaimer\n\n*The code and data in this repository are for research purpose only. No representation or warranty whatsoever, expressed or implied, is made as to its accuracy, reliability or completeness. We assume no liability and are not responsible for any misuse or damage caused by the code and data. Your use of the code and data are subject to applicable laws and your use of them is at your own risk.*\n\n\n\u003cbr\u003e\n\n\u003cdiv\u003e\n\u003cdiv align=left\u003e\u003cimg src=\"figures/optimization.png\" alt=\"teaser\" width=80%\u003e\n\u003c/div\u003e\n\n\u003cdiv\u003e\n\u003cdiv align=left\u003e\u003cimg src=\"figures/HD_UV_norm.png\" alt=\"teaser\" width=80%\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n## Directories\n\n**3DMM**: \\\n    our constructed 3DMM from 200 high-fidelity East-Asian face models, see the README.md file inside the folder for more details. \n\n**data_prepare**:  \\\n    code for face detection, landmark detection and face segmentation to preprocess image and generate necessary information for 3DMM optimization. \n\n**optimization**: \\\n    3DMM optimization code for facial geometry modeling. \n\n**texture**:  \\\n    code for facial reflectance synthesis. \n    \n**third_party**:  \\\n    code from the other repositories. \n    \n\n## Prerequisites\n\n- **Dependencies**\n    - Linux\n    - python 3.6\n    - tensorflow 1.8 (also tested in tensorflow 1.15)\n    - g++ 4.8 (other version might also work)\n\n- **Installation**\n\n    Please run the follow commands for installation.\n    ```\n    git clone https://github.com/tencent-ailab/hifi3dface.git\n    cd hifi3dface\n    bash install.sh\n    ```\n    The script in **install.sh** installs all packages required to run our code, and compiles the c++ kernel of the differentiable renderer provided by [tf_mesh_renderer](https://github.com/google/tf_mesh_renderer).\n    Pay attention to finding the correct path to **TF_INC** and **TF_LIB**. The path specified in **install.sh** might not suit your case. If it does not work, please try to fix them manually. If your g++ version is greater than 4, please delete \"-D_GLIBCXX_USE_CXX11_ABI=0\" ( Line 9 ) in **install.sh**.\n    You can also compile the code using the method provided by [**tf_mesh_renderer**](https://github.com/google/tf_mesh_renderer). \n\n- **Downloads**\n\n    You will need to download the following files to run our code: \n\n    - Download our **3DMM** (5.4GB) and unzip it to the **./3DMM** floder: [Google Drive](https://drive.google.com/file/d/1vSb2EpduRJuIEUOc_aRsbjASSOUAk7tG/view?usp=sharing), or [**腾讯微云**](https://share.weiyun.com/ByrUzsJG)\n    - Download the **resources.zip** (2GB) into the repo root directory **./** and unzip it: [Google Drive](https://drive.google.com/file/d/1FoBbj1qWdf2LyJkxLYnMJ4h4woTQ6QL_/view?usp=sharing), or [**腾讯微云**](https://share.weiyun.com/Hgov20It)\n    - Download the **test_data.zip** (80MB) into the repo root directory **./** and unzip it: [Google Drive](https://drive.google.com/file/d/1f5imMCO0ug_ozj0cDcNYtizom5lBu2PH/view?usp=sharing), or [**腾讯微云**](https://share.weiyun.com/5PCy1rID)\n\n\n## Running The Code\n\n### 1. RGBD Face Reconstruction \n\nTo reproduce the results in our paper, please follow the steps provided below.\n\n\n**1.1 Data acquisition:**\n\nYou may use our [RGBD data capture code](https://github.com/lxk121lalala/RGBD_data_capture) (need an iPhone with TrueDepth camera) to capture RGB-D selfies. The color and depth images are in JPG and PNG formats respectively. The resolution of the color image is 1504 x 1128, and the depth image is of size 640 x 480. If you want to use your own capturing system, please modify the code to suit your case. Below are example RGB-D images exported from our capturing app:\n\u003cdiv\u003e\n\u003cdiv align=left\u003e\u003cimg src=\"figures/RGBD_data_example.png\" alt=\"RGBD_data_example\" width=80%\u003e\n\u003c/div\u003e\n\n**1.2 Configuration:**\n\nModify **run_rgbd.sh** as following:\n\n1. Line 6: define the path to **3DMM** as where you download the 3DMM model. \n    ```bash\n    basic_path=$(pwd)/3DMM/files/;\n    ```\n2. Line 7: define the path to **resources** as where you download the data. \n    ```bash\n    resources_path=$(pwd)/resources/;\n    ```\n3. Line 24: define the path to **test data**. If you capture your own data, please specify the path to the video sequence or 4 pairs of (color, depth) images with different poses (mid-left-right-up).\n    ```bash\n    ROOT_DIR=$(pwd)/test_data/RGBD/test1/;\n    ```\n4. Line 23: define whether testing data is a video sequence. \n    ```bash\n    is_only_four_frame=\"False\" #  True for four frames, False for a video sequence\n    ```\n5. Line 121-136: configure optimization parameters.\n    ```bash\n    log_step=10\n    learning_rate=0.05\n    lr_decay_step=10\n    lr_decay_rate=0.9\n    photo_weight=100\n    gray_photo_weight=80\n    reg_shape_weight=0.4\n    reg_tex_weight=0.0001\n    depth_weight=1000\n    id_weight=1.8\n    real_86pt_lmk3d_weight=0.01\n    lmk_struct_weight=0\n    train_step=100\n    is_fixed_pose=\"False\"\n    is_add_head_mirrow=\"False\" #  True: output symmetrical head completion result\n    is_add_head_male=\"True\" #  True: complete head with a male template\n    ```\n\n**1.3 Run:**\n\nPlease run the follow command to generate the results.\n```bash\nbash run_opt_rgbd.sh\n```\n\nThe RGBD data will be used for shape optimization and HD texture/normal maps generation. We highly recommend that you use our 3DMM version **AI-NExT-Shape.mat** in order to achieve the same results as reported in our paper. You can also run our code with **BFM**, but the texture map and normal map will not be generated if you use BFM, as BFM does not provide UV maps.\n\n**1.4 Results:**\n\nThe produced results will be saved in the **results** floder in data path, including: \n\n    - head.obj: the final mesh file\n    - albedo.png: the final albedo map file\n    - normal.png: the final normal map file\n    - test.mtl: a material description file for simple rendering in meshlab\n\nAn example of the resulting files is shown below. Note that the our 3DMM version **AI-NExT-Shape.mat** only contains face region, and our code consists of a head completion step. \n\n\u003cdiv\u003e\n\u003cdiv align=left\u003e\u003cimg src=\"figures/RGBD_example.png\" alt=\"RGBD_example\" width=80%\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n\n### 2. RGB Face Reconstruction\n\nOur code also supports RGB inputs without depth data, but **note** that running with only RGB inputs cannot achieve comparable reconstruction accuracy as RGBD inputs.\n\n**2.1 Data acquisition:**\nYou may use one image or multiple images of the same person to run our code. Please note that all data of the same identity should be placed into the same folder.\n\n**2.2 Configuration:**\n\nPlease modify **run_rgb.sh** as the following:\n\n1. Line 6: define the path to **3DMM** as where you download the 3DMM model. \n  ```bash\n  basic_path=$(pwd)/3DMM/files/;\n  ```\n2. Line 7: define the path to **resources**. \n  ```bash\n  resources_path=$(pwd)/resources/;\n  ```\n3. Line 22: define the path to **test data**. You may also specify the path to your own input rgb data folder.\n  ```bash\n  ROOT_DIR=$(pwd)/test_data/RGB/test1/single_img/;\n  ```\n4. Line 54-70: configure optimization parameters.\n  ```bash\n  train_step=120\n  log_step=20\n  learning_rate=0.05\n  lr_decay_step=20\n  lr_decay_rate=0.9\n\n  photo_weight=100.0\n  gray_photo_weight=80.0\n  reg_shape_weight=4.0\n  reg_tex_weight=8.0\n  id_weight=1.5\n  real_86pt_lmk3d_weight=5.0\n  real_68pt_lmk2d_weight=5.0\n  lmk_struct_weight=0\n\n  num_of_img=1 # The number of input images\n  project_type=\"Pers\" # Use \"Pers\" for selfie photos, and \"Orth\" for face photos captured in a far distance\n  ```\n\n**2.3 Run:**\n\nPlease run the follow command to generate the results.\n```bash\nbash run_opt_rgb.sh\n```\n\nNote that if you run the script with more than one images, only the first image would be used for generating the texture map and normal map. Besides, you can also run the code with **BFM**, but the texture map and normal map will not be generated in this case. We highly recommend that you use the our 3DMM version **AI-NExT-Shape-NoAug.mat** for the RGB inputs, as there are no reliable geometric constraints without depth data inputs. \n\n\n**2.4 Results:**\n \nThe produced results will be saved in the **results** floder in data path, including: \n\n    - head.obj: the final mesh file\n    - albedo.png: the final albedo map file\n    - normal.png: the final normal map file\n    - test.mtl: a material description file for simple rendering in meshlab\n\nAn example of the resulting files is shown below. Note that the shape model obtained with **AI-NExT-Shape-NoAug.mat** is already a full head model. \n\u003cdiv\u003e\n\u003cdiv align=left\u003e\u003cimg src=\"figures/RGB_example.png\" alt=\"RGB_example\" width=80%\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n\n### 3. Rendering\n\nThe produced mesh and albedo/normal map can be used to render a realistic digital human head in a physically based rendering engine like [Unreal Engine 4](https://docs.unrealengine.com/en-US/Resources/Showcases/DigitalHumans/index.html) (UE4). The results in our paper are rendered using UE4 engine, with the skin material settings the same as the [digital human rendering showcase \"Mike\"](https://docs.unrealengine.com/en-US/Resources/Showcases/DigitalHumans/index.html) provided by UE4. \n\nFor simpler visualization, we here provide two other rendering ways as follows. \n\n  - **Rendering with Meshlab**: You can directly use [Meshlab](https://www.meshlab.net/) to open the obj file, with the mtl file in a same folder. Note that normal maps are not used in the rendering. \n  - **Rendering with Three.js**: Please follow the instructions in this repo(https://github.com/cyj907/simple-mesh-visualizer). Normal maps can be used for rendering in this way. \n\nThe rendering results are as follows (eyeballs are attached in the UE4 rendering): \n\n\u003cdiv\u003e\n\u003cdiv align=left\u003e\u003cimg src=\"figures/render_results.png\" alt=\"render_results\" width=80%\u003e\n\u003c/div\u003e\n\n\u003e Note: The renderings are with different lighting settings. The mesh and albedo/normal map are the same (from RGBD reconstruction) in the renderings. \n\n\u003cbr\u003e\n\n\n## Contact\n\nIf you have any questions, please contact [Linchao Bao](https://linchaobao.github.io/).\n\n\n## Acknowledgement\n\nOur code uses the code from the following repos. We thank them for providing the code.\n\n- **tf_mesh_renderer**: [https://github.com/google/tf_mesh_renderer](https://github.com/google/tf_mesh_renderer)\n- **vgg face**: [https://github.com/ZZUTK/Tensorflow-VGG-face](https://github.com/ZZUTK/Tensorflow-VGG-face)\n- **pix2pix**: [https://github.com/affinelayer/pix2pix-tensorflow](https://github.com/affinelayer/pix2pix-tensorflow)\n- **face segmentation**: [https://github.com/switchablenorms/CelebAMask-HQ](https://github.com/switchablenorms/CelebAMask-HQ)\n- **N-ICP**: [https://github.com/charlienash/nricp](https://github.com/charlienash/nricp)\n\n\n## Citation\n\nIf you use the code or 3DMM provided in this repository, please cite our paper as follows.\n\n```\n@article{hifi3dface2021tencentailab,\n  title={High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies},\n  author={Bao, Linchao and Lin, Xiangkai and Chen, Yajing and Zhang, Haoxian and Wang, Sheng and Zhe, Xuefei and Kang, Di and Huang, Haozhi and Jiang, Xinwei and Wang, Jue and Yu, Dong and Zhang, Zhengyou},\n  journal={ACM Transactions on Graphics},\n  year={2021}\n}\n```\n\n## License\n\n```\nMIT License\n\nCopyright (c) [2020-2021] [Tencent AI Lab]\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencent-ailab%2Fhifi3dface","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftencent-ailab%2Fhifi3dface","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencent-ailab%2Fhifi3dface/lists"}