{"id":18488382,"url":"https://github.com/choyingw/synergynet","last_synced_at":"2025-04-08T12:08:58.616Z","repository":{"id":37653123,"uuid":"419189286","full_name":"choyingw/SynergyNet","owner":"choyingw","description":"3DV 2021: Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry","archived":false,"fork":false,"pushed_at":"2024-12-11T10:06:40.000Z","size":41077,"stargazers_count":395,"open_issues_count":14,"forks_count":57,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-04-01T11:02:05.969Z","etag":null,"topics":["2d-3d","3d","3d-face-alignment","3d-face-reconstruction","3dv2021","3dvision","computer-graphics","computer-vision","deep-neural-networks","facial-keypoints","facial-landmarks","head-pose-estimation","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/choyingw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-20T04:51:29.000Z","updated_at":"2025-03-17T08:25:22.000Z","dependencies_parsed_at":"2022-07-20T14:32:35.864Z","dependency_job_id":"93a5ba23-2cda-4e24-98d7-794b246eb1d5","html_url":"https://github.com/choyingw/SynergyNet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choyingw%2FSynergyNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choyingw%2FSynergyNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choyingw%2FSynergyNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/choyingw%2FSynergyNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/choyingw","download_url":"https://codeload.github.com/choyingw/SynergyNet/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247838444,"owners_count":21004580,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["2d-3d","3d","3d-face-alignment","3d-face-reconstruction","3dv2021","3dvision","computer-graphics","computer-vision","deep-neural-networks","facial-keypoints","facial-landmarks","head-pose-estimation","pytorch"],"created_at":"2024-11-06T12:51:38.508Z","updated_at":"2025-04-08T12:08:58.585Z","avatar_url":"https://github.com/choyingw.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"#  \u003cdiv align=\"center\"\u003e SynergyNet\u003c/div\u003e\n3DV 2021: Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry\n\nCho-Ying Wu, Qiangeng Xu, Ulrich Neumann, CGIT Lab at University of Souther California\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/synergy-between-3dmm-and-3d-landmarks-for/face-alignment-on-aflw)](https://paperswithcode.com/sota/face-alignment-on-aflw?p=synergy-between-3dmm-and-3d-landmarks-for)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/synergy-between-3dmm-and-3d-landmarks-for/head-pose-estimation-on-aflw2000)](https://paperswithcode.com/sota/head-pose-estimation-on-aflw2000?p=synergy-between-3dmm-and-3d-landmarks-for)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/synergy-between-3dmm-and-3d-landmarks-for/face-alignment-on-aflw2000-3d)](https://paperswithcode.com/sota/face-alignment-on-aflw2000-3d?p=synergy-between-3dmm-and-3d-landmarks-for)\n\n[\u003ca href=\"https://arxiv.org/abs/2110.09772\"\u003epaper\u003c/a\u003e] [\u003ca href=\"https://youtu.be/i1Y8U2Z20ko\"\u003evideo\u003c/a\u003e] [\u003ca href=\"https://choyingw.github.io/works/SynergyNet/index.html\"\u003eproject page\u003c/a\u003e]\n\nNews [Dec 11, 2024]: Add details of training UV-texture GAN. See the section \"Training UV-texture GAN\"\n\nNews [Jul 10, 2022]: Add simplified api for getting 3d landmarks, face mesh, and face pose in only one line. See \"Simplified API\" It's convenient if you simply want to plug in this method in your work.\n\nNews: Add Colab demo \n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1q9HRLA3wGxz4IFIseZFK1maOyH0wutYk?usp=sharing)\n\nNews: Our new work [\u003ca href=\"https://github.com/choyingw/Voice2Mesh\"\u003eCross-Modal Perceptionist\u003c/a\u003e] is accepted to CVPR 2022, which is based on this SynergyNet project.\u003cbr\u003e\n\n\u003cimg src='demo/demo.gif'\u003e\n\n\u003cimg src='demo/teaser.png'\u003e\n\n\u003cimg src='demo/multiple.png'\u003e\n\n\u003cimg src='demo/single.png'\u003e\n\n## \u003cdiv align=\"center\"\u003e Advantages\u003c/div\u003e\n\n:+1: SOTA on all 3D facial alignment, face orientation estimation, and 3D face modeling.\u003cbr\u003e\u003cbr\u003e\n:+1: Fast inference with 3000fps on a laptop RTX 2080.\u003cbr\u003e\u003cbr\u003e\n:+1: Simple implementation with only widely used operations.\u003cbr\u003e\u003cbr\u003e\n\n(This project is built/tested on Python 3.8 and PyTorch 1.9 on a compatible GPU)\n\n## \u003cdiv align=\"center\"\u003e Single Image Inference Demo\u003c/div\u003e\n\n1. Clone\n\n    ```git clone https://github.com/choyingw/SynergyNet```\n\n    ```cd SynergyNet ```\n\n2. Use conda\n\n    ```conda create --name SynergyNet```\n\n    ```conda activate SynergyNet```\n\n3. Install pre-requisite common packages\n\n    ```PyTorch 1.9 (should also be compatiable with 1.0+ versions), Torchvision, Opencv, Scipy, Matplotlib, Cython ```\n\n4. Download data [\u003ca href=\"https://drive.google.com/file/d/1YVBRcXmCeO1t5Bepv67KVr_QKcOur3Yy/view?usp=sharing\"\u003ehere\u003c/a\u003e] and\n[\u003ca href=\"https://drive.google.com/file/d/1SQsMhvAmpD1O8Hm0yEGom0C0rXtA0qs8/view?usp=sharing\"\u003ehere\u003c/a\u003e]. Extract these data under the repo root.\n\nThese data are processed from [\u003ca href=\"https://github.com/cleardusk/3DDFA\"\u003e3DDFA\u003c/a\u003e] and [\u003ca href=\"https://github.com/shamangary/FSA-Net\"\u003eFSA-Net\u003c/a\u003e].\n\nDownload pretrained weights [\u003ca href=\"https://drive.google.com/file/d/1BVHbiLTfX6iTeJcNbh-jgHjWDoemfrzG/view?usp=sharing\"\u003ehere\u003c/a\u003e]. Put the model under 'pretrained/'\n\n5. Compile Sim3DR and FaceBoxes:\n\n    ```cd Sim3DR```\n\n    ```./build_sim3dr.sh```\n\n    ```cd ../FaceBoxes```\n\n    ```./build_cpu_nms.sh```\n\n    ```cd ..```\n\n6. Inference\n\n    ```python singleImage.py -f img```\n\nThe default inference requires a compatible GPU to run. If you would like to run on a CPU, please comment the .cuda() and load the pretrained weights into cpu.\n\n## \u003cdiv align=\"center\"\u003e Simplified API\u003c/div\u003e\n\nWe provide a simple API for convenient usage if you want to plug in this method into your work.\n\n```python\nimport cv2\nfrom synergy3DMM import SynergyNet\nmodel = SynergyNet()\nI = cv2.imread(\u003cyour image path\u003e)\n# get landmark [[y, x, z], 68 (points)], mesh [[y, x, z], 53215 (points)], and face pose (Euler angles [yaw, pitch, roll] and translation [y, x, z])\nlmk3d, mesh, pose = model.get_all_outputs(I)\n```\nWe provide a simple script in singleImage_simple.py\n\nWe also provide a setup.py file. Run \u003ccode\u003epip install -e . \u003c/code\u003e You can do \u003ccode\u003efrom synergy3DMM import SynergyNet\u003c/code\u003e in other directory. Note that [\u003ca href=\"https://drive.google.com/file/d/1SQsMhvAmpD1O8Hm0yEGom0C0rXtA0qs8/view?usp=sharing\"\u003e3dmm_data\u003c/a\u003e] and [\u003ca href=\"https://drive.google.com/file/d/1BVHbiLTfX6iTeJcNbh-jgHjWDoemfrzG/view?usp=sharing\"\u003epretrained weight\u003c/a\u003e] (Put the model under 'pretrained/') need to be present.\n\n## \u003cdiv align=\"center\"\u003eBenchmark Evaluation\u003c/div\u003e\n\n1. Follow Single Image Inference Demo: Step 1-4\n\n2. Benchmarking\n\n    ```python benchmark.py -w pretrained/best.pth.tar```\n\nPrint-out results and visualization fo first-50 examples are stored under 'results/' (see 'demo/' for some pre-generated samples as references) are shown.\n\nUpdates: Best head pose estimation [\u003ca href=\"https://drive.google.com/file/d/1XN74PIMWDue1UNJuDJd16BH8fvZpfwDp/view?usp=sharing\"\u003epretrained model\u003c/a\u003e]  (Mean MAE: 3.31) that is better than number reported in paper (3.35). Use -w to load different pretrained models.\n\n## \u003cdiv align=\"center\"\u003eTraining\u003c/div\u003e\n\n1. Follow Single Image Inference Demo: Step 1-4.\n\n2. Download training data from [\u003ca href=\"https://github.com/cleardusk/3DDFA\"\u003e3DDFA\u003c/a\u003e]: train_aug_120x120.zip and extract the zip file under the root folder (Containing about 680K images).\n\n3. \n    ```bash train_script.sh```\n\n4. Please refer to train_script for hyperparameters, such as learning rate, epochs, or GPU device. The default settings take ~19G on a 3090 GPU and about 6 hours for training. If your GPU is less than this size, please decrease the batch size and learning rate proportionally.\n\n## \u003cdiv align=\"center\"\u003eTextured Artistic Face Meshes\u003c/div\u003e\n\n1. Follow Single Image Inference Demo: Step 1-5.\n\n2. Download artistic faces data [\u003ca href=\"https://drive.google.com/file/d/1yYR5aqSCUGnggjhTbwU4vEwHxV1xq9ko/view?usp=sharing\"\u003ehere\u003c/a\u003e], which are from [\u003ca href=\"https://faculty.idc.ac.il/arik/site/foa/artistic-faces-dataset.asp\"\u003eAF-Dataset\u003c/a\u003e]. Download our predicted UV maps [\u003ca href=\"https://drive.google.com/file/d/1TWJiitXAfZD_AwoJLw58XBPgOdanqgcG/view?usp=sharing\"\u003ehere\u003c/a\u003e] by UV-texture GAN. Extract them under the root folder.\n\n3.\n    ```python artistic.py -f art-all --png```(whole folder)\n    \n    ```python artistic.py -f art-all/122.png```(single image)\n    \n\nNote that this artistic face dataset contains many different level/style face abstration. If a testing image is close to real, the result is much better than those of highly abstract samples. \n\n## \u003cdiv align=\"center\"\u003eTextured Real Face Renderings\u003c/div\u003e\n\n1. Follow Single Image Inference Demo: Step 1-5.\n\n2. Download our predicted UV maps and real face images for AFLW2000-3D [\u003ca href=\"https://drive.google.com/file/d/12QCzkzBCKIEA3DSn6Kx5seeCeoUXKISc/view?usp=sharing\"\u003ehere\u003c/a\u003e] by UV-texture GAN. Extract them under the root folder.\n\n3.\n    ```python uv_texture_realFaces.py -f texture_data/real --png``` (whole folder)\n\n    ```python uv_texture_realFaces.py -f texture_data/real/image00002_real_A.png``` (single image) \n\nThe results (3D meshes and renderings) are stored under 'inference_output'\n\n## \u003cdiv align=\"center\"\u003eTraining UV-texture GAN\u003c/div\u003e\n\n1. Acquire AFLW2000-3D dataset and use [\u003ca href=\"https://github.com/jiaxiangshang/MGCNet/tree/master\"\u003eMGC-Net\u003c/a\u003e] test pipeline to get UV-texture for the AFLW2000 images.\n\n2. Use [\u003ca href=\"https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix\"\u003ePix2Pix\u003c/a\u003e] and train LSGAN with un-paired loss by their training recipe. In the input layer, concat the mean UV-texture and image and also shortcut add the mean texture at the output of generator.\n\n3. The mean UV-texture can be got from original BFM set or from [\u003ca href=\"https://github.com/yfeng95/face3d/blob/master/examples/Data/uv_texture_map.jpg\"\u003eface3d\u003c/a\u003e]\n\n## \u003cdiv align=\"center\"\u003eMore Results\u003c/div\u003e\n\nWe show a comparison with [\u003ca href=\"https://github.com/YadiraF/DECA\"\u003eDECA\u003c/a\u003e] using the top-3 largest roll angle samples in AFLW2000-3D.\n\n\u003cimg src='demo/comparison-deca.png'\u003e\n\n\nFacial alignemnt on AFLW2000-3D (NME of facial landmarks):\n\n\u003cimg src='demo/alignment.png'\u003e\n\nFace orientation estimation on AFLW2000-3D (MAE of Euler angles):\n\n\u003cimg src='demo/orientation.png'\u003e\n\nResults on artistic faces: \n\n\u003cimg src='demo/AF-1.png'\u003e\n\n\u003cimg src='demo/AF-2.png'\u003e\n\n**Related Project**\n\n[\u003ca href=\"https://github.com/choyingw/Voice2Mesh\"\u003eCross-Modal Perceptionist\u003c/a\u003e] (analysis on relation for voice and 3D face)\n\n**Bibtex**\n\nIf you find our work useful, please consider to cite our work \n\n    @INPROCEEDINGS{wu2021synergy,\n      author={Wu, Cho-Ying and Xu, Qiangeng and Neumann, Ulrich},\n      booktitle={2021 International Conference on 3D Vision (3DV)}, \n      title={Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry}, \n      year={2021}\n      }\n\n**Acknowledgement**\n\nThe project is developed on [\u003ca href=\"https://github.com/cleardusk/3DDFA\"\u003e3DDFA\u003c/a\u003e] and [\u003ca href=\"https://github.com/shamangary/FSA-Net\"\u003eFSA-Net\u003c/a\u003e]. Thank them for their wonderful work. Thank [\u003ca href=\"https://github.com/cleardusk/3DDFA_V2\"\u003e3DDFA-V2\u003c/a\u003e] for the face detector and rendering codes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchoyingw%2Fsynergynet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchoyingw%2Fsynergynet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchoyingw%2Fsynergynet/lists"}