{"id":15027494,"url":"https://github.com/cleardusk/3ddfa","last_synced_at":"2025-05-14T23:07:35.698Z","repository":{"id":39421943,"uuid":"139158036","full_name":"cleardusk/3DDFA","owner":"cleardusk","description":"The PyTorch improved version of TPAMI 2017 paper: Face Alignment in Full Pose Range: A 3D Total Solution.","archived":false,"fork":false,"pushed_at":"2022-05-14T12:12:21.000Z","size":83387,"stargazers_count":3653,"open_issues_count":65,"forks_count":651,"subscribers_count":117,"default_branch":"master","last_synced_at":"2025-05-14T23:07:24.606Z","etag":null,"topics":["3d","3d-face","3dmm","computer-vision","deep-learning","face-alignment","python","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cleardusk.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-29T14:19:21.000Z","updated_at":"2025-05-09T09:37:59.000Z","dependencies_parsed_at":"2022-07-14T08:49:40.104Z","dependency_job_id":null,"html_url":"https://github.com/cleardusk/3DDFA","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleardusk%2F3DDFA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleardusk%2F3DDFA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleardusk%2F3DDFA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleardusk%2F3DDFA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cleardusk","download_url":"https://codeload.github.com/cleardusk/3DDFA/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254243362,"owners_count":22038046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","3d-face","3dmm","computer-vision","deep-learning","face-alignment","python","pytorch"],"created_at":"2024-09-24T20:06:32.458Z","updated_at":"2025-05-14T23:07:30.687Z","avatar_url":"https://github.com/cleardusk.png","language":"Python","readme":"# Face Alignment in Full Pose Range: A 3D Total Solution\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n![stars](https://img.shields.io/github/stars/cleardusk/3DDFA.svg?style=flat)\n![GitHub issues](https://img.shields.io/github/issues/cleardusk/3DDFA.svg)\n![GitHub repo size](https://img.shields.io/github/repo-size/cleardusk/3DDFA.svg)\n\n\u003c!-- By [Jianzhu Guo](https://guojianzhu.com/aboutme.html). --\u003e\nBy [Jianzhu Guo](http://guojianzhu.com).\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"samples/obama_three_styles.gif\" alt=\"obama\"\u003e\n\u003c/p\u003e\n\n**\\[Updates\\]**\n - `2022.5.14`: Recommend a python implementation of face profiling: [face_pose_augmentation](https://github.com/hhj1897/face_pose_augmentation).\n - `2020.8.30`: The pre-trained model and code of ECCV-20 are made public on [3DDFA_V2](https://github.com/cleardusk/3DDFA_V2), the copyright is explained by Jianzhu Guo and the CBSR group.\n - `2020.8.2`: Update a \u003cstrong\u003e[simple c++ port](./c++/readme.md)\u003c/strong\u003e of this project.\n - `2020.7.3`: The extended work \u003cstrong\u003e[Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://guojianzhu.com/assets/pdfs/3162.pdf)\u003c/strong\u003e is accepted by [ECCV 2020](https://eccv2020.eu/). See [my page](https://guojianzhu.com) for more details.\n - `2019.9.15`: Some updates, see the commits for details.\n - `2019.6.17`: Adding a [video demo](./video_demo.py) contributed by [zjjMaiMai](https://github.com/zjjMaiMai).\n - `2019.5.2`: Evaluating inference speed on CPU with PyTorch v1.1.0, see [here](#CPU) and [speed_cpu.py](./speed_cpu.py).\n - `2019.4.27`: A simple render pipeline running at ~25ms/frame (720p), see [rendering.py](demo@obama/rendering.py) for more details.\n - `2019.4.24`: Providing the demo building of obama, see [demo@obama/readme.md](demo@obama/readme.md) for more details.\n - `2019.3.28`: Some updates.\n - `2018.12.23`: **Add several features: depth image estimation, PNCC, PAF feature and obj serialization.** See `dump_depth`, `dump_pncc`, `dump_paf`, `dump_obj` options for more details.\n - `2018.12.2`: Support landmark-free face cropping, see `dlib_landmark` option.\n - `2018.12.1`: Refine code and add pose estimation feature, see [utils/estimate_pose.py](./utils/estimate_pose.py) for more details.\n - `2018.11.17`: Refine code and map the 3d vertex to original image space.\n - `2018.11.11`: **Update end-to-end inference pipeline: infer/serialize 3D face shape and 68 landmarks given one arbitrary image, please see readme.md below for more details.**\n - `2018.10.4`: Add Matlab face mesh rendering demo in [visualize](./visualize).\n - `2018.9.9`: Add pre-process of face cropping in [benchmark](./benchmark).\n\n**\\[Todo\\]**\n\n- [x] Add c++ port.\n- [x] Depth image estimation.\n- [x] PNCC (Projected Normalized Coordinate Code).\n- [x] PAF (Pose Adaptive Feature).\n- [x] Obj serialization with sampled texture.\n- [x] Recommendation of fast face detectors: [FaceBoxes.PyTorch](https://github.com/zisianw/FaceBoxes.PyTorch), [libfacedetection](https://github.com/ShiqiYu/libfacedetection), [ZQCNN](https://github.com/zuoqing1988/ZQCNN)\n- [x] [Training details](#Training-details)\n- [x] Face Profiling: [Official Matlab code](https://drive.google.com/open?id=1f4686yak4lGHLD6MfIiX5knVtq8a762d), [Python version](https://github.com/hhj1897/face_pose_augmentation) \u003c!--(http://www.cbsr.ia.ac.cn/users/xiangyuzhu/projects/3DDFA/Code/FaceProfilingRelease_v1.1.zip) --\u003e\n\n## Introduction\nThis repo holds the pytorch improved version of the paper: [Face Alignment in Full Pose Range: A 3D Total Solution](https://arxiv.org/abs/1804.01005). Several works beyond the original paper are added, including the real-time training, training strategies. Therefore, this repo is an improved version of the original work. As far, this repo releases the pre-trained first-stage pytorch models of MobileNet-V1 structure, the pre-processed training\u0026testing dataset and codebase. Note that the inference time is about **0.27ms per image** (input batch with 128 images as an input batch) on GeForce GTX TITAN X.\n\u003c!-- Note that if your academic work use the code of this repo, you should cite this repo not the original paper.--\u003e\n\u003c!-- One related blog will be published for some important technique details in future. --\u003e\n\u003c!-- Why not evaluate it on single image? Because most time for single image is spent on function call. The inference speed is equal to MobileNet-V1 with 120x120x3 tensor as input, therefore it is possible to convert to mobile devices. --\u003e\n\n**This repo will keep updating in my spare time, and any meaningful issues and PR are welcomed.**\n\nSeveral results on ALFW-2000 dataset (inferenced from model *phase1_wpdc_vdc.pth.tar*) are shown below.\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"imgs/landmark_3d.jpg\" alt=\"Landmark 3D\" width=\"1000px\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"imgs/vertex_3d.jpg\" alt=\"Vertex 3D\" width=\"750px\"\u003e\n\u003c/p\u003e\n\n## Applications \u0026 Features\n#### 1. Face Alignment\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"samples/dapeng_3DDFA_trim.gif\" alt=\"dapeng\"\u003e\n\u003c/p\u003e\n\n#### 2. Face Reconstruction\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"samples/5.png\" alt=\"demo\" width=\"750px\"\u003e\n\u003c/p\u003e\n\n#### 3. 3D Pose Estimation\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"samples/pose.png\" alt=\"tongliya\" width=\"750px\"\u003e\n\u003c/p\u003e\n\n#### 4. Depth Image Estimation\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"samples/demo_depth.jpg\" alt=\"demo_depth\" width=\"750px\"\u003e\n\u003c/p\u003e\n\n#### 5. PNCC \u0026 PAF Features\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"samples/demo_pncc_paf.jpg\" alt=\"demo_pncc_paf\" width=\"800px\"\u003e\n\u003c/p\u003e\n\n## Getting started\n### Requirements\n - PyTorch \u003e= 0.4.1 (**PyTorch v1.1.0** is tested successfully on macOS and Linux.)\n - Python \u003e= 3.6 (Numpy, Scipy, Matplotlib)\n - Dlib (Dlib is optionally for face and landmarks detection. There is no need to use Dlib if you can provide face bouding bbox and landmarks. Besides, you can try the two-step inference strategy without initialized landmarks.)\n - OpenCV (Python version, for image IO operations.)\n - Cython (For accelerating depth and PNCC render.)\n - Platform: Linux or macOS (Windows is not tested.)\n\n ```\n # installation structions\n sudo pip3 install torch torchvision # for cpu version. more option to see https://pytorch.org\n sudo pip3 install numpy scipy matplotlib\n sudo pip3 install dlib==19.5.0 # 19.15+ version may cause conflict with pytorch in Linux, this may take several minutes. If 19.5 version raises errors, you may try 19.15+ version.\n sudo pip3 install opencv-python\n sudo pip3 install cython\n ```\n\nIn addition, I strongly recommend using Python3.6+ instead of older version for its better design.\n\n### Usage\n\n1. Clone this repo (this may take some time as it is a little big)\n    ```\n    git clone https://github.com/cleardusk/3DDFA.git  # or git@github.com:cleardusk/3DDFA.git\n    cd 3DDFA\n    ```\n\n   Then, download dlib landmark pre-trained model in [Google Drive](https://drive.google.com/open?id=1kxgOZSds1HuUIlvo5sRH3PJv377qZAkE) or [Baidu Yun](https://pan.baidu.com/s/1bx-GxGf50-KDk4xz3bCYcw), and put it into `models` directory. (To reduce this repo's size, I remove some large size binary files including this model, so you should download it : ) )\n\n\n2. Build cython module (just one line for building)\n   ```\n   cd utils/cython\n   python3 setup.py build_ext -i\n   ```\n   This is for accelerating depth estimation and PNCC render since Python is too slow in for loop.\n   \n    \n3. Run the `main.py` with arbitrary image as input\n    ```\n    python3 main.py -f samples/test1.jpg\n    ```\n    If you can see these output log in terminal, you run it successfully.\n    ```\n    Dump tp samples/test1_0.ply\n    Save 68 3d landmarks to samples/test1_0.txt\n    Dump obj with sampled texture to samples/test1_0.obj\n    Dump tp samples/test1_1.ply\n    Save 68 3d landmarks to samples/test1_1.txt\n    Dump obj with sampled texture to samples/test1_1.obj\n    Dump to samples/test1_pose.jpg\n    Dump to samples/test1_depth.png\n    Dump to samples/test1_pncc.png\n    Save visualization result to samples/test1_3DDFA.jpg\n    ```\n\n    Because `test1.jpg` has two faces, there are two `.ply` and `.obj` files (can be rendered by Meshlab or Microsoft 3D Builder) predicted. Depth, PNCC, PAF and pose estimation are all set true by default. Please run `python3 main.py -h` or review the code for more details.\n\n    The 68 landmarks visualization result `samples/test1_3DDFA.jpg` and pose estimation result `samples/test1_pose.jpg` are shown below:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"samples/test1_3DDFA.jpg\" alt=\"samples\" width=\"650px\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"samples/test1_pose.jpg\" alt=\"samples\" width=\"650px\"\u003e\n\u003c/p\u003e\n\n4. Additional example\n\n    ```\n    python3 ./main.py -f samples/emma_input.jpg --bbox_init=two --dlib_bbox=false\n    ```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"samples/emma_input_3DDFA.jpg\" alt=\"samples\" width=\"750px\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"samples/emma_input_pose.jpg\" alt=\"samples\" width=\"750px\"\u003e\n\u003c/p\u003e\n\n\n## Inference speed\n### CPU\nJust run\n```\npython3 speed_cpu.py\n```\n\nOn my MBP (i5-8259U CPU @ 2.30GHz on 13-inch MacBook Pro), based on **PyTorch v1.1.0**, with a single input, the running output is:\n```\nInference speed: 14.50±0.11 ms\n```\n\n\u003c!-- [speed_cpu.py](./speed_cpu.py) --\u003e\n\n\n### GPU\nWhen input batch size is 128, the total inference time of MobileNet-V1 takes about 34.7ms. The average speed is about **0.27ms/pic**.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"imgs/inference_speed.png\" alt=\"Inference speed\" width=\"600px\"\u003e\n\u003c/p\u003e\n\n## Training details\nThe training scripts lie in `training` directory. The related resources are in below table.\n\n| Data | Download Link | Description |\n|:-:|:-:|:-:|\n| train.configs | [BaiduYun](https://pan.baidu.com/s/1ozZVs26-xE49sF7nystrKQ) or [Google Drive](https://drive.google.com/open?id=1dzwQNZNMppFVShLYoLEfU3EOj3tCeXOD), 217M | The directory containing 3DMM params and filelists of training dataset |\n| train_aug_120x120.zip | [BaiduYun](https://pan.baidu.com/s/19QNGst2E1pRKL7Dtx_L1MA) or [Google Drive](https://drive.google.com/open?id=17LfvBZFAeXt0ACPnVckfdrLTMHUpIQqE), 2.15G | The cropped images of augmentation training dataset |\n| test.data.zip | [BaiduYun](https://pan.baidu.com/s/1DTVGCG5k0jjjhOc8GcSLOw) or [Google Drive](https://drive.google.com/file/d/1r_ciJ1M0BSRTwndIBt42GlPFRv6CvvEP/view?usp=sharing), 151M | The cropped images of AFLW and ALFW-2000-3D testset |\n\nAfter preparing the training dataset and configuration files, go into `training` directory and run the bash scripts to train. `train_wpdc.sh`, `train_vdc.sh` and `train_pdc.sh` are examples of training scripts. After configuring the training and testing sets, just run them for training. Take `train_wpdc.sh` for example as below:\n\n```\n#!/usr/bin/env bash\n\nLOG_ALIAS=$1\nLOG_DIR=\"logs\"\nmkdir -p ${LOG_DIR}\n\nLOG_FILE=\"${LOG_DIR}/${LOG_ALIAS}_`date +'%Y-%m-%d_%H:%M.%S'`.log\"\n#echo $LOG_FILE\n\n./train.py --arch=\"mobilenet_1\" \\\n    --start-epoch=1 \\\n    --loss=wpdc \\\n    --snapshot=\"snapshot/phase1_wpdc\" \\\n    --param-fp-train='../train.configs/param_all_norm.pkl' \\\n    --param-fp-val='../train.configs/param_all_norm_val.pkl' \\\n    --warmup=5 \\\n    --opt-style=resample \\\n    --resample-num=132 \\\n    --batch-size=512 \\\n    --base-lr=0.02 \\\n    --epochs=50 \\\n    --milestones=30,40 \\\n    --print-freq=50 \\\n    --devices-id=0,1 \\\n    --workers=8 \\\n    --filelists-train=\"../train.configs/train_aug_120x120.list.train\" \\\n    --filelists-val=\"../train.configs/train_aug_120x120.list.val\" \\\n    --root=\"/path/to//train_aug_120x120\" \\\n    --log-file=\"${LOG_FILE}\"\n```\n\nThe specific training parameters are all presented in bash scripts, including learning rate, mini-batch size, epochs and so on.\n\n## Evaluation\nFirst, you should download the cropped testset ALFW and ALFW-2000-3D in [test.data.zip](https://pan.baidu.com/s/1DTVGCG5k0jjjhOc8GcSLOw), then unzip it and put it in the root directory.\nNext, run the benchmark code by providing trained model path.\nI have already provided five pre-trained models in `models` directory (seen in below table). These models are trained using different loss in the first stage. The model size is about 13M due to the high efficiency of MobileNet-V1 structure.\n```\npython3 ./benchmark.py -c models/phase1_wpdc_vdc.pth.tar\n```\n\nThe performances of pre-trained models are shown below. In the first stage, the effectiveness of different loss is in order: WPDC \u003e VDC \u003e PDC. While the strategy using VDC to finetune WPDC achieves the best result.\n\n| Model | AFLW (21 pts) | AFLW 2000-3D (68 pts) | Download Link |\n|:-:|:-:|:-:| :-: |\n| *phase1_pdc.pth.tar*  | 6.956±0.981 | 5.644±1.323 | [Baidu Yun](https://pan.baidu.com/s/1xeyZa4rxVazd_QGWx6QXFw) or [Google Drive](https://drive.google.com/open?id=18UQfDkGNzotKoFV0Lh_O-HnXsp1ABdjl) |\n| *phase1_vdc.pth.tar*  | 6.717±0.924 | 5.030±1.044 | [Baidu Yun](https://pan.baidu.com/s/10-0YpYKj1_efJYqC1q-aNQ) or [Google Drive](https://drive.google.com/open?id=1iHADYNIQR2Jqvt4nwmnh5n3Axe-HXMRR) |\n| *phase1_wpdc.pth.tar* | 6.348±0.929 | 4.759±0.996 | [Baidu Yun](https://pan.baidu.com/s/1yqaJ3S3MNpYBgyA5BYtHuw) or [Google Drive](https://drive.google.com/open?id=1ebwkOWjaQ7U4mpA89ldfmjeQdfDDdFS-) |\n| *phase1_wpdc_vdc.pth.tar* | **5.401±0.754** | **4.252±0.976** | In this repo. |\n\n### About the performance\nBelieve me that the framework of this repo can achieve better performance than [PRNet](https://github.com/YadiraF/PRNet) without increasing any computation budget. Related work is under review and code will be released upon acceptance.\n\n## FQA\n1. Face bounding box initialization\n\n    The original paper shows that using detected bounding box instead of ground truth box will cause a little performance drop. Thus the current face cropping method is robustest. Quantitative results are shown in below table.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"imgs/bouding_box_init.png\" alt=\"bounding box\" width=\"500px\"\u003e\n\u003c/p\u003e\n\n2. Face reconstruction\n   \n   The texture of non-visible area is distorted due to self-occlusion, therefore the non-visible face region may appear strange (a little horrible).\n\n3. About shape and expression parameters clipping\n   \n    The parameters clipping accelerates the training and reconstruction, but degrades the accuracy especially the details like closing eyes. Below is an image, with parameters dimension 40+10, 60+29 and 199+29 (the original one). Compared to shape, expression clipping has more effect on reconstruction accuracy when emotion is involved. Therefore, you can choose a trade-off between the speed/parameter-size and the accuracy. A recommendation of clipping trade-off is 60+29.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"imgs/params_clip.jpg\" alt=\"bounding box\" width=\"600px\"\u003e\n\u003c/p\u003e\n\n## Acknowledgement\n - Thanks for [Yao Feng](https://github.com/YadiraF)'s fantastic works [PRNet](https://github.com/YadiraF/PRNet) and [face3d](https://github.com/YadiraF/face3d).\n - Thanks for this [tweet](https://twitter.com/PyTorch/status/1066064914249367552) of PyTorch.\n\n\nThanks for your interest in this repo. If your work or research benefits from this repo, star it 😃\n\nWelcome to focus on my 3D face related works: [MeGlass](https://github.com/cleardusk/MeGlass) and [Face Anti-Spoofing](https://arxiv.org/abs/1901.00488).\n\n## Citation\n**If your work benefits from this repo, please cite three bibs below.**\n\n    @misc{3ddfa_cleardusk,\n      author =       {Guo, Jianzhu and Zhu, Xiangyu and Lei, Zhen},\n      title =        {3DDFA},\n      howpublished = {\\url{https://github.com/cleardusk/3DDFA}},\n      year =         {2018}\n    }\n    \n    @inproceedings{guo2020towards,\n      title=        {Towards Fast, Accurate and Stable 3D Dense Face Alignment},\n      author=       {Guo, Jianzhu and Zhu, Xiangyu and Yang, Yang and Yang, Fan and Lei, Zhen and Li, Stan Z},\n      booktitle=    {Proceedings of the European Conference on Computer Vision (ECCV)},\n      year=         {2020}\n    }\n\n    @article{zhu2017face,\n      title=      {Face alignment in full pose range: A 3d total solution},\n      author=     {Zhu, Xiangyu and Liu, Xiaoming and Lei, Zhen and Li, Stan Z},\n      journal=    {IEEE transactions on pattern analysis and machine intelligence},\n      year=       {2017},\n      publisher=  {IEEE}\n    }\n\n\n## Contact\n**Jianzhu Guo (郭建珠)** [[Homepage](http://guojianzhu.com), [Google Scholar](https://scholar.google.com/citations?user=W8_JzNcAAAAJ\u0026hl=en\u0026oi=ao)]:  **jianzhu.guo@nlpr.ia.ac.cn** or **guojianzhu1994@foxmail.com**.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcleardusk%2F3ddfa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcleardusk%2F3ddfa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcleardusk%2F3ddfa/lists"}