{"id":15035108,"url":"https://github.com/cleardusk/3ddfa_v2","last_synced_at":"2025-05-15T03:06:19.534Z","repository":{"id":37381007,"uuid":"290171579","full_name":"cleardusk/3DDFA_V2","owner":"cleardusk","description":"The official PyTorch implementation of Towards Fast, Accurate and Stable 3D Dense Face Alignment, ECCV 2020.","archived":false,"fork":false,"pushed_at":"2024-02-02T06:54:17.000Z","size":76579,"stargazers_count":2983,"open_issues_count":99,"forks_count":521,"subscribers_count":67,"default_branch":"master","last_synced_at":"2025-05-09T16:42:14.053Z","etag":null,"topics":["3d","3d-face","3d-face-alignment","3d-landmarks","3dmm","alignment","computer-vision","eccv","eccv-2020","face-alignment","onnx","pytorch","single-image-reconstruction"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cleardusk.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-25T09:25:30.000Z","updated_at":"2025-05-08T08:01:41.000Z","dependencies_parsed_at":"2023-01-31T10:46:05.624Z","dependency_job_id":"c9699560-9e78-4812-a846-38db1859445a","html_url":"https://github.com/cleardusk/3DDFA_V2","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleardusk%2F3DDFA_V2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleardusk%2F3DDFA_V2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleardusk%2F3DDFA_V2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleardusk%2F3DDFA_V2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cleardusk","download_url":"https://codeload.github.com/cleardusk/3DDFA_V2/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254264765,"owners_count":22041793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","3d-face","3d-face-alignment","3d-landmarks","3dmm","alignment","computer-vision","eccv","eccv-2020","face-alignment","onnx","pytorch","single-image-reconstruction"],"created_at":"2024-09-24T20:27:32.448Z","updated_at":"2025-05-15T03:06:14.522Z","avatar_url":"https://github.com/cleardusk.png","language":"Python","readme":"# Towards Fast, Accurate and Stable 3D Dense Face Alignment\n\n[![License](https://img.shields.io/badge/license-MIT-yellow.svg)](LICENSE)\n![GitHub repo size](https://img.shields.io/github/repo-size/cleardusk/3DDFA_V2.svg)\n[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1OKciI0ETCpWdRjP-VOGpBulDJojYfgWv)\n\nBy [Jianzhu Guo](https://guojianzhu.com), [Xiangyu Zhu](http://www.cbsr.ia.ac.cn/users/xiangyuzhu/), [Yang Yang](http://www.cbsr.ia.ac.cn/users/yyang/main.htm), Fan Yang, [Zhen Lei](http://www.cbsr.ia.ac.cn/users/zlei/) and [Stan Z. Li](https://scholar.google.com/citations?user=Y-nyLGIAAAAJ).\nThe code repo is owned and maintained by **[Jianzhu Guo](https://guojianzhu.com)**.\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/images/webcam.gif\" alt=\"demo\" width=\"512px\"\u003e\n\u003c/p\u003e\n\n\n**\\[Updates\\]**\n - `2021.7.10`: Run 3DDFA_V2 online on [Gradio](https://gradio.app/hub/AK391/3DDFA_V2).\n - `2021.1.15`: Borrow the implementation of [Dense-Head-Pose-Estimation](https://github.com/1996scarlet/Dense-Head-Pose-Estimation) for the faster mesh rendering (speedup about 3x, 15ms -\u003e 4ms), see [utils/render_ctypes.py](./utils/render_ctypes.py) for details.\n - `2020.10.7`: Add the latency evaluation of the full pipeline in [latency.py](./latency.py), just run by `python3 latency.py --onnx`, see [Latency](#Latency) evaluation for details.\n - `2020.10.6`: Add onnxruntime support for FaceBoxes to reduce the face detection latency, just append the `--onnx` action to activate it, see [FaceBoxes_ONNX.py](FaceBoxes/FaceBoxes_ONNX.py) for details.\n - `2020.10.2`: **Add onnxruntime support to greatly reduce the 3dmm parameters inference latency**, just append the `--onnx` action when running `demo.py`, see [TDDFA_ONNX.py](./TDDFA_ONNX.py) for details.\n - `2020.9.20`: Add features including pose estimation and serializations to .ply and .obj, see `pose`, `ply`, `obj` options in [demo.py](./demo.py).\n - `2020.9.19`: Add PNCC (Projected Normalized Coordinate Code), uv texture mapping features, see `pncc`, `uv_tex` options in [demo.py](./demo.py).\n\n\n## Introduction\n\nThis work extends [3DDFA](https://github.com/cleardusk/3DDFA), named **3DDFA_V2**, titled [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://guojianzhu.com/assets/pdfs/3162.pdf), accepted by [ECCV 2020](https://eccv2020.eu/). The supplementary material is [here](https://guojianzhu.com/assets/pdfs/3162-supp.pdf). The [gif](./docs/images/webcam.gif) above shows a webcam demo of the tracking result, in the scenario of my lab. This repo is the official implementation of 3DDFA_V2.\n\nCompared to [3DDFA](https://github.com/cleardusk/3DDFA), 3DDFA_V2 achieves better performance and stability. Besides, 3DDFA_V2 incorporates the fast face detector [FaceBoxes](https://github.com/zisianw/FaceBoxes.PyTorch) instead of Dlib. A simple 3D render written by c++ and cython is also included. This repo supports the onnxruntime, and the latency of regressing 3DMM parameters using the default backbone is about **1.35ms/image on CPU** with a single image as input. If you are interested in this repo, just try it on this **[google colab](https://colab.research.google.com/drive/1OKciI0ETCpWdRjP-VOGpBulDJojYfgWv)**! Welcome for valuable issues, PRs and discussions 😄\n\n\u003c!-- Currently, the pre-trained model, inference code and some utilities are released.  --\u003e\n\n## Getting started\n\n### Requirements\nSee [requirements.txt](./requirements.txt), tested on macOS and Linux platforms. The Windows users may refer to [FQA](#FQA) for building issues. Note that this repo uses Python3. The major dependencies are PyTorch, numpy, opencv-python and onnxruntime, etc. If you run the demos with `--onnx` flag to do acceleration, you may need to install `libomp` first, i.e., `brew install libomp` on macOS.\n\n### Usage\n\n1. Clone this repo\n   \n```shell script\ngit clone https://github.com/cleardusk/3DDFA_V2.git\ncd 3DDFA_V2\n```\n\n2. Build the cython version of NMS, Sim3DR, and the faster mesh render\n\u003c!-- ```shell script\ncd FaceBoxes\nsh ./build_cpu_nms.sh\ncd ..\n\ncd Sim3DR\nsh ./build_sim3dr.sh\ncd ..\n\n# the faster mesh render\ncd utils/asset\ngcc -shared -Wall -O3 render.c -o render.so -fPIC\ncd ../..\n```\n\nor simply build them by --\u003e\n```shell script\nsh ./build.sh\n```\n\n3. Run demos\n\n```shell script\n# 1. running on still image, the options include: 2d_sparse, 2d_dense, 3d, depth, pncc, pose, uv_tex, ply, obj\npython3 demo.py -f examples/inputs/emma.jpg --onnx # -o [2d_sparse, 2d_dense, 3d, depth, pncc, pose, uv_tex, ply, obj]\n\n# 2. running on videos\npython3 demo_video.py -f examples/inputs/videos/214.avi --onnx\n\n# 3. running on videos smoothly by looking ahead by `n_next` frames\npython3 demo_video_smooth.py -f examples/inputs/videos/214.avi --onnx\n\n# 4. running on webcam\npython3 demo_webcam_smooth.py --onnx\n```\n\nThe implementation of tracking is simply by alignment. If the head pose \u003e 90° or the motion is too fast, the alignment may fail. A threshold is used to trickly check the tracking state, but it is unstable.\n\nYou can refer to [demo.ipynb](./demo.ipynb) or [google colab](https://colab.research.google.com/drive/1OKciI0ETCpWdRjP-VOGpBulDJojYfgWv) for the step-by-step tutorial of running on the still image.\n\nFor example, running `python3 demo.py -f examples/inputs/emma.jpg -o 3d` will give the result below:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/images/emma_3d.jpg\" alt=\"demo\" width=\"640px\"\u003e\n\u003c/p\u003e\n\nAnother example:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/images/trump_biden_3d.jpg\" alt=\"demo\" width=\"640px\"\u003e\n\u003c/p\u003e\n\nRunning on a video will give:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/images/out.gif\" alt=\"demo\" width=\"512px\"\u003e\n\u003c/p\u003e\n\nMore results or demos to see: [Hathaway](https://guojianzhu.com/assets/videos/hathaway_3ddfa_v2.mp4).\n\n\u003c!-- Obviously, the eyes parts are not good. --\u003e\n\n### Features (up to now)\n\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth\u003e2D sparse\u003c/th\u003e\n    \u003cth\u003e2D dense\u003c/th\u003e\n    \u003cth\u003e3D\u003c/th\u003e\n  \u003c/tr\u003e\n\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/images/trump_hillary_2d_sparse.jpg\" width=\"360\" alt=\"2d sparse\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/images/trump_hillary_2d_dense.jpg\"  width=\"360\" alt=\"2d dense\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/images/trump_hillary_3d.jpg\"        width=\"360\" alt=\"3d\"\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\n  \u003ctr\u003e\n    \u003cth\u003eDepth\u003c/th\u003e\n    \u003cth\u003ePNCC\u003c/th\u003e\n    \u003cth\u003eUV texture\u003c/th\u003e\n  \u003c/tr\u003e\n\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/images/trump_hillary_depth.jpg\"     width=\"360\" alt=\"depth\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/images/trump_hillary_pncc.jpg\"      width=\"360\" alt=\"pncc\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/images/trump_hillary_uv_tex.jpg\"    width=\"360\" alt=\"uv_tex\"\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\n  \u003ctr\u003e\n    \u003cth\u003ePose\u003c/th\u003e\n    \u003cth\u003eSerialization to .ply\u003c/th\u003e\n    \u003cth\u003eSerialization to .obj\u003c/th\u003e\n  \u003c/tr\u003e\n\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/images/trump_hillary_pose.jpg\"      width=\"360\" alt=\"pose\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/images/ply.jpg\"                     width=\"360\" alt=\"ply\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/images/obj.jpg\"                     width=\"360\" alt=\"obj\"\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\n\u003c/table\u003e\n\n### Configs\n\nThe default backbone is MobileNet_V1 with input size 120x120 and the default pre-trained weight is `weights/mb1_120x120.pth`, shown in [configs/mb1_120x120.yml](configs/mb1_120x120.yml). This repo provides another config in [configs/mb05_120x120.yml](configs/mb05_120x120.yml), with the widen factor 0.5, being smaller and faster. You can specify the config by `-c` or `--config` option. The released models are shown in the below table. Note that the inference time on CPU in the paper is evaluated using TensorFlow.\n\n| Model | Input | #Params | #Macs | Inference (TF) |\n| :-: | :-: | :-: | :-: | :-: |\n| MobileNet  | 120x120 | 3.27M | 183.5M | ~6.2ms |\n| MobileNet x0.5 | 120x120 | 0.85M | 49.5M | ~2.9ms |\n\n\n**Surprisingly**, the latency of [onnxruntime](https://github.com/microsoft/onnxruntime) is much smaller. The inference time on CPU with different threads is shown below. The results are tested on my MBP (i5-8259U CPU @ 2.30GHz on 13-inch MacBook Pro), with the `1.5.1` version of onnxruntime. The thread number is set by `os.environ[\"OMP_NUM_THREADS\"]`, see [speed_cpu.py](./speed_cpu.py) for more details.\n\n| Model | THREAD=1 | THREAD=2 | THREAD=4 |\n| :-: | :-: | :-: | :-: |\n| MobileNet  | 4.4ms  | 2.25ms | 1.35ms |\n| MobileNet x0.5 | 1.37ms | 0.7ms | 0.5ms |\n\n### Latency\n\nThe `onnx` option greatly reduces the overall **CPU** latency, but face detection still takes up most of the latency time, e.g., 15ms for a 720p image. 3DMM parameters regression takes about 1~2ms for one face, and the dense reconstruction (more than 30,000 points, i.e. 38,365) is about 1ms for one face. Tracking applications may benefit from the fast 3DMM regression speed, since detection is not needed for every frame. The latency is tested using my 13-inch MacBook Pro (i5-8259U CPU @ 2.30GHz).\n\nThe default `OMP_NUM_THREADS` is set 4, you can specify it by setting `os.environ['OMP_NUM_THREADS'] = '$NUM'` or inserting `export OMP_NUM_THREADS=$NUM` before running the python script.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/images/latency.gif\" alt=\"demo\" width=\"640px\"\u003e\n\u003c/p\u003e\n\n## FQA\n\n1. What is the training data?\n\n    We use [300W-LP](https://drive.google.com/file/d/0B7OEHD3T4eCkVGs0TkhUWFN6N1k/view?usp=sharing) for training. You can refer to our [paper](https://guojianzhu.com/assets/pdfs/3162.pdf) for more details about the training. Since few images are closed-eyes in the training data 300W-LP, the landmarks of eyes are not accurate when closing. The eyes part of the webcam demo are also not good.\n\n2. Running on Windows.\n\n    You can refer to [this comment](https://github.com/cleardusk/3DDFA_V2/issues/12#issuecomment-697479173) for building NMS on Windows.\n\n## Acknowledgement\n\n* The FaceBoxes module is modified from [FaceBoxes.PyTorch](https://github.com/zisianw/FaceBoxes.PyTorch).\n* A list of previous works on 3D dense face alignment or reconstruction: [3DDFA](https://github.com/cleardusk/3DDFA), [face3d](https://github.com/YadiraF/face3d), [PRNet](https://github.com/YadiraF/PRNet).\n* Thank [AK391](https://github.com/AK391) for hosting the Gradio web app.\n\n## Other implementations or applications\n\n* [Dense-Head-Pose-Estimation](https://github.com/1996scarlet/Dense-Head-Pose-Estimation): Tensorflow Lite framework for face mesh, head pose, landmarks, and more.\n* [HeadPoseEstimate](https://github.com/bubingy/HeadPoseEstimate): Head pose estimation system based on 3d facial landmarks.\n* [img2pose](https://github.com/vitoralbiero/img2pose): Borrow the renderer implementation of Sim3DR in this repo.\n\n## Citation\n\nIf your work or research benefits from this repo, please cite two bibs below : ) and 🌟 this repo.\n\n    @inproceedings{guo2020towards,\n        title =        {Towards Fast, Accurate and Stable 3D Dense Face Alignment},\n        author =       {Guo, Jianzhu and Zhu, Xiangyu and Yang, Yang and Yang, Fan and Lei, Zhen and Li, Stan Z},\n        booktitle =    {Proceedings of the European Conference on Computer Vision (ECCV)},\n        year =         {2020}\n    }\n\n    @misc{3ddfa_cleardusk,\n        author =       {Guo, Jianzhu and Zhu, Xiangyu and Lei, Zhen},\n        title =        {3DDFA},\n        howpublished = {\\url{https://github.com/cleardusk/3DDFA}},\n        year =         {2018}\n    }\n\n## Contact\n**Jianzhu Guo (郭建珠)** [[Homepage](https://guojianzhu.com), [Google Scholar](https://scholar.google.com/citations?user=W8_JzNcAAAAJ\u0026hl=en\u0026oi=ao)]: **guojianzhu1994@foxmail.com** or **guojianzhu1994@gmail.com** or **jianzhu.guo@nlpr.ia.ac.cn** (this email will be invalid soon).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcleardusk%2F3ddfa_v2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcleardusk%2F3ddfa_v2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcleardusk%2F3ddfa_v2/lists"}