https://github.com/tomaszrewak/face-landmarking
Real time face landmarking using decision trees and NN autoencoders
https://github.com/tomaszrewak/face-landmarking
autoencoder computer-vision decision-trees face-detection face-landmarking lips-movement-detection machine-learning opencv real-time
Last synced: 12 months ago
JSON representation
Real time face landmarking using decision trees and NN autoencoders
- Host: GitHub
- URL: https://github.com/tomaszrewak/face-landmarking
- Owner: TomaszRewak
- License: mit
- Created: 2017-12-29T15:27:03.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2019-02-28T21:05:53.000Z (over 7 years ago)
- Last Synced: 2025-06-22T19:04:28.749Z (about 1 year ago)
- Topics: autoencoder, computer-vision, decision-trees, face-detection, face-landmarking, lips-movement-detection, machine-learning, opencv, real-time
- Language: C++
- Homepage:
- Size: 24.3 MB
- Stars: 70
- Watchers: 8
- Forks: 6
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# FaceLandmarking
Real time face landmarking using decision trees and NN autoencoders.
The approach to the face landmarking problem used in this project is briefly described on my blog: https://blog.tomasz-rewak.com/face-landmarking/. The blog post contains also some information on how I’ve achieved the real time performance.
## About
This software is written (mostly) in C++ (with some ML parts written in Python). For video capturing and processing it uses OpenCV (OpenCV is also used for initial face detection).
In this project, the face landmarking is an iterative process which updates positions of key face points each frame using simple filters and decision trees. At the same time NN autoencoders ensure that the overall shape of the face stays correct.
The algorithm maps 194 points on all of the detected faces each frame. The ML models have been thought using the HELEN dataset: http://www.ifp.illinois.edu/~vuongle2/helen/
It's just a pet project of mine, so it still requires some work. In particular I didn't have that much time (nor will :D) to conduct extended experiments. Most of the parameters (like shape and number of filters, size of NN etc.) are just mine educated guesses.
## Setup
This repo contains all the code required to extract features from the dataset, generate ML models and use the face landmarker. Things that are missing are: dependencies (like OpenCV which has to be installed separately) and learning data.
First: in the main directory of the cloned repo create a `Data` directory with following subdirectories: `annotation`, `images`, `haar`, `mask`, `autoencoder`, `features` and `regressors`.
The dataset can be downloaded from the HELEN project website: http://www.ifp.illinois.edu/~vuongle2/helen/
All of the annotation files (`1.txt` to `2330.txt`) have to be extracted into the `annotation` directory and all of the images (`232194_1.jpg` to `3266693323_1.jpg`) into the `images` dir.
Also, as the software uses pretrained haar filters for initial face detection (at the first frame only), all of the .xml files from https://github.com/opencv/opencv/tree/master/data/haarcascades have to be copied into the previously created `haar` directory.
## Feature extraction
To extract features from the dataset the compiled program has to be ran with a `-type features` flag. This process will create files with learning examples in two directories: `autoencoder` and `features`. Also a `maks/avg-face.mask` file, that contains an average face shape computed based on the entire dataset, will be generated.
Make sure that when you perform this step (or any other of the following steps) the `./../Data` dir is visible to the program. This is the default configuration of the VS project.
The process might take some time.
## Learning process
To generate decision trees and the NN autoencoder, the `autoencoder.py` and the `regressor.py` scripts from the `FaceLandmarking.LearningProcess` dir have to be run.
This process populates the `regressors` dir.
## Usage
If all of the previous steps finished successfully, the software should be ready to use.
The best way to test it is to run the program with a `-type example` flag. It will load examples (one by one) from the dataset and display them. On each spacebar hit the program will perform one step of mask adjustment. Any other key stroke will change the image to the next one.
Finally, a `-type video` flag can be used to load a video for a file (provided with a `-video [path]` flag). Unfortunately I didn't have time to play with video settings, so the program doesn't read information about the orientation of the video. It (as well as the size of the video) might have to be adjusted manually (using `-transform`, `-transform-rotate`, `-transform-width` and `-transform-height` flags - see `main.cpp` for their default values). To initialize face landmarking process simply hit space – it will run haar filters on given frame to detect the initial position of the face.
## More
If you have any questions fell free to contact me.