Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/powermobileweb/arfacefilter

Javascript/WebGL lightweight face tracking library designed for augmented reality webcam filters. Features : multiple faces detection, rotation, mouth opening. Various integration examples are provided (Three.js, Babylon.js, FaceSwap, Canvas2D, CSS3D...).
https://github.com/powermobileweb/arfacefilter

3d augmented-reality babylonjs deep-learning detection face face-detection face-filters face-tracking faceswap javascript library msqrd picojs snapchat threejs tracking trackingjs webcam webgl

Last synced: 10 days ago
JSON representation

Javascript/WebGL lightweight face tracking library designed for augmented reality webcam filters. Features : multiple faces detection, rotation, mouth opening. Various integration examples are provided (Three.js, Babylon.js, FaceSwap, Canvas2D, CSS3D...).

Awesome Lists containing this project

README 

        
# JavaScript/WebGL lightweight and robust face tracking library designed for augmented reality face filters



This JavaScript library detects and tracks the face in real time from the webcam video feed captured with WebRTC. Then it is possible to overlay 3D content for augmented reality applications. We provide various demonstrations using main WebGL 3D engines. We have included in this repository the release versions of the 3D engines to work with a determined version (they are in `/libs//`).



This library is lightweight and it does not include any 3D engine or third party library. We want to keep it framework agnostic so the outputs of the library are raw: if the a face is detected or not, the position and the scale of the detected face and the rotation Euler angles. But thanks to the featured helpers, examples and boilerplates, you can quickly deal with a higher level context (for motion head tracking, for face filter or face replacement...). We continuously add new demontrations, so stay tuned ! Also, feel free to open an issue if you have any question or suggestion.



## Table of contents



* [Features](#features)

* [Architecture](#architecture)

* [Demonstrations](#demonstrations)

* [Specifications](#specifications)

  * [Get started](#get-started)

  * [Optionnal init arguments](#optionnal-init-arguments)

  * [Error codes](#error-codes)

  * [The returned objects](#the-returned-objects)

  * [Miscellaneous methods](#miscellaneous-methods)

  * [Multiple faces](#multiple-faces)

  * [Optimization](#optimization)

  * [Changing the 3D engine](#changing-the-3d-engine)

  * [Changing the neural network](#changing-the-neural-network)

  * [Using ES6](#using-es6)

* [Hosting](#hosting)

  * [The development server](#the-development-server)  

  * [Hosting optimization](#hosting-optimization)

* [About the tech](#about-the-tech)

  * [Under the hood](#under-the-hood)

  * [Compatibility](#compatibility)

* [Articles and tutorials](#articles-and-tutorials)

* [License](#license)

* [See also](#see-also)

* [References](#references)










## Features



Here are the main features of the library :



* face detection,

* face tracking,

* face rotation detection,

* mouth opening detection,

* multiple faces detection and tracking,

* very robust for all lighting conditions,

* video acquisition with HD video ability,

* mobile friendly,

* interfaced with 3D engines like THREE.JS, BABYLON.JS, A-FRAME,

* interfaced with more accessible APIs like CANVAS, CSS3D.



## Architecture



* `/demos/`: source code of the demonstrations, sorted by 2D/3D engine used,

* `/dist/`: heart of the library: 

  * `jeelizFaceFilter.js`: main minified script,

  * `jeelizFaceFilterES6.js`: main minified script for ES6 use (with `import` or `require`),

  * `NNC.json`: file storing the neural network parameters, loaded by the main script,

  * `NNC.json` : alternative neural network models,

* `/helpers/`: scripts which can help you to use this library in some specific use cases,

* `/libs/`: 3rd party libraries and 3D engines used in the demos.



## Demonstrations



You will probably find among these demos the perfect starting point to build your own face based augmented reality application :



* BABYLON.JS based demos :

  * Boilerplate (displays a cube on the user's head) : [live demo](https://jeeliz.com/demos/faceFilter/demos/babylonjs/cube/), [source code](/demos/babylonjs/cube/)



* THREE.JS based demos - [specific README about THREE.js based demo problems](demos/threejs/) :

  * Boilerplates :

    * Boilerplate (displays a cube on the user's head): [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/cubeWithHelper/), [source code](/demos/threejs/cubeWithHelper/)

    * Boilerplate with ES6 : [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/cubeES6/), [source code](/demos/threejs/cubeES6/)

    * Boilerplate with 2 canvas : 1 for FaceFilter and 1 for THREE.JS: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/cube2cv/), [source code](/demos/threejs/cube2cv/)

    * Boilerplate using `dist/NNClight.json` as neural net: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/cubeNNClight/), [source code](/demos/threejs/cubeNNClight/)

    * Boilerplate using `dist/NNCwideAngles.json` as neural net: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/cubeNNCwideAngles/), [source code](/demos/threejs/cubeNNCwideAngles/)

    * Boilerplate without JeelizThreeHelper: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/cube/), [source code](/demos/threejs/cube/)

    * Multiple face tracking: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/multiCubes/), [source code](/demos/threejs/multiCubes/)

    * GLTF fullscreen demo with HD video: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/gltf_fullScreen/), [source code](/demos/threejs/gltf_fullScreen/)

  

  * AR 3D demos :

    * Daft Punk (put the iconic helmet): [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/daft_punk/)

    * Star Wars: Darth Vader: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/star_wars/)

    * Harry Potter (say "Lumos!"): [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/harry_potter/)

    * Halloween Spiders (you've got a spider in your mouth): [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/halloween_spider/)

    * Werewolf (turn yourself into a werewolf): [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/werewolf/)

    * Angel/Demon (discover who of the angel or demon will win in this animated scene): [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/angel_demon/)

    * Anonymous mask and video effect: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/anonymous/), [source code](/demos/threejs/anonymouse)

    * Rupy Motorcycle Helmet VTO: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/rupy_helmet/)

    * Dog-Face filter: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/dog_face/), [source code](/demos/threejs/dog_face/)

    * Butterflies animation: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/butterflies/), [source code](/demos/threejs/butterflies/)

    * Clouds above the head: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/cloud/), [source code](/demos/threejs/cloud/)

    * Casa-de-Papel mask: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/casa_de_papel/), [source code](/demos/threejs/casa_de_papel/)

    * Miel Pops glasses and bees: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/miel_pops/), [source code](/demos/threejs/miel_pops/)

    * Football makeup: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/football_makeup/), [source code](/demos/threejs/football_makeup/)

    * Tiger face filter with mouth opening detection (strong WTF effect): [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/tiger/), [source code](/demos/threejs/tiger/)

    * Fireworks - particules: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/fireworks/), [source code](/demos/threejs/fireworks/)

  

  * face painting or deformation :

    * Face deformation: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/faceDeform/), [source code](/demos/threejs/faceDeform/)

    * Face cel shading: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/celFace/), [source code](/demos/threejs/celFace/)



  * demos linked with tutorials :

    * Luffy's Hat: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/luffys_hat_part2/), [source code part 1](/demos/threejs/luffys_hat_part1/), [tutorial part 1](https://jeeliz.com/blog/creating-a-snapchat-like-filter-with-jeelizs-facefilter-api-part-1-creating-your-first-filter/), [source code part 2](/demos/threejs/luffys_hat_part2/), [tutorial part 2](https://jeeliz.com/blog/creating-a-snapchat-like-filter-with-jeelizs-facefilter-api-part-2-user-interactions-and-particles/)

    * Statue Of Liberty: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/multiLiberty/), [source code](/demos/threejs/multiLiberty/), [interactive tutorial](https://webglacademy.jeeliz.com/courses.php?courses=19_25_27_33_34#34)

    * Matrix: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/matrix/), [source code](/demos/threejs/matrix/), [tutorial in French](https://xavierbourry.developpez.com/filtre-facial-webcam/), [tutorial in English](https://jeeliz.com/blog/tutorial-javascript-webgl-webcam-facial-filter-on-the-theme-of-matrix/)



  * head motion control :

    * Head controlled navigation: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/headControls/), [source code](/demos/threejs/headControls/)



* A-FRAME based demos :

  * Boilerplate (displays a cube on the user's head): [live demo](https://jeeliz.com/demos/faceFilter/demos/aFrame/cube/), [source code](/demos/aFrame/cube/)

  

* CSS3D based demos :

  * Boilerplate (displays a `
` element on the user's head): [live demo](https://jeeliz.com/demos/faceFilter/demos/CSS3D/div/), [source code](/demos/CSS3D/div/)

  * Comedy glasses demo: [live demo](https://jeeliz.com/demos/faceFilter/demos/CSS3D/comedy-glasses/), [source code](/demos/CSS3D/comedy-glasses/)



* Canvas2D based demos :

  * Draw on the face with the mouse: [live demo](https://jeeliz.com/demos/faceFilter/demos/canvas2D/faceDraw/), [source code](/demos/canvas2D/faceDraw/)

  * 2D face detection and tracking - 30 lines of code only !: [live demo](https://jeeliz.com/demos/faceFilter/demos/canvas2D/faceTrack/), [source code](/demos/canvas2D/faceTrack/), [JSfiddle](https://jsfiddle.net/jeeliz/2p34hbeh/)

  * 2D face detection and tracking from a video file instead of webcam video: [live demo](https://jeeliz.com/demos/faceFilter/demos/canvas2D/fromVideoFile/), [source code](/demos/canvas2D/fromVideoFile/)



* CESIUM.JS based demos :

  * 3D view of the Earth with head controlled navigation: [live demo](https://jeeliz.com/demos/faceFilter/demos/cesium/headControls/), [source code](/demos/cesium/headControls/), [article about the demo](https://cesium.com/blog/2018/03/22/jeeliz-and-cesium/)



* Face replacement demos :

  * Insert your face into portrait art painting or film posters: [live demo](https://jeeliz.com/demos/faceFilter/demos/faceReplacement/image), [source code](demos/faceReplacement/image/)

  * Insert your face into an animated gif: [live demo](https://jeeliz.com/demos/faceFilter/demos/faceReplacement/gif), [specific README: [live demo](demos/faceReplacement/gif/), [source code](demos/faceReplacement/gif/)

  * The traditional faceSwap, fullscreen and with color correction: [live demo](https://jeeliz.com/demos/faceFilter/demos/faceReplacement/faceSwap/), [source code](demos/faceReplacement/faceSwap/)



* Head motion control :

  * PACMAN game with head controlled navigation: [live demo](https://jeeliz.com/demos/faceFilter/demos/pacman/), [source code](demos/pacman/)

  * Head controlled mouse cursor: [live demo](https://jeeliz.com/demos/faceFilter/demos/headCursor/), [source code](demos/headCursor/)



[comment]:![giphy-downsized-large](https://user-images.githubusercontent.com/11960872/37475622-6a602cf6-2873-11e8-83f0-134b6c1ba666.gif)



If you have not bought a webcam yet, a screenshot video of some of these examples is available [on Youtube](https://youtu.be/jQkaJoMGinQ). You can also subscribe to the [Jeeliz Youtube channel](https://www.youtube.com/channel/UC3XmXH1T3d1XFyOhrRiiUeA) or to the [@StartupJeeliz Twitter account](https://twitter.com/StartupJeeliz) to be kept informed of our cutting edge developments.



If you have developped an application or a fun demo using this library, we would love to see it and insert a link here ! Just contact us on [Twitter @StartupJeeliz](https://twitter.com/StartupJeeliz) or [LinkedIn](https://www.linkedin.com/company/jeeliz).



## Specifications

Here we describe how to use this library. Although we planned to add new features, we will keep it backward compatible.



### Get started

On your HTML page, you first need to include the main script between the tags `` and `` :

```html

 

```

Then you should include a `` HTML element in the DOM, between the tags `` and ``. The `width` and `height` properties of the `` element should be set. They define the resolution of the canvas and the final rendering will be computed using this resolution. Be careful to not enlarge too much the canvas size using its CSS properties without increasing its resolution, otherwise it may look blurry or pixelated. We advise to fix the resolution to the actual canvas size. Do not forget to call `JEEFACEFILTERAPI.resize()` if you resize the canvas after the initialization step. We strongly encourage you to use our helper `/helpers/JeelizResizer.js` to set the width and height of the canvas (see [Optimization/Canvas and video resolutions](#optimization) section).

```html



```

This canvas will be used by WebGL both for the computation and the 3D rendering. When your page is loaded you should launch this function :

```javascript

JEEFACEFILTERAPI.init({

    canvasId: 'jeeFaceFilterCanvas',

    NNCpath: '../../../dist/', //path to JSON neural network model (NNC.json by default)

    callbackReady: function(errCode, spec){

        if (errCode){

            console.log('AN ERROR HAPPENS. ERROR CODE =', errCode);

            return;

        }

        [init scene with spec...]

        console.log('INFO: JEEFACEFILTERAPI IS READY');

    }, //end callbackReady()



    //called at each render iteration (drawing loop)

    callbackTrack: function(detectState){

        //render your scene here

        [... do something with detectState]

    } //end callbackTrack()

});//end init call

```



### Optionnal init arguments

* ` maxFacesDetected`: Only for multiple face detection - maximum number of faces which can be detected and tracked. Should be between `1` (no multiple detection) and `8`,

* ` animateDelay`: It is used only in normal rendering mode (not in slow rendering mode). With this statement you can set accurately the number of milliseconds during which the browser wait at the end of the rendering loop before starting another detection. If you use the canvas of this API as a secondary element (for example in *PACMAN* or *EARTH NAVIGATION* demos) you should set a small `animateDelay` value (for example 2 milliseconds) in order to avoid rendering lags.

* ` onWebcamAsk`: Function launched just before asking for the user to allow its webcam sharing,

* ` onWebcamGet`: Function launched just after the user has accepted to share its video. It is called with the video element as argument,

* ` videoSetting`: override WebRTC specified video settings, which are by default :

```javascript

{

  'videoElement' //not set by default.  element used

   //If you specify this parameter,

   //all other settings will be useless

   //it means that you fully handle the video aspect



  'deviceId'             //not set by default

  'facingMode': 'user', //to use the rear camera, set to 'environment'



  'idealWidth': 800,  //ideal video width in pixels

  'idealHeight': 600, //ideal video height in pixels

  'minWidth': 480,    //min video width in pixels

  'maxWidth': 1280,   //max video width in pixels

  'minHeight': 480,   //min video height in pixels

  'maxHeight': 1280   //max video height in pixels

}

```

If the user has a mobile device in portrait display mode, the width and height of these parameters are automatically inverted for the first camera request. If it does not succeed, we invert the width and height.



### Error codes

The initialization function ( `callbackReady` in the code snippet ) will be called with an error code ( `errCode` ). It can have these values :

* `false`: no error occurs,

* `"GL_INCOMPATIBLE"`: WebGL is not available, or this WebGL configuration is not enough (there is no WebGL2, or there is WebGL1 without OES_TEXTURE_FLOAT or OES_TEXTURE_HALF_FLOAT extension),

* `"ALREADY_INITIALIZED"`: the API has been already initialized,

* `"NO_CANVASID"`: no canvas ID was specified,

* `"INVALID_CANVASID"`: cannot found the `` element in the DOM,

* `"INVALID_CANVASDIMENSIONS"`: the dimensions `width` and `height` of the canvas are not specified,

* `"WEBCAM_UNAVAILABLE"`: cannot get access to the webcam (the user has no webcam, or it has not accepted to share the device, or the webcam is already busy),

* `"GLCONTEXT_LOST"`: The WebGL context was lost. If the context is lost after the initialization, the `callbackReady` function will be launched a second time with this value as error code,

* `"MAXFACES_TOOHIGH"`: The maximum number of detected and tracked faces, specified by the optional init argument `maxFacesDetected`, is too high.



### The returned objects

We detail here the arguments of the callback functions like `callbackReady` or `callbackTrack`. The reference of these objects do not change for memory optimization purpose. So you should copy their property values if you want to keep them unchanged outside the callback functions scopes.



#### The initialization returned object

The initialization callback function ( `callbackReady` in the code snippet ) is called with a second argument, `spec`, if there is no error. `spec` is a dictionnary having these properties :

* `GL`: the WebGL context. The rendering 3D engine should use this WebGL context,

* `canvasElement`: the `` element,

* `videoTexture`: a WebGL texture displaying the webcam video. It matches the dimensions of the canvas. It can be used as a background,

* `maxFacesDetected`: the maximum number of detected faces. 



#### The detection state

At each render iteration a callback function is executed ( `callbackTrack` in the code snippet ). It has one argument ( `detectState` ) which is a dictionnary with these properties :

* `detected`: the face detection probability, between `0` and `1`,

* `x`, `y`: The 2D coordinates of the center of the detection frame in the viewport (each between -1 and 1, `x` from left to right and `y` from bottom to top),

* `s`: the scale along the horizontal axis of the detection frame, between 0 and 1 (1 for the full width). The detection frame is always square,

* `rx`, `ry`, `rz`: the Euler angles of the head rotation in radians.

* `expressions`: `Float32Array` listing the facial expression coefficients :

    * `expressions[0]`: mouth opening coefficient (`0` → mouth closed, `1` → mouth fully opened)



### Miscellaneous methods

After the initialization (ie after that `callbackReady` is launched ) , these methods are available :



* `JEEFACEFILTERAPI.resize()`: should be called after resizing the `` element to adapt the cut of the video,



* `JEEFACEFILTERAPI.toggle_pause( isPause)`: pause/resume,



* `JEEFACEFILTERAPI.toggle_slow( isSlow)`: toggle the slow rendering mode: because this API consumes a lot of GPU resources, it may slow down other elements of the application. If the user opens a CSS menu for example, the CSS transitions and the DOM update can be slow. With this function you can slow down the rendering in order to relieve the GPU. Unfortunately the tracking and the 3D rendering will also be slower but this is not a problem is the user is focusing on other elements of the application. We encourage to enable the slow mode as soon as a the user's attention is focused on a different part of the canvas,



* `JEEFACEFILTERAPI.set_animateDelay( delay)`: Change the `animateDelay` (see `init()` arguments),



* `set_inputTexture( tex,  width,  height)`: Change the video input by a WebGL Texture instance. The dimensions of the texture, in pixels, should be provided,



* `reset_inputTexture()`: Come back to the user's video as input texture,



* `get_videoDevices( callback)`: Should be called before the `init` method. 2 arguments are provided to the callback function:

  * ` mediaDevices`: an array with all the devices founds. Each device is a javascript object having a `deviceId` string attribute. This value can be provided to the `init` method to use a specific webcam. If an error happens, this value is set to `false`,

  * ` errorLabel`: if an error happens, the label of the error. It can be: `NOTSUPPORTED`, `NODEVICESFOUND` or `PROMISEREJECTED`.



### Optimization



#### Canvas and video resolutions



We strongly recommend the use of the `JeelizResizer` helper in order to size the canvas to the display size in order to not compute more pixels than required. This helper also computes the best camera resolution, which is the closer to the canvas actual size. If the camera resolution is too high compared to the canvas resolution, your application will be unnecessarily slowed because it is quite costly to refresh the WebGL texture for each video frame. And if the video resolution is too low compared to the canvas resolution, the image will be blurry. You can take a look at the THREE.js boilerplate to see how it is used. To use the helper, you first need to include it in the HTML code :

```



```

Then in your main script, before initializing Jeeliz FaceFilter, you should call it to size the canvas to the best resolution and to find the optimal video resolution :

```

JeelizResizer.size_canvas({

  canvasId: 'jeeFaceFilterCanvas',

    callback: function(isError, bestVideoSettings){

        JEEFACEFILTERAPI.init({

          videoSettings: bestVideoSettings,

          //...

          //...

        });

    }

});

```

Take a look at the source code of this helper (in [helpers/JeelizResize.js](helpers/JeelizResize.js)) to get more information.



#### Misc



A few tips :

* In term of optimisation, the WebGL based demos are more optimized than Canvas2D demos, which are still more optimized than CSS3D demos.

* Try to use lighter resources as possibles. Each texture image should have the lowest resolution as possible, use mipmapping for texture minification filtering.

* The more effects you use, the slower it will be. Add the 3D effects gradually to check that they do not penalize too much the frame rate.

* Use low polygon meshes.



### Multiple faces

It is possible to detect and track several faces at the same time. To enable this feature, you only have to specify the optional init parameter `maxFacesDetected`. Its maximum value is `8`. Indeed, if you are tracking for example 8 faces at the same time, the detection will be slower because there is 8 times less computing power per tracked face. If you have set this value to `8` but if there is only `1` face detected, it should not slow down too much compared to the single face tracking.



If multiple face tracking is enabled, the `callbackTrack` function is called with an array of detection states (instead of being executed with a simple detection state). The detection state format is still the same.



You can use our `Three.js` multiple faces detection helper, `helpers/JeelizThreejsHelper.js` to get started and test [this example](https://jeeliz.com/demos/faceFilter/demos/threejs/multiCubes/). The [main script](demos/threejs/multiCubes/demo_multiCubes.js) has only 60 lines of code !



### Changing the 3D engine

It is possible to use another 3D engine than BABYLON.JS or THREE.JS. If you have accomplished this work, we would be interested to add your demonstration in this repository (or link to your code). Just open a pull request.



The 3D engine should share the WebGL context with FaceFilter API. The WebGL context is created by Jeeliz Face Filter. The background video texture is given directly as a `WebGLTexture` object, so it is usable only on the FaceFilter WebGL context. It would be more costly in term of computating time to have a second WebGL context for the 3D rendering, because at each new video frame we should transfert the video data from the `` element to the 2 webgl contexts: the Jeeliz Face Filter WebGL context for processing, and the 3D engine WebGL Context for rendering. Fortunately, with BABYLON.JS or THREE.JS, it is easy to specify an already initialized WebGL context.



### Changing the neural network

Since July 2018 it is possible to change the neural network. When calling `JEEFACEFILTERAPI.init({...})` with `NNCpath: ` you set NNCpath value to a specific neural network file :



```javascript

  JEEFACEFILTERAPI.init({

    NNCpath: '../../dist/NNClight.json'

    //...

  })

```



We provide several neural network models :

* `dist/NNC.json`: this is the default neural network. Good tradeoff between size and performances,

* `dist/NNCwideAngles.json`: this neural network is better to detect wide head angles (but less accurate for small angles),

* `dist/NNClight.json`: this is a light version of the neural network. The file is twice lighter and it runs faster but it is less accurate for large head rotation angles,

* `dist/NNCveryLight.json`: even lighter than the previous version: 250Kbytes, and very fast. But not very accurate and robust to all lighting conditions,

* `dist/NNCdeprecated.json`: this is a deprecated version of the neural network (since 2018-07-25).



### Using ES6

`/dist/jeelizFaceFilterES6.js` is exactly the same than `/dist/jeelizFaceFilter.js` except that it works with ES6, so you can import it directly using :



```javascript

import 'dist/jeelizFaceFilterES6.js'

```



or using `require`.



## Hosting



This API requires the user's webcam video feed through `MediaStream API`. So your application should be hosted by a HTTPS server (even with a self-signed certificate). It won't work at all with unsecure HTTP, even locally with some web browsers.



### The development server

For development purpose we provide a simple and minimalist HTTPS server in order to check out the demos or develop your very own filters. To launch it, execute in the bash console :



```bash

  python2 httpsServer.py

```

It requires Python 2.X. Then open in your web browser [https://localhost:4443](https://localhost:4443).



### Hosting optimization

You can use our hosted and up to date version of the library, available here :

```

https://appstatic.jeeliz.com/faceFilter/jeelizFaceFilter.js

```

It uses the neuron network `NNC.json` hosted in the same path. The helpers used in these demos (all scripts in [/helpers/](helpers/)) are also hosted on `https://appstatic.jeeliz.com/faceFilter/`.



It is served through a content delivery network (CDN) using gzip compression.

If you host the scripts by yourself, be careful to enable gzip HTTP/HTTPS compression for JSON and JS files. Indeed, the neuron network JSON file, `dist/NNC.json` is quite heavy, but very well compressed with GZIP. You can check the gzip compression of your server [here](https://checkgzipcompression.com/).



The neuron network file, `dist/NNC.json` is loaded using an ajax `XMLHttpRequest` after calling `JEEFACEFILTER.init()`. This loading is proceeded after the user has accepted to share its camera. So we won't load this quite heavy file if the user refuses to share it or if there is no webcam available. The loading can be faster if you systematically preload `dist/NNC.json` using a service worker or a simple raw `XMLHttpRequest` just after the HTML page loading. Then the file will be already in the browser cache when Jeeliz Facefilter API will request it.



## About the tech

### Under the hood

This API uses Jeeliz WebGL Deep Learning technology to detect and track the user's face using a neural network. The accuracy is adaptative: the best is the hardware, the more detections are processed per second. All is done client-side.



### Compatibility

* If `WebGL2` is available, it uses `WebGL2` and no specific extension is required,

* If `WebGL2` is not available but `WebGL1`, we require either `OES_TEXTURE_FLOAT` extension or `OES_TEXTURE_HALF_FLOAT` extension,

* If `WebGL2` is not available, and if `WebGL1` is not available or neither `OES_TEXTURE_FLOAT` or `OES_HALF_TEXTURE_FLOAT` are implemented, the user is not compatible.



In all cases, WebRTC should be implemented in the web browser, otherwise FaceFilter API will not be able to get the webcam video feed. Here are the compatibility tables from [caniuse.com](https://caniuse.com/) here: [WebGL1](https://caniuse.com/#feat=webgl), [WebGL2](https://caniuse.com/#feat=webgl2), [WebRTC](https://caniuse.com/#feat=stream).



If a compatibility error is triggered, please post an issue on this repository. If this is a problem with the webcam access, please first retry after closing all applications which could use your device (Skype, Messenger, other browser tabs and windows, ...). Please include :

* a screenshot of [webglreport.com - WebGL1](http://webglreport.com/?v=1) (about your `WebGL1` implementation),

* a screenshot of [webglreport.com - WebGL2](http://webglreport.com/?v=2) (about your `WebGL2` implementation),

* the log from the web console,

* the steps to reproduce the bug, and screenshots.



## Articles and tutorials

We are currently writing a series of tutorial for the API, starting by building some very basic filters and moving to harder ones.



* Creating a Snapchat-like face filter using Jeeliz FaceFilter API and THREE.JS :

  * Part 1: [Creating your first filter](https://jeeliz.com/blog/creating-a-snapchat-like-filter-with-jeelizs-facefilter-api-part-1-creating-your-first-filter/)

  * Part 2: [ User interactions and particles](https://jeeliz.com/blog/creating-a-snapchat-like-filter-with-jeelizs-facefilter-api-part-2-user-interactions-and-particles/)



* [Flying around the Globe with Cesium and Your Head](https://cesium.com/blog/2018/03/22/jeeliz-and-cesium/)



* [Build a multifacial face filter](https://webglacademy.jeeliz.com/courses.php?courses=19_25_27_33_34#34): Interactive step by step tutorial hosted on [WebGL Academy](http://www.webglacademy.com) where you learn to build a Statue of Liberty using THREE.js and this library



* [JSFiddle: Face detection and tracking in 25 lines](https://jsfiddle.net/jeeliz/2p34hbeh/)



* Tutorial: Matrix theme face filter: [French version](https://xavierbourry.developpez.com/filtre-facial-webcam/), [English translation](https://jeeliz.com/blog/tutorial-javascript-webgl-webcam-facial-filter-on-the-theme-of-matrix/)



## License

[Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0.html). This application is free for both commercial and non-commercial use.



We appreciate attribution by including the [Jeeliz logo](https://jeeliz.com/wp-content/uploads/2018/01/LOGO_JEELIZ_BLUE.png) and a link to your application or desktop website. Of course we do not expect a large link to Jeeliz over your face filter, but if you can put the link in the credits/about/help/footer section it would be great.



## See also

Our newest deep learning based library is called *Weboji*. It detects 11 facial expressions in real time from the webcam video feed. Then they are reproduced on an avatar, either in 3D with a THREE.JS renderer or in 2D with a SVG renderer (so you can use it even if you are not a 3D developer). You can access to the github repository.



If you just want to detect if the user is looking at the screen or not, [Jeeliz Glance Tracker]is what you are looking for. It can be useful to play and pause a video whether the user is watching or not. This library needs fewer resources and the neural network file is much lighter.



If you want to use this library for glasses virtual try-on (sunglasses, spectacles, ski masks), you can take a look at [Jeeliz VTO widget]. It includes a high quality and lightweight 3D engine which implements the following features: deferred shading, PBR, raytraced shadows, normal mapping, ... It also reconstructs the lighting environment around the user (ambient and directional lighting). But the glasses comes from a database hosted in our servers. If you want to add some models, please contact us.