Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/enoxsoftware/hololenswithdlibfacelandmarkdetectorexample

HoloLens With DlibFaceLandmarkDetector Example (Support for Hololens1 and Hololens2)
https://github.com/enoxsoftware/hololenswithdlibfacelandmarkdetectorexample

augmented-reality dlib-face-detection face-detection face-landmarks hololens hololens2 mixed-reality mrtk opencv unity

Last synced: 6 days ago
JSON representation

HoloLens With DlibFaceLandmarkDetector Example (Support for Hololens1 and Hololens2)

Host: GitHub
URL: https://github.com/enoxsoftware/hololenswithdlibfacelandmarkdetectorexample
Owner: EnoxSoftware
Created: 2017-05-13T19:19:41.000Z (over 7 years ago)
Default Branch: master
Last Pushed: 2023-01-15T05:48:47.000Z (almost 2 years ago)
Last Synced: 2024-03-15T04:51:57.192Z (8 months ago)
Topics: augmented-reality, dlib-face-detection, face-detection, face-landmarks, hololens, hololens2, mixed-reality, mrtk, opencv, unity
Language: C#
Homepage:
Size: 338 MB
Stars: 28
Watchers: 4
Forks: 11
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# HoloLens With DlibFaceLandmarkDetector Example

## Demo Video (old version)
[![](http://img.youtube.com/vi/egMZB_3DGVc/0.jpg)](https://youtu.be/egMZB_3DGVc)

## Environment
* HoloLens1 10.0.17763.316 / HoloLens2 22621.1399
* Windows 10 SDK 10.0.19041.0 / 10.0.22621.0
* Unity 2019.4.40f1 (Built-in Render Pipeline / LegacyXR / MRTK 2.8.3 / DirectX 11 / Visual Studio 2019 MSVC v142)
* Unity 2021.3.35f1 (Built-in Render Pipeline / OpenXR 1.11.1 / MRTK 2.8.3 / DirectX 11 / Visual Studio 2022 MSVC v143)
* Unity 2021.3.35f1 (Built-in Render Pipeline / OpenXR 1.11.1 / MRTK 3.2.2 / DirectX 11 / Visual Studio 2022 MSVC v143)
* [Mixed Reality Feature Tool](https://learn.microsoft.com/en-us/windows/mixed-reality/develop/unity/welcome-to-mr-feature-tool)
* [OpenCV for Unity](https://assetstore.unity.com/packages/tools/integration/opencv-for-unity-21088?aid=1011l4ehR) 2.6.3+
* [Dlib FaceLandmarkDetector](https://assetstore.unity.com/packages/tools/integration/dlib-facelandmark-detector-64314?aid=1011l4ehR) 1.4.0+
* [EnoxSoftware/HoloLensCameraStream](https://github.com/EnoxSoftware/HoloLensCameraStream)

## Setup (Common)
1. Download the latest release unitypackage. [HoloLensWithDlibFaceLandmarkDetectorExample.unitypackage](https://github.com/EnoxSoftware/HoloLensWithDlibFaceLandmarkDetectorExample/releases)
1. Create a new project. (`HoloLensWithDlibFaceLandmarkDetectorExample`)
* Change the platform to `UWP` in the "Build Settings" window.
1. Import and setup the Microsoft Mixed Reality Toolkit. (Recommend using [MixedRealityFeatureTool](https://www.microsoft.com/en-us/download/details.aspx?id=102778))
1. Import the OpenCVForUnity.
* Select MenuItem[Tools/OpenCV for Unity/Open Setup Tools].
* Click the [Move StreamingAssets Folder] button.
* Leave the following files and delete the rest. ("StreamingAssets/OpenCVForUnity/objdetect/haarcascade_frontalface_alt.xml", "lbpcascade_ frontalface.xml")
1. Import the DlibFaceLandmarkDetector.
* Select MenuItem[Tools/Dlib FaceLandmark Detector/Open Setup Tools].
* Click the [Move StreamingAssets Folder] button.
* Leave the following files and delete the rest. ("StreamingAssets/DlibFaceLandmarkDetector/sp_human_face_68.dat", "sp_human_face_68_for_mobile.dat", "sp_human_face_17.dat", "sp_human_face_17_for_mobile.dat" and "sp_human_face_6.dat")
1. Clone HoloLensCameraStream repository.
* Copy the "HoloLensCameraStream/HoloLensVideoCaptureExample/Assets/CamStream/" folder to the "Assets/" folder.
1. Import the HoloLensWithDlibFaceLandmarkDetectorExample.unitypackage.
1. Add the "Assets/HoloLensWithDlibFaceLandmarkDetectorExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
1. Configure settings in the "Project Settings" window.
* Add Define Symbols: the following to `Scripting Define Symbols` depending on the XR system used in your project; Legacy built-in XR: `BUILTIN_XR`; XR Plugin Management (Windows Mixed Reality): `XR_PLUGIN_WINDOWSMR`; XR Plugin Management (OpenXR):`XR_PLUGIN_OPENXR`.
* Enable `WebCam` Capabilties in Publishing settings tab.
1. (Optional) Setup a performance environment suitable for Holorens. (See [https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/recommended-settings-for-unity](https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/recommended-settings-for-unity))
1. **Build the project:** You can now build the Unity project, which generates a Visual Studio Solution (which you will then have to also build). With the Build Settings window still open, click **Build**. In the explorer window that appears, make a new folder called `App`, which should live as a sibling next to the 'Assets` folder. Then click Select Folder to generate the VS solution in that folder. Then wait for Unity to build the solution.
1. **Open the VS Solution:** When the solution is built, a Windows explorer folder will open. Open the newly-built VS solution, which lives in `App/HoloLensWithDlibFaceLandmarkDetectorExample.sln`. This is the solution that ultimately gets deployed to your HoloLens.
1. **Configure the deploy settings:** In the Visual Studio toolbar, change the solution platform from `ARM` to `x86` if you are building for Hololens1 or to `ARM64` if you are building for Hololens2; Change the deploy target (the green play button) to `Device` (if your HoloLens is plugged into your computer), or `Remote Machine` (if your HoloLens is connected via WiFi).
1. **Run the app:** Go to **Debug > Start Debugging**. Once the app is deployed to the HoloLens, you should see some confirmation output in the Output window.
* (Print the AR marker "CanonicalMarker-d10-i1-sp500-bb1.pdf" on an A4 size paper)

* [Choosing a Unity version and XR plugin](https://learn.microsoft.com/en-us/windows/mixed-reality/develop/unity/choosing-unity-version)
* [Known issues in Unity versions and packages](https://learn.microsoft.com/en-us/windows/mixed-reality/develop/unity/known-issues)

## Setup (HoloLens1 / Unity 2019 (Built-in Render Pipeline / LegacyXR / MRTK 2 / DirectX 11 / Visual Studio 2019))
1. Download the latest release unitypackage.
1. Create a new project. (`HoloLensWithDlibFaceLandmarkDetectorExample`)
1. Import the Microsoft Mixed Reality Toolkit.
* Add MRTK2 (Mixed Reality Toolkit - Mixed Reality Toolkit Foundation) to the project using "Mixed Reality Feature Tool".
* Follow the MRTK2 configuration dialog to set up the project. (XR System: Legacy Built-in XR)
1. Import the OpenCVForUnity.
1. Import the DlibFaceLandmarkDetector.
1. Import the HoloLensCameraStream.
1. Import the HoloLensWithDlibFaceLandmarkDetectorExampleMRTK2.unitypackage.
1. Add the "Assets/HoloLensWithDlibFaceLandmarkDetectorExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
1. Configure settings in the "Project Settings" window.
* Add `BUILTIN_XR` to the list of Define Symbols.
* Enable `WebCam` Capabilties in Publishing settings tab.
1. Build the project; open it in VS2019, set the deployment target to `x86` and deploy it to the Hololens1 actual device.

## Setup (Unity 2021 (Built-in Render Pipeline / OpenXR / MRTK 2 / DirectX 11 / Visual Studio 2022))
1. Download the latest release unitypackage.
1. Create a new project. (`HoloLensWithDlibFaceLandmarkDetectorExample`)
1. Import the Microsoft Mixed Reality Toolkit.
* Add MRTK2 (Mixed Reality Toolkit - Mixed Reality Toolkit Foundation) and OpenXR(Platform Support - Mixed Reality OpenXR Plugin) to the project using "Mixed Reality Feature Tool".
* Follow the MRTK2 configuration dialog to set up the project. (XR System: OpenXR)
1. Import the OpenCVForUnity.
1. Import the DlibFaceLandmarkDetector.
1. Import the HoloLensCameraStream.
1. Import the HoloLensWithDlibFaceLandmarkDetectorExampleMRTK2.unitypackage.
1. Add the "Assets/HoloLensWithDlibFaceLandmarkDetectorExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
1. Configure settings in the "Project Settings" window.
* Add `XR_PLUGIN_OPENXR` to the list of Define Symbols.
* Enable `WebCam` Capabilties in Publishing settings tab.
1. Build the project; open it in VS2022, set the deployment target to `ARM64` and deploy it to the Hololens2 actual device.

* [Set up a new OpenXR project with MRTK](https://learn.microsoft.com/en-us/windows/mixed-reality/develop/unity/new-openxr-project-with-mrtk)

## Setup (Unity 2021 (Built-in Render Pipeline / OpenXR / MRTK 3 / DirectX 11 / Visual Studio 2022))
1. Download the latest release unitypackage.
1. Create a new project. (`HoloLensWithDlibFaceLandmarkDetectorExample`)
1. Import the Microsoft Mixed Reality Toolkit.
* Add MRTK3 (MRTK3 - MRTK Input, MRTK UX Components, MRTK UX Components (Non-Canvas)) and OpenXR(Platform Support - Mixed Reality OpenXR Plugin) to the project using "Mixed Reality Feature Tool".
* * Set up the MRTK3. (XR System: OpenXR)
1. Import the OpenCVForUnity.
1. Import the DlibFaceLandmarkDetector.
1. Import the HoloLensCameraStream.
1. Import the HoloLensWithDlibFaceLandmarkDetectorExampleMRTK3.unitypackage.
1. Add the "Assets/HoloLensWithDlibFaceLandmarkDetectorExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
1. Configure settings in the "Project Settings" window.
* Add `XR_PLUGIN_OPENXR` to the list of Define Symbols.
* Enable `WebCam` Capabilties in Publishing settings tab.
1. Build the project; open it in VS2022, set the deployment target to `ARM64` and deploy it to the Hololens2 actual device.

* [Setting up a new Unity project with MRTK3](https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk3-overview/getting-started/setting-up/setup-new-project)

|Project Assets|Build Settings|
|---|---|
|![ProjectAssets.jpg](ProjectAssets.jpg)|![BuildSettings.jpg](BuildSettings.jpg)|

|Player Settings|
|---|
|![PlayerSettings.jpg](PlayerSettings.jpg)|

## ScreenShot (old version)
![screenshot01.jpg](screenshot01.jpg)

![screenshot02.jpg](screenshot02.jpg)

![screenshot03.jpg](screenshot03.jpg)