Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/scrn-vrc/svm-face-and-object-detection-shader

SVM using HOG descriptors implemented in fragment shaders
https://github.com/scrn-vrc/svm-face-and-object-detection-shader

face-detection hlsl hog-features opencv shaderlab svm svm-rbf unity unity3d vrchat

Last synced: 27 days ago
JSON representation

SVM using HOG descriptors implemented in fragment shaders

Awesome Lists containing this project

README 

        





# Face and Object Detection Shader



A binary classifier implemented with a support vector machine (SVM) written in C++ and converted to HLSL to be used inside VRChat. I have included a program that allows you to train your own detector. You can use it to detect other objects as well.



## Overview




My implementation has three camera inputs to detect varying sizes 480 x 270, 240 x 135, 120 x 67. A 64 x 64 sliding window is applied with a stride of 16 pixels for the largest camera inputs to 8 pixels for the smallest.



HOG features are extracted by binning by the magnitude of the gradient according to the direction of the gradient into 8 bins. The features are normalized to account for differences in lighting.



The radial basis function (RBF) kernel is applied to each of the 1568 features per 64 x 64 image block and the support vectors. At the end, classification is done by doing the summation of all RBF calculations per 64 x 64 image block.



## Problems



I have fixed most of the problems I listed in the previous implementation. However,

1. In VRChat, cameras are friends only, you can only show this to your friends.

2. SVMs needs to store data of the hyperplane that does the classification. So more training = more complex hyper plane = lag in game.



## Setup



No prerequisites, either clone the repository or download the latest [Release](https://github.com/SCRN-VRC/SVM-Face-and-Object-Detection-Shader/releases) and open it in Unity



## Training



These steps are only for the people who want to train their own detector.

1. Get the svm-detector.exe in [Release](https://github.com/SCRN-VRC/SVM-Face-and-Object-Detection-Shader/releases) or compile it yourself. **The .exe only works with x64 bit machines, anything else you'll have to compile it yourself. Compiling the C++ code requires OpenCV 4.0.1.**

2. Layout your training data like this example for the Faces folder. **All pictures must be 64 x 64 in size.**




3. Run svm-detector.exe with the path to the folder. For example, ```svm-detector.exe D:\Storage\Datasets\Faces 100``` will look for images in the Faces folder and train for 100 iterations.

4. After the program finishes, it will output a ```svm-out.xml``` file containing the support vectors for classification.

5. To use it inside VRChat, drop the ```svm-out.xml``` file into the Unity project with this repository.

6. Navigate the menu Tool -> SCRN -> Bake Support Vectors, put in the .xml file and hit Bake!




7. After a few seconds there should be a new ```SupportVectors.asset``` file in the SVM Detector folder. If a ```SupportVectors.asset``` already exists, it will be overwritten.

8. In the SVM prefab, find the game object called SVM place the new ```SupportVectors.asset``` into the Support Vectors texture slot.




9. Click on the ```svm-out.xml``` file and look at the contents in the Inspector, remember the sv_total number.




10. Locate ```svmhelper.cginc``` inside the Shaders folder.

11. Change number after SV_NUM in ```svmhelper.cginc``` to the number you saw for sv_total inside ```svm-out.xml```.




12. Done!



## Contact



Contact me on Discord if you have any questions or suggestions: **SCRN#8008**



Thanks to [Merlin](https://github.com/MerlinVR) and 1001 for making this possible.