https://github.com/syanenko/magichand
Hand pose detection with shaders effects
https://github.com/syanenko/magichand
c deep-learning glsl glsl-shaders glut gpu opencv opengl
Last synced: about 2 months ago
JSON representation
Hand pose detection with shaders effects
- Host: GitHub
- URL: https://github.com/syanenko/magichand
- Owner: syanenko
- License: gpl-3.0
- Created: 2020-12-17T15:42:43.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2024-01-21T17:10:26.000Z (over 2 years ago)
- Last Synced: 2025-07-31T02:30:05.053Z (11 months ago)
- Topics: c, deep-learning, glsl, glsl-shaders, glut, gpu, opencv, opengl
- Language: GLSL
- Homepage:
- Size: 7.77 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Magic hand
-- What
Hand pose detection with OpenCV/DNN, applying shaders effects with OpenGL/GLUT/GLSL.
-- Why
Explore OpenCV, DNN, OpenGL interoperability, play with GLSL
-- How
* OpenGL/GLUT/GLSL part based on Peter R. Elespuru code: https://github.com/elespuru/elespuru-cvglsl
* OpenCV hand pose detection based on this example: https://github.com/opencv/opencv/blob/master/samples/dnn/openpose.cpp
#### Requirements
* *Hardware:* Nvidia Geforce RTX 2060 or higher (OpenCV build for CUDA compute capability 7.5, you can rebuild it for GPU you have).
More about CUDA architecture versions: https://developer.nvidia.com/cuda-gpus
* *Software:* Windows 7 or higher
#### Running from pre-built binaries
1. Download DNN model: https://drive.google.com/file/d/1nms4M5plslVEG1RSzLQVydi8Df7KiaLs/view?usp=sharing (or use getModels.sh)
and unzip to `\MagicHand\model`.
1. Download required DLLs: https://drive.google.com/file/d/115ty4cpAiCNTgYNGquw9NbH4c7g5JztJ/view?usp=sharing
and unzip to `\MagicHand\bin`.
1. Run:
```
cd \MagicHand
.\bin\MagicHand.exe
```
#### Compiling from sources
1. Install NVIDIA CUDA Toolkit 10.2: https://developer.nvidia.com/cuda-10.2-download-archive.
1. Install cuDNN for CUDA 10.2: https://developer.nvidia.com/rdp/cudnn-download.
1. Download and build OpenCV: https://github.com/opencv/opencv with options WITH_CUDA=ON, OPENCV_DNN_CUDA=ON,
detailed instructions are here: https://www.pyimagesearch.com/2020/02/03/how-to-use-opencvs-dnn-module-with-nvidia-gpus-cuda-and-cudnn/.
1. Download and install FreeGlut: http://freeglut.sourceforge.net/.
1. Download and install Glew: http://glew.sourceforge.net/.
1. Open MagicHand solution in Visual Studio 2017 or later, then open MagicHand project's properties.
1. Change appropriate include directories in section `C/C++/General/Additional Include Directories` and library directories in section `Linker/General/Additional Library Directories` according to locations of above dependencies.
1. Build desired configuration.
1. Be sure that `freeglut.dll` and `glew32.dll` are in the path or place them in `\MagicHand\bin`.
1. Run: `Ctrl+F5` or
```
cd \MagicHand
.\bin\MagicHand.exe
```
#### Help
```
Usage: MagicHand [-h][-v [`]]
-h: Show this help
-v: Save video to , will be 'MagicHand.avi' if not specified
```
#### Control
| Keys | Actions |
|------|---------|
| PgUp/PgDn | switch shaders |
| +/- | zooms in/out |
|Arrows|pan|
|P / p|change number of passes|
|a|draw axes|
|t or , | decrease net threshold|
|T or . | increase net threshold|
|d|show debug window|
|j| draw joints (focus on debug window)|
|r| draw areas (focus on debug window)|
|l| draw lines (focus on debug window)|
|z| decrease area size (focus on debug window)|
|x| increase area size (focus on debug window)|
|Enter| set default views|
|Esc|exit|
#### Technical details
Following uniforms are accessible in shaders:
```
uniform sampler2D u_texture; // - Image, captured from camera
uniform vec2 u_resolution; // - Image size
uniform float u_time; // - Shader running time in seconds
uniform vec2 u_mouse_move; // - Free mouse move coordinates as retuned by glutPassiveMotionFunc()
uniform vec4 u_mouse_click; // - Mouse click position and state as retuned by glutMouseFunc()
uniform vec2 u_hand[22]; // - Hand points, detected by DNN module
```
Glut functions reference:
https://www.opengl.org/resources/libraries/glut/spec3/node51.html
https://www.opengl.org/resources/libraries/glut/spec3/node50.html
#### Hand pose detection background
General idea: https://www.learnopencv.com/hand-keypoint-detection-using-deep-learning-and-opencv/
Original paper: "Hand Keypoint Detection in Single Images using Multiview Bootstrapping" by Tomas, Simon Hanbyul Joo, Iain Matthews, Yaser Sheikh from Carnegie Mellon University (https://github.com/syanenko/MagicHand/blob/main/doc/1704.07809.pdf).
Contact me at [LinkedIn](https://www.linkedin.com/in/sergey-yanenko-57b21a96/).