https://github.com/der3318/head-mapped-key
Java Console App - Control Keyboard Using Head With Webcam
https://github.com/der3318/head-mapped-key
automation keyboard onnxruntime pose webcam
Last synced: about 1 year ago
JSON representation
Java Console App - Control Keyboard Using Head With Webcam
- Host: GitHub
- URL: https://github.com/der3318/head-mapped-key
- Owner: der3318
- License: mit
- Created: 2023-10-07T08:49:04.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-10-22T06:32:16.000Z (over 2 years ago)
- Last Synced: 2025-03-03T07:13:59.998Z (over 1 year ago)
- Topics: automation, keyboard, onnxruntime, pose, webcam
- Language: Java
- Homepage:
- Size: 21.6 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## ⌨️ Head Pose Keyboard Mapping
[](https://github.com/sarxos/webcam-capture)
[](https://github.com/microsoft/onnxruntime)
A Java program that can trigger keyboard actions using head poses streamed from PC camera, leveraging the end-to-end landmark detection ONNX model (ref: [atksh/onnx-facial-lmk-detector](https://github.com/atksh/onnx-facial-lmk-detector)):
### How to Run
| Using Portable Version |
| :- |
| For Windows AMD64 devices, an all-in-one app package [head-mapped-key-portable-x64.zip](https://github.com/der3318/head-mapped-key/releases/download/2023.10.16/head-mapped-key-portable-x64.zip) is available. Unzip and double-click `launch.bat` to turn camera on and start the tool. |
| Using JRE 17+ |
| :- |
| Download fat JAR [head-mapped-key-all.jar](https://github.com/der3318/head-mapped-key/releases/download/2023.10.16/head-mapped-key-all.jar), [model.onnx](https://github.com/der3318/head-mapped-key/blob/main/model.onnx) and [app.properties](https://github.com/der3318/head-mapped-key/blob/main/app.properties) from repo. Put them together and run `java -Duser.language=en -Dfile.encoding=UTF8 -jar head-mapped-key-all.jar` in CLI. |
| Using Gradle Wrapper |
| :- |
| Clone the repository and use `gradlew run` command. |
Debug messages will be dumpped directly to the console. Press CTRL+C to stop listening. A JVM shutdown hook will help release resouces gracefully, including camera handle.
Also noted that the responsiveness of the tracking mechanism highly depends on the actual CPU speed. For example, on a 4-Core 1.90GHz Intel laptop, each frame takes about 900ms to process, which makes the overall experience a bit lagging.
### Customizing Settings
The editable `app.properties` (plain text file) provides basic values that can be consumed during runtime:
```txt
# preferred camera (will be prioritized if any)
camera.keyword=Camera Front
# how many ms should the worker wait to take next webcam capture
worker.delay=100
# https://docs.oracle.com/en/java/javase/17/docs/api/java.desktop/java/awt/event/KeyEvent.html
keyname.head.left=VK_LEFT
keyname.head.right=VK_RIGHT
```
Key mappings based on personal need would be the most useful one.
### How to Build & Redistribute
To compile the sources, JDK 17+ should be either accessible via environment variable `JAVA_HOME`, or the Java excutables are avaiable under `PATH`. This is the only prerequisite.
Command `gradlew shadowJar` will download Gradle v8.3 and use it to build the redistributable JAR: `build/libs/head-mapped-key-all.jar`, including all the dependencies.
### Inference Logic Behind the Scene
Nothing fancy. Given the horizontal locations of eyes and nose, the head turn can be roughly inferred.
For instance, `(Nose.X - LeftEye.X) : (RightEye.X - Nose.X) = (3 or higher) : 1` implies the head is facing relatively left (mirrored) to the sensor, and vice versa. Otherwise, the pose is considered centered.