Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/kiyoon/nvim-hand-gesture
Write programs with hand gestures
https://github.com/kiyoon/nvim-hand-gesture
action-recognition computer-vision gesture-recognition neovim nvim nvim-plugin vim
Last synced: 3 months ago
JSON representation
Write programs with hand gestures
- Host: GitHub
- URL: https://github.com/kiyoon/nvim-hand-gesture
- Owner: kiyoon
- License: mit
- Created: 2022-12-20T05:18:05.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2023-03-21T18:01:18.000Z (almost 2 years ago)
- Last Synced: 2024-10-12T15:54:30.306Z (4 months ago)
- Topics: action-recognition, computer-vision, gesture-recognition, neovim, nvim, nvim-plugin, vim
- Language: Python
- Homepage:
- Size: 36.1 KB
- Stars: 90
- Watchers: 2
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Hand Gesture for Neovim
Use your webcam to unlock hand gesture recognition to control Neovim!
Highly experimental. Is it practical? Maybe not, but some use cases are not bad. Should you try it? Why not, it's fun!
Highly customisable. Just change the lua scripts to your own custom commands!
The demo uses [GitHub Copilot](https://github.com/github/copilot.vim) (AI code suggestion), [Nvim-Tree](https://github.com/nvim-tree/nvim-tree.lua) (file explorer), and [nvim-treesitter-textobjects](https://github.com/nvim-treesitter/nvim-treesitter-textobjects) (peep function definition).
## Prerequisites
You need an Nvidia GPU on a Linux computer. I tested with Ubuntu 22.04, GTX 1080 Ti and RTX 3060. The Nvidia driver has to support CUDA 11.3 (see `nvidia-smi` command).
I'm sorry, no Windows or Mac support as of yet.
## Installation
This plugin depends on a number of softwares so there's no easy way to install using plugin managers.
The easiest way to get started is using Docker.
You need to install [Nvidia-docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). From their official guide,
```bash
curl https://get.docker.com | sh \
&& sudo systemctl --now enable docker
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
&& curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker
```
### Pull the repo and the docker container.
Warning: this will download around 7GB. This program has many dependencies that makes it huge.
```bash
git clone https://github.com/kiyoon/nvim-hand-gesture
docker pull kiyoon/nvim-hand-gesture
```
### Before testing with Neovim, see if the program works.
```bash
# You may need to change the variables
WEBCAM=/dev/video0
REPO_PATH=~/nvim-hand-gesture
docker run --gpus all --rm -it \
--device=$WEBCAM:/dev/video0 \
-v "$REPO_PATH":/workspace kiyoon/nvim-hand-gesture
```
To enable the GUI demo,
```bash
xhost +local:docker
docker run --gpus all --rm -it \
--device=$WEBCAM:/dev/video0 \
-e DISPLAY=${DISPLAY} -v /tmp/.X11-unix:/tmp/.X11-unix \
-v "$REPO_PATH":/workspace kiyoon/nvim-hand-gesture --gui
```
### Once you saw it working, make vim bindings.
```vim
" With GUI demo
nmap G call system("docker run --gpus all --rm --device=/dev/video0:/dev/video0 -e DISPLAY=${DISPLAY} -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/project/nvim-hand-gesture:/workspace -v /run/user:/run/user kiyoon/nvim-hand-gesture --gui --nvim_socket_path " . v:servername . " &")
" Without GUI
nmap g call system("docker run --gpus all --rm --device=/dev/video0:/dev/video0 -v ~/project/nvim-hand-gesture:/workspace -v /run/user:/run/user kiyoon/nvim-hand-gesture --nvim_socket_path " . v:servername . " &")
" Quit running process
nmap g let g:quit_nvim_hand_gesture = 1
```
Note that you need to remove `-it` option, and may change the webcam device and nvim-hand-gesture repo path to the correct one.
`-v /run/user` mount is because Neovim's `v:servername` is usually `/run/user/$USER/nvim.$pid.0`. It may depend on the system.
## Customisation
Change lua files in [gesture_mappings](./gesture_mappings) to your likings. If the action is in [persistent_gestures.txt](./persistent_gestures.txt), the lua script will keep being executed until you stop that action. If not, it will only be executed once and be ignored for the rest of the duration.
### Setup tips
You may not want to make a gesture command intrusive to your original neovim workflow. To make it as practical as possible, try to avoid moving your cursor or amending text.
Instead, some example use cases include:
- Opening / closing a side bar (e.g. nvim-tree) without focussing on it
- Opening a floating window (e.g. peek function definition).
- Adjusting size of splits
I'd be happy if anyone shares their own use cases and good sets of defaults!
## Acknowledgement
This is a porting of [TSM demo](https://github.com/mit-han-lab/temporal-shift-module/tree/master/online_demo) in order to use it to control Neovim.