{"id":23480592,"url":"https://github.com/farazkh80/airmouse","last_synced_at":"2026-05-20T10:02:22.789Z","repository":{"id":156658471,"uuid":"387307412","full_name":"farazkh80/AirMouse","owner":"farazkh80","description":"A hands-free PC control module using live hand gesture detection and finger movement tracking","archived":false,"fork":false,"pushed_at":"2021-09-10T17:18:18.000Z","size":77818,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-16T08:28:23.254Z","etag":null,"topics":["classification","computer-vision","gesture-detection","machine-learning","mediapipe"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/farazkh80.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-19T01:40:13.000Z","updated_at":"2024-01-21T11:25:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"83b7dd68-b55e-4d1f-bd58-a9b3e99a0db8","html_url":"https://github.com/farazkh80/AirMouse","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farazkh80%2FAirMouse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farazkh80%2FAirMouse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farazkh80%2FAirMouse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/farazkh80%2FAirMouse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/farazkh80","download_url":"https://codeload.github.com/farazkh80/AirMouse/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248773299,"owners_count":21159446,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["classification","computer-vision","gesture-detection","machine-learning","mediapipe"],"created_at":"2024-12-24T20:17:25.389Z","updated_at":"2026-05-20T10:02:17.749Z","avatar_url":"https://github.com/farazkh80.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AirMouse\n\n![demo.gif](visuals/demo.gif)\n\nA hands-free PC control module using live hand gesture detection and finger movement tracking.\nWith AirMouse, basic mouse operations such as clicking, moving cursor, scrolling and zooming are possible without even touching the mouse. Moreover, AirMouse' gesture detection extension adds on the possibility of user defined gestures and computer operations such as playing your favorite Spotify playlist by showing a thumps up!\n\n# Table of Contents\n\n- [AirMouse](#airmouse)\n- [Table of Contents](#table-of-contents)\n- [Features](#features)\n  - [1. Basic Mode Features](#1-basic-mode-features)\n  - [2. Gesture Mode Features](#2-gesture-mode-features)\n- [Installation/Usage](#installationusage)\n- [Motivation](#motivation)\n- [Project Overview](#project-overview)\n- [Current Issues](#current-issues)\n- [Contributing](#contributing)\n- [Roadmap](#roadmap)\n- [Acknowledgements](#acknowledgements)\n- [License](#license)\n\n# Features\n\nAirMouse features consist of two main modes, **Basic** mode used for basic mouse functionality such as moving the cursor and clicking, and **Gesture** mode used for custom gesture-based operation such as playing a music playlist or browsing a specific webpage.\n\n## 1. Basic Mode Features\n\n- 1.1 Starting AirMouse ✅\n- 1.2 Moving the Cursor 👆\n- 1.3 Clicking click 🖱\n- 1.4 Scrolling Up and Down ⬆⬇\n- 1.5 Enabling Gesture Mode 🚀\n- 1.6 Stopping AirMouse ❌\n\n## 2. Gesture Mode Features\n\n- 2.1 Zoom In 🔍\n- 2.2 Zoom Out 🔎\n- 2.3 Play Happy Songs 👍\n- 2.4 Play Sad Songs 👎\n- 2.5 Open Swag Picture on Browser 🤙\n- 2.6 Exit Gesture Mode ❌\n\n# Installation/Usage\n\n1. Clone the repository and change the directory to the path the cloned repository is located at\n\n```bash\ngit clone https://github.com/farazkh80/AirMouse.git\ncd AirMouse\n```\n\n2. Create a virtual environment and activate it\n\n```bash\nvirtualenv venv\nvenv/Scripts/activate\n```\n\n3. Install the project requirements\n\n```bash\npip install -r requirements.txt\n```\n\n4. To start the AirMouse webcam simply run\n\n```bash\npython MouseController.py\n```\n\n**Note: For proper starting and usage of AirMouse refer to [Features](#features).**\n\n# Motivation\n\nAirMouse is a personal project inspired by Google's Mediapipe Machine Learning solutions for hand detection. Using the coordination data collected by Mediapipe solutions, AirMouse takes the human-computer interaction experience to the next level by making computer control more personalized and creative.\n\n# Project Overview\n\nAirMouse file structure:\n\n```\nAirMouse\n│   README.md\n│   MouseController.py #main file with definitions for each mode\n|   HandDetectorModule.py #hand detector and finger distance calculator\n|   GestureDetectorScikit.py #scikit learn gesture classification model\n|   GestureDetectorTorch.py #Pytorch gesture classification DNN\n│\n└───data\n│   │   hands-coords.csv #collected hand landmark coordinates\n│\n│\n└───models #pre-trained model checkpoints\n|   |   gb-14-14-39-01-08-2021\n|   |   torch-mc-12-18-38-08-08-2021\n|\n|\n│\n└───prototypes\n    |   GestureClassifier.ipynb #scikit learn classifier prototype for AirMouse\n    |   GestureClassifierNet.ipynb #pytorch DNN prototype for AirMouse\n    |   HandDataCollector.ipynb #hand landmark coordinate collector\n    |   HolisticDataCollector.ipynb #holistic landmark coordinate collector\n```\n\n`MouseController.py` is the main controller that calls upon `HandDetectorModule.py` for hand detection and finger position tracking when in Basic Mode and calls either of `GestureClassifierTorch.py` and `GestureClassifierScikit.py` modules for detecting the gesture being shown when in Gesture Mode.\n\nWhen in **Basic** Mode:\n`HandDetectorModule.py` is responsible for detecting the position of the hand using OpenCV and then extract the hand landmark coordinates using Mediapipe Machine Learning Solutions `HandDetectorModule.py` can then determine the finger tip position and whether the fingers are up or down for automating basic operations such as moving mouse cursor around with the movement of index finger or clicking when index finger and middle finger are close to each other.\n\nWhen in **Gesture** Mode:\nany of `GestureClassifierTorch.py` and `GestureClassifierScikit.py` can use `/data/hands-coords.csv` data for training a Multi Classification Neural Network Model(Pytorch) or a pre-defined Classification model such as Logistic Regression or Random Forest Classifier(Scikit-learn) and predicting the gesture being captured by OpenCV webcam. Upon predicting the gesture class, the `MouseController.py` can perform personalized computer operations such as playing a specific video or showing the current weather outside.\n\n# Current Issues\n\n- Searching dataset initialization takes place before each new search, reducing the searching speed\n- Model loading from checkpoint also happens before each summarization task\n- Constant, non-configurable `text_max_token_len` of 512 and `summary_max_token_len` of 128 as suggested by the original [T5](https://arxiv.org/abs/1910.10683) paper, limit the word-range of input and output text\n\n# Contributing\n\nContributions are always welcome!\n\nA detailed guide about project specifics and in-depth architecture analysis will be released soon.\n\n# Roadmap\n\n- Fixing Issues:\n\n  - Avoid searching dataset initialization from happening before each search by either running both the text summarization and searching engines in a parallel setup or storing search dataset in a database.\n  - Load model from checkpoint only once at the beginning of the program or just before the first summarization prompt\n  - Add configurable `text_max_token_len` and `summary_max_token_len` to increase text word-range flexibility.\n\n- New Features:\n  - Fine-tune T5 Models with higher number of parameters such as `t5-large` and `t5-3B` for better results.\n  - Add other NLP models such as `BERT` and `ALBERT` for performance comparison.\n  - Add a web news scrapper to maintain an up-to-date version of latest most popular `n` news.\n\n# Acknowledgements\n\nT5 Text Summarizer\n\n- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683)\n- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)\n- [Hugging Face T5 API](https://huggingface.co/transformers/model_doc/t5.html)\n- [PyTorch Lightning Trainer](https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html)\n- [Text Summarization with T5](https://www.youtube.com/watch?v=KMyZUIraHio)\n\nSearch Engine\n\n- [Building a Search Engine with C++](https://www.education-ecosystem.com/nikos_tsiougkranas/ljJg5-how-to-build-a-search-engine-in-c/yDd46-intro-how-to-build-a-search-engine-in-c/)\n\n# License\n\nThis project is licensed under the terms of the [MIT](https://en.wikipedia.org/wiki/MIT_License) license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffarazkh80%2Fairmouse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffarazkh80%2Fairmouse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffarazkh80%2Fairmouse/lists"}