{"id":20346675,"url":"https://github.com/arthurfdlr/drone-gesture-control","last_synced_at":"2025-07-27T21:33:40.439Z","repository":{"id":112736028,"uuid":"396425090","full_name":"ArthurFDLR/drone-gesture-control","owner":"ArthurFDLR","description":"🚁 Robust proof-of-concept of a gesture-controlled drone: augmenting an ArduPilot flight controller with a Jetson Nano!","archived":false,"fork":false,"pushed_at":"2022-09-13T23:58:51.000Z","size":14864,"stargazers_count":49,"open_issues_count":0,"forks_count":17,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-19T08:51:20.653Z","etag":null,"topics":["ardupilot","drone","gesture-recognition","jetson-nano"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ArthurFDLR.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-15T17:01:28.000Z","updated_at":"2025-04-08T03:35:25.000Z","dependencies_parsed_at":"2023-09-13T12:45:43.561Z","dependency_job_id":null,"html_url":"https://github.com/ArthurFDLR/drone-gesture-control","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ArthurFDLR/drone-gesture-control","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurFDLR%2Fdrone-gesture-control","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurFDLR%2Fdrone-gesture-control/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurFDLR%2Fdrone-gesture-control/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurFDLR%2Fdrone-gesture-control/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ArthurFDLR","download_url":"https://codeload.github.com/ArthurFDLR/drone-gesture-control/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurFDLR%2Fdrone-gesture-control/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267427338,"owners_count":24085510,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-27T02:00:11.917Z","response_time":82,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ardupilot","drone","gesture-recognition","jetson-nano"],"created_at":"2024-11-14T22:13:47.921Z","updated_at":"2025-07-27T21:33:40.182Z","avatar_url":"https://github.com/ArthurFDLR.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align = \"center\"\u003e Vision-Based Gesture-Controlled Drone \u003c/h1\u003e\n\n[![MIT License][license-shield]][license-url]\n[![LinkedIn][linkedin-shield]][linkedin-url]\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/ArthurFDLR/drone-gesture-control/blob/main/.github/Github_Banner.jpg?raw=true\" alt=\"Banner\" width=\"100%\" style=\"border-radius: 5px;\"\u003e\n\u003c/p\u003e\n\nThis project leverages the Jetson Nano's computational power to augment a drone with computer vision capabilities and allow gesture control. The deep learning model deployed here is part of a larger project, a [**Pose Classification Kit**](https://github.com/ArthurFDLR/pose-classification-kit), focusing on pose estimation/classification applications toward new human-machine interfaces.\n\n- [Demonstration \u0026 Processing pipeline description](#demonstration--processing-pipeline-description)\n- [Getting Started](#getting-started)\n  - [Step 1 - Install Dependencies](#step-1---install-dependencies)\n  - [Step 2 - Install the gesture control pipeline](#step-2---install-the-gesture-control-pipeline)\n  - [Step 3 - Hardware setup](#step-3---hardware-setup)\n  - [Step 4 - Launch the system](#step-4---launch-the-system)\n- [Usage](#usage)\n- [System performance](#system-performance)\n- [Additional resources](#additional-resources)\n- [Citation](#citation)\n- [License](#license)\n- [Acknowledgments](#acknowledgments)\n\n## Demonstration \u0026 Processing pipeline description\n\n\u003ca href=\"https://youtu.be/FZAUPmKiSXg\"\u003e\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/ArthurFDLR/drone-gesture-control/blob/main/.github/video_embed.PNG?raw=true\" alt=\"Demonstration video\" width=\"80%\" style=\"border-radius: 5px;\"\u003e\n\u003c/p\u003e\u003c/a\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/ArthurFDLR/drone-gesture-control/blob/main/.github/pipeline.png?raw=true\" alt=\"pipeline\" width=\"95%\"\u003e\n\u003c/p\u003e\n\n## Getting Started\n\n### Step 1 - Install Dependencies\n\n1. Install PyTorch and Torchvision - see [PyTorch for Jetson Nano](https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-9-0-now-available/72048).\n2. Install TensorFlow - see [Installing TensorFlow For Jetson Platform](https://docs.nvidia.com/deeplearning/frameworks/install-tf-jetson-platform/index.html). Note that TensorFlow is already installed on JetPack.\n3. Install [torch2trt](https://github.com/NVIDIA-AI-IOT/torch2trt)\n    ```\n    git clone https://github.com/NVIDIA-AI-IOT/torch2trt\n    cd torch2trt\n    sudo python3 setup.py install --plugins\n    ```\n4. Install other miscellaneous packages\n    ```\n    sudo pip3 install numpy Pillow pymavlink dronekit \n    ```\n\n\n### Step 2 - Install the gesture control pipeline\n\n1. Clone this repository on your Jetson Nano\n    ```\n    git clone https://github.com/ArthurFDLR/drone-gesture-control\n    cd drone-gesture-control\n    ```\n2. Download and place the TensorRT-Pose pre-trained model [resnet18_baseline_att_224x224_A_epoch_249.pth](https://drive.google.com/file/d/1XYDdCUdiF2xxx4rznmLb62SdOUZuoNbd/view) in the folder [`.\\drone-gesture-control\\models`](https://github.com/ArthurFDLR/drone-gesture-control/tree/main/drone-gesture-control/models)\n3. Run the installation procedure. This operation can take a while.\n    ```\n    sudo python3 install.py\n    ```\n\n### Step 3 - Hardware setup \n\n1. Wire the UART ports _D15 (RX) - D14 (TX)_ on the _J41_ expansion header pins of the carrier board of the Jetson Nano to a MAVLink enabled serial port on your flight controller. See bellow a setup example using the Pixhawk 4 flight controller. The default baud rate is 57600.\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/ArthurFDLR/drone-gesture-control/blob/main/.github/schematic.png?raw=true\" alt=\"Banner\" width=\"50%\" style=\"border-radius: 5px;\"\u003e\n\u003c/p\u003e\n\n1. Disable the Serial Console trigger on the serial connection - see [Jetson Nano – UART](https://www.jetsonhacks.com/2019/10/10/jetson-nano-uart/).\n    ```\n    systemctl stop nvgetty\n    systemctl disable nvgetty\n    udevadm trigger\n    ```\n\n2. Connect your camera to the CSI port 0. You might have to adapt the GStreamer pipeline for your camera - see `gstream_pipeline` in [`.\\drone-gesture-control\\__main__.py`](https://github.com/ArthurFDLR/drone-gesture-control/blob/main/drone-gesture-control/__main__.py). The camera used for development is an Arducam IMX477 mini.\n\n### Step 4 - Launch the system\n```\npython3 drone-gesture-control\n```\n\n\u003c!-- USAGE EXAMPLES --\u003e\n## Usage\n\nThe gesture control system currently supports only basics - yet essential - commands:\n- _T_: Arm the drone if it is disarmed and landed; Disarm the drone if it is armed and landed.\n- _Traffic\\_AllStop_: Take-off at an altitude of 1.8m if the drone is armed and landed; Land if the drone is in flight.\n- _Traffic\\_RightTurn_: Move 4m to the right if the drone is armed.\n- _Traffic\\_LeftTurn_: Move 4m to the left if the drone is armed.\n- _Traffic\\_BackFrontStop_: Move 2m backward if the drone is armed.\n- _Traffic\\_FrontStop_: Move 2m forward if the drone is armed.\n- _Yoga\\_UpwardSalute_: Return to Launch (RTL).\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/ArthurFDLR/drone-gesture-control/blob/main/.github/class.png?raw=true\" alt=\"Body classes\" width=\"80%\" style=\"border-radius: 5px;\"\u003e\n\u003c/p\u003e\n\nFor security purposes, the system only transmits orders to the flight controller if it is in _GUIDED_ mode. We recommend binding a switch of your radio controller to select this mode for ease of use. \n\n## System performance\n\nThe classification model used in this project is the best performing of the [Pose Classification Kit (PCK)](https://github.com/ArthurFDLR/pose-classification-kit). This model yields great results both in terms of accuracy and inference time. During flights, it is pretty common for the embedded camera only to record a person's upper body. The system thus has to be highly reliable even on partial inputs. The model is tested on two datasets to ensure this property: the original PCK dataset and the same samples with missing keypoints. **The testing accuracies on these datasets respectively reach 98.3% and 95.1%.** As shown in the confusion matrices bellow (left: original dataset - right: partial inputs), even poses that are hardly distinguishable by humans (only looking at the upper-body) are almost perfectly classified by the model. After optimization (see [`.\\install.py`](https://github.com/ArthurFDLR/drone-gesture-control/blob/main/install.py)), **the whole processing pipeline - from image capture to drone control - consistently run at speed from 9.5Hz to 13Hz.**\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/ArthurFDLR/drone-gesture-control/blob/main/.github/cms.PNG?raw=true\" alt=\"Confusion matrices\" width=\"100%\" style=\"border-radius: 5px;\"\u003e\n\u003c/p\u003e\n\n## Additional resources\n\n- [**Vision-Based Gesture-Controlled Drone:A Comprehensive Python Package to DeployEmbedded Pose Recognition Systems, Arthur Findelair, August 11 2021**](https://github.com/ArthurFDLR/drone-gesture-control/raw/main/.github/Project_Slides_Presentation.pdf)\n- [**Presentation slides of the project**](https://github.com/ArthurFDLR/drone-gesture-control/raw/main/.github/Project_Slides_Presentation.pdf)\n\n\n## Citation\n\n```\n@inproceedings{9813802,\n    author    = {Findelair, Arthur and Yu, Xinrui and Saniie, Jafar},\n    booktitle = {2022 IEEE International Conference on Electro Information Technology (eIT)},\n    title     = {Design Flow and Implementation of a Vision-Based Gesture-Controlled Drone},\n    year      = {2022},\n    pages     = {320-324},\n    doi       = {10.1109/eIT53891.2022.9813802}\n}\n```\n\n\u003c!-- LICENSE --\u003e\n## License\n\nDistributed under the MIT License. See [`LICENSE`](https://github.com/ArthurFDLR/drone-gesture-control/blob/main/LICENSE) for more information.\n\n## Acknowledgments\n\nMany thanks to the [ECASP Laboratory](http://ecasp.ece.iit.edu/) from the Illinois Institute of Technology that has provided all the  necessary hardware to develop this project.\n\n\u003c!-- MARKDOWN LINKS \u0026 IMAGES --\u003e\n[license-shield]: https://img.shields.io/github/license/ArthurFDLR/drone-gesture-control?style=for-the-badge\n[license-url]: https://github.com/ArthurFDLR/drone-gesture-control/blob/master/LICENSE\n[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge\u0026logo=linkedin\u0026colorB=555\n[linkedin-url]: https://linkedin.com/in/arthurfdlr/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthurfdlr%2Fdrone-gesture-control","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farthurfdlr%2Fdrone-gesture-control","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthurfdlr%2Fdrone-gesture-control/lists"}