{"id":37311924,"url":"https://github.com/kinivi/tello-gesture-control","last_synced_at":"2026-01-16T03:01:46.885Z","repository":{"id":40385743,"uuid":"335333837","full_name":"kinivi/tello-gesture-control","owner":"kinivi","description":"Control  DJI Tello 🛸 using hand gesture recognition on drone`s camera video-stream. Feel free to contribute!","archived":false,"fork":false,"pushed_at":"2023-08-29T09:25:53.000Z","size":295,"stargazers_count":263,"open_issues_count":3,"forks_count":83,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-04-21T12:26:03.503Z","etag":null,"topics":["dji-tello","dji-tello-edu","gesture-control","gesture-controller","gesture-recognition","gesture-recognizer","mediapipe","notebook-jupyter","python","tello","tello-drone","tello-sdk","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kinivi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-02T15:25:34.000Z","updated_at":"2024-04-15T17:08:48.000Z","dependencies_parsed_at":"2022-08-09T19:00:38.724Z","dependency_job_id":null,"html_url":"https://github.com/kinivi/tello-gesture-control","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/kinivi/tello-gesture-control","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinivi%2Ftello-gesture-control","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinivi%2Ftello-gesture-control/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinivi%2Ftello-gesture-control/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinivi%2Ftello-gesture-control/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kinivi","download_url":"https://codeload.github.com/kinivi/tello-gesture-control/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kinivi%2Ftello-gesture-control/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28476673,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T00:15:39.755Z","status":"online","status_checked_at":"2026-01-16T02:00:07.781Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dji-tello","dji-tello-edu","gesture-control","gesture-controller","gesture-recognition","gesture-recognizer","mediapipe","notebook-jupyter","python","tello","tello-drone","tello-sdk","tensorflow"],"created_at":"2026-01-16T03:01:46.818Z","updated_at":"2026-01-16T03:01:46.879Z","avatar_url":"https://github.com/kinivi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DJI Tello Hand Gesture control\n-------\n\n🏆 This project featured in [Official Google Dev Blog](https://developers.googleblog.com/2021/09/drone-control-via-gestures-using-mediapipe-hands.html)\n\n-------\nThe main goal of this project is to control the drone using hand gestures without any gloves or additional equipment.\nJust camera on the drone or your smartphone(soon), laptop and human hand.\u003cbr\u003e\n\n\u003cimg alt=\"demo_gif\" src=\"https://user-images.githubusercontent.com/13486777/111168690-fb2e9280-85aa-11eb-894f-fe70633072fd.gif\"\u003e\n\n\n## Index\n1. [Introduction](#Introduction)\n2. [Setup](#Setup)\n    1. [Install pip packages](#1.-Installing-pip-packages)\n    2. [Connect and test Tello](#2.-Connect-Tello)\n3. [Usage](#Usage)\n    * [Keyboard control](##Keyboard-control)\n    * [Gesture control](#Gesture-control)\n4. [Adding new gestures](#Adding-new-gestures)\n    * [Technical description](#Technical-details-of-gesture-detector)\n    * [Creating dataset](#Creating-dataset-with-new-gestures)\n    * [Retrain model](#Notebook-for-retraining-model)\n5. [Repository structure](#Repository-structure)\n\n## Introduction\nThis project relies on two main parts - DJI Tello drone and Mediapipe fast hand keypoints recognition.\n\nDJI Tello is a perfect drone for any kind of programming experiments. It has a rich Python API (also Swift is available) which helps to almost fully control a drone, create drone swarms and utilise its camera for Computer vision.\n\nMediapipe is an amazing ML platform with many robust solutions like Face mesh, Hand Keypoints detection and Objectron. Moreover, their model can be used on the mobile platforms with on-device acceleration.\n\nHere is a starter-pack that you need:\n\n\u003cimg alt=\"starter_pack\" width=\"80%\" src=\"https://user-images.githubusercontent.com/13486777/111294166-b65e3680-8652-11eb-8225-c1fb1e5b867d.JPG\"\u003e\n\n## Setup\n### 1. Installing pip packages\nFirst, we need to install python dependencies. Make sure you that you are using `python3.7`\n\nList of packages\n```sh\nConfigArgParse == 1.2.3\ndjitellopy == 1.5\nnumpy == 1.19.3\nopencv_python == 4.5.1.48\ntensorflow == 2.4.1\nmediapipe == 0.8.2\n```\n\nInstall\n```sh\npip3 install -r requirements.txt\n```\n### 2. Connect Tello\nTurn on drone and connect computer to its WiFi\n\n\u003cimg width=\"346\" alt=\"wifi_connection\" src=\"https://user-images.githubusercontent.com/13486777/110932822-a7b30f00-8334-11eb-9759-864c3dce652d.png\"\u003e\n\n\nNext, run the following code to verify connectivity\n\n```sh\npython3 tests/connection_test.py\n```\n\nOn successful connection\n\n```json\n1. Connection test:\nSend command: command\nResponse: b'ok'\n\n\n2. Video stream test:\nSend command: streamon\nResponse: b'ok'\n```\n\nIf you get such output, you may need to check your connection with the drone\n\n```json\n1. Connection test:\nSend command: command\nTimeout exceed on command command\nCommand command was unsuccessful. Message: False\n\n\n2. Video stream test:\nSend command: streamon\nTimeout exceed on command streamon\nCommand streamon was unsuccessful. Message: False\n```\n\n## Usage\nThe most interesting part is demo. There are 2 types of control: keyboard and gesture. You can change between control types during the flight. Below is a complete description of both types.\n\nRun the following command to start the tello control :\n\n```sh\npython3 main.py\n```\n\nThis script will start the python window with visualization like this:\n\n\u003cimg width=\"60%\" alt=\"window\" src=\"https://user-images.githubusercontent.com/13486777/111294470-09d08480-8653-11eb-895d-a8ca9f6a288d.png\"\u003e\n\n\n### Keyboard control\nTo control the drone with your keyboard at any time - press the `k` key.\n\nThe following is a list of keys and action description -\n\n* `k` -\u003e Toggle Keyboard controls\n* `g` -\u003e Toggle Gesture controls\n* `Space` -\u003e Take off drone(if landed) **OR** Land drone(if in flight)\n* `w` -\u003e Move forward\n* `s` -\u003e Move back\n* `a` -\u003e Move left\n* `d` -\u003e Move right\n* `e` -\u003e Rotate clockwise\n* `q` -\u003e Rotate counter-clockwise\n* `r` -\u003e Move up\n* `f` -\u003e Move down\n* `Esc` -\u003e End program and land the drone \n\n\n### Gesture control \n\nBy pressing `g` you activate gesture control mode. Here is a full list of gestures that are available now.\n\n\u003cimg alt=\"gestures_list\" width=\"80%\" src=\"https://user-images.githubusercontent.com/13486777/110933057-f1035e80-8334-11eb-8458-988af973804e.JPG\"\u003e\n\n## Adding new gestures\nHand recognition detector can add and change training data to retrain the model on the own gestures. But before this,\nthere are technical details of the detector to understand how it works and how it can be improved\n### Technical details of gesture detector\nMediapipe Hand keypoints recognition is returning 3D coordinated of 20 hand landmarks. For our\nmodel we will use only 2D coordinates.\n\n\u003cimg alt=\"gestures_list\" width=\"80%\" src=\"https://user-images.githubusercontent.com/13486777/110933339-49d2f700-8335-11eb-9588-5f68a2677ff0.png\"\u003e\n\n\nThen, these points are preprocessed for training the model in the following way.\n\n\u003cimg alt=\"preprocessing\" width=\"80%\" src=\"https://user-images.githubusercontent.com/13486777/111294503-11902900-8653-11eb-9856-a50fe96e750e.png\"\u003e\n\n\nAfter that, we can use data to train our model. Keypoint classifier is a simple Neural network with such \nstructure\n\n\u003cimg alt=\"model_structure\" width=\"80%\" src=\"https://user-images.githubusercontent.com/13486777/112172879-c0a5a500-8bfd-11eb-85b3-34ccfa256ec3.jpg\"\u003e\n\n\n\n_check [here](#Grid-Search) to understand how the architecture was selected_\n### Creating dataset with new gestures\nFirst, pull datasets from Git LFS. [Here](https://github.com/git-lfs/git-lfs/wiki/Installation) is the instruction of how \nto install LFS. Then, run the command to pull default csv files\n```sh\ngit lfs install\ngit lfs pull\n```\n\nAfter that, run `main.py` and press \"n\" to enter the mode to save key points\n(displayed as **MODE:Logging Key Point**）\n\n\u003cimg width=\"60%\" alt=\"writing_mode\" src=\"https://user-images.githubusercontent.com/13486777/111301228-a185a100-865a-11eb-8a3c-fa4d9ee96d6a.png\"\u003e\n\n\nIf you press \"0\" to \"9\", the key points will be added to [model/keypoint_classifier/keypoint.csv](model/keypoint_classifier/keypoint.csv) as shown below.\u003cbr\u003e\n1st column: Pressed number (class ID), 2nd and subsequent columns: Keypoint coordinates\n\n\u003cimg width=\"90%\" alt=\"keypoints_table\" src=\"https://user-images.githubusercontent.com/13486777/111295338-ec4fea80-8653-11eb-9bb3-4d27b519a14f.png\"\u003e\n\nIn the initial state, 7 types of learning data are included as was shown [here](#Gesture-control). If necessary, add 3 or later, or delete the existing data of csv to prepare the training data.\n### Notebook for retraining model\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kinivi/tello-gesture-control/blob/main/Keypoint_model_training.ipynb)\n\nOpen [Keypoint_model_training.ipynb](Keypoint_model_training.ipynb) in Jupyter Notebook or Google Colab.\nChange the number of training data classes,the value of **NUM_CLASSES = 3**, and path to the dataset. Then, execute all cells\nand download `.tflite` model\n\n\u003cimg width=\"60%\" alt=\"notebook_gif\" src=\"https://user-images.githubusercontent.com/13486777/111295516-1ef9e300-8654-11eb-9f59-6f7a85b99076.gif\"\u003e\n\n\nDo not forget to modify or add labels in `\"model/keypoint_classifier/keypoint_classifier_label.csv\"`\n\n#### Grid Search\n❗️ Important ❗️ The last part of the notebook is an experimental part of the notebook which main functionality is to test hyperparameters of the model structure. In a nutshell: grid search using TensorBoard visualization. Feel free to use it for your experiments.\n\n\n\u003cimg width=\"70%\" alt=\"grid_search\" src=\"https://user-images.githubusercontent.com/13486777/111295521-228d6a00-8654-11eb-937f-a15796a3024c.png\"\u003e\n\n\n## Repository structure\n\u003cpre\u003e\n│  main.py\n│  Keypoint_model_training.ipynb\n│  config.txt\n│  requirements.txt\n│  \n├─model\n│  └─keypoint_classifier\n│      │  keypoint.csv\n│      │  keypoint_classifier.hdf5\n│      │  keypoint_classifier.py\n│      │  keypoint_classifier.tflite\n│      └─ keypoint_classifier_label.csv\n│ \n├─gestures\n│   │  gesture_recognition.py\n│   │  tello_gesture_controller.py\n│   └─ tello_keyboard_controller.py\n│          \n├─tests\n│   └─connection_test.py\n│ \n└─utils\n    └─cvfpscalc.py\n\u003c/pre\u003e\n### app.py\nMain app which controls the functionality of drone control and gesture recognition\u003cbr\u003e\nApp also includes mode to collect training data for adding new gestures.\u003cbr\u003e\n\n### keypoint_classification.ipynb\nThis is a model training script for hand sign recognition.\n\n### model/keypoint_classifier\nThis directory stores files related to gesture recognition.\u003cbr\u003e\n\n* Training data(keypoint.csv)\n* Trained model(keypoint_classifier.tflite)\n* Label data(keypoint_classifier_label.csv)\n* Inference module(keypoint_classifier.py)\n\n### gestures/\nThis directory stores files related to drone controllers and gesture modules.\u003cbr\u003e\n\n* Keyboard controller (tello_keyboard_controller.py)\n* Gesture controller(tello_keyboard_controller.py)\n* Gesture recognition module(keypoint_classifier_label.csv)\n\n### utils/cvfpscalc.py\nModule for FPS measurement.\n\n# TODO\n- [ ] Motion gesture support (LSTM)\n- [ ] Web UI for mobile on-device gesture control\n- [ ] Add [Holistic model](https://google.github.io/mediapipe/solutions/holistic) support\n\n# Reference\n* [MediaPipe](https://github.com/google/mediapipe)\n* [MediaPipe Hand gesture recognition (by Kazuhito00)](https://github.com/Kazuhito00/hand-gesture-recognition-using-mediapipe)\n* [Tello SDK python interface](https://github.com/damiafuentes/DJITelloPy)\n\n# Author\nNikita Kiselov(https://github.com/kinivi)\n \n# License \ntello-gesture-control is under [Apache-2.0 License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkinivi%2Ftello-gesture-control","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkinivi%2Ftello-gesture-control","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkinivi%2Ftello-gesture-control/lists"}