{"id":13710967,"url":"https://github.com/carlo98/tello-ai-features","last_synced_at":"2025-05-06T20:31:24.148Z","repository":{"id":49470868,"uuid":"328141428","full_name":"carlo98/tello-ai-features","owner":"carlo98","description":"Tello AI features. Face recognition and tracking,  collision avoidance, reinforcement learning and more.","archived":false,"fork":false,"pushed_at":"2023-05-12T21:27:35.000Z","size":79908,"stargazers_count":9,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-13T21:44:45.889Z","etag":null,"topics":["ai","camera-calibration","collision-avoidance","face-recognition","reinforcement-learning","tello","tracker"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/carlo98.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-09T11:39:55.000Z","updated_at":"2024-08-20T23:22:32.000Z","dependencies_parsed_at":"2024-11-13T21:35:19.174Z","dependency_job_id":"7ccc180d-660e-401f-aa63-d3431ab8b5a8","html_url":"https://github.com/carlo98/tello-ai-features","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carlo98%2Ftello-ai-features","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carlo98%2Ftello-ai-features/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carlo98%2Ftello-ai-features/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carlo98%2Ftello-ai-features/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/carlo98","download_url":"https://codeload.github.com/carlo98/tello-ai-features/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252764293,"owners_count":21800668,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","camera-calibration","collision-avoidance","face-recognition","reinforcement-learning","tello","tracker"],"created_at":"2024-08-02T23:01:02.776Z","updated_at":"2025-05-06T20:31:24.102Z","avatar_url":"https://github.com/carlo98.png","language":"Python","funding_links":[],"categories":["Examples"],"sub_categories":[],"readme":"# Tello AI Features\nAutonomous tracker for Tello drones. It works on python 3.7 and 3.6 and uses [TelloPy](https://github.com/hanyazou/TelloPy).\n\nThe starting point is [Ubotica telloCV](https://github.com/Ubotica/telloCV), Nov 6 2018.\n\nThe initial idea was to code a face recognition and tracker, then a lot of ideas have arrived and still are coming, so this repository will continue to grow, with the goal of providing a package of AI algorithms for Tello drones.\n\n# Table of contents\n1. [Installation](#installation)\n2. [Control Commands](#control)\n3. [Collision Avoidance](#ca)\n    1. [Reinforcement Learning](#rl)\n4. [Face Recognition](#fr)\n5. [Camera Calibration](#cc)\n6. [Files Description](#f)\n7. [References](#ref)\n\n## Installation \u003ca name=\"installation\"\u003e\u003c/a\u003e\nInstall anaconda and then:\n```\nsudo apt update\nsudo apt install python3-opencv cmake build-essential\nconda create --name \u003cenv\u003e python=3.7 # or python=3.6\nconda activate \u003cenv\u003e\nconda install pip\npip install numpy av pynput face-recognition scikit-learn torch jupyter\n```\n\nTelloPy:\n```\ngit clone https://github.com/hanyazou/TelloPy\ncd TelloPy\npython setup.py bdist_wheel\npip install dist/tellopy-*.dev*.whl --upgrade\n```\n\nFor collision avoidance\n```\nmkdir Collision_Avoidance/saved_models\nmkdir Collision_Avoidance/data\nmkdir Collision_Avoidance/data/blocked\nmkdir Collision_Avoidance/data/free\nmkdir Collision_Avoidance/saliency\nmkdir Collision_Avoidance/saliency/blocked\nmkdir Collision_Avoidance/saliency/free\n```\n\nFor RL training:\n```\nmkdir Collision_Avoidance/rl_saved_models\n```\n\nFor face recognition:\n```\nmkdir Face_Recognition/train_dir\n```\n\nFor camera calibration\n```\nmkdir Camera_Calibration/chessboards\n```\n\n## Control commands \u003ca name=\"control\"\u003e\u003c/a\u003e\nAll control commands are described in telloCV.py.\n\nFor features:\n- '1': To toggle collision avoidance\n- '2': To toggle face recognition\n- '3': To toggle reinforcement learning training\n- 'x': To end/start episode of RL\n- 'f': To take a picture and label it as 'free'\n- 'b': To take a picture and label it as 'blocked'\n\n## Collision avoidance \u003ca name=\"ca\"\u003e\u003c/a\u003e\nFrom repository's root folder:\n```\nconda activate \u003cenv\u003e\npython3 telloCV.py\n```\nIn order to start/stop press '1'.\n\nThe images can be acquired from telloCV.py ('f' for image to be labelled as 'free', 'b' for image to be labelled as 'blocked').\n\nOnce the NN has been trained and saved in folder 'Collision_Avoidance/saved_models' as best_model.pth.\n\nThe NN provided in 'Collision_Avoidance/saved_models' has not been fully trained so pay attention please, it's there only to provide a starting point for transfer learning.\n\nIn order to provide fast inference for collision avoidance also in pc without GPU a small NN is provided, it accepts in input the saliency maps computed from raw images; this approach has led to a reduction in compution time of approximately 1/2 - 2/3 on intel-i5, with respect to AlexNet.\n\nThe drawback is the amount of training images required, due to a partial loss of transfer learning weights.\n\nIn order to perform collision avoidance a neural network is required, train it with 'Collision_Avoidance/train_model.ipynb', it needs to be launched from 'Collision_Avoidance' folder.\n\nFrom Collision_Avoidance folder:\n```\nconda activate \u003cenv\u003e\njupyter notebook\n```\n\nIMPORTANT: At the moment only one between face recognition and collision avoidance can be active.\n\n### Reinforcement Learning Training \u003ca name=\"rl\"\u003e\u003c/a\u003e\nFrom repository's root folder:\n```\nconda activate \u003cenv\u003e\npython3 telloCV.py\n```\nIn order to start/stop press '3'.\n\nIt is possible to further train the collision avoidance model with online reinforcement learning, this relies on the user to detect collisions by pressing 'x'. If no collision is detected by the user each episode will terminate after a given amount of steps (default: 100, change in 'Collision_Avoidance/RL.py').\n\nEvery time an episode ends the drone stops, giving you the time to move it to another position while the network is training; in order to restart inference press 'x' a second time.\n\nReward: +1/max_steps_per_episode if agent decides to go forward, 0 if it turns, -1 for collisions.\n\nDo not attempt to train a full model with this method because it requires a lot of time and it would seem nearly impossible, first get a collision avoidance model trained with 'train_model.pynb' as satisfactory as possible and then proceed with this.\n\nThe model trained by RL is saved into the folder 'Collision_Avoidance/rl_saved_models'.\n\n## Face recognition \u003ca name=\"fr\"\u003e\u003c/a\u003e\nFrom repository's root folder:\n```\nconda activate \u003cenv\u003e\npython3 telloCV.py\n```\nIn order to start/stop press '2'.\n\nThe binarized svm in the repo, \"Face_Recognition/svm_fam.bin\", should be replaced with an svm fitted with your images, using the script \"Face_Recognition/svm.py\".\nFrom Face Recognition folder:\n```\nconda activate \u003cenv\u003e\npython3 svm.py\n```\n\nInstructions on how to organize the images are available in the script.\n\nBy changing the two parameters at the beginning of the python script 'Face_Recognition/face_rec_tracker.py' one can choose which person the tello should track and the ratio between recall and speed, for face detection and recognition.\n\nIMPORTANT: At the moment only one between face recognition and collision avoidance can be active.\n\n## Camera Calibration \u003ca name=\"cc\"\u003e\u003c/a\u003e\nSave 15-20 images of a chessboard, made with the camera of tello, in the folder 'Camera_Calibration/chessboards' and call them n.jpg, (n=0, 1, ...).\n\nUse the jupyter notebook 'Camera_Calibration/camera_calibration.ipynb' to compute the parameters and then copy and paste them in the python script 'Camera_Calibration/process_image.py'.\n\nFrom Camera Calibration folder:\n```\nconda activate \u003cenv\u003e\njupyter notebook\n```\n\n## Files Description \u003ca name=\"f\"\u003e\u003c/a\u003e\ntelloCV.py: controller\n\nFace_Recognition/face_rec_tracker.py: uses a SVM and python's face_recognition to recognize faces, the binary SVM can be computed with \"svm.py\" DANGER!\n\nFace_Recognition/svm.py: Creates a SVM capable of recognize faces, at the top of the file is shown how one should organize the images. [Face recognition](https://github.com/ageitgey/face_recognition)\n\nCollision_Avoidance/collision_avoidance.py: It uses the saliency map generator provided [here](https://github.com/tobybreckon/DoG-saliency) to generate lower dimensional inputs to use in a neural network which performs collision avoidance, chooses whether to go forward or turn. DANGER!\n\nCollision_Avoidance/train_model.ipynb: Can be used to train a NN using images in folder \"data\" and saving the NN in folder \"saved_models\". In order to save images look at the commands in telloCV.py\n\nCamera_Calibration/camera_calibration.ipynb: Can be used to compute the camera parameters.\n\nCamera_Calibration/process_image.py: Provides a class which computes undistorted images, given camera parameters.\n\nCollision_Avoidance/model.py: NN model.\n\nCollision_Avoidance/RL.py: Reinforcement learning script, change parameters in the 'init' method to change saving/training frequencies and other RL parameters.\n\n## References \u003ca name=\"ref\"\u003e\u003c/a\u003e\n\nIf you are making use of this work in any way please reference the following articles in any report, publication, presentation, software release or any other associated materials:\n\n[Real-time Visual Saliency by Division of Gaussians](https://breckon.org/toby/publications/papers/katramados11salient.pdf)\n(Katramados, Breckon), In Proc. International Conference on Image Processing, IEEE, 2011.\n```\n@InProceedings{katramados11salient,\n  author    =    {Katramados, I. and Breckon, T.P.},\n  title     = \t {Real-time Visual Saliency by Division of Gaussians},\n  booktitle = \t {Proc. Int. Conf. on Image Processing},\n  pages     = \t {1741-1744},\n  year      = \t {2011},\n  month     = \t {September},\n  publisher =    {IEEE},\n  url       = \t {https://breckon.org/toby/publications/papers/katramados11salient.pdf},\n  doi       = \t {10.1109/ICIP.2011.6115785},\n}\n```\n\nFor non-commercial use (i.e. academic, non-for-profit and research) the (very permissive) terms of the MIT free software [LICENSE](LICENSE) must be adhered to.\n\nFor commercial use, the Division of Gaussians (DIVoG / DoG) saliency detection algorithm is patented (WIPO reference: [WO2013034878A3](https://patents.google.com/patent/WO2013034878A3/)) and available for licensing via [Cranfield University](https://www.cranfield.ac.uk/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcarlo98%2Ftello-ai-features","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcarlo98%2Ftello-ai-features","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcarlo98%2Ftello-ai-features/lists"}