{"id":14977900,"url":"https://github.com/preste-ai/ai_whiteboard","last_synced_at":"2025-10-28T07:30:36.163Z","repository":{"id":167651583,"uuid":"333422239","full_name":"preste-ai/AI_whiteboard","owner":"preste-ai","description":"Transform any wall to an intelligent whiteboard","archived":false,"fork":false,"pushed_at":"2021-02-11T14:18:07.000Z","size":6376,"stargazers_count":16,"open_issues_count":2,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-01T12:44:08.969Z","etag":null,"topics":["fingertips-detector","hand-detector","jetson-devices","jetson-xavier-nx","raspberry-pi-camera","tensorrt-engine","whiteboard"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/preste-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-27T13:02:56.000Z","updated_at":"2024-11-10T07:45:04.000Z","dependencies_parsed_at":"2023-07-25T05:02:23.308Z","dependency_job_id":null,"html_url":"https://github.com/preste-ai/AI_whiteboard","commit_stats":{"total_commits":49,"total_committers":3,"mean_commits":"16.333333333333332","dds":0.08163265306122447,"last_synced_commit":"edb89ec3621f23de3b3b27133419a72f630c0b1e"},"previous_names":["preste-ai/ai_whiteboard"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/preste-ai%2FAI_whiteboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/preste-ai%2FAI_whiteboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/preste-ai%2FAI_whiteboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/preste-ai%2FAI_whiteboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/preste-ai","download_url":"https://codeload.github.com/preste-ai/AI_whiteboard/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238608332,"owners_count":19500360,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fingertips-detector","hand-detector","jetson-devices","jetson-xavier-nx","raspberry-pi-camera","tensorrt-engine","whiteboard"],"created_at":"2024-09-24T13:56:30.786Z","updated_at":"2025-10-28T07:30:35.400Z","avatar_url":"https://github.com/preste-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI_whiteboard\n\n![](images/ai_whiteboard.gif)\n\n## Idea\n\nThe idea of this project is to transform any wall or surface into an interactive whiteboard just with an ordinary RGB camera and your hand.\nI hope you'll find it interesting !\n\n## Hardware\n\n- Jetson Xavier NX **JetPack 4.4**\n- Raspberry Pi Camera + ArduCam (8MP IMX219 Sensor Module)\n\n**Note:** The system works also on Jetson Nano, TX2\n\n## Details\n\nTo use AI whiteboard correctly you need to find a wall or flat surface and place a camera at a distance of about 1 meter. It can be any wall/surface but the system works more accurately with the dark or light monotonous walls/surfaces.\nWe capture an image from a camera. Then we crop this image into a square. Next, we use **a hand detector[1]  (YOLO[3] - deep neural network)**,to find a hand in the image. If there is a hand in the image, we crop that hand out of the image and feed it to **a Fingertip detector[1]  (modified VGG16 - deep neural network)**. Next, if we can detect fingertips, we use their coordinates to control the whiteboard (See the control section below). \n\n## Launch AI Whiteboard\n\n##### 1. Set up your NVIDIA Jetson Device or use your PC.\n- [Jetson Xavier NX](https://developer.nvidia.com/embedded/learn/get-started-jetson-xavier-nx-devkit) with [JetPack 4.4](https://developer.nvidia.com/jetpack-sdk-44-archive) (CUDA 10.2, TensorRT 7.1.3, cuDNN 8.0)\n- [Install Tensorflow 1.15.3](https://docs.nvidia.com/deeplearning/frameworks/install-tf-jetson-platform/index.html)  \n\n##### 2. Download AI Whiteboard project. `$ git clone https://github.com/preste-ai/camera_ai_whiteboard.git `\n \n##### 3. Install packages\n\nYou can download needed packages via pip using the `requirements.txt` file:\n\n```python\n  pip3 install -r requirements.txt\n```\n\n##### 4. Download [weights or TensorRT engines](https://drive.google.com/drive/folders/1eDBqbZfoY7XJ3fYv8FEMJ5AZe_3n0sjU?usp=sharing) and put it to `weights` or `weights/engines`.\n\n**Note:** The current TensorRT engines work correctly **only** on Jetson Xavier NX devices as TensorRT runs device-specific profiling during the optimization phase.If you want to use this models(engines) on others Jetson devices please convert .h5 model with `h5_to_trt.py` script on your platform. \n\n##### 5. Set up the power mode (ID=2, 15W 6 cores) `$ sudo /usr/sbin/nvpmodel -m 2`\n\n##### 6. Launch device's fan `sudo jetson_clocks --fan` \n\n##### 7. Run AI whiteboard script. \n\nCheck `config.py` file and set up needed parameters.\n- whiteboard_w : 200 - whiteboard width (px) (displayed on camera caputed image)\n- whiteboard_h : 200 - whiteboard height (px) (displayed on camera caputed image)\n- cam_w       : 320 - width (px) of a captured image \n- cam_h       : 240 - height (px) of a captured image\n- framerate   : 60 - camera capture framerate (for Raspberry Pi Camera)\n- zoom_koef   : 2 - zoom coefficient to resize whiteboard_w and whiteboard_h\n- confidence_ft_threshold : 0.5 - confidence threshold of Fingertips detector\n- confidence_hd_threshold : 0.8 - confidence threshold of Hand detector      \n\n---\nRun from a project root directory:\n\n**Jetson Devices**\n```python \n  python3 ai_whiteboard.py --rpc --trt \n```\n- rpc : If you want to use a Raspberry Pi Camera. Default: False\n- trt : If you want to use TensorRT engines. Default: False\n\n**Laptop**\n```python\n  python3 ai_whiteboard.py \n```\n\n###### Control gesture (combination)\n| To draw | To move | To erase | To clean | To save | \n|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|\n|![](images/to_paint.jpg)|![](images/to_move.jpg)|![](images/to_erase.jpg)|![](images/to_clean.jpg)|![](images/to_save.jpg)|\n \n\n## Training Hand-detector\n\n#### Custom Dataset\n\nA [custom dataset](https://drive.google.com/drive/folders/1rFHtl6A4EKokuOQk-9vqvWV0WiKyRfco?usp=sharing) was collected and labeled (12,000 images) for training. For labeling I used [CVAT](https://github.com/openvinotoolkit/cvat).\n\n- Train: 9,500 images\n- Validation: 1000 images\n- Test : 1500 images\n\n|  |  |  |  |  | \n|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|\n|![](images/1.jpg)|![](images/2.jpg)|![](images/3.jpg)|![](images/4.jpg)|![](images/5.jpg)|\n|![](images/6.jpg)|![](images/7.jpg)|![](images/8.jpg)|![](images/9.jpg)|![](images/10.jpg)|\n \n\nRun from a project root directory:\n\n```python\n python3 yolo_train.py\n```\n\n\n## Testing Hand-detector\n\nRun from a project root directory:\n\n```python\n python3 yolo_test.py\n```\n\n## Convert .h5 model to TensorRT engine [2]\n\nThe transformation takes place in 3 stages:\n1. Freeze graph and remove training nodes (.h5 -\u003e .pb)\n2. Convert frozen graph to onnx (.pb -\u003e .onnx)\n3. Convert onnx model to TensorRT engine (.onnx -\u003e .engine)\n\nRun from a project root directory:\n\n```python\n  python3 h5_to_trt.py --folder weights --weights_file yolo --fp 16\n```\n\n- folder weights : path to the folder with model\n- weights_file : weights file name (**without .h5**)\n- fp : TensorRT engine precision (16 or 32)\n\nMetrics for **Hand detection** after model conversion. \n\nIn order to determine the correctness of the detection, we use the value of [IOU](https://medium.com/towards-artificial-intelligence/understanding-iou-metric-in-object-detection-1e5532f06a76). If the value of IOU is more than 0.5 then the detector predicts a hand correctly otherwise - no. The results are given below. \n\n\n|  | keras model before training | keras model after training | TensorRT engine (fp32) | TensorRT engine (fp16) | \n|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|\n| Accuracy    | 72.68 % | 89.14 % | 89.14 % | 89.07 % |\n| Precision   | 84.80 % | 99.45 % | 99.45 % | 99.45 % |\n| Recall      | 50.78 % | 77.24 % | 77.24 % | 77.10 % |\n\n\n## Solution performance (Hand detector + Fingertips detector)\n\nCaptured image shape : 320x240\nJetson Xavier NX: power mode ID 2: 15W 6 cores\n\n|  | keras model | TensorRT engine (fp32) | TensorRT engine (fp16) | \n|:---------------:|:---------------:|:---------------:|:---------------:|\n| Average FPS | 12 | 33 | 60 |\n\n\n\n## References\n1. Unified Gesture and Fingertip Detection : https://github.com/MahmudulAlam/Unified-Gesture-and-Fingertip-Detection\n2. TensorRT guide: https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#python_topics\n3. YOLO9000: Better, Faster, Stronger : https://arxiv.org/abs/1612.08242\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpreste-ai%2Fai_whiteboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpreste-ai%2Fai_whiteboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpreste-ai%2Fai_whiteboard/lists"}