{"id":18895487,"url":"https://github.com/daleonpz/poc_cv_tinyml","last_synced_at":"2025-04-12T03:50:59.495Z","repository":{"id":249426419,"uuid":"729471681","full_name":"daleonpz/POC_CV_tinyml","owner":"daleonpz","description":"Screw type detection using ESP-EYE and TensorFlow Lite Micro for real-time classification on ESP32.","archived":false,"fork":false,"pushed_at":"2024-11-13T18:17:28.000Z","size":123601,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T23:23:44.433Z","etag":null,"topics":["computer-vision","esp32","machne-learning","tinyml","yolo"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/daleonpz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-09T10:41:47.000Z","updated_at":"2025-01-28T22:55:56.000Z","dependencies_parsed_at":"2024-08-10T14:02:47.600Z","dependency_job_id":"05348c62-d7a4-4343-8409-a57e2071594d","html_url":"https://github.com/daleonpz/POC_CV_tinyml","commit_stats":null,"previous_names":["daleonpz/poc_anomaly_manufacturing","daleonpz/poc_cv_tinyml"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daleonpz%2FPOC_CV_tinyml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daleonpz%2FPOC_CV_tinyml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daleonpz%2FPOC_CV_tinyml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daleonpz%2FPOC_CV_tinyml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/daleonpz","download_url":"https://codeload.github.com/daleonpz/POC_CV_tinyml/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248514209,"owners_count":21116899,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","esp32","machne-learning","tinyml","yolo"],"created_at":"2024-11-08T08:28:25.195Z","updated_at":"2025-04-12T03:50:59.476Z","avatar_url":"https://github.com/daleonpz.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Screw Type Detection Using ESP-EYE and TensorFlow Lite Micro\n\nThis project focuses on using the ESP-EYE and a computer vision algorithm running TensorFlow Lite Micro to detect different types of screws: \"Big,\" \"Medium,\" and \"Black.\" The project consists of two main parts:\n\n1. Data Collection and Model Training\n2. Model Deployment on ESP32\n\n## 1. Setup the Development Environment\n### Install the ESP-IDF\n\nFollow the [ESP-IDF Get Started Guide](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/linux-macos-setup.html) to set up the toolchain and install the ESP-IDF framework.\n\n## 2. Data Collection \nNavigate to the `data_collector` folder.\n\n### Flashing the ESP32\n\n1. Navigate to the `esp32` directory, activate the ESP32 virtual environment, and clean the build directory:\n\n```bash\ncd esp32/\nget_idf\nidf.py fullclean\n```\n\n2. Set up your Wi-Fi credentials by running:\n\n```bash\nidf.py menuconfig\n```\n\nNavigate to **Example Connection Configuration** and input your Wi-Fi SSID and password.\n\n3. Build and flash the application to the ESP32:\n\n```bash\nidf.py build\nidf.py -p /dev/ttyUSB0 flash\nidf.py -p /dev/ttyUSB0 monitor\n```\n\nAfter flashing, the ESP32’s IP address will be displayed. This IP address will be used to access the web server.\n\n```bash\nI (4180) WIFI_MODULE: got ip:X.X.X.X\nI (4184) WIFI_MODULE: connected to ap \u003cCONFIGURED_SSID\u003e password:\u003cCONFIGURED_PASSWORD\u003e\n```\n\n### Running the Web Server\n\nThe web server receives images from the ESP32 and stores them in the uploads folder. To start the web server:\n\n1. Set the ESP32’s IP address as an environment variable:\n\n```bash\nexport ESP32_SERVER_URL=http://\u003cESP32_IP\u003e:81\n```\n\n2. Start the web server:\n\n```bash\ncd webserver/\npython3 webserver.py\n```\n\n3. In a new terminal, start capturing images:\n\n```bash\npython3 capture.py\n```\n\nCaptured images will be saved in the `static/uploads` directory.\n\n### Labeling the Data\n\n1. Use [makesense.ai](https://www.makesense.ai/) for labeling the collected images.\n2. Upload the images from the `webserver/static/uploads/` directory to the platform.\n3. Load the labels from `labels.txt` and begin labeling your images.\nThe labels should be \"Big,\" \"Medium,\" and \"Black.\" and the images should be labeled accordingly. The interface should look like this:\n\n![Interface](readme_extras/labeling_makesense.png)\n\n4. After labeling, export the annotations in YOLO format. The files should follow the format `XXX.txt`, where XXX corresponds to the image number.\n\nIf you are not familiar with the YOLO format, here is an example of the format for a file named `000.txt`:\n\n```bash\n0 0.509715 0.331606 0.190415 0.272021\n1 0.591321 0.654793 0.107513 0.224093\n```\n\nThe first number is the **label**, and the next four numbers are the bounding box coordinates in the format `x_center y_center width height`, scaled to the image size. \n\nFor example, if the image size is 640x480, the bounding box \n\n```bash\n0.509715 0.331606 0.190415 0.272021\n```\n\n would be \n```bash\nx_center=0.509715*640=326.29 \ny_center=0.331606*480=159.14 \nwidth=0.190415*640=121.86 \nheight=0.272021*480=130.57\n```\n\n### Dataset Preparation\n\nTo split the labeled dataset into training and testing sets:\n\n1. Move the labeled images and annotation files into the appropriate directories:\n\n```bash\ncd datasets/\nrm -rf images labels\nmkdir images labels\nmv \u003cpath_to_labels_zip_file\u003e labels/\ncp ../webserver/static/uploads/* images/\n```\n\n2. Split the dataset:\n\n```bash\npython3 split_dataset.py\n```\n\nYou should now have inside `images/` and `labels/` the dataset split into `train`, `val`, and `test` directories.\n\n## 3. Model Training\n### Training the YOLOv5 Model\nFor the model training the [YOLOv5](https://github.com/ultralytics/yolov5) repository from [ultralytics](https://ultralytics.com) will be used. The following steps will guide you through the model training process:\n\n1. Navigate to the YOLOv5 Directory:\n\n```bash\ncd yolov5/\n```\n\n2. Train the Model:\n\nRun the following command to start training:\n\n```bash\npython train.py --img 96 --cfg ../data_collector/model.yaml --batch 32 --epochs 300 --data ../data_collector/model_data.yaml --name my_run\n```\n\nThis command trains the model with the specified image size, configuration file, batch size, number of epochs, and data configuration. The trained model will be saved in the `runs/train/my_run/weights` directory by default.\n\n**Note:** Model training may take a significant amount of time depending on your hardware.\n\n3. Testing the Model:\n\nAfter training, you can test the model on the test dataset with the following command:\n\n```bash\npython detect.py --source ../data_collector/datasets/images/test/ --weights runs/train/my_run/weights/best.pt --img 96 --name my_run --data ../data_collector/model_data.yaml\n```\n\nThis command runs detection on images from the test dataset and saves the results in the `runs/detect/my_run` directory.\n\n### Model Quantization and Conversion using YOLOv5\n\n1. Convert and Quantize the Model:\n\nExport the trained model to TensorFlow Lite (TFLite) format and apply quantization with the following command:\n\n```bash\npython export.py --weights runs/train/my_run/weights/best.pt --include saved_model tflite --img 96 --data ../data_collector/model_data.yaml\n```\n\nThe quantized model will be saved in the `runs/train/my_run/weights` directory.\n\n**Note:** The output directory will vary each time you retrain the model, so ensure that you update the path as needed.\n\n### Model Quantization and Conversion using my own script\n\nConvert the Model to a C Array:\n\nNavigate to the `data_collector/datasets/` directory and run:\n\n```bash\npython convert_model_to_tflite.py ../../yolov5/runs/train/my_run/weights/best_saved_model/ images/test/\u003cimage_name\u003e.jpg\n```\n\nThis command converts the TFLite model to a C array format, generating a `model.cc` file. You can use this file to deploy the model on the ESP32, while the `.tflite` file can be used for testing the model on your computer.\n\n**Note:** this script generates a different `.tflite` as `export.py`. I don't use the exported from `export.py` because it's a bit larger than the one generated by `convert_model_to_tflite.py`, but I didn't test if it works with the ESP32.\n\n## 4. Model Deployment on ESP32\n\nAfter training and converting your model, you’ll need to deploy it to the ESP32. Follow these steps to integrate the model and flash it onto the ESP32:\n\n### Integrate the Model into the ESP32 Codebase\n\n1. Copy the Model File:\n\nCopy the `model.cc` file generated from the previous steps to the `application/main` directory of your ESP32 project.\n\n2. Update Label Settings:\n\nIf your labels differ from the default labels, update the `model_settings.cc` file in the `application/main` directory to reflect the new labels.\n\n### Build and Flash the Application\n\n1. Navigate to the Application Directory:\n\n```bash\ncd application/\n```\n\n2. Set Up the ESP-IDF Environment:\n\n```bash\nget_idf\n```\n\n3. Clean, Build, and Flash the Application:\n\n```bash\nidf.py fullclean\nidf.py build\nidf.py -p /dev/ttyUSB0 flash\nidf.py -p /dev/ttyUSB0 monitor\n```\nThese commands will clean any previous build artifacts, build the application, flash it to the ESP32, and open the serial monitor to view the ESP32's output.\n\n## TODOs\n- Send Results to BLE Client: Implement functionality to send the detected screw type results to a BLE client.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaleonpz%2Fpoc_cv_tinyml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaleonpz%2Fpoc_cv_tinyml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaleonpz%2Fpoc_cv_tinyml/lists"}