{"id":31588514,"url":"https://github.com/nliaudat/esphome_ai_component","last_synced_at":"2025-10-06T02:11:23.421Z","repository":{"id":316725844,"uuid":"1064572798","full_name":"nliaudat/esphome_ai_component","owner":"nliaudat","description":"A component to run AI model on esphome","archived":false,"fork":false,"pushed_at":"2025-10-03T10:33:40.000Z","size":1641,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-03T12:25:28.799Z","etag":null,"topics":["camera","esp32","esphome-component","meter-reading","tflite"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nliaudat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["nliaudat"]}},"created_at":"2025-09-26T08:28:05.000Z","updated_at":"2025-10-03T10:33:19.000Z","dependencies_parsed_at":"2025-09-26T11:35:14.843Z","dependency_job_id":null,"html_url":"https://github.com/nliaudat/esphome_ai_component","commit_stats":null,"previous_names":["nliaudat/esphome_ai_component"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nliaudat/esphome_ai_component","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nliaudat%2Fesphome_ai_component","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nliaudat%2Fesphome_ai_component/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nliaudat%2Fesphome_ai_component/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nliaudat%2Fesphome_ai_component/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nliaudat","download_url":"https://codeload.github.com/nliaudat/esphome_ai_component/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nliaudat%2Fesphome_ai_component/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278547821,"owners_count":26004775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["camera","esp32","esphome-component","meter-reading","tflite"],"created_at":"2025-10-06T02:10:17.547Z","updated_at":"2025-10-06T02:11:23.414Z","avatar_url":"https://github.com/nliaudat.png","language":"C++","funding_links":["https://github.com/sponsors/nliaudat"],"categories":[],"sub_categories":[],"readme":"# ESPHome Meter Reader TFLite Component\n\n\u003e General-purpose TensorFlow Lite Micro implementation with camera support for esphome\n\n[![ESPHome](https://img.shields.io/badge/ESPHome-Compatible-brightgreen)](https://esphome.io/)\n\n\n## 🚀 What is this?\n\nA general-purpose ESPHome component for running TensorFlow Lite Micro models on ESP32 devices.\nWhile originally designed for meter reading, it can be used for any computer vision task including object detection, image classification, and custom AI applications.\n\n\n## ✨ Key Features\n\n- **🤖 TensorFlow Lite Micro**: Full TFLite Micro runtime support (with operators detection and auto loading)\n- **📷 Camera Integration**: State of the art ESP32 camera integration with TrackedBuffer for better memory management and windowing for OV2640 like camera\n- **🖼️ Image Preprocessing**: Automatic cropping, scaling, and format conversion\n- **⚡ Optimized Performance**: ESP-NN accelerated operations (full image processing and 8 inferences takes less than 3 seconds)\n- **🎯 Multi-Zone Processing**: Process multiple regions of interest\n- **🔧 Flexible Configuration**: Support for various model types and input formats\n- **🐛 Advanced Debugging**: Real-time image analysis and model output inspection\n\n## 🏁 Quick Start\n\n### 1. Installation\n\n***A detailed procedure for setting up is available in [wiki/setup page](https://github.com/nliaudat/esphome_ai_component/wiki/Setup)*** \n\nAdd this to your ESPHome configuration:\n\n```yaml\nexternal_components:\n  - source: \n      type: git\n      url: https://github.com/nliaudat/esphome_ai_component\n      ref: main\n    components: [meter_reader_tflite]\n```\n\n### 2. Basic Configuration\n\n```yaml\n# Configure camera\nesp32_camera:\n  id: my_camera\n  name: \"AI Camera\"\n  resolution: 640x480\n  pixel_format: JPEG\n\n# Configure TFLite component\nmeter_reader_tflite:\n  id: tflite_processor\n  model: \"model.tflite\"  # Your TensorFlow Lite model\n  camera_id: my_camera\n  update_interval: 60s\n  debug: false\n  debug_image: false # static embedded image (debug.jpg)\n  debug_image_out_serial: false\n  crop_zones_global: crop_zones  # Reference the global variables set in globals_AI_component.yaml | To update, use http://[hassio]/developer-tools/action \u003eesphome.[s3cam_tflite]_set_crop_zones | to get, use the tools\u003edraw_regions.py\n  flash_light : ${id_prefix}flash # optional : flash light at taking image\n  flash_pre_time: 7000    # 7 seconds before update (time to stabilize)\n  flash_post_time: 2000   # 2 seconds after update\n  # ### zoom availale for OV2640, OV3660, OV5640, SC101IOT, SC030IOT from crop zones (explicit)\n  # camera_window:  # can be set in http://[hassio]/developer-tools/action \u003e esphome.[s3cam_tflite]_set_camera_window\n  #  offset_x: 928 # multiple of 4\n  #  offset_y: 480 # multiple of 4\n  #  width: 448 # multiple of 4 and 4:3 proportions with height for best quality\n  #  height: 88 #  multiple of 4 and 4:3 proportions with width for best quality\n  # ### Enhanced validation parameters\n  allow_negative_rates: false    # Prevent meter rollbacks\n  max_absolute_diff: 300         # Maximum allowed absolute difference\n```\n\n### 3. Add Your Model\n\nPlace your trained `.tflite` model file in the same directory as your ESPHome configuration.\n\n## 📋 Prerequisites\n\n- **ESP32 board** with camera support\n- **ESPHome 2025.09** or newer\n- **TensorFlow Lite model** (quantized recommended)\n\n## 🎯 Use Cases\n\n### Computer Vision Applications\n- **Object Detection**: Identify objects in camera frames\n- **Image Classification**: Categorize images into classes\n- **Anomaly Detection**: Detect unusual patterns or events\n- **Quality Control**: Inspect products or components\n- **Gesture Recognition**: Recognize hand gestures or movements\n\n### Meter Reading (Original Purpose)\n- Water, electricity, gas meter digit recognition\n- Analog gauge reading\n- Digital display extraction\n\n## ⚙️ Configuration Examples\n\n### Basic Object Detection\n\n```yaml\nmeter_reader_tflite:\n  id: object_detector\n  model: \"object_model.tflite\" # Must be in same directory as YAML\n  camera_id: my_camera\n  update_interval: 30s\n  confidence_threshold: 0.7\n```\n\n### Image Classification\n\n```yaml\nmeter_reader_tflite:\n  id: image_classifier\n  model: \"dig-class100-0180-s2-q.tflite\"  # Must be in same directory as YAML\n  camera_id: my_camera\n  update_interval: 60s\n  \n  # Custom sensor for classification results\nsensor:\n  - platform: template\n    name: \"Meter Reading\"\n    id: meter_value\n    unit_of_measurement: \"units\"\n    accuracy_decimals: 0\n    icon: \"mdi:counter\"\n    update_interval: 30s\n    lambda: |-\n      auto reader = id(meter_reader);\n      if (reader != nullptr) {\n        return reader-\u003eget_last_reading();\n      }\n      return 0.0;\n\n  - platform: template\n    name: \"Meter Reading Confidence\"\n    id: meter_confidence\n    unit_of_measurement: \"%\"\n    accuracy_decimals: 0\n    icon: \"mdi:percent\"\n    update_interval: 30s\n    lambda: |-\n      auto reader = id(meter_reader);\n      if (reader != nullptr) {\n        return reader-\u003eget_last_confidence() * 100.0; // Convert to percentage\n      }\n      return 0.0;\n```\n\n### Multi-Zone Processing\n\n```yaml\nglobals:\n  - id: detection_zones\n    type: string\n    initial_value: '[[0,0,400,300],[400,0,800,300],[0,300,400,600],[400,300,800,600]]'\n\nmeter_reader_tflite:\n  id: multi_zone_detector\n  model: \"detection_model.tflite\"\n  camera_id: my_camera\n  crop_zones_global: globals.detection_zones\n```\n\n### Advanced Configuration with Debugging\n\n```yaml\nmeter_reader_tflite:\n  id: advanced_processor\n  model: \"custom_model.tflite\"\n  camera_id: my_camera\n  tensor_arena_size: 512KB\n  update_interval: 60s\n  debug: true\n  debug_image: true # static embedded image\n  debug_image_out_serial: true\n  confidence_threshold: 0.8\n```\n\n## 🔧 Model Configuration\n\n### Supported Model Types\n\nThe component automatically detects and configures for various model architectures:\n\n```cpp\n// Built-in model configurations (model_config.h)\n{\"class100-0180\", ...}     // 100-class classification (0.0-9.9) - 512KB arena\n{\"class100-0173\", ...}     // 100-class classification (0.0-9.9) - 512KB arena  \n{\"class10-0900\", ...}      // 10-class classification (0-9) - 800KB arena\n{\"class10-0810\", ...}      // 10-class classification (0-9) - 800KB arena\n{\"mnist\", ...}             // MNIST-style grayscale models - 900KB arena\n```\n\n## Image Processing Flowchart\n```mermaid\ngraph TD\n\n    A[Camera Capture] --\u003e B{Image Format?}\n    B --\u003e|JPEG| C[JPEG Decoding]\n    B --\u003e|RGB888/RGB565| D[Direct Processing]\n    \n    C --\u003e E[Convert to RGB888]\n    D --\u003e F[Crop Zones]\n    E --\u003e F\n    \n    F --\u003e G[Scale to Model Input]\n    G --\u003e H[Format Conversion]\n    H --\u003e I{Model Input Type?}\n    \n    I --\u003e|float32| J[Normalize 0-255 to 0.0-1.0]\n    I --\u003e|uint8| K[Direct Copy]\n    \n    J --\u003e L[TensorFlow Lite Inference]\n    K --\u003e L\n    \n    L --\u003e M[Output Processing]\n    M --\u003e N[Confidence Check]\n    N --\u003e O[Publish Results]\n```\n    \n### Processing Steps Explained\n\n1. Image Acquisition: Capture frame from ESP32 camera (JPEG, RGB888, RGB565, or Grayscale)\n2. Format Conversion: Convert all formats to standardized RGB888\n3. Zone Cropping: Extract regions of interest based on configured crop zones\n4. Scaling: Resize cropped regions to match model input dimensions\n5. Normalization: Convert pixel values to model-expected range (0-255 or 0.0-1.0)\n6. Inference: Run TensorFlow Lite model on prepared input\n7. Post-processing: Convert raw model outputs to usable values and confidence scores\n8. Result Publication: Send processed results to ESPHome sensors\n\n### Custom Model Support\n\nFor custom models, the component auto-detects:\n- Input dimensions (width, height, channels)\n- Data type (float32, uint8)\n- Output processing requirements\n\n### Input Formats\n- **RGB888**: 3-channel color images\n- **GRAYSCALE**: 1-channel monochrome\n- **JPEG**: Automatic decoding to RGB888\n- **RGB565**: Automatic conversion\n\n\n## ⚡ Performance Optimization\n\n### Memory Settings\n\n```yaml\nmeter_reader_tflite:\n  tensor_arena_size: 512KB  # Default, adjust based on model size or if you get bad results\n  \n  # For larger models:\n  # tensor_arena_size: 768KB\n  # tensor_arena_size: 1024KB\n```\n\n### Camera Optimization\n\n```yaml\nesp32_camera:\n  resolution: 640x480      # Lower resolution for faster processing\n  pixel_format: JPEG       # Convert to RGB888\n  jpeg_quality: 20         # If using JPEG, lower quality for speed\n  framerate: 1 fps         # Reduce frame rate for periodic processing\n```\n\n## 🐛 Debugging \u0026 Development\n\n### Enable Debug Output\n\n```yaml\nmeter_reader_tflite:\n  debug: true # the tflite model operators are set manually (model_handler.cpp)\n  debug_image: true # static embedded image debug.jpg\n  debug_image_out_serial: true #output image to serial console, can be reconstructed with tools/analyze_serial_output.py\n```\n\nDebug output includes:\n- Model input/output values\n- Processing timing statistics\n- Memory usage reports\n- Image analysis data\n- Zone processing details\n\n### Serial Monitoring\n\n```bash\n# Monitor debug output\nesphome logs your-config.yaml\n\n# Expected debug output example:\n# DEBUG: Model input: 32x20x3, output: 10 classes\n# DEBUG: Processing time: 45ms\n# DEBUG: Zone 1 confidence: 0.92\n# DEBUG: Memory usage: 412KB/512KB\n```\n\n## 🔍 Troubleshooting\n\n### Common Issues \u0026 Solutions\n\n**❌ Model loading fails**\n```yaml\n# Increase tensor arena size\nmeter_reader_tflite:\n  tensor_arena_size: 768KB  # Increase from 512KB\n```\n\n**❌ Poor inference results**\n```yaml\n# Check model compatibility and preprocessing\nmeter_reader_tflite:\n  debug: true  # Enable debug to see input data\n# Increase tensor_arena_size\n    tensor_arena_size: 768KB \n```\n\n**❌ Camera frame issues**\n```yaml\n# Adjust camera settings\nesp32_camera:\n  resolution: 320x240  # Lower resolution\n  framerate: 1 fps     # Reduce frame rate\n```\n\n### Performance Tips\n\n1. **Use quantized models** (int8) for better performance\n2. **Match input dimensions** to your actual use case\n3. **Enable ESP-NN optimizations** (enabled by default)\n4. **Use appropriate tensor_arena_size** (start with 512KB)\n5. **Optimize update interval** based on your application needs\n\n## 📊 Technical Specifications\n\n- **TensorFlow Lite Version**: Micro 1.3.4\n- **ESP-NN Optimization**: Enabled by default\n- **Memory Usage**: 512KB tensor arena (auto detected by ./tools/check_tflite_model.py or configurable)\n- **Processing Speed**: 10-1000ms per inference (model-dependent)\n- **Supported Operations**: Conv2D, FullyConnected, Pooling, Activation functions, etc.\n- **Input Types**: float32, int8, uint8\n\n## 🤝 Contributing\n\nThis component is designed to be extensible for various TFLite applications. Contributions welcome!\n\n### Extension Ideas\n- Object detection bounding box support\n- Multi-model switching\n- Custom preprocessing pipelines\n- Cloud model updates\n- Advanced output decoding\n\n## 📄 License\n\n* Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC-BY-NC-SA)\n* No commercial use\n* The AI models from [haverland](https://github.com/haverland/Tenth-of-step-of-a-meter-digit) is under Apache Licence\n\n\n---\n\n**Transform your ESP32 camera into an AI vision system!** 🤖✨\n\n\n*For questions and support, check the GitHub repository discussions section.*\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnliaudat%2Fesphome_ai_component","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnliaudat%2Fesphome_ai_component","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnliaudat%2Fesphome_ai_component/lists"}