{"id":26507383,"url":"https://github.com/cnadler86/mp_esp_dl_models","last_synced_at":"2026-04-22T04:34:26.958Z","repository":{"id":281351295,"uuid":"944518466","full_name":"cnadler86/mp_esp_dl_models","owner":"cnadler86","description":"Micropython binding for the ESP32 DL AI vision models like face detection / recognition, imagenet classifier or pedestrian (human) detection","archived":false,"fork":false,"pushed_at":"2025-09-10T03:58:34.000Z","size":84,"stargazers_count":10,"open_issues_count":1,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-12-19T12:29:09.507Z","etag":null,"topics":["ai","esp32","face-detection","face-recognition","imagenet-classifier","micropython","pedestrian-detection"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cnadler86.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"cnadler86"}},"created_at":"2025-03-07T13:42:42.000Z","updated_at":"2025-12-12T04:44:51.000Z","dependencies_parsed_at":"2025-07-16T15:08:47.859Z","dependency_job_id":null,"html_url":"https://github.com/cnadler86/mp_esp_dl_models","commit_stats":null,"previous_names":["cnadler86/mp_esp_face_detector","cnadler86/mp_esp_dl_models"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cnadler86/mp_esp_dl_models","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnadler86%2Fmp_esp_dl_models","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnadler86%2Fmp_esp_dl_models/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnadler86%2Fmp_esp_dl_models/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnadler86%2Fmp_esp_dl_models/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cnadler86","download_url":"https://codeload.github.com/cnadler86/mp_esp_dl_models/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnadler86%2Fmp_esp_dl_models/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32121091,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-22T00:31:26.853Z","status":"online","status_checked_at":"2026-04-22T02:00:05.693Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","esp32","face-detection","face-recognition","imagenet-classifier","micropython","pedestrian-detection"],"created_at":"2025-03-20T23:19:08.088Z","updated_at":"2026-04-22T04:34:26.940Z","avatar_url":"https://github.com/cnadler86.png","language":"C++","funding_links":["https://github.com/sponsors/cnadler86","https://www.paypal.me/cnadler"],"categories":["Libraries","Recently Updated"],"sub_categories":["AI","[Mar 25, 2025](/content/2025/03/25/README.md)"],"readme":"# ESP DL MicroPython Binding\r\n\r\nThis is a MicroPython binding for ESP-DL (Deep Learning) models that enables face detection, face recognition, human detection, cat detection, and image classification on ESP32 devices.\r\n\r\n## Donate\r\n\r\nI spent a lot of time and effort to make this. If you find this project useful, please consider donating to support my work.\r\n[![Donate](https://img.shields.io/badge/Donate-PayPal-blue.svg)](https://www.paypal.me/cnadler)\r\n\r\n## Available Models\r\n\r\n- `FaceDetector`: Detects faces in images and provides bounding boxes and facial features\r\n- `FaceRecognizer`: Recognizes enrolled faces and manages a face database\r\n- `HumanDetector`: Detects people in images and provides bounding boxes\r\n- `CatDetector`: Detects cats in images and provides bounding boxes\r\n- `ImageNet`: Classifies images into predefined categories\r\n- `CocoDetector`: Detects objects in images using COCO dataset categories\r\n\r\n## Installation \u0026 Building\r\n\r\n### Requirements\r\n\r\n- ESP-IDF:\r\n  - Version 5.4.2 with MicroPython \u003e=1.26.0\r\n- Make sure you have the complete ESP32 build environment set up\r\n\r\n### Precompiled Images\r\n\r\nYou can find precompiled images in two ways:\r\n1. In the Actions section for passed workflows under artifacts\r\n2. By forking the repo and manually starting the action\r\n\r\n### Building from Source\r\n\r\n1. Clone the required repositories:\r\n```sh\r\ngit clone --recursive https://github.com/cnadler86/mp_esp_dl_models.git\r\ngit clone https://github.com/cnadler86/micropython-camera-API.git\r\ngit clone https://github.com/cnadler86/mp_jpeg.git\r\n```\r\n\r\n2. Build the firmware:\r\nThere are two ways to enable the different models:\r\n\r\na) Using mpconfigvariant files (recommended):\r\nThe models can be enabled in the board's mpconfigvariant files (e.g., mpconfigvariant_FLASH_16M.cmake). The following flags are available:\r\n- MP_DL_FACE_DETECTOR_ENABLED\r\n- MP_DL_FACE_RECOGNITION_ENABLED\r\n- MP_DL_PEDESTRIAN_DETECTOR_ENABLED\r\n- MP_DL_IMAGENET_CLS_ENABLED\r\n- MP_DL_COCO_DETECTOR_ENABLED\r\n- MP_DL_CAT_DETECTOR_ENABLED\r\n\r\nb) Using command line flags:\r\nYou can enable models directly through the idf.py command using -D flags:\r\n```sh\r\nidf.py -D MP_DL_FACE_RECOGNITION_ENABLED=1 -D MP_DL_CAT_DETECTOR_ENABLED=1 [other flags...]\r\n```\r\n\r\nBasic build command:\r\n```sh\r\ncd mp_esp_dl_models/boards/\r\nidf.py -D MICROPY_DIR=\u003cmicropython-dir\u003e -D MICROPY_BOARD=\u003cBOARD_NAME\u003e -D MICROPY_BOARD_VARIANT=\u003cBOARD_VARIANT\u003e -B build-\u003cyour-build-name\u003e build\r\ncd build-\u003cyour-build-name\u003e\r\npython ~/micropython/ports/esp32/makeimg.py sdkconfig bootloader/bootloader.bin partition_table/partition-table.bin micropython.bin firmware.bin micropython.uf2\r\n```\r\n\r\n## Module Usage\r\n\r\n### Common Requirements\r\n\r\nAll models support various input pixel formats including RGB888 (default), RGB565, and others supported by ESP-DL. You can use [mp_jpeg](https://github.com/cnadler86/mp_jpeg/) to decode camera images to the correct format.\r\n\r\nThe pixel format can be set through the constructor's `pixel_format` parameter. This value matches the ESP-DL image format definitions.\r\n\r\n### Pixel Formats\r\n- `espdl.RGB888` (default)\r\n- `espdl.RGB565`\r\n- `espdl.GRAYSCALE`\r\n\r\n### FaceDetector\r\n\r\nThe FaceDetector module detects faces in images and can optionally provide facial feature points.\r\n\r\n#### Constructor\r\n```python\r\nFaceDetector(width=320, height=240, pixel_format=espdl.RGB888, features=True)\r\n```\r\n\r\n**Parameters:**\r\n- `width` (int, optional): Input image width. Default: 320\r\n- `height` (int, optional): Input image height. Default: 240\r\n- `pixel_format` (int, optional): Input image pixel format. Default: espdl.RGB888\r\n- `features` (bool, optional): Whether to return facial feature points. Default: True\r\n\r\n#### Methods\r\n\r\n- **run(framebuffer)**\r\n  \r\n  Detects faces in the provided image.\r\n\r\n  **Parameters:**\r\n  - `framebuffer`: image data (required)\r\n\r\n  **Returns:**\r\n  List of dictionaries with detection results, each containing:\r\n  - `score`: Detection confidence (float)\r\n  - `box`: Bounding box coordinates [x1, y1, x2, y2]\r\n  - `features`: Facial feature points [(x,y) coordinates for: left eye, right eye, nose, left mouth, right mouth] if enabled, None otherwise\r\n\r\n### FaceRecognizer\r\n\r\nThe FaceRecognizer module manages a database of faces and can recognize previously enrolled faces.\r\n\r\n#### Constructor\r\n```python\r\nFaceRecognizer(width=320, height=240, pixel_format=espdl.RGB888, features=True, db_path=\"face.db\", model=None)\r\n```\r\n\r\n**Parameters:**\r\n- `width` (int, optional): Input image width. Default: 320\r\n- `height` (int, optional): Input image height. Default: 240\r\n- `pixel_format` (int, optional): Input image pixel format. Default: espdl.RGB888\r\n- `features` (bool, optional): Whether to return facial feature points. Default: True\r\n- `db_path` (str, optional): Path to the face database file. Default: \"face.db\"\r\n- `model` (str, optional): Feature extraction model to use (\"MBF\" or \"MFN\"). Default: None (uses default model)\r\n\r\n#### Methods\r\n\r\n- **run(framebuffer)**\r\n  \r\n  Detects and recognizes faces in the provided image.\r\n\r\n  **Parameters:**\r\n  - `framebuffer`: image data (required)\r\n\r\n  **Returns:**\r\n  List of dictionaries with recognition results, each containing:\r\n  - `score`: Detection confidence\r\n  - `box`: Bounding box coordinates [x1, y1, x2, y2]\r\n  - `features`: Facial feature points (if enabled)\r\n  - `person`: Recognition result containing:\r\n    - `id`: Face ID\r\n    - `similarity`: Match confidence (0-1)\r\n    - `name`: Person name (if provided during enrollment)\r\n\r\n- **enroll(framebuffer, validate=False, name=None)**\r\n  \r\n  Enrolls a new face in the database.\r\n\r\n  **Parameters:**\r\n  - `framebuffer`: image data\r\n  - `validate` (bool, optional): Check if face is already enrolled. Default: False\r\n  - `name` (str, optional): Name to associate with the face. Default: None\r\n\r\n  **Returns:**\r\n  - ID of the enrolled face\r\n\r\n- **delete_face(id)**\r\n  \r\n  Deletes a face from the database.\r\n\r\n  **Parameters:**\r\n  - `id` (int): ID of the face to delete\r\n\r\n- **print_database()**\r\n  \r\n  Prints the contents of the face database.\r\n\r\n### HumanDetector and Cat Detector\r\n\r\nThe HumanDetector module detects people in images. The CatDetector does it for cats. Both modules provide bounding boxes for detected objects.\r\n\r\n#### Constructor\r\n```python\r\nHumanDetector(width=320, height=240, pixel_format=espdl.RGB888) #For cats use CatDetector\r\n```\r\n\r\n**Parameters:**\r\n- `width` (int, optional): Input image width. Default: 320\r\n- `height` (int, optional): Input image height. Default: 240\r\n- `pixel_format` (int, optional): Input image pixel format. Default: espdl.RGB888\r\n\r\n#### Methods\r\n\r\n- **run(framebuffer)**\r\n  \r\n  Detects people in the provided image.\r\n\r\n  **Parameters:**\r\n  - `framebuffer`: image data\r\n\r\n  **Returns:**\r\n  List of dictionaries with detection results, each containing:\r\n  - `score`: Detection confidence\r\n  - `box`: Bounding box coordinates [x1, y1, x2, y2]\r\n\r\n### ImageNet\r\n\r\nThe ImageNet module classifies images into predefined categories.\r\n\r\n#### Constructor\r\n```python\r\nImageNet(width=320, height=240, pixel_format=espdl.RGB888)\r\n```\r\n\r\n**Parameters:**\r\n- `width` (int, optional): Input image width. Default: 320\r\n- `height` (int, optional): Input image height. Default: 240\r\n- `pixel_format` (int, optional): Input image pixel format. Default: espdl.RGB888\r\n\r\n#### Methods\r\n\r\n- **run(framebuffer)**\r\n  \r\n  Classifies the provided image.\r\n\r\n  **Parameters:**\r\n  - `framebuffer`: image data\r\n\r\n  **Returns:**\r\n  List alternating between class names and confidence scores:\r\n  `[class1, score1, class2, score2, ...]`\r\n\r\n### COCO detect\r\n\r\nThe COCO detect module detects objects in images using the COCO dataset.\r\n\r\n#### Constructor\r\n```python\r\nCOCODetector(width=320, height=240, pixel_format=espdl.RGB888, model=CONFIG_DEFAULT_COCO_DETECT_MODEL)\r\n```\r\n\r\n**Parameters:**\r\n- `width` (int, optional): Input image width. Default: 320\r\n- `height` (int, optional): Input image height. Default: 240\r\n- `pixel_format` (int, optional): Input image pixel format. Default: espdl.RGB888\r\n- `model` (int, optional): COCO detection model to use. Default: CONFIG_DEFAULT_COCO_DETECT_MODEL\r\n\r\n#### Methods\r\n- **run(framebuffer)**\r\n  \r\n  Detects objects in the provided image.\r\n\r\n  **Parameters:**\r\n  - `framebuffer`: image data\r\n\r\n  **Returns:**\r\n  List of dictionaries with detection results, each containing:\r\n  - `score`: Detection confidence\r\n  - `box`: Bounding box coordinates [x1, y1, x2, y2]\r\n  - `category`: Detected object class id\r\n\r\n## Usage Examples\r\n\r\n### Face Detection Example\r\n```python\r\nfrom espdl import FaceDetector\r\nimport camera\r\nfrom jpeg import Decoder\r\n\r\n# Initialize components\r\ncam = camera.Camera()\r\ndecoder = Decoder(pixel_format=\"RGB888\")\r\nface_detector = FaceDetector()\r\n\r\n# Capture and process image\r\nimg = cam.capture()\r\nframebuffer = decoder.decode(img)  # Convert to RGB888\r\nresults = face_detector.run(framebuffer)\r\n\r\nif results:\r\n    for face in results:\r\n        print(f\"Face detected with confidence: {face['score']}\")\r\n        print(f\"Bounding box: {face['box']}\")\r\n        if face['features']:\r\n            print(f\"Facial features: {face['features']}\")\r\n```\r\n\r\n### Face Recognition Example\r\n```python\r\nfrom espdl import FaceRecognizer\r\nimport camera\r\nfrom jpeg import Decoder\r\n\r\n# Initialize components\r\ncam = camera.Camera()\r\ndecoder = Decoder(pixel_format=\"RGB888\")\r\nrecognizer = FaceRecognizer(db_path=\"/faces.db\")\r\n\r\n# Enroll a face\r\nimg = cam.capture()\r\nframebuffer = decoder.decode(img)\r\nface_id = recognizer.enroll(framebuffer, name=\"John\")\r\nprint(f\"Enrolled face with ID: {face_id}\")\r\n\r\n# Later, recognize faces\r\nimg = cam.capture()\r\nframebuffer = decoder.decode(img)\r\nresults = recognizer.run(framebuffer)\r\n\r\nif results:\r\n    for face in results:\r\n        if face['person']:\r\n            print(f\"Recognized {face['person']['name']} (ID: {face['person']['id']})\")\r\n            print(f\"Similarity: {face['person']['similarity']}\")\r\n```\r\n\r\n## Benchmark results\r\nThe following table shows the frames per second (fps) for different image sizes and models. The results are based on a test with a 2MP camera and a ESP32S3.\r\n\r\n| Frame Size  | FaceDetector | HumanDetector |\r\n|-------------|--------------|---------------|\r\n| QQVGA       | 14.5         | 6.6           |\r\n| R128x128    | 21           | 6.6           |\r\n| QCIF        | 19.7         | 6.5           |\r\n| HQVGA       | 18           | 6.3           |\r\n| R240X240    | 16.7         | 6.1           |\r\n| QVGA        | 15.2         | 6.6           |\r\n| CIF         | 13           | 5.5           |\r\n| HVGA        | 11.9         | 5.3           |\r\n| VGA         | 8.2          | 4.4           |\r\n| SVGA        | 6.2          | 3.8           |\r\n| XGA         | 4.1          | 2.8           |\r\n| HD          | 3.6          | 2.6           |\r\n\r\n## Notes \u0026 Best Practices\r\n\r\n1. **Image Format**: Always ensure input images are in the right format. Use mp_jpeg for JPEG decoding from camera.\r\n\r\n2. **Memory Management**: \r\n   - Close/delete detector objects when no longer needed\r\n   - Consider memory constraints when choosing image dimensions\r\n\r\n3. **Face Recognition**:\r\n   - Enroll faces in good lighting conditions\r\n   - Multiple enrollments of the same person can improve recognition\r\n   - Use `validate=True` during enrollment to avoid duplicates\r\n\r\n4. **Storage**:\r\n   - Face database is persistent across reboots\r\n   - Consider backing up the face database file\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcnadler86%2Fmp_esp_dl_models","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcnadler86%2Fmp_esp_dl_models","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcnadler86%2Fmp_esp_dl_models/lists"}