{"id":20373182,"url":"https://github.com/st235/hse.cvformobiledevices.facialprocessing","last_synced_at":"2026-04-29T23:33:51.600Z","repository":{"id":222207483,"uuid":"756046103","full_name":"st235/HSE.CVforMobileDevices.FacialProcessing","owner":"st235","description":"Detects and extracts faces from the images from your gallery 👱‍♂️","archived":false,"fork":false,"pushed_at":"2024-03-03T19:39:26.000Z","size":80865,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-04T20:44:05.342Z","etag":null,"topics":["android","android-compose","celeba-dataset","kotlin","kotlin-coroutines","tensorflow","tflite"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/st235.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-11T20:23:37.000Z","updated_at":"2024-03-03T19:14:29.000Z","dependencies_parsed_at":"2025-01-15T05:55:27.286Z","dependency_job_id":null,"html_url":"https://github.com/st235/HSE.CVforMobileDevices.FacialProcessing","commit_stats":null,"previous_names":["st235/hse.cvformobiledevices.facialprocessing"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/st235/HSE.CVforMobileDevices.FacialProcessing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/st235%2FHSE.CVforMobileDevices.FacialProcessing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/st235%2FHSE.CVforMobileDevices.FacialProcessing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/st235%2FHSE.CVforMobileDevices.FacialProcessing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/st235%2FHSE.CVforMobileDevices.FacialProcessing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/st235","download_url":"https://codeload.github.com/st235/HSE.CVforMobileDevices.FacialProcessing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/st235%2FHSE.CVforMobileDevices.FacialProcessing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32448412,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T22:27:22.272Z","status":"ssl_error","status_checked_at":"2026-04-29T22:10:49.234Z","response_time":110,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["android","android-compose","celeba-dataset","kotlin","kotlin-coroutines","tensorflow","tflite"],"created_at":"2024-11-15T01:17:01.656Z","updated_at":"2026-04-29T23:33:46.590Z","avatar_url":"https://github.com/st235.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"/images/logo.png\" width=\"72\" height=\"72\"\u003e\n\n# Faces Gallery\n\nExtracts faces from images inside your gallery and offers an overview of the analysis. \n\n| Feed                                     | Attributes                                    | Attributes search                                           | Detailed view                                                |\n|------------------------------------------|-----------------------------------------------|-------------------------------------------------------------|--------------------------------------------------------------|\n| ![Feed](./images/faces_gallery_main.png) | ![Attributes](./images/faces_gallery_tag.png) | ![Attributes search](./images/faces_gallery_eyeglasses.png) | ![Detailed view](./images/faces_gallery_detailed_person.png) |\n\n## Technologies\n\nThe application runs on **Android Lollipop and above**.\n\nThe main technologies used in the project:\n\n- Kotlin + [Coroutines](https://kotlinlang.org/docs/coroutines-overview.html)\n- [Jetpack Compose](https://developer.android.com/jetpack/compose): a robust technology to build UI\n- [Room](https://developer.android.com/training/data-storage/room): Android persistence library\n- [Tensorflow Lite](https://www.tensorflow.org/lite/android): Framework for running ML models on mobile\n\n## Pre-work\n\nFacial Attributes extraction with [CelebA](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).\n\nThere are not much pretrained weights/models that are able to extract facial attributes. Moreover, I had\na few additional hard and soft requirements: it should be small _(below 40Mb)_ as I run it on edge devices\nand big models will likely lead to [OOM](https://developer.android.com/topic/performance/memory), it should\nextract facial attributes I am interested in _(at least beard or eyeglasses), and, ideally, it should be\na Keras or TFLite model. At least, I was not able to find any that satisfy the criteria 😅\n\nAt this point I decided to train my own model. I decided to stop on 4 facial attributes from CelebA:\n**🕶️ eyeglasses, 🧔‍♂️ beard, 🙂 smiling, and mustache**.\n\n### CelebA preprocessing\n\nRaw CelebA images are too noisy and contains too much details, therefore I preprocessed the dataset.\nI used OpenCV and DLib to detect faces, and saved them in a separate folder. Some images **had more than\none face**, for example, a person and a bodyguard. There were about 50 of such images and I removed them\nfrom the final dataset.\n\nYou can see the examples of processed images below:\n\n| Example 1                                      | Example 2                                      | Example 3                                      | Example 4                                      |\n|------------------------------------------------|------------------------------------------------|------------------------------------------------|------------------------------------------------|\n| ![Example 1](./images/celeba_pretrained_1.jpg) | ![Example 2](./images/celeba_pretrained_2.jpg) | ![Example 3](./images/celeba_pretrained_3.jpg) | ![Example 4](./images/celeba_pretrained_4.jpg) |\n\n### Training\n\nI trained the model in the following settings:\n\n- [MobileNetV3 Large](https://keras.io/api/applications/mobilenet/)\n- No pretrained weights\n- 100 epochs\n\n### Results\n\n- Accuracy: 0.9551\n- F1 Score: 0.9359\n- Precision: 0.9478\n- Recall: 0.9257\n\n| Example 1                                     | Example 2                                     | Example 3                                     |\n|-----------------------------------------------|-----------------------------------------------|-----------------------------------------------|\n| ![Example 1](./images/results_example_1.jpeg) | ![Example 2](./images/results_example_2.jpeg) | ![Example 3](./images/results_example_3.jpeg) |\n\n\n⚠️ The notebooks with pre-processing and training scripts can be found in [my gists](https://gist.github.com/st235/b3e658f383404acce551d8d7374a61bc).\n\n⚠️⚠️ Pre-trained weights and TF Lite models can be found in [my Google Drive](https://drive.google.com/drive/folders/1y8XXubvsFeie4qDxjQEMzf3dV7uQFq8D?usp=sharing).\n\n## Application\n\n### General Overview\n\nThe application scans for the following attributes:\n- Age\n- Gender\n- Emotion: anger, disgust, fear, happiness, neutral, sadness and surprise\n- Smiling\n- Beard\n- Mustache\n- Eyeglasses\n\nThe attributes are extracted with a few different models, specially trained for the tasks. You can find\n  all of these pretrained models under [`raw` folder](./app/src/main/res/raw):\n- Face Detection: [MobileNet V2 MediaPipe, Full Range](https://developers.google.com/mediapipe/solutions/vision/face_detector)\n- Age: MobileNet MediaPipe\n- Gender: MobileNet MediaPipe\n- Emotions: MobileNet\n- Facial Attributes: [MobileNet V3 Large, was trained in the scope of this project](#pre-work)\n- Face Embeddings: FaceNet Mobile\n\n### Cache\n\nCaching is done via [SQLite](https://developer.android.com/training/data-storage/sqlite) as it allows\nto organise data in a convenient manner and queries this data efficiently.\n\nAs an ORM framework I used Room [(see Technologies section for details)](#technologies).\n\nThere are 3 tables:\n- Face\n- Media File: I store them separately to save the processed images with **no faces** on them\n- Clusters\n\nThe implementation of the persistent layer can be found under [`data` folder](./app/src/main/java/github/com/st235/facialprocessing/data).\n\n### Clustering\n\nOnce the embeddings are extracted, they are clustered together based on the classical clustering algorithms.\n\nThere are 2 algorithms implemented for image clustering. All algorithms can be found under [`domain/clustering` folder](./app/src/main/java/github/com/st235/facialprocessing/domain/clustering):\n- [DBSCAN](https://en.wikipedia.org/wiki/DBSCAN): [implementation](./app/src/main/java/github/com/st235/facialprocessing/domain/clustering/dbscan/DbscanClusterer.kt)\n- [HDBSCAN](https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html): [implementation](./app/src/main/java/github/com/st235/facialprocessing/domain/clustering/hdbscan/HdbscanClusterer.kt)\n\nThe final implementation of the app is using **HDBSCAN**.\n\n## Misc\n\n### Faces Gallery Youtube Playlist\n\nPlaylist with all demo videos is available here: [https://youtube.com/playlist?list=PLucKuGqiOAE9fzt8duUc5qnq8mX8c6ReO\u0026si=ZIqqumO2ytc7DrRv](https://youtube.com/playlist?list=PLucKuGqiOAE9fzt8duUc5qnq8mX8c6ReO\u0026si=ZIqqumO2ytc7DrRv)\n\nVideos from the playlist are:\n\n- **Main Flow Demo**: https://youtu.be/IaZfJUDsUow?si=5CkpuchyM09qzpxH\n- **Photos review**: https://youtu.be/652KHTUCkZk?si=dqetxw9qznth_iVh\n- **Adding a new photo**: https://youtu.be/o1CulOjHrEo?si=A9HS7jy2v5NVejZu\n- **Attributes Search**: https://youtu.be/iPUsTKsxL9M?si=XegyJUpExNBxjUi8\n- **Person Search**: https://youtu.be/uAqx7uraQlM?si=PyOueC05wW4tqO0a\n\n### Evaluation Criteria List\n\n\u003e 0-5: Processing of all photos from a gallery is supported\n\n✅\n\n\u003e 0-5: The results of processing are stored in a file, so that it is not necessary to process the whole gallery after restart of the application\n\n✅ See [Cache section](#cache)\n\n\u003e 0-5: The neural network models from examples provided for this course are used\n\n✅ Pre-trained MobileNet for emotions recognition, see [General overview](#general-overview) \n\n\u003e 0-10: At least one extra model (not presented in the examples provided for this course) for facial attribute recognition is used\n\n✅ MediaPipe models for age and gender, custom pre-trained MobileNet V3 for 4 attributes (beard, eyeglasses, mustache, smiling), see [General overview](#general-overview)\n\n\u003e 0-10: Clustering identical faces by using face recognition model is supported\n\n✅\n\n\u003e 0-5: At least one extra model (not presented in the examples provided for this course) for face recognition is used\n\n✅ Mobile FaceNet, see [General overview](#general-overview)\n\n0-15: Special face clustering methods that use specific of faces are implemented, e.g., https://personal.ie.cuhk.edu.hk/~ccloy/files/aaai_2018_merge.pdf\n\n✅ HDBSCAN, see [Clustering section](#clustering)\n\n\u003e 0-15: Visualization of the results by grouping the photos with identical value of selected attribute\n\n✅\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fst235%2Fhse.cvformobiledevices.facialprocessing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fst235%2Fhse.cvformobiledevices.facialprocessing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fst235%2Fhse.cvformobiledevices.facialprocessing/lists"}