{"id":21458080,"url":"https://github.com/shubham0204/facerecognition_with_facenet_android","last_synced_at":"2025-04-05T22:08:56.663Z","repository":{"id":40431326,"uuid":"273630336","full_name":"shubham0204/FaceRecognition_With_FaceNet_Android","owner":"shubham0204","description":"Face Recognition using the FaceNet model and MLKit on Android.","archived":false,"fork":false,"pushed_at":"2024-07-27T16:05:47.000Z","size":386878,"stargazers_count":278,"open_issues_count":10,"forks_count":92,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-29T21:07:05.363Z","etag":null,"topics":["android","android-application","android-studio","embeddings","face-detection","face-recognition","facenet-model","firebase-mlkit","kotlin","kotlin-android","machine-learning","tensorflow2"],"latest_commit_sha":null,"homepage":"https://towardsdatascience.com/using-facenet-for-on-device-face-recognition-with-android-f84e36e19761","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shubham0204.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"shubham0204","custom":["https://www.paypal.me/ShubhamPanchalDev"]}},"created_at":"2020-06-20T03:19:32.000Z","updated_at":"2025-03-27T21:22:53.000Z","dependencies_parsed_at":"2024-07-27T17:39:28.333Z","dependency_job_id":null,"html_url":"https://github.com/shubham0204/FaceRecognition_With_FaceNet_Android","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FFaceRecognition_With_FaceNet_Android","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FFaceRecognition_With_FaceNet_Android/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FFaceRecognition_With_FaceNet_Android/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FFaceRecognition_With_FaceNet_Android/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shubham0204","download_url":"https://codeload.github.com/shubham0204/FaceRecognition_With_FaceNet_Android/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247406091,"owners_count":20933803,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["android","android-application","android-studio","embeddings","face-detection","face-recognition","facenet-model","firebase-mlkit","kotlin","kotlin-android","machine-learning","tensorflow2"],"created_at":"2024-11-23T06:17:36.909Z","updated_at":"2025-04-05T22:08:56.635Z","avatar_url":"https://github.com/shubham0204.png","language":"Kotlin","funding_links":["https://github.com/sponsors/shubham0204","https://www.paypal.me/ShubhamPanchalDev"],"categories":[],"sub_categories":[],"readme":"\n\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003eFace Recognition and Classification With FaceNet On Android\u003c/h1\u003e\n\u003c/div\u003e\n\n\u003e **Store images of people who you would like to recognize and the app, using these images, will classify those people. \nWe don't need to modify the app/retrain any ML model to add more people ( subjects ) for classification**  \n\n![banner](https://github.com/shubham0204/FaceRecognition_With_FaceNet_Android/assets/41076823/5922415d-ff61-4276-9817-7e472ebac7c4)\n\nDo check an enhanced version of this project,\n\n* [On-Device Face Recognition In Android](https://github.com/shubham0204/OnDevice-Face-Recognition-Android)  \n \n \n### Features\n\n* Asynchronous processing with [Kotlin Coroutines](https://developer.android.com/kotlin/coroutines)\n* Quick labelling of faces with C++ computation (See [`native`](https://github.com/shubham0204/FaceRecognition_With_FaceNet_Android/tree/native) branch)\n* Use of latest Android development practices with configurable camera facing, GPU usage and mask detection.\n\n---\n![Working of the app](images/app_1.gif)\n\nIf you're ML developer, you might have heard about FaceNet, Google's state-of-the-art model for generating face embeddings. In this   \nproject, we'll use the FaceNet model on Android and generate embeddings ( fixed size vectors ) which hold information of the face.  \n  \n\u003e The accuracy of the face detection system ( with FaceNet ) may not have a considerable accuracy. Make sure you explore other options as well while considering your app's production.  \n  \n## FaceNet\n\n![Working of the FaceNet model](images/fig_1.png)\n\nSo, the aim of the FaceNet model is to generate a 128 dimensional vector of a given face. It takes in an 160 * 160 RGB image and   \noutputs an array with 128 elements. How is it going to help us in our face recognition project?   \nWell, the FaceNet model generates similar face vectors for similar faces. Here, by the term \"similar\", we mean   \nthe vectors which point out in the same direction.\nIn this app, we'll generate two such vectors and use a suitable metric to compare them ( either L2norm or cosine similarity ). \nThe one which is the closest will form our desired output.  \n  \nYou can download the FaceNet Keras `.h5` file from this [repo](https://github.com/nyoki-mtl/keras-facenet) and TFLite model \nfrom the [`assets`](https://github.com/shubham0204/FaceRecognition_With_FaceNet_Android/tree/master/app/src/main/assets) folder.\n  \n## Usage\n\n![Intended File Structure](images/fig_2.png)\n  \nSo, an user can store images in his/her device in a specific folder. If, for instance, the user wants the app to recognize  \ntwo people namely \"Rahul\" and \"Neeta\". So the user needs to store the images by creating two directories namely \"Rahul\" and \"Neeta\"   \nand store their images inside of these directories. For instance, the file structure for the working example ( as shown above in the GIF ),\n\n![Intended File Structure](images/fig_4.png)\n\nThe app will then process these images and classify these people thereafter. For face recognition, Firebase MLKit is used which   \nfetches bounding boxes for all the faces present in the camera frame.  \n  \n\u003e For better performance, we recommend developers to use more images of the subjects, they need to recognize.\n\n## Working  \n\n![Sample Prediction](images/fig_3.png)\n  \nThe app's working is described in the steps below:\n  \n1. Scan the `images` folder present in the internal storage. Next, parse all the images present within `images` folder and store   \nthe names of sub directories within `images`. For every image, collect bounding box coordinates ( as a `Rect` ) using MLKit.\n   Crop the face from the image ( the one which was collected from user's storage ) using the bounding box coordinates.   \n  \n2. Finally, we have a list of cropped `Bitmap` of the faces present in the images. Next, feed the cropped `Bitmap` to the FaceNet   \nmodel and get the embeddings ( as `FloatArray` ). Now, we create a `HashMap\u003cString,FloatArray\u003e` object where we store the names of   \nthe sub directories as keys and the embeddings as their corresponding values. \n   \nSee [`MainActivity.kt`](https://github.com/shubham0204/FaceRecognition_With_FaceNet_Android/blob/master/app/src/main/java/com/ml/quaterion/facenetdetection/MainActivity.kt) and [`FileReader.kt`](https://github.com/shubham0204/FaceRecognition_With_FaceNet_Android/blob/master/app/src/main/java/com/ml/quaterion/facenetdetection/FileReader.kt) for the code.\n  \nThe above procedure is carried out only on the app's startup. The steps below will execute on each camera frame.  \n  \n1. Using `androidx.camera.core.ImageAnalysis`, we construct a `FrameAnalyser` class which processes the camera frames. Now, for a   \ngiven frame, we first get the bounding box coordinates ( as a `Rect` ) of all the faces present in the frame. Crop the face from   \nthe frame using these boxes.  \n2. Feed the cropped faces to the FaceNet model to generate embeddings for them. We compare the embedding with a suitable metric and\nform clusters for each user. We compute the average score for each cluster. The cluster with the best score is our output.\nThe final output is then stored as a `Prediction` and passed to the `BoundingBoxOverlay` which draws boxes and   \ntext.  \n3. For multiple images for a single user, we compute the score for each image. An average score is computed for each group.\n  The group with the best score is chosen as the output. See `FrameAnalyser.kt`.\n\n```  \nimages -\u003e  \n    Rahul -\u003e \n         image_rahul_1.png -\u003e score=0.6 --- | average = 0.65 --- |\n         image_rahul_2.png -\u003e score=0.5 ----|                    | --- output -\u003e \"Rahul\"\n    Neeta -\u003e                                                     |\n         image_neeta_1.png -\u003e score=0.4 --- | average = 0.35 --- |\n         image_neeta_2.png -\u003e score=0.3 ----|             \n ```\n\nSee [`FaceNetModel.kt`](https://github.com/shubham0204/FaceRecognition_With_FaceNet_Android/blob/master/app/src/main/java/com/ml/quaterion/facenetdetection/FaceNetModel.kt) and [`FrameAnalyser.kt`](https://github.com/shubham0204/FaceRecognition_With_FaceNet_Android/blob/master/app/src/main/java/com/ml/quaterion/facenetdetection/FrameAnalyser.kt) for the code.\n  \n## Limitations  \n  \nPredictions may go wrong as FaceNet does not always produce similar embeddings for the same person. \nConsider the accuracy of the FaceNet model while using it in your apps. In that case, you may learn to use the `FaceNetModel` class separating for using FaceNet in some other tasks.  \n\n## Important Resources  \n\n- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://arxiv.org/abs/1503.03832)\n- [MLKit](https://developers.google.com/ml-kit/vision/face-detection) for face recognition.  \n- [TensorFlow Lite Android](https://www.tensorflow.org/lite)  \n- [TensorFlow Lite Android Support Library](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/experimental/support/java)  \n- [CameraX](https://developer.android.com/training/camerax)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshubham0204%2Ffacerecognition_with_facenet_android","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshubham0204%2Ffacerecognition_with_facenet_android","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshubham0204%2Ffacerecognition_with_facenet_android/lists"}