{"id":15640595,"url":"https://github.com/rishit-dagli/ml-with-android-11","last_synced_at":"2025-04-30T08:13:05.785Z","repository":{"id":105751200,"uuid":"284226663","full_name":"Rishit-dagli/ML-with-Android-11","owner":"Rishit-dagli","description":"A repository demonstrating all that's new in Android 11 for ML and how you could try it out for your own use-cases ","archived":false,"fork":false,"pushed_at":"2020-12-14T12:40:36.000Z","size":42902,"stargazers_count":75,"open_issues_count":1,"forks_count":12,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-30T08:12:55.175Z","etag":null,"topics":["ai","android","android11","kotlin","ml","mlkit","mlkit-android","model-binding","tensorflow","tensorflow-lite"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rishit-dagli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":["https://www.buymeacoffee.com/rishitdagli","http://paypal.me/alpadagli"]}},"created_at":"2020-08-01T09:13:08.000Z","updated_at":"2025-02-21T15:52:54.000Z","dependencies_parsed_at":"2023-06-13T14:31:06.484Z","dependency_job_id":null,"html_url":"https://github.com/Rishit-dagli/ML-with-Android-11","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FML-with-Android-11","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FML-with-Android-11/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FML-with-Android-11/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FML-with-Android-11/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rishit-dagli","download_url":"https://codeload.github.com/Rishit-dagli/ML-with-Android-11/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251666338,"owners_count":21624295,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","android","android11","kotlin","ml","mlkit","mlkit-android","model-binding","tensorflow","tensorflow-lite"],"created_at":"2024-10-03T11:38:15.748Z","updated_at":"2025-04-30T08:13:05.778Z","avatar_url":"https://github.com/Rishit-dagli.png","language":"Java","funding_links":["https://www.buymeacoffee.com/rishitdagli","http://paypal.me/alpadagli"],"categories":[],"sub_categories":[],"readme":"# Machine Learning with Android 11 [![Twitter URL](https://img.shields.io/twitter/url?style=social\u0026url=https%3A%2F%2Fgithub.com%2FRishit-dagli%2FML-with-Android-11)](https://twitter.com/intent/tweet?text=Wow:\u0026url=https%3A%2F%2Fgithub.com%2FRishit-dagli%2FML-with-Android-11)\n\n[![Join the chat at https://gitter.im/ML-with-Android-11/ML-with-Android11](https://badges.gitter.im/ML-with-Android-11/ML-with-Android11.svg)](https://gitter.im/ML-with-Android-11/ML-with-Android11?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n[![GitHub followers](https://img.shields.io/github/followers/Rishit-dagli?label=Follow\u0026style=social)](https://github.com/Rishit-dagli)\n[![Twitter Follow](https://img.shields.io/twitter/follow/rishit_dagli?style=social)](https://twitter.com/intent/follow?screen_name=rishit_dagli)\n\n|![Build ML-Example-App](https://github.com/Rishit-dagli/ML-with-Android-11/workflows/Build%20ML-Example-App/badge.svg)|![Build MLKitCustom](https://github.com/Rishit-dagli/ML-with-Android-11/workflows/Build%20MLKitCustom/badge.svg)|\n|---|---|\n|Camera based Rock, Paper, Scissor Classifier|Camera based OCR and Vision models|\n\nThis repository demonstrates how you can get started with on-device ML with \ntools or plugins specifically launched with \n[Android 11](https://developer.android.com/11). If you have earlier worked with \nML in Android, you will explore easier ways to integrate your ML applications \nwith your Android apps. In this repository I majorly demonstrate the two \nbiggest updates: \n[ML Model Binding Plugin](https://developer.android.com/studio/preview/features#tensor-flow-lite-models)\nand the [new ML Kit](https://g.co/mlkit)\n.\n\nI would also urge you to check out this \n[blog by me](https://medium.com/@rishit.dagli/machine-learning-with-android-11-whats-new-e8c829e9452) \non this topic.\nYou can know more or watch the talks I gave on this topic at \n[talks.md](https://github.com/Rishit-dagli/ML-with-Android-11/blob/master/talks.md) and also take a look at the slides \n\u003ca href=\"http://bit.ly/dc-apac-slides\"\u003e\u003cimg src=\"images/speakerdeck_button.svg\" height=\"20\"\u003e\u003c/a\u003e\n.\n\n## Table of Contents\n\n- [Why care about on-device ML in Android?](#why-care-about-on-device-ml-in-android-)\n  * [The idea behind on-device ML](#the-idea-behind-on-device-ml)\n  * [Advantages of on-device ML](#advantages-of-on-device-ml)\n- [ML Model Binding Plugin](#ml-model-binding-plugin)\n  * [What does Model Binding Plugin focus on?](#what-does-model-binding-plugin-focus-on-)\n  * [Using the Model Binding Plugin](#using-the-model-binding-plugin)\n    + [Import a TF Lite Model](#import-a-tf-lite-model)\n    + [Using the TF Lite Model](#using-the-tf-lite-model)\n  * [Leveraging GPU Acceleration](#leveraging-gpu-acceleration)\n- [New ML Kit](#new-ml-kit)\n  * [What does the new ML Kit focus on?](#what-does-the-new-ml-kit-focus-on-)\n  * [Using the ML Kit](#using-the-ml-kit)\n- [Finding Models](#finding-models)\n- [Wish to contribute to this repository? :sparkles:](#wish-to-contribute-to-this-repository-sparkles)\n  * [1. Fork this repository :fork_and_knife:](#1-fork-this-repository-fork_and_knife)\n  * [2. Clone this repository :busts_in_silhouette:](#2-clone-this-repository-busts_in_silhouette)\n  * [3. Set it up :wrench:](#3-set-it-up-wrench)\n  * [4. Ready, Set, Go!!! :turtle::rabbit2:](#4-ready-set-go-turtlerabbit2)\n  * [5. Create a new branch :bangbang:](#5-create-a-new-branch-bangbang)\n  * [6. Push your awesome work to your remote repository :rocket:](#6-push-your-awesome-work-to-your-remote-repository-rocket)\n  * [7. Finally, make a PR! :fire:](#7-finally-make-a-pr-fire)\n\n## Why care about on-device ML in Android?\n\nAs you might have noticed we majorly focus on on-device ML here, Android 11 has \na lot of cool updates for on-device ML but let's talk in brief about why you \nshould care about it, you will also understand why there is such a hype about \non-device ML or ML on edge.\n\n### The idea behind on-device ML\n\nWhile performing on-device ML as opposed to the traditional approach you no longer \nsend data to a server or some cloud-based system which does the ML part for you \nand then returns me the outputs. So as an example if you were classifying if an \nimage as an example if the image contains a cat or dog, you would no longer send \nthe data here the image to a server. You would instead do the inference on the \ndata on the device itself, do all the computation on the device itself.\n\n![](images/on-device-ml-idea.jpg)\n\nYou would not directly use the model for your edge device. You would need to \ncompress it or optimize the model so you can run it on the edge device as it \nhas limited computation power, network availability, and disk space. In this\ndocument, however, we will not be discussing about the optimization process. We \nwill be deploying a `.tflite` model file. You can read more about \n[TensorFlow Lite](https://www.tensorflow.org/lite/) and the \n[Model Optimization process](https://www.tensorflow.org/lite/performance/model_optimization)\nwith TensorFlow Lite.\n\n### Advantages of on-device ML\n\nHere I have listed some advantages of using on-device ML:\n\n- Power consumption\n\nSo the first thing that would come to your mind is power consumption, you spend \na lot of power sending or streaming video data continuously to a server and \nsometimes it becomes infeasible to do so. However, also worth a mention \nsometimes the opposite could also be true when you employ heavy pre-processing.\n\n- Inference time\n\nAnother important thing to consider is the time it takes me to get the output or\nessentially run the model. For real-time applications, this is a pretty important\naspect to consider. Without sending the data and the having to receive it back \nI speed up my inference time too.\n\n- Network availability \n\nUsing the traditional approach is also expensive in terms of network \navailability. I should have the bandwidth or network to continuously send the \ndata and receive inferences from the server. \n\n- Security \n\nAnd finally security I no longer send data to a server or cloud-based system, \nI no longer send data out of the device at all thus enforcing security.\n\n## ML Model Binding Plugin\n\n### What does Model Binding Plugin focus on?\n\n\u003e Note: You need Android Studio 4.1 or above to be able to use the Model \n\u003e Binding Plugin\n\nYou can make a fair enough guess from the name “Model Building” so as to what \nthe [ML Model Binding Plugin](https://developer.android.com/studio/preview/features#tensor-flow-lite-models)\nwould do allow us to use custom TF Lite Models \nvery easily. This lets developers import any TFLite model, read the input / \noutput signature of the model, and use it with just a few lines of code that \ncalls the open-source TensorFlow Lite Android Support Library.\n\nThe ML model binding plugin makes it super easy for you to use a TF model in \nyour app. You essentially have a lot less code to write that calls the \nTensorFlow Lite Android Support Library. If you have worked with TensorFlow \nLite models you maybe know that you first need to convert everything to a \n`ByteArray` you no longer have to convert everything to `ByteArray` anymore with\nML Model Binding Plugin.\n\nWhat I also love about this new plugin is you can easily use make use of GPUs \nand the NN API very easily. With the model binding plugin using them has never \nbeen easier. Using them is now just a dependency call and a single line of code\naway isn’t that cool what you can do with Model Binding plugin. With Android 11\nThe Neural Network API you also have unsigned integer weight support and a new \nQuality of Service (QOS) API too supporting even more edge scenarios. And Of \ncourse, this would make your development a lot more faster with the features we \njust talked about.\n\n### Using the Model Binding Plugin\n\nLet us now see how we can implement all that we talked about.\n\n#### Import a TF Lite Model\n\nSo the first step is to import a TensorFlow Lite model with metadata. \nAndroid Studio now has a new option for importing TensorFlow model, \njust right click on the module you want to import it in and you will see an \noption under `others` called the `TF Lite model`. \n\n![](images/import-model-opt.jpg)\n\nYou can now just pass in the path of your `tflite` model, it will import the model \nfor you in a directory in the module you selected earlier called `ml` from where\nyou will be able to use the model. Adding the dependencies and GPU acceleration \ntoo is just a click away.\n\n![](images/import-model-dialog-box.jpg)\n\nSo now from my model metadata, I can also know the input, output shapes, and a \nlot more that I would need to use it, you can see this info by opening the \n`tflite` model file in Android Studio. So in this screenshot, I am using an \nopen-source model made by me to classify between rock, paper, and scissors. So \nyou just show your hand in front of the camera and it identifies if it's a rock \npaper or scissor, and that's what I demonstrate here too.\n\n![](images/model-metadata.jpg)\n\n#### Using the TF Lite Model\n\nLet’s finally start using the model, so for a streaming inference which is most\nprobably what you would want to do; live image classification. The easiest way \nwould be to use Camera X and pass each frame to a function which can perform the\ninference. So what I’m interested as of now is the function which does the \ninference. You will see how easy it is to do this, a sample code for this is also \nseem when you import a TF Lite Model which you can use.\n\n```kotlin\nprivate val rpsModel = RPSModel.newInstance(ctx)\n```\n\nSo we’ll start by instantiating an `rps` model short for a rock papers scissors \nmodel and pass it the context. With the plugin, my model name was \n`RPS Model.tflite` so a class of the exact same name would be made for you so I \nhave a class called `RPS Model`.\n\n```kotlin\nval tfImage = TensorImage.fromBitmap(toBitmap(imageProxy))\n```\n\nOnce you do this you need to convert your data into a form which we can use so \nwe’ll convert it to a `Tensor Image` from `bitmap` if you used the TF \nInterpreter, you know that you need to convert your image to a `ByteArray`, \nyou don't need to do that anymore and you’ll feed in an image proxy\n\n```kotlin\nval outputs = rpsModel.process(tfImage)\n    .probabilityAsCategoryList.apply {\n        sortByDescending { it.score } // Sort with highest confidence first\n    }.take(MAX_RESULT_DISPLAY) // take the top results\n```\n\nSo now we will pass in the data to the model so first, we will process the image \nfrom the model and get the outputs we will essentially get an array of \nprobabilities and perform a descending sort on it as we want to show the label \nwhich has the most probability and then picks first `n` results to show.\n\n```kotlin\nfor (output in outputs) {\n    items.add(\n        Recognition(\n            output.label,\n            output.score\n        )\n    )\n}\n```\n\nAnd finally, I want to show users the labels so I will add the label \ncorresponding to each entry in the outputs. And that’s all you need :rocket:.\n\n### Leveraging GPU Acceleration\n\nIf you want to use GPU acceleration again it is made very easy for you so you \nwill make an `options` object where I specify it to use GPU and build it. In \nthe instantiation part, I would just pass this in as an argument and you can use\nthe GPU. It also makes it very easy to use the NN API for acceleration to do \neven more and with Android 11.\n\n```kotlin\nprivate val options = Model.Options.Builder().setDevice(Model.Device.GPU).build()\nprivate val rpsModel = rpsModel.newInstance(ctx, options)\n```\n\n## New ML Kit\n\n### What does the new ML Kit focus on?\n\nThe other notable update Another way to implement a TensorFlow Lite model is \nvia [ML Kit](https://g.co/mlkit). And before I move on ML Kit is now available \neven without having to use a Firebase project, you can now use ML Kit even \nwithout a Firebase project.\n\nAs I mentioned earlier a lot of updates in Android 11 are focused on on-device \nML due to the benefits I mentioned earlier. The new ML Kit now has better \nusability for on-device ML. The ML Kit \n[image classification](https://developers.google.com/ml-kit/vision/image-labeling/custom-models/android) \nand \n[object detection and tracking (ODT)](https://developers.google.com/ml-kit/vision/object-detection/custom-models/android) \nnow also support custom models, which means now you can also have a `tflite` \nmodel file along with this. This also means if you are working on some generic \nuse case like a specific kind of object detection ML Kit is the best thing to \nuse.\n\n### Using the ML Kit\n\nLet's see this in code and see an example of this.\nSo here as an example I build a model which can classify different food items, \n\n```kotlin\nprivate localModel = LocalModel.Builder()\n    .setAssetFilePath(\"lite-model_aiy_vision_classifier_food_V1_1.tflite\").\n    .build()\n```\n\nSo I will first start off by setting the model and specifying the `tflite`\nmodel file path for it.\n\n```kotlin\nprivate val customObjectDetectorOptions = CustomObjectDetectorOptions\n    .Builder(localModel)\n    .setDetectorMode(CustomObjectDetectorOptions.STREAM_MODE) \n    .setClassificationConfidenceThreshold(0.8f) \n    .build()\n```\n\nThis `tflite` model will then run on top of the Object detection model with ML \nKit so you can customize these options a bit. Here I have specifically used the \n`STREAM_MODE` as I want to work with streaming input and also specify the \nconfidence threshold.\n\n```kotlin\nprivate val objectDetector = ObjectDetection.getClient(customObjectDetectorOptions) objectDetector.process(image) \n    .addOnFailureListener(Log.d(...)) \n    .addOnSuccessListener{ \n        graphicsOverlay.clear() \n        for (detectedObject in it){ \n            graphicsOverlay.add(ObjectGraphic(graphicsOverlay, detectedObject))\n        } \n        graphicsOverlay.postInvalidate()} \n    .addOnCompleteListenerl imageProxy.close() } \n```\n\nSo let us get to the part where we run the model so you might see some syntax\nsimilar to the previous example here. I will process my image and a thing to \nnote here is all of these listeners that are on failure or on success are \nessentially tasks so they need to be attached for every run. \nAnd that is all you need to do, we are done :rocket:\n\n## Finding Models\n\nWe talked a lot about what after making a model let us take a look at how you \ncan find models for your use-cases.\n\n- TF Lite Model Maker\n\nTF Lite Model Maker too was announced by The TensorFlow Team earlier in 2020. \nThis makes making good models super easy to use, gives a high performance and, \nalso allows for a good amount of customization. You can simply pass in the data\nand use little code to build a `tflite` model. You can take a look at the \n[TensorFlow Lite Model Maker Example](https://github.com/Rishit-dagli/ML-with-Android-11/blob/dev/TensorFlow_Lite_Model_Maker_example.ipynb) \npresent in this repo.\n\n- TensorFlow Hub\n\nTensorFlow Hub is an open-source repository of state of the art and well \ndocuemnted, models. The food classification app we built with ML Kit is also \npresent on TF Hub. You get to use models from the community. You can find these \nat [tfhub.dev](https//tfhub.dev).\n\n![](images/tf-hub.png)\n\nAnd many more! There are a lot of services like \n[Teachable Machine](https://teachablemachine.withgoogle.com/), \n[AutoML](https://cloud.google.com/automl) and many more but these are the major\nones.\n## Wish to contribute to this repository? :sparkles:\n\nDo you notice a bug or want to add a feature, you can very easily contribute to \nthis repo. I also advise you to take a look at \n[CODE OF CONDUCT](https://github.com/Rishit-dagli/ML-with-Android-11/blob/master/CODE_OF_CONDUCT.md) \n. Here are the steps to help you along :scroll:\n\n### 1. Fork this repository :fork_and_knife:\n\nYou can get your own fork/copy of \n[ML with Android 11](https://github.com/Rishit-dagli/ML-with-Android-11) by \nusing the \u003ca href=\"https://github.com/Rishit-dagli/ML-with-Android-11/new/master?readme=1#fork-destination-box\"\u003e\u003ckbd\u003e\u003cb\u003eFork\u003c/b\u003e\u003c/kbd\u003e\u003c/a\u003e \nbutton or clicking [here](https://github.com/Rishit-dagli/ML-with-Android-11/new/master?readme=1#fork-destination-box).\n[![Fork Button](https://help.github.com/assets/images/help/repository/fork_button.jpg)](https://github.com/CoderJolly/IPU-Engineering-Notes) \n \n### 2. Clone this repository :busts_in_silhouette:\n\nYou need to clone (download a copy of) this repository to your local machine using:\n```sh\n$ git clone https://github.com/Your_Username/ML-with-Android-11.git\n```\n\u003e This makes a local copy of repository in your machine.\nOnce you have cloned the `ML-with-Android-11` repository in GitHub, move to that folder first using change directory command on linux and Mac.\n```sh\n# This will change directory to a folder ML-with-Android-11\n$ cd ML-with-Android-11\n```\nMove to this folder for all other commands.\n\n### 3. Set it up :wrench:\nRun the following commands to see that *your local copy* has a reference to *your forked remote repository* in GitHub :octocat:\n```sh\n$ git remote -v\norigin  https://github.com/Your_Username/ML-with-Android-11.git (fetch)\norigin  https://github.com/Your_Username/ML-with-Android-11.git (push)\n```\nNow, lets add a reference to the original [ML-with-Android-11](https://github.com/Rishit-dagli/ML-with-Android-11) repository using\n```sh\n$ git remote add upstream https://github.com/Rishit-dagli/ML-with-Android-11.git\n```\n\u003e This adds a new remote named ***upstream***.\nSee the changes using\n```sh\n$ git remote -v\norigin    https://github.com/Your_Username/ML-with-Android-11.git (fetch)\norigin    https://github.com/Your_Username/ML-with-Android-11.git (push)\nupstream  https://github.com/Rishit-dagli/ML-with-Android-11.git (fetch)\nupstream  https://github.com/Rishit-dagli/ML-with-Android-11.git (push)\n```\n\n### 4. Ready, Set, Go!!! :turtle::rabbit2:\nOnce you have completed these steps, you are ready to start contributing by checking our `Help Wanted` Issues and creating [pull requests](https://github.com/Rishit-dagli/ML-with-Android-11.git/pulls).\n\n### 5. Create a new branch :bangbang:\nWhenever you are going to make contribution. Please create separate branch using command and keep your `master` branch clean (i.e. synced with remote branch).\n```sh\n# It will create a new branch with name Branch_Name and switch to branch Folder_Name\n$ git checkout -b Folder_Name\n```\nCreate a separate branch for contribution and try to use same name of branch as of folder.\nTo switch to desired branch\n```sh\n# To switch from one folder to other\n$ git checkout Folder_Name\n```\nTo add the changes to the branch. Use\n```sh\n# To add all files to branch Folder_Name\n$ git add .\n```\nType in a message relevant for the code reviewer using\n```sh\n# This message get associated with all files you have changed\n$ git commit -m 'relevant message'\n```\n### 6. Push your awesome work to your remote repository :rocket:\n```sh\n# To push your work to your remote repository\n$ git push -u origin Folder_Name\n```\n### 7. Finally, make a PR! :fire:\nGo to your repository in browser and click on `compare and pull requests`.\nThen add a title and description to your pull request that explains your precious effort.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frishit-dagli%2Fml-with-android-11","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frishit-dagli%2Fml-with-android-11","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frishit-dagli%2Fml-with-android-11/lists"}