{"id":13710368,"url":"https://github.com/jimbobbennett/Xam.Plugins.OnDeviceCustomVision","last_synced_at":"2025-05-06T19:30:54.915Z","repository":{"id":81417309,"uuid":"116733166","full_name":"jimbobbennett/Xam.Plugins.OnDeviceCustomVision","owner":"jimbobbennett","description":"Plugin to allow custom vision models to be used from mobile devices","archived":false,"fork":false,"pushed_at":"2019-10-28T20:54:23.000Z","size":8480,"stargazers_count":57,"open_issues_count":2,"forks_count":13,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-30T11:56:38.062Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jimbobbennett.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-08T21:59:42.000Z","updated_at":"2023-06-26T01:39:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"99ab79cc-dc63-40f7-9796-31e31a9a7e8a","html_url":"https://github.com/jimbobbennett/Xam.Plugins.OnDeviceCustomVision","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jimbobbennett%2FXam.Plugins.OnDeviceCustomVision","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jimbobbennett%2FXam.Plugins.OnDeviceCustomVision/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jimbobbennett%2FXam.Plugins.OnDeviceCustomVision/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jimbobbennett%2FXam.Plugins.OnDeviceCustomVision/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jimbobbennett","download_url":"https://codeload.github.com/jimbobbennett/Xam.Plugins.OnDeviceCustomVision/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252753195,"owners_count":21798930,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T23:00:55.080Z","updated_at":"2025-05-06T19:30:49.901Z","avatar_url":"https://github.com/jimbobbennett.png","language":"C#","readme":"# Xam.Plugins.OnDeviceCustomVision\n\nThe [Azure Custom Vision service](https://customvision.ai/?WT.mc_id=customvision-github-jabenn) is able to create models that can be exported as CoreML, TensorFlow or ONNX models to do image classification on device.\n\nThis plugin makes it easy to download and use these models offline from inside your mobile app, using CoreML on iOS, TensorFlow on Android and Windows ML on Windows. These models can then be called from a .NET standard library, using something like Xam.Plugins.Media to take photos for classification.\n\n## Setup\n\n* Available on NuGet: https://www.nuget.org/packages/Xam.Plugins.OnDeviceCustomVision/ [![NuGet](https://img.shields.io/nuget/v/Xam.Plugins.OnDeviceCustomVision.svg?label=NuGet)](https://www.nuget.org/packages/Xam.Plugins.OnDeviceCustomVision/)\n* Install into your .NET Standard project and iOS, Android and UWP client projects.\n\n## Platform Support\n\n|Platform|Version|\n| ------------------- | :------------------: |\n|Xamarin.iOS|iOS 11+|\n|Xamarin.Android|API 21+|\n|UWP|Windows SDK 17110+|\n\n## Usage\n\nBefore you can use this API, you need to initialize it with the model file downloaded from CustomVision. Trying to classify an image without calling `Init` will result in a `ImageClassifierException` being thrown.\n\n\u003e Each platform has it's own platform-specific `Init` method on a platform-specific static class. This is due to differences in the way each platform handles the models.\n\n### iOS\n\nExport, then download the Core ML model from the Custom Vision service.\n\n#### Pre-compiled models\n\nModels can be compiled before being used, or compiled on the device. To use a pre-compiled model, compile the downloaded model using:\n\n```bash\nxcrun coremlcompiler compile \u003cmodel_file_name\u003e.mlmodel \u003cmodel_name\u003e.mlmodelc\n```\n\nThis will spit out a folder called `\u003cmodel_name\u003e.mlmodelc` containing a number of files. Add this entire folder to the `Resources` folder in your iOS app. Once this has been added, add a call to `Init` to your app delegate, passing in the name of your compiled model without the extension (i.e. the name of the model folder __without__ `mlmodelc`):\n\n```cs\nusing Xam.Plugins.OnDeviceCustomVision;\n...\npublic override bool FinishedLaunching(UIApplication uiApplication, NSDictionary launchOptions)\n{\n   ...\n   iOSImageClassifier.Init(\"\u003cmodel_name\u003e\");\n   return base.FinishedLaunching(uiApplication, launchOptions);\n}\n```\n\n#### Uncompiled models\n\nAdd the downloaded model, called `\u003cmodel_name\u003e.mlmodel`, to the `Resources` folder in your iOS app.Once this has been added, add a call to `Init` to your app delegate, passing in the name of your model without the extension (i.e. the name of the model folder __without__ `mlmodel`):\n\n```cs\nusing Xam.Plugins.OnDeviceCustomVision;\n...\npublic override bool FinishedLaunching(UIApplication uiApplication, NSDictionary launchOptions)\n{\n   ...\n   iOSImageClassifier.Init(\"\u003cmodel_name\u003e\");\n   return base.FinishedLaunching(uiApplication, launchOptions);\n}\n```\n\nThe call to `Init` will attempt to compile the model, throwing a `ImageClassifierException` if the compile fails.\n\n### Android\n\nExport, then download the TensorFlow model from the Custom Vision service. This will be a zip file, and when unzipped this will be a folder containing two files.\n\n* `labels.txt`\n* `model.pb`\n\nAdd both these files to the `Assets` folder in your Android app. Once these are added, add a call to `Init` to your main activity passing in the name of the model file, name of the labels file and the type of model downloaded from the custom vision service. The model and labels file names have default arguments of `\"models.pb\"` and `\"labels.txt\"`, so you only need to set these if you have changed the names of these files.\n\n```cs\nusing Xam.Plugins.OnDeviceCustomVision;\n...\nprotected override void OnCreate(Bundle savedInstanceState)\n{\n   ...\n   AndroidImageClassifier.Init(\"model.pb\", \"labels.txt\", ModelType.General);\n}\n```\n\n\u003e The model type is used to provide adjustments to the image color scheme, and this is only necessary for models [trained before 7th May 2018](https://github.com/Azure-Samples/cognitive-services-android-customvision-sample#compatibility), models trained after this date do not need any adjustments. The library will work out for you if adjustments need to be made, but if the model was generated after 7th May 2018 you can leave the model type as the default value.\n\n### Windows\n\nExport, then download the ONNX model from the Custom Vision service. Add the downloaded model to the `Assets` folder in your UWP app and ensure the **Build Action** is set to **Content**. A `\u003cmodel_name\u003e.cs` file will be created in the root folder of your app to provide a wrapper around the model, but this wrapper is not needed as this plugin provides its own wrapper.\n\n\u003e Don't delete the wrapper class yet, as you will need it to get the labels for the model.\n\nYou need to pass the name of the model, along with a list of the models labels in the order that the tags have been defined in the model. You can get this order by opening the auto-generated `\u003cmodel_name\u003e.cs` file and copying the order from there. The wrapper will have a class called `\u003cmodel_name\u003eModelOutput` and in the constructor for this class will be some code to create a dictionary called `loss`:\n\n```cs\nthis.loss = new Dictionary\u003cstring, float\u003e()\n{\n    { \"\u003clabel 1\u003e\", float.NaN },\n    { \"\u003clabel 2\u003e\", float.NaN },\n    ...\n};\n```\n\nThis defines the labels in the correct order. Pass them to the `Init` method along with the model name inside the `MainPage` class in your UWP app, or similar page class. The `Init` method is `async` so will need to be awaited in an appropriate place, such as by overriding `OnNavigatedTo`.\n\n```cs\nusing Xam.Plugins.OnDeviceCustomVision;\n...\nprotected override async void OnNavigatedTo(NavigationEventArgs e)\n{\n    await WindowsImageClassifier.Init(\"Currency\", new[] { \"\u003clabel 1\u003e\", \"\u003clabel 2\u003e\", ... });\n    base.OnNavigatedTo(e);\n}\n```\n\n### Calling this from your .NET Standard library\n\nTo classify an image, call:\n\n```cs\nvar tags = await CrossImageClassifier.Current.ClassifyImage(stream);\n```\n\nPassing in an image as a stream. You can use a library like [Xam.Plugins.Media](https://github.com/jamesmontemagno/MediaPlugin) to get an image as a stream from the camera or image library.\n\nThis will return a list of `ImageClassification` instances, one per tag in the model with the probability that the image matches that tag. Probabilities are doubles in the range of 0 - 1, with 1 being 100% probability that the image matches the tag. To find the most likely classification use:\n\n```cs\ntags.OrderByDescending(t =\u003e t.Probability)\n    .First().Tag;\n```\n\n### Using with an IoC container\n\n`CrossImageClassifier.Current` returns an instance of the `IImageClassifier` interface, and this can be stored inside your IoC container and injected where required.\n","funding_links":[],"categories":["Plugins"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjimbobbennett%2FXam.Plugins.OnDeviceCustomVision","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjimbobbennett%2FXam.Plugins.OnDeviceCustomVision","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjimbobbennett%2FXam.Plugins.OnDeviceCustomVision/lists"}