{"id":13677920,"url":"https://github.com/homuler/MediaPipeUnityPlugin","last_synced_at":"2025-04-29T12:32:24.388Z","repository":{"id":37464847,"uuid":"278114481","full_name":"homuler/MediaPipeUnityPlugin","owner":"homuler","description":"Unity plugin to run MediaPipe","archived":false,"fork":false,"pushed_at":"2024-10-20T07:32:20.000Z","size":4499,"stargazers_count":1798,"open_issues_count":47,"forks_count":467,"subscribers_count":32,"default_branch":"master","last_synced_at":"2024-10-29T17:42:24.021Z","etag":null,"topics":["mediapipe","unity"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/homuler.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":"homuler","tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2020-07-08T14:40:12.000Z","updated_at":"2024-10-29T05:42:39.000Z","dependencies_parsed_at":"2023-02-18T17:30:23.340Z","dependency_job_id":"7b88f561-bcd1-424d-b737-4ef833495d4c","html_url":"https://github.com/homuler/MediaPipeUnityPlugin","commit_stats":null,"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/homuler%2FMediaPipeUnityPlugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/homuler%2FMediaPipeUnityPlugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/homuler%2FMediaPipeUnityPlugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/homuler%2FMediaPipeUnityPlugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/homuler","download_url":"https://codeload.github.com/homuler/MediaPipeUnityPlugin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224173314,"owners_count":17268096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mediapipe","unity"],"created_at":"2024-08-02T13:00:48.535Z","updated_at":"2025-04-29T12:32:24.360Z","avatar_url":"https://github.com/homuler.png","language":"C#","readme":"# MediaPipe Unity Plugin\n\nThis is a Unity (\u003e= 2022.3) [Native Plugin](https://docs.unity3d.com/Manual/NativePlugins.html) to use [MediaPipe](https://github.com/google/mediapipe) (0.10.20).\n\nThe goal of this project is to port the MediaPipe API (C++) _one by one_ to C# so that it can be called from Unity.\\\nThis approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.\n\nWith this plugin, you can\n\n- Write MediaPipe code in C#.\n- Run MediaPipe's official solution on Unity.\n- Run your custom `Calculator` and `CalculatorGraph` on Unity.\n  - :warning: Depending on the type of input/output, you may need to write C++ code.\n\n## :smile_cat: Hello World!\n\nHere is a Hello World! example.\\\nCompare it with [the official code](https://github.com/google/mediapipe/blob/cf101e62a9d49a51be76836b2b8e5ba5c06b5da0/mediapipe/examples/desktop/hello_world/hello_world.cc)!\n\n```cs\nusing Mediapipe;\nusing UnityEngine;\n\npublic sealed class HelloWorld : MonoBehaviour\n{\n    private const string _ConfigText = @\"\ninput_stream: \"\"in\"\"\noutput_stream: \"\"out\"\"\nnode {\n  calculator: \"\"PassThroughCalculator\"\"\n  input_stream: \"\"in\"\"\n  output_stream: \"\"out1\"\"\n}\nnode {\n  calculator: \"\"PassThroughCalculator\"\"\n  input_stream: \"\"out1\"\"\n  output_stream: \"\"out\"\"\n}\n\";\n\n    private void Start()\n    {\n        using var graph = new CalculatorGraph(_ConfigText);\n        using var poller = graph.AddOutputStreamPoller\u003cstring\u003e(\"out\");\n        graph.StartRun();\n\n        for (var i = 0; i \u003c 10; i++)\n        {\n            graph.AddPacketToInputStream(\"in\", Packet.CreateStringAt(\"Hello World!\", i));\n        }\n\n        graph.CloseInputStream(\"in\");\n        var packet = new Packet\u003cstring\u003e();\n\n        while (poller.Next(packet))\n        {\n            Debug.Log(packet.Get());\n        }\n        graph.WaitUntilDone();\n    }\n}\n```\n\nFor more detailed usage, see [the API Overview](https://github.com/homuler/MediaPipeUnityPlugin/wiki/API-Overview) page or [the tutorials](./docs).\n\n## :hammer_and_wrench: Installation\n\nPlease first download the pre-built package from the [releases page](https://github.com/homuler/MediaPipeUnityPlugin/releases).\n\n| file                                  | contents                                                                 |\n| ------------------------------------- | ------------------------------------------------------------------------ |\n| MediaPipeUnityPlugin-all.zip          | All the source code with required libraries.                             |\n| MediaPipeUnityPlugin-all-stripped.zip | Same as `MediaPipeUnityPlugin-all.zip` but the symbols are stripped.     |\n| com.github.homuler.mediapipe-\\*.tgz   | A [tarball package](https://docs.unity3d.com/Manual/upm-ui-tarball.html) |\n| MediaPipeUnityPlugin.\\*.unitypackage  | A .unitypackage file                                                     |\n\nIf you need to run sample scenes on your mobile devices, prefer `MediaPipeUnityPlugin-all.zip` or `MediaPipeUnityPlugin-all-stripped.zip`.\\\nTo run sample scenes on your mobile devices, you need to place required models properly, but most required setup is already done in `MediaPipeUnityPlugin-all.zip`.\n\n## Build the plugin by yourself\n\n\u003e :warning: In most cases, you don't need to build the plugin by yourself. Only if the pre-built package doesn't work for you, please build the plugin by yourself.\n\nThis repository **doesn't include required libraries or models**, so if you clone this repository, you need to build the plugin by yourself.\\\nSee [the build guide](./docs/Build.md) for more details.\n\n## Build a package by yourself\n\nIf you want, you can also build the plugin by yourself using `MediaPipeUnityPlugin-all(-stripped).zip`.\n\n### Build a unity package\n\n1. Open this project\n1. Click `Tools \u003e Export Unitypackage`\n   ![export-unity-package](https://user-images.githubusercontent.com/4690128/163669270-2d5365eb-eac1-46b1-aed5-83c28a377090.png)\n\n- `MediaPipeUnity.[version].unitypackage` file will be created at the project root.\n\n### Build a local tarball file\n\n1. Install `npm` command\n1. Build a tarball file\n\n```sh\ncd Packages/com.github.homuler.mediapipe\nnpm pack\n# com.github.homuler.mediapipe-[version].tgz will be created\n\nmv com.github.homuler.mediapipe-[version].tgz your/favorite/path\n```\n\n## Supported Platforms\n\n\u003e :warning: GPU mode is not supported on macOS and Windows.\n\n|                            |       Editor       |   Linux (x86_64)   |   macOS (x86_64)   |   macOS (ARM64)    |  Windows (x86_64)  |      Android       |        iOS         | WebGL |\n| :------------------------: | :----------------: | :----------------: | :----------------: | :----------------: | :----------------: | :----------------: | :----------------: | :---: |\n|     Linux (AMD64) [^1]     | :heavy_check_mark: | :heavy_check_mark: |                    |                    |                    | :heavy_check_mark: |                    |       |\n|         Intel Mac          | :heavy_check_mark: |                    | :heavy_check_mark: |                    |                    | :heavy_check_mark: | :heavy_check_mark: |       |\n|           M1 Mac           | :heavy_check_mark: |                    |                    | :heavy_check_mark: |                    | :heavy_check_mark: | :heavy_check_mark: |       |\n| Windows 10/11 (AMD64) [^2] | :heavy_check_mark: |                    |                    |                    | :heavy_check_mark: | :heavy_check_mark: |                    |       |\n\n[^1]: Tested on Arch Linux.\n[^2]: Running MediaPipe on Windows is [experimental](https://ai.google.dev/edge/mediapipe/framework/getting_started/install#installing_on_windows).\n\n## Supported Solutions\n\nThis plugin implements the following [MediaPipe Tasks](https://ai.google.dev/edge/mediapipe/solutions/tasks) C# APIs.\n\ncf. [The official available solutions](https://ai.google.dev/edge/mediapipe/solutions/guide#available_solutions)\n\n|         Solution         |      Android       |        iOS         |       Linux        |       macOS        |      Windows       |\n| :----------------------: | :----------------: | :----------------: | :----------------: | :----------------: | :----------------: |\n|    LLM Inference API     |                    |                    |                    |                    |\n|     Object detection     | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |\n|   Image classification   |                    |                    |                    |                    |                    |\n|    Image segmentation    | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |\n| Interactive segmentation |                    |                    |                    |                    |                    |\n| Hand landmark detection  | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |\n|   Gesture recognition    | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |\n|     Image embedding      |                    |                    |                    |                    |                    |\n|      Face detection      | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |\n| Face landmark detection  | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |\n|     Face stylization     |                    |                    |                    |                    |                    |\n| Pose landmark detection  | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |\n|     Image generation     |                    |                    |                    |                    |                    |\n|   Text classification    |                    |                    |                    |                    |                    |\n|      Text embedding      |                    |                    |                    |                    |                    |\n|    Language detector     |                    |                    |                    |                    |                    |\n|   Audio classification   | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |\n\n### Legacy Solutions\n\nYou can also use [MediaPipe Framework](https://ai.google.dev/edge/mediapipe/framework), which allows you to run [Legacy Solutions](https://ai.google.dev/edge/mediapipe/solutions/guide#legacy). However, please note that support for these solutions has ended.\n\n# :book: Usage\n\nOnce you've downloaded the pre-built package, please import the plugin into your project.\n\n- [.unitypackage](https://docs.unity3d.com/Manual/upm-ui-import.html)\n- [.tar.gz](https://docs.unity3d.com/Manual/upm-ui-tarball.html)\n\n## For Android\n\n:skull_and_crossbones: If you need to build your app for Android, **please ensure you include `libstdc++_shared.so` in your APK**[^3], otherwise `DllNotFoundException` will be thrown at runtime.\n\n[^3]: `mediapipe_android.aar` contains `libopencv_java4.so` and it depends on `libstdc++_shared.so`. However, some project or plugins may already include `libstdc++_shared.so`, so we don't include `libstdc++_shared.so` in `mediapipe_android.aar`.\n\nThe easiest way to include `libstdc++_shared.so` in your APK is to place it in the `Assets/Plugins/Android` directory of your project.\n\nYou can also include `libstdc++_shared.so` at build time by adding the following code to your [`mainTemplate.gradle` file](https://docs.unity3d.com/Manual/gradle-templates.html), and the sample project is using this method.\n\n```gradle\n// Include libc++_shared.so\ntask copyLibcppShared(type: Copy) {\n    def ndkDir = android.ndkDirectory\n    from(\"$ndkDir/sources/cxx-stl/llvm-libc++/libs\") { include '**/libc++_shared.so' }\n    into(\"$projectDir/src/main/jniLibs\")\n}\nclean.dependsOn 'cleanCopyLibcppShared'\n\ntasks.whenTaskAdded { task -\u003e\n    if (task.name == \"mergeDebugJniLibFolders\" || task.name == \"mergeReleaseJniLibFolders\") {\n        task.dependsOn(\"copyLibcppShared\")\n    }\n}\n```\n\n## :plate_with_cutlery: Try the sample app\n\nBefore using the plugin in your project, it's strongly recommended that you check if sample scenes work.\n\n![test-face-mesh](https://user-images.githubusercontent.com/4690128/163668702-26357605-c1f2-4678-8fce-3adc258a9aa1.png)\n\n### Example Solutions\n\nSome solutions (including Legacy solutions) can be tested using the sample app.\nPlease check [`Assets/MediaPipeUnity/Samples/Scenes`](https://github.com/homuler/MediaPipeUnityPlugin/tree/master/Assets/MediaPipeUnity/Samples/Scenes) to see which solutions have samples.\n\n### UnityEditor\n\nSelect any scenes under `Assets/MediaPipeUnity/Samples/Scenes` and play.\n\n### Desktop, Android, iOS\n\nSelect proper Inference Mode and Asset Loader Type from the Inspector Window.\n\n#### Preferable Inference Mode\n\nIf the native libraries are built for CPU (i.e. `--desktop cpu`), select `CPU` for inference mode.\\\nFor the libraries distributed on the release page, only the CPU is available for use on Windows and macOS.\n\n![preferable-inference-mode](https://github.com/homuler/MediaPipeUnityPlugin/assets/4690128/129d18be-8184-43f7-8ac8-56db4df9f9a7)\n\n#### Asset Loader Type\n\nThe default Asset Loader Type is set to `Local`, which only works on UnityEditor.\\\nTo run it on your devices, switch it to `StreamingAssets` and copy the required resources under [`StreamingAssets`](https://docs.unity3d.com/2022.3/Documentation/Manual/StreamingAssets.html) (if you're using `MediaPipeUnityPlugin-all.zip`, the `StreamingAssets` directory already contains them).\n\n![asset-loader-type](https://github.com/homuler/MediaPipeUnityPlugin/assets/4690128/f7059140-4da9-4201-a232-83ff07cd63df)\n\n## :warning: Technical Limitations\n\n### UnityEditor / Your application may crash!\n\nSince this plugin uses native libraries under the hood, if there is a bug in those libraries, the UnityEditor or the application may crash at runtime.\n\nAdditionally, in some cases, MediaPipe may crash the entire program by sending a `SIGABRT` signal instead of throwing an exception.\n\nThis may not be a problem in production since it usually happens when there's a fatal bug in the application code, and such bugs are probably fixed before release.\\\nHowever, in a development environment, it is very annoying since the UnityEditor crashes.\n\nOn Linux and macOS, this plugin avoids UnityEditor crashing by handling `SIGABRT`, so if UnityEditor crashes, please let us know!\\\nOn Windows, there seem to be no ways to handle `SIGABRT` properly, so if you cannot tolerate this, use a different OS.\n\n### Graphics API\n\nIf you want to run inference using a GPU, you cannot use OpenGL Core API.\nOtherwise, you will encounter an error like the following:\n\n```txt\nInternalException: INTERNAL: ; eglMakeCurrent() returned error 0x3000_mediapipe/mediapipe/gpu/gl_context_egl.cc:261)\n```\n\nIn practice, this error only occurs on PC standalone builds, and in such cases, please switch the Graphics API to Vulkan.\n\n## :scroll: LICENSE\n\n[MIT](https://github.com/homuler/MediaPipeUnityPlugin/blob/master/LICENSE)\n\nNote that some files are distributed under other licenses.\n\n- MediaPipe ([Apache Licence 2.0](https://github.com/google/mediapipe/blob/e6c19885c6d3c6f410c730952aeed2852790d306/LICENSE))\n- emscripten ([MIT](https://github.com/emscripten-core/emscripten/blob/7c873832e933e86855f5ef5f7c6438f0e457c94e/LICENSE))\n  - `third_party/mediapipe_emscripten_patch.diff` contains code copied from emscripten\n- FontAwesome ([LICENSE](https://github.com/FortAwesome/Font-Awesome/blob/7cbd7f9951be31f9d06b6ac97739a700320b9130/LICENSE.txt))\n  - Sample scenes use Font Awesome fonts\n\nSee also [Third Party Notices.md](https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Third%20Party%20Notices.md).\n","funding_links":["https://ko-fi.com/homuler"],"categories":["Game-Design-Tool","Open Source Repositories","Libraries, Platforms and Development Platform-specific Resources"],"sub_categories":["Machine Learning","Development Platform"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhomuler%2FMediaPipeUnityPlugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhomuler%2FMediaPipeUnityPlugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhomuler%2FMediaPipeUnityPlugin/lists"}