{"id":29738690,"url":"https://github.com/saimgulay/unity-rtml-toolkit","last_synced_at":"2026-05-11T07:04:05.383Z","repository":{"id":302058928,"uuid":"1011100163","full_name":"saimgulay/Unity-RTML-ToolKit","owner":"saimgulay","description":"🎛️ Real-Time ML in Unity. RTML Tool Kit is a lightweight, OSC-controllable machine learning framework for Unity, supporting Linear Regression, KNN, and DTW — designed for Mixed Reality and mobile deployment.","archived":false,"fork":false,"pushed_at":"2025-07-01T14:20:31.000Z","size":234,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-26T00:04:49.703Z","etag":null,"topics":["android","artificial-intelligence","audio-visual","dtw","knn-classification","linear-regression","machine-learning","mixed-reality","osc","unity"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saimgulay.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-30T09:52:57.000Z","updated_at":"2025-07-10T13:29:07.000Z","dependencies_parsed_at":"2025-06-30T22:41:21.486Z","dependency_job_id":null,"html_url":"https://github.com/saimgulay/Unity-RTML-ToolKit","commit_stats":null,"previous_names":["saimgulay/rtml-toolkit","saimgulay/unity-rtml-toolkit"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/saimgulay/Unity-RTML-ToolKit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saimgulay%2FUnity-RTML-ToolKit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saimgulay%2FUnity-RTML-ToolKit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saimgulay%2FUnity-RTML-ToolKit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saimgulay%2FUnity-RTML-ToolKit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saimgulay","download_url":"https://codeload.github.com/saimgulay/Unity-RTML-ToolKit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saimgulay%2FUnity-RTML-ToolKit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272805572,"owners_count":24995916,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["android","artificial-intelligence","audio-visual","dtw","knn-classification","linear-regression","machine-learning","mixed-reality","osc","unity"],"created_at":"2025-07-25T19:01:47.039Z","updated_at":"2026-05-11T07:04:05.290Z","avatar_url":"https://github.com/saimgulay.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Unity - RTML Tool Kit: **A Real-Time Machine Learning Framework for Unity**  \n\n\nLightweight, modular, and open-source. Designed for live performance, Mixed Reality, and mobile deployment.\n\n---\n\n## ✨ Overview\n\nRTML Tool Kit is a real-time, OSC-controllable machine learning framework for Unity, inspired by [Wekinator](http://wekinator.org).  \nWhile Wekinator pioneered GUI-based interactive machine learning, its reliance on desktop environments and lack of runtime portability made it unsuitable for mobile and XR pipelines. RTML addresses this gap by offering a Unity-native runtime alternative with Android support and Inspector-based integration.\n\n- 🎛️ **Inspector-driven**: no external GUI or dependencies\n- 🧠 **Built-in models**: Linear Regression, K-Nearest Neighbours, DTW placeholder\n- 💾 **Runtime model saving/loading**\n- 🎯 **Designed for mobile and Mixed Reality workflows**\n- 🛰️ **Full OSC I/O** for external tools (TouchDesigner, Max/MSP, etc.)\n- 🧪 **Runtime-safe** with Unity Test Framework support\n\n---\n\n## 📁 Project Structure\n\n```\nAssets/\n└── RTMLToolKit/\n    ├── Core/           # Engine: RTMLCore, IModel, model classes\n    ├── IO/             # OSCReceiver, OSCSender, optional command parser\n    ├── Util/           # Logger, normalisers, math helpers\n    ├── Editor/         # Custom Inspector tools\n    ├── Runtime/        # Prefabs for scene-based setups (optional)\n    ├── SavedModels/    # Serialized models (JSON)\n    ├── Tests/          # Unit + runtime verification\n    └── ExampleScenes/  # Demo scenes + controller scripts\n```\n\n---\n\n## 🚀 Quick Start\n\n1. Import RTML Tool Kit into your Unity project  \n2. Create a GameObject and add the `RTMLCore` script  \n3. Configure `inputSize`, `outputSize`, and `modelType` in the Inspector (`RTMLCore.cs`)\n4. Feed input data manually or via OSC\n5. Use Inspector toggles to Record → Train → Run  \n6. Use the buttons “Save Model” and “Load Model” to persist models across sessions\n\n---\nVoice Controller Demonstration: [![Voice Controller Demonstration](https://img.youtube.com/vi/L1AZWkIZCNk/hqdefault.jpg)](https://youtu.be/L1AZWkIZCNk)\n---\n\n## 🧠 How to Use RTML Tool Kit (For Absolute Beginners)\n\nThis tool lets you teach your game objects how to learn and react — with zero coding required. Here's how:\n\n---\n\n### 🟢 Step 1 — Add RTML to Your Scene\n\n1. Open your Unity project.\n2. Go to **Assets/RTMLToolKit/Runtime/**.\n3. Drag the **RTMLRunner.prefab** into your scene.\n\nBoom! RTML is now active.\n\n---\n\n### 🎚 Step 2 — Feed It Some Input\n\nYou need to send numbers to RTML so it can learn.\n\nYou can:\n- Connect it to sliders, sensors, or code.\n- Or, just modify `SphereController.cs` in the **ExampleScene** to simulate input.\n\nYour input must be a list of numbers (e.g., `[0.5, 0.2, 0.9]`).\n\n---\n\n### 📊 Step 3 — Choose a Model\n\nIn the **Inspector** of `RTMLRunner`, you'll see a dropdown called:\n\n\u003e `Model Type`\n\nPick one:\n- `LinearRegression` (best for smooth control)\n- `KNNClassifier` (great for categories)\n- `DTWRecognizer` (best for gestures or time-based input)\n\n---\n\n### 🎓 Step 4 — Train the Model\n\nStill in Inspector:\n1. Click **Record** — move the object, send input, etc.\n2. Choose a **Label** (e.g., “jump”, “left”, “red”).\n3. Click **Add Sample**.\n4. Repeat for as many labels as you want.\n5. Click **Train** — the model learns!\n\n---\n\n### 🧪 Step 5 — Use It in Runtime\n\n1. Hit **Play**.\n2. RTML will analyse the input in real-time and output a result.\n3. You can use `RTMLCore.Output` to drive game logic, move objects, trigger events, and more.\n\n---\n\n### 💾 Step 6 — Save or Load Model\n\n\n\n---\n\n### 💡 TL;DR\n\n✅ Drag RTMLRunner  \n✅ Choose a model  \n✅ Record → Add Samples → Train  \n✅ Hit Play and react to output  \n✅ Save model for next time\n\n\nRTML Tool Kit allows you to persist trained model data and reuse it later — both during development and in runtime builds.  \nYou can save and load models either through **Unity Inspector** or via **C# code**.\n\n---\n\n### 🔧 Option 1 — Using the Unity Inspector (No Coding Required)\n\n1. **Add `RTMLCore` component** to any GameObject (or use the one attached to `RTMLRunner.prefab`).\n2. In the **Inspector**, scroll to the section titled `Model Save/Load`.\n3. Type a model name in the input field (e.g., `myModel`).\n4. Click **Save Model** to export the current trained state.\n5. Click **Load Model** to restore a previously saved model.\n\nSaved model files will appear under:\n\n```\nAssets/RTMLToolKit/SavedModels/\n```\n\nIf the file exists, it will be loaded instantly. If not, you'll see a warning in the Console.\n\n---\n\n### 🧑‍💻 Option 2 — Saving \u0026 Loading via Script\n\nYou can also control saving/loading behaviour programmatically:\n\n```csharp\nRTMLCore core = GetComponent\u003cRTMLCore\u003e();\n\n// Save current model state\ncore.SaveModel(\"myModel\");\n\n// Load previously saved state\ncore.LoadModel(\"myModel\");\n```\n\n\u003e If no name is passed, a default fallback name will be used (e.g., `\"defaultModel\"`).\n\n---\n\n### 📦 What Gets Saved?\n\nThe exact contents depend on the model type:\n\n| Model Type       | Saved Data Description                        |\n|------------------|-----------------------------------------------|\n| Linear Regression | Weight vector and bias                        |\n| KNN Classifier    | All training samples and associated labels    |\n| DTW Recogniser    | Template time-series gestures (per label)     |\n\nModels are serialised as `.json` files and can be inspected or version-controlled if needed.\n\n---\n\n\n---\n\nRTML turns numbers into intelligence — so your game objects don’t have to be dumb anymore.\n\n\n## 💡 Supported Models\n\n| Model Type        | Use Case                        | Notes                        |\n|-------------------|----------------------------------|------------------------------|\n| LinearRegression  | Continuous regression            | Fast and lightweight         |\n| KNNClassifier     | Classification or regression     | Memory-based, non-parametric |\n| DTWRecognizer     | Gesture/time-series matching     | Fast and lightweight         |\n\nAll models implement the `IModel` interface with consistent `Train()` and `Predict()` methods.\n\n---\n\n### ⚠️ Notes \u0026 Best Practices\n\n- Always ensure your `RTMLRunner` prefab is active in the scene.\n- For mobile platforms, make sure persistent data paths are correctly mapped when building runtime save/load functionality.\n- Overwriting files will replace previous training data.\n\n---\n\nThis system allows a seamless workflow where training once is enough — even across sessions, devices, and builds.\n\n## 🌐 OSC Integration\n\nRTML supports OSC (Open Sound Control) for external communication.\n\n| OSC Role | Address                  | Value          |\n|----------|--------------------------|----------------|\n| Input    | `/rtml/inputs`           | float array    |\n| Output   | `/rtml/outputs`          | float array    |\n| Record   | `/rtml/control/record`   | float 0/1      |\n| Train    | `/rtml/control/train`    | float 0/1      |\n| Run      | `/rtml/control/run`      | float 0/1      |\n\nEnable OSC in the Inspector by toggling `useOsc`.\n\n---\n\n## ✅ Tests \u0026 Scenes\n\nRTML is tested with Unity's Test Framework (NUnit-compatible).  \nScenes and tests include:\n\n- `LinearRegressionTest.cs`\n- `KNNClassifierTest.cs`\n- `RuntimeSmokeTest.cs`\n- `AutomatedModelTester.cs`\n\nSample scenes:\n- `LinearRegressionTest.scene` (more coming soon)\n\nRun tests via: **Window \u003e General \u003e Test Runner**\n\n---\n\n## 📌 TODO\n\n- [ ] Visualise real-time I/O streams and model state (black-box → glass-box)\n- [ ] Extend DTW to true dynamic time warping (sequence comparison)\n- [ ] Add optional Barracuda backend for neural models\n- [ ] Modular command parsing (custom OSC handlers)\n- [ ] Android build with Meta Quest Mixed Reality support\n- [ ] Refactor Inspector into collapsible widget system\n\n---\n\n## 🤝 Credits \u0026 License\n\nRTML Tool Kit is authored by **Saim Gülay**.  \nIt draws conceptual inspiration from Wekinator but is written from scratch to support modern XR workflows and embedded deployment in Unity-based systems.\n\nThis project was originally developed for academic research in **real-time, embodied machine learning interfaces**.\n\n📄 Licensed under the **Apache License 2.0** – see [LICENSE](./LICENSE) for full terms.\n\n---\n\n## 🧾 Citation\n\nIf you use RTML Tool Kit in your academic work, please cite:\n\n```\nSaim Gülay. \"RTML Tool Kit: A Real-Time, Unity-Native Machine Learning Framework for XR.\" (2025). [arXiv link coming soon]\n```\n\n---\nRTML Tool Kit empowers artists, performers, and researchers with a plug-and-play ML interface that works **where traditional tools can't** — inside Unity, in real time, across platforms.\n\n## Whitepaper\n\n# RTML Tool Kit: A Lightweight, Platform-Agnostic Real-Time ML Framework for Unity\n\n## Abstract\n\nReal-time, interactive machine learning (ML) workflows are increasingly vital in creative coding, live performance, and immersive media applications. While frameworks such as Wekinator have pioneered GUI-based ML pipelines for artists and designers, their reliance on desktop interfaces and external runtimes limits their applicability in mobile and Mixed Reality (MR) contexts.  \n**RTML Tool Kit** addresses this gap by providing a lightweight, Unity-native ML runtime with full OSC integration, modular architecture, and support for Android deployment. This paper introduces the design rationale, core architecture, and preliminary performance benchmarks of the system.\n\n---\n\n## 1. Introduction\n\nCreative technologists often require rapid iteration and live adaptation of ML systems during rehearsals, performances, or interactive installations. Tools like [Wekinator](http://wekinator.org) have enabled accessible workflows by exposing supervised learning through graphical interfaces. However, Wekinator's desktop-centric architecture, reliance on external GUIs, and lack of runtime serialisation make it incompatible with mobile or embedded deployments.\n\n**RTML Tool Kit** is designed as a real-time, in-Editor and runtime-capable alternative. Built entirely in C# for the Unity engine, RTML allows users to:\n- Train, run, and export ML models directly in Unity\n- Integrate OSC I/O for external tools (TouchDesigner, Max/MSP)\n- Deploy to mobile devices (e.g., Android-based MR headsets)\n- Replace GUI-based ML interaction with Inspector-driven pipelines\n\nThis paper documents the motivation, architectural design, implemented models, and evaluation results of the RTML Tool Kit.\n\n---\n\n## 2. Related Work\n\n- **Wekinator** (Fiebrink, 2011): GUI-based interactive ML tool for artists. Lacks mobile support and requires external OSC bridge.\n- **RunwayML**: ML-as-a-service for creatives. High latency, cloud-dependent, not suitable for embedded performance workflows.\n- **Unity ML-Agents**: RL-oriented framework for agents in Unity simulations. Not intended for low-latency interactive ML.\n- **Encodex (TouchDesigner)**: Custom CHOPs allow real-time learning, but lack portability and in-scene serialisation.\n\nRTML Tool Kit situates itself in this landscape by targeting *low-latency, local, supervised ML* within Unity itself.\n\n---\n\n## 3. System Architecture\n\nRTML is composed of modular components grouped under functional domains:\n\n- **RTMLCore.cs**: Central controller, manages data, model lifecycle, and OSC routing\n- **IModel.cs**: Shared interface defining `Train()` and `Predict()` for all models\n- **Model Classes**: `LinearRegression`, `KNNClassifier`, `DTWRecognizer`\n- **SampleBuffer.cs**: Circular buffer for collecting and replaying training samples\n- **OSCReceiver / OSCSender**: Bi-directional OSC input/output\n- **Inspector Tools**: Save/Load model buttons, record/train/run toggles\n\nModels can be swapped at runtime, and serialised via Unity's `JsonUtility` to `Assets/RTMLToolKit/SavedModels/`.\n\n---\n\n## 4. Implemented Models\n\n| Model            | Type           | Purpose                        | Current Status |\n|------------------|----------------|--------------------------------|----------------|\n| LinearRegression | Supervised     | Continuous regression          | ✅ Fully tested |\n| KNNClassifier    | Supervised     | Lazy classification/regression | ⚠ Integrated only |\n| DTWRecognizer    | Sequence-based | Temporal gesture recognition   |  ✅ Fully tested |\n\nAll models conform to a shared `IModel` interface and are selectable via an Inspector dropdown.  \nOnly `LinearRegression` has been evaluated in real runtime scenarios thus far.\n\n---\n\n## 5. Evaluation\n\n### 5.1 Test Conditions\n\n- Unity 6 on macOS (Intel chip)\n- Editor-only tests using Inspector UI\n- InputSize: 5, OutputSize: 12\n- Sample count: 2 to 50 training samples\n\n### 5.2 Results (Linear Regression only)\n\n| Metric                        | Value       |\n|-------------------------------|-------------|\n| Training latency              | ~2 ms       |\n| Prediction latency            | \u003c1 ms       |\n| Save/Load roundtrip           | Verified    |\n| Serialized model file size    | ~120 KB     |\n\nNo evaluation has yet been conducted for `KNNClassifier` or `DTWRecognizer`. They are structurally integrated and will be validated in future updates.\n\n---\n\n## 6. Case Study: Linear Regression in Unity Runtime\n\nA sample scene (`LinearRegressionTest.scene`) was built to demonstrate real-time interaction between OSC inputs and Unity parameters. Inputs were streamed via TouchDesigner and processed by RTML to control the position of GameObjects. The pipeline required no external ML servers and performed entirely within Unity’s runtime loop.\n\n---\n\n## RTML vs Existing ML Tools\n\nWhile various tools exist for interactive or creative machine learning workflows, none offer a fully Unity-native, runtime-safe supervised ML pipeline. Below is a comparison of commonly cited alternatives:\n\n| System         | RTML Comparison                                                                 |\n|----------------|----------------------------------------------------------------------------------|\n| **Wekinator**  | Requires external GUI, desktop-only, no JSON export, no mobile/XR deployment    |\n| **RunwayML**   | Cloud-based, high latency, cannot be embedded into Unity, no runtime training   |\n| **Encodex (TD)** | Real-time via CHOPs in TouchDesigner, but cannot be embedded into Unity        |\n| **ml.lib**     | Works in Max/MSP; requires OSC bridge for Unity, no native integration          |\n| **ml5.js**     | Browser-based, JavaScript only; not designed for Unity or runtime deployment    |\n| **Magenta.js** | Focused on deep learning and music; web-first and not Unity runtime-compatible  |\n| **Unity ML-Agents** | Reinforcement learning only, no support for supervised runtime ML, no OSC/UI |\n\n---\n\n## What Makes RTML Tool Kit Unique?\n\n**Unity-native**  \nAll training, inference, and model management are handled directly through the Unity Inspector. No external software, no GUI bridges, no extra scripts required.\n\n**Runtime-capable**  \nUnlike Wekinator or RunwayML, RTML allows training and inference while the application is running. Models can be trained live and saved/loaded during runtime.\n\n**OSC Input/Output**  \nBidirectional OSC communication enables seamless integration with creative tools like TouchDesigner, Max/MSP, and even modular synth environments.\n\n**Mobile \u0026 XR-Ready**  \nDesigned for Unity’s Android pipeline — including Meta Quest MR builds. Models and logic are deployable via `.apk` without any desktop dependency.\n\n**No Code Required**  \nArtists and non-programmers can use `RTMLRunner.prefab` with the custom Inspector UI to train models without writing a single line of code.\n\n---\n\nBy combining Unity-native execution, runtime model control, and external OSC support, RTML Tool Kit fills a critical gap in real-time ML workflows for mobile, MR, and interactive art installations.\n\n## 7. Conclusion \u0026 Future Work\n\nRTML Tool Kit presents a viable direction for GUI-free, real-time ML in Unity, enabling new workflows in mobile and XR contexts. Future directions include:\n\n- Full evaluation of `KNNClassifier` and `DTWRecognizer`\n- Android .apk deployment and latency profiling\n- Optional support for Barracuda or ONNX backends\n- Visual monitoring of input/output signals in-scene\n\nWe hope this project helps bridge the gap between creative coding and deployable, low-latency ML.\n\n---\n\n## References\n\n- Fiebrink, R. et al. (2011). *Wekinator: Machine Learning for Musicians and Artists*. NIME Proceedings.\n- Unity ML-Agents Toolkit. https://github.com/Unity-Technologies/ml-agents\n- RunwayML. https://runwayml.com\n- Open Sound Control. http://opensoundcontrol.org\n- Gülay, S. (2025). *RTML Tool Kit Repository*. https://github.com/saimgulay/RTMLToolKit\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaimgulay%2Funity-rtml-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaimgulay%2Funity-rtml-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaimgulay%2Funity-rtml-toolkit/lists"}