{"id":15034790,"url":"https://github.com/getnamo/tensorflow-unreal","last_synced_at":"2026-10-19T14:37:50.096Z","repository":{"id":38354953,"uuid":"75088875","full_name":"getnamo/TensorFlow-Unreal","owner":"getnamo","description":"TensorFlow plugin for the Unreal Engine.","archived":false,"fork":false,"pushed_at":"2024-12-05T03:30:36.000Z","size":509,"stargazers_count":1218,"open_issues_count":36,"forks_count":207,"subscribers_count":83,"default_branch":"main","last_synced_at":"2026-10-05T02:29:33.649Z","etag":null,"topics":["blueprint","machine-learning","python","tensorflow","ue4","ue5","unreal-engine"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/getnamo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-11-29T14:26:00.000Z","updated_at":"2026-10-01T17:35:01.000Z","dependencies_parsed_at":"2024-01-16T21:46:47.293Z","dependency_job_id":"796e9317-d10c-4262-a5f8-8c19011c7780","html_url":"https://github.com/getnamo/TensorFlow-Unreal","commit_stats":null,"previous_names":["getnamo/tensorflow-ue4"],"tags_count":29,"template":false,"template_full_name":null,"purl":"pkg:github/getnamo/TensorFlow-Unreal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getnamo%2FTensorFlow-Unreal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getnamo%2FTensorFlow-Unreal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getnamo%2FTensorFlow-Unreal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getnamo%2FTensorFlow-Unreal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/getnamo","download_url":"https://codeload.github.com/getnamo/TensorFlow-Unreal/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getnamo%2FTensorFlow-Unreal/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":344872684,"owners_count":38358137,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-10-03T21:59:58.778Z","status":"online","status_checked_at":"2026-10-19T02:00:06.414Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blueprint","machine-learning","python","tensorflow","ue4","ue5","unreal-engine"],"created_at":"2024-09-24T20:26:20.517Z","updated_at":"2026-10-19T14:37:50.089Z","avatar_url":"https://github.com/getnamo.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TensorFlow Unreal Plugin\n\n[![GitHub release](https://img.shields.io/github/release/getnamo/TensorFlow-Unreal/all.svg)](https://github.com/getnamo/TensorFlow-Unreal/releases)\n[![Github All Releases](https://img.shields.io/github/downloads/getnamo/TensorFlow-Unreal/total.svg)](https://github.com/getnamo/TensorFlow-Unreal/releases)\n\n[\u003cimg src=\"https://i.imgur.com/qn73w1u.png\" alt=\"badge\" width=\"100\"/\u003e](https://www.unrealengine.com/en-US/blog/epic-megagrants-reaches-13-million-milestone-in-2019)\n\n[TensorFlow](https://www.tensorflow.org/) and [Keras](https://keras.io/) for [Unreal Engine](https://www.unrealengine.com) 5.8, from Blueprints and C++. You write the model in Python. The plugin runs it, trains it, feeds it inputs, render targets and audio, and returns results and events to Unreal.\n\n**2.0 is a full rewrite.**\n- Python runs out of process, in an embedded interpreter with a per-project virtual environment, through [PythonCLI](https://github.com/getnamo/PythonCLI-Unreal) on top of [CLISystem](https://github.com/getnamo/CLISystem-Unreal).\n- A crash in TensorFlow never takes the editor down, and the GIL never touches the game thread.\n- MachineLearningRemote, socket.io and UnrealEnginePython are gone.\n- The Python API stays the same: `TFPluginAPI` scripts keep working. See [Migrating from 1.x](#migrating-from-1x).\n\nStatus: **2.0.0**. Tested on Win64 in the editor and in packaged builds, and on Linux x86_64 in packaged builds.\n\n[Discord Server](https://discord.gg/qfJUyxaW4s)\n\n## PyTorch Alternative\n\nIf you find TensorFlow GPU support on Windows lacking or would like to use a more modern python ML library with unreal, consider using the [PyTorch-Unreal](https://github.com/getnamo/pytorch-unreal) plugin instead.\n\n\n## Setup\n\n1. Put these three plugins in your project's `Plugins/`:\n   - `TensorFlow-Unreal`\n   - [PythonCLI-Unreal](https://github.com/getnamo/PythonCLI-Unreal)\n   - [CLISystem-Unreal](https://github.com/getnamo/CLISystem-Unreal)\n\n   Release archives include all three, with the Python interpreter. From git, fetch the interpreter with PythonCLI's `Tools/FetchRuntime.ps1` (or `fetch_runtime.sh`).\n2. Start the editor. In the background, PythonCLI creates `Intermediate/PythonCLI/venv` and installs the TensorFlow profile into it:\n   - The download is about 350 MB, once.\n   - Later starts skip setup.\n   - A rebuilt venv comes from uv's cache in seconds.\n3. Add a **TensorFlow** component to an actor. Set **Script Params \u003e Script** to your script, for example `my_model` for `Content/Scripts/my_model.py`, or a bundled example such as `examples/mnist_keras_cnn`.\n\n### Environment profiles\n\nChoose one in **Project Settings \u003e Plugins \u003e TensorFlow \u003e Profile**:\n\n| Profile | Installs | Use |\n|---|---|---|\n| TensorFlow (CPU), the default | `tensorflow-cpu` 2.21 | Windows and Linux. Windows has had no native TensorFlow GPU build since 2.10. |\n| TensorFlow (CUDA, Linux) | `tensorflow[and-cuda]` 2.21 | Linux with an NVIDIA GPU (a few GB of CUDA wheels). |\n| Keras on PyTorch (GPU on Windows) | `keras` 3 + `torch` (CUDA build) | GPU on Windows: Keras 3 on the torch backend (`KERAS_BACKEND=torch`). |\n| None | nothing | List your own packages in **Python CLI \u003e Requirements**. |\n\n- **Your own packages:** add them in **Project Settings \u003e Plugins \u003e Python CLI \u003e Requirements**, or in `Content/Scripts/requirements.txt`.\n- **Long paths:** keep the project path short or enable long paths on Windows; TensorFlow's files are deep.\n- **Log level:** TensorFlow's C++ logging uses Profile's TensorFlow Log Level (`TF_CPP_MIN_LOG_LEVEL`).\n\n### Packaging\n\nPackaged games (Win64, Linux) take the interpreter, the scripts and, by default, the installed packages with them. PythonCLI installs the environment's packages for the target platform while packaging, so a Linux build packaged on Windows gets Linux wheels. The tensorflow-cpu profile adds about 1.2 GB to a Linux build and 1.3 GB to a Win64 build.\n\nTo keep TensorFlow out of the build, set **Project Settings \u003e Plugins \u003e Python CLI \u003e Packaged Env Mode** to **Install On First Run**: the game then installs it on its first launch. See [PythonCLI's packaging notes](https://github.com/getnamo/PythonCLI-Unreal#packaging).\n\n## Python API\n\n```python\nimport unreal_bridge as ue\nfrom unreal_bridge.ml import TFPluginAPI   # or MLPluginAPI: the same class, snake_case hooks\n\nclass Classifier(TFPluginAPI):\n    def on_setup(self):                    # once, when the script starts: build or load the model\n        import keras\n        self.model = keras.saving.load_model(ue.content_path(\"Models\", \"mnist.keras\"))\n\n    def on_json_input(self, data):         # Send Json Input -\u003e On Input Results\n        return {\"prediction\": 3}\n\n    def on_tensor_input(self, image):      # Send Tensor Input / Texture / Render Target: a numpy array\n        scores = self.model(image[None], training=False)\n        return {\"prediction\": int(scores.numpy().argmax())}\n\n    def on_begin_training(self):           # Begin Training: runs on its own thread\n        self.model.fit(x, y, epochs=5, callbacks=[ue.keras.StopOnRequest(self),    # Stop Training ends fit()\n                                                  ue.keras.ProgressEvents(self)])   # -\u003e On Training Progress\n        self.call_event(\"Saved\", {\"path\": \"...\"})                                   # -\u003e On Custom Event\n        return {\"accuracy\": 0.99}           # -\u003e On Training Complete\n\ndef get_api():\n    return Classifier.get_instance()\n```\n\n**The API class:**\n- **Hooks:** `on_setup`, `on_json_input`, `on_float_array_input`, `on_tensor_input`, `on_begin_training`, `on_stop_training`, `on_save_model` and `on_load_model`. The camelCase forms from 1.x work too (`onJsonInput`, `onBeginTraining`, …).\n- **Custom functions:** any other method can be called with **Call Custom Function**.\n- **Stopping:** `self.should_stop` (or `shouldStop`) turns true on Stop Training, a script stop or reload, and EndPlay.\n- **Events:** `self.call_event(name, data)` (or `callEvent`) reaches On Custom Event. `self.report_progress(data)` reaches On Training Progress.\n- **Concurrency:** inputs are answered while training runs. Everything else for a script runs in order, on that script's own thread.\n\n**The rest of `unreal_bridge`** (`ue.log`, `ue.emit`, `@ue.on_stream`, …) is in the [PythonCLI README](https://github.com/getnamo/PythonCLI-Unreal#python-api-unreal_bridge).\n\n## Blueprint API\n\n**TensorFlow component.** It is a Python ML component, and runs in PythonCLI's shared Python process by default.\n\n| Calls | |\n|---|---|\n| Send Json Input / Send String Input | `on_json_input(data)`, or a named function; the result arrives in On Input Results |\n| Send Tensor Input | a tensor (Make Tensor From Floats, Texture To Tensor, …) to `on_tensor_input` |\n| Send Texture Input / Send Render Target Input | an image to `on_tensor_input` as a float32 (H, W, C) tensor (grayscale/RGB/RGBA, resize, invert, normalization) |\n| Send Json Input Texture | 1.x: `{\"pixels\": [grayscale], \"size\": {\"x\", \"y\"}}` to `on_json_input` |\n| Send Float Array Input | `on_float_array_input(list)`; the result arrives in On Float Array Results |\n| Call Custom Function | any method of the API object, with JSON arguments |\n| Begin Training / Stop Training, Save Model / Load Model | |\n| **Send Json Input (Async)**, **Send Tensor Input (Async)**, **Call Custom Function (Async)** | async nodes that return the result (or the error) instead of firing the events |\n\n| Events | |\n|---|---|\n| On Setup Complete | `on_setup` finished |\n| On Input Results (JsonResult, FunctionName), On Float Array Results | |\n| On Training Started / Progress (Json) / Complete (JsonSummary, bStoppedEarly) | |\n| On Custom Event (EventName, Data) | `call_event` |\n| On Script Error (Message, Traceback) | load failures, exceptions in calls and training, crashes |\n\n**Properties:**\n- **Train On Begin Play**\n- **Stop Training On End Play**\n- **Script Params:** Script, Script Root and Args (`ue.args`).\n- **Use Shared Process:** off gives the component its own process.\n\n**Reading results:** use **Get Json Number / String / Floats** with a path such as `\"prediction\"`, `\"scores.3\"` or `\"action.move\"`.\n\n**Images:**\n- **Texture To Tensor** and **Render Target To Tensor** produce float32 tensors (H, W, C), or (C, H, W) when channels-first. **Tensor To Texture** goes the other way.\n- The 1.x conversions are kept: To Float Array, To Grayscale Float Array, To Texture2D, Invert Float Array.\n- **Texture reading** uses, in order:\n  - the editor's source image;\n  - a CPU copy (texture **Availability: CPU**);\n  - uncompressed platform data (Compression **UserInterface2D** or **Grayscale**).\n- Render targets are read with a blocking call.\n\n**Streams:** add a **Python Render Target Stream** next to the component, for every frame of a scene capture without stalls. **Python Audio Stream** does the same for audio and **Python Observation Stream** for gameplay data. See [examples/camera_stream_classify](Content/Scripts/examples/camera_stream_classify.py).\n\n**TF Audio Capture (1.x):** microphone capture as raw PCM bytes (On Audio Data, On Capture Finished, WAV helpers), now on Unreal's AudioCapture for every platform. New projects should use a Python Audio Stream.\n\n## Example map\n\nOpen **/TensorFlow/Examples/MnistExample** (Content Browser: Plugins \u003e TensorFlow Content \u003e Examples) and press Play.\n\n![MNIST example map](Resources/MnistExample.png)\n\nHow it works:\n- `BP_MnistExample` runs `examples/mnist_demo` on a TensorFlow component.\n- On the first play it trains a small CNN on MNIST (about 30 s on a CPU) with live status lines. It saves `Saved/TensorFlow/mnist.keras`, and later plays load that model instead.\n- Every 3 seconds it asks Python for a test digit and draws it into a render target.\n- A **Python Render Target Stream** next to the component streams that render target back to the script, and the model reads what the camera sees.\n- Keys: **T** trains, **S** stops training, **N** shows the next digit.\n\n**Event graph:** everything comes from events.\n- **On Custom Event** (`Status`, `Digit`, `Prediction`) drives the TextRender components and the canvas.\n- **On Script Error** shows Python errors.\n- The keys call Begin Training, Stop Training and Call Custom Function.\n\n**Regenerating:** the example content is generated by [BlueprintOracle](https://github.com/getnamo/BlueprintOracle-Unreal) from [Tools/Examples/mnist_example.json](Tools/Examples/mnist_example.json):\n\n`UnrealEditor-Cmd.exe \u003cProject\u003e.uproject -run=BlueprintOracle -migrate -spec=Plugins/TensorFlow-Unreal/Tools/Examples/mnist_example.json -commit -unattended -nullrhi -nopause`\n\n## Examples\n\nThese are bundled in `Content/Scripts/examples/` and resolve by name, for example `Script = examples/mnist_keras_cnn`:\n\n| Script | |\n|---|---|\n| `add_example` | the smallest TF2 script: Send Json Input `{\"a\": [1, 2], \"b\": [3, 4]}` → `{\"c\": [4, 6]}` |\n| `mnist_keras_cnn` | trains a Keras CNN on MNIST with progress events and early stop, then saves and loads `Saved/TensorFlow/mnist.keras`. Predicts tensors and 1.x JSON pixels. |\n| `mnist_spawn_samples` | streams MNIST images to Unreal as events (Call Custom Function `spawn_samples`, or Begin Training until stopped) |\n| `camera_stream_classify` | with a Python Render Target Stream named `camera`: classifies the digit the render target shows, as it streams |\n| `mnist_demo` | the example map's script: `mnist_keras_cnn` plus display-ready `Status` / `Digit` / `Prediction` events and live classification of the `camera` stream |\n\nThe [examples project](https://github.com/getnamo/TensorFlow-Unreal-examples) has maps for 1.x.\n\n## Migrating from 1.x\n\nThe backend changed completely; there is no MachineLearningRemote, socket.io or UnrealEnginePython mode. What carries over:\n\n- **Python scripts:** `TFPluginAPI` subclasses with camelCase hooks, `getApi()`, `callEvent`, `shouldStop`, `import unreal_engine as ue` (`log`, `get_content_dir`). `from TFPluginAPI import TFPluginAPI` still works. Scripts using TF1 sessions and placeholders need porting to TF2 / Keras 3.\n- **Script names:** set **Script Params \u003e Script** to your module name; the `.py` is optional.\n- **Blueprints:** the component is now C++ (`TensorFlowComponent`).\n  - A CoreRedirect maps the old Blueprint class, so references load, but its variables (Tensor Flow Module, Should Train On Begin Play) need setting again.\n  - The function names are the same: Begin Training, Stop Training, Send Json Input, Send Json Input Texture. The events are On Input Results, On Training Complete, and On Custom Event (1.x's On Event).\n- **Dependencies:** `upymodule.json` is gone. Use the environment profile, Python CLI Requirements, or `Content/Scripts/requirements.txt`.\n- **Old code:** the last 1.x commit is tagged `v1.6.0beta-legacy`.\n\n## Tests\n\n`UnrealEditor-Cmd.exe \u003cProject\u003e.uproject -ExecCmds=\"Automation RunTests TensorFlow; Quit\" -unattended -nullrhi -nopause`\n\n- The tests cover the profile, image conversions, the add example, and MNIST training, stop and prediction.\n- Two tests need a real RHI (use `-RenderOffscreen` instead of `-nullrhi`):\n  - `TensorFlow.Mnist.WithCamera` draws a test digit into a render target, streams it, and checks the classification.\n  - `TensorFlow.Examples.MnistMap` plays the example map in PIE. It waits until the camera reads the digit the map drew, presses N, and saves a screenshot to `Saved/Screenshots`.\n\n## License\n\nMIT; see LICENSE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetnamo%2Ftensorflow-unreal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgetnamo%2Ftensorflow-unreal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetnamo%2Ftensorflow-unreal/lists"}