Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/finger563/unreal-rtsp-display
Unreal Engine 5 Example for receiving video streams over RTSP (using FSockets and FRunnables) and displaying their image data in a texture.
https://github.com/finger563/unreal-rtsp-display
mjpeg mjpeg-player rtsp-client rtsp-player rtsp-stream runnable ue5 unreal-engine
Last synced: 2 months ago
JSON representation
Unreal Engine 5 Example for receiving video streams over RTSP (using FSockets and FRunnables) and displaying their image data in a texture.
- Host: GitHub
- URL: https://github.com/finger563/unreal-rtsp-display
- Owner: finger563
- License: mit
- Created: 2023-06-09T12:33:14.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-07-30T14:52:02.000Z (over 1 year ago)
- Last Synced: 2024-10-20T08:00:39.090Z (2 months ago)
- Topics: mjpeg, mjpeg-player, rtsp-client, rtsp-player, rtsp-stream, runnable, ue5, unreal-engine
- Language: C++
- Homepage:
- Size: 207 KB
- Stars: 18
- Watchers: 2
- Forks: 8
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# unreal-rtsp-display
Example code for receiving video streams over RTSP using Unreal Engine FSockets
and displaying them in real time.
https://github.com/finger563/unreal-rtsp-display/assets/213467/64270d95-f223-43ef-885b-599e607d48b1
When running this example, you will need to configure the address, port, and
path of your RTSP server on the actor:
Note: this example currently only supports MJPEG streams over RTSP, which are
parsed with the RtpJpegPacket class into JpegFrames. This example uses Unreal
Engine's built-in image decoding to decode the jpeg frames into uncompressed
image data for use in a UTexture2D.
This example contains a few components:
1. The `RtspClientComponent` class: This component can be added to an actor and
exposes some functions for connecting to an RTSP server and configuring /
controlling the stream. Inside its TickComponent function, it waits for new
images (decompressed) to be available and if so, creates a new transient
UTexture2D and writes the decompressed data to the new texture. It then
broadcasts this new texture using the multicast delegate to any registered
listeners.
2. The `RtpPacket`, `RtpJpegPacket`, `JpegHeader`, and `JpegFrame` classes which
handle the parsing of the media data (as RTP over UDP from the server to the
client) and reassembling of multiple networks packets into a single jpeg
frame.
3. The `MyRunnable` class: used by the RtspClientComponent when it connects to a
server it spawns two runnables (Unreal Engine threads) for receiving data
from the server on the RTP/UDP socket and the RTCP/UDP socket. The RTP
runnable runs a bound function from the RtspClientComponent class which
receives the raw data, parses it into jpeg frames, decompresses the jpeg
frames, and then updates some mutex-protected data to inform the game thread
(RtspClientComponent::TickComponent) that new data is available. This class
is also used to allow the FSocket::Connect (TCP connection from RTSP Client
to RTSP Server) to run without blocking the main / game thread.
4. `M_Display` and `M_Display_Inst`: these assets in the Content/Materials
directory are simple materials which render a texture parameter with optional
configuration for the UV mapping of the texture. This material instance is
the base for the dynamic material instance that is created at runtime in the
RtspDisplay blueprint.
5. `RtspDisplay`: This blueprint actor contains an RtspClientComponent and a
Plane static mesh component. On BeginPlay it creates a dynamic material
instance of the M_Display_Inst which it stores a reference to so that it can
dynamically update the texture parameter that the material is rendering. It
sets this material on the plane static mesh component. It binds an event to
the OnFrameReceived event from the RtspClientComponent and when it receives a
message from that event, it sets the new texture to be the dynamic material
instance's texture parameter.
6. `W_RtspDisplay`: This user widget contains the UI (2D) for interacting with a
RtspClientComponent. It is configured by the `RtspDisplayMap`'s level
blueprint to be added as the UI to the viewport for the first player
controller and to control the RtspClientComponent of the RtspDisplay actor in
the level. It provides the UI (display image, URL textbox, and buttons for
connect, disconnect, play, and pause) for the RtspClientComponent.
Image of the running example in the editor:
Disconnected (With text box to write URI and connect button):
Connected:
Playing:
### Details
#### RtspDisplay Actor Blueprint
#### M_Display Material
#### RtspDisplay User Widget
Note the image has been set with angle 180 and X scale of -1 so that it matches the camera image that I'm currently sending. May need to be changed for other streams depending on camera orientation.
Its blueprint:
#### Level Blueprint
### Setup for Android App
Follow the setup instructions
[here](https://docs.unrealengine.com/5.2/en-US/how-to-set-up-android-sdk-and-ndk-for-your-unreal-engine-development-environment/).
Note: you will likely have to modify the `/Users/Shared/Epic\
Games/UE_5.2/Engine/Extras/Android/SetupAndroid.command` file - possibly to
point to the right `JAVA_HOME` directory. In my case I had to modify the
JAVA_HOME export in the `SetupAndroid.command` file to point to
`/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home` and had
to install jdk8 specifically.
You will need to set the environment variables (under `Android SDK`)
appropriately, e.g.:
- `Android SDK` : `/Users/bob/Library/Android/sdk`
- `Android NDK` : `/Users/bob/Library/Android/sdk/ndk/25.1.8937393`
- `Location of JAVA` : `/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home`
- `SDK API Level` : `android-32`
- `NDK API Level` : `android-32`
For that version of java (jdk 8) which is required to successfully build for
android, you can (on macos) install it via:
``` sh
brew install --cask adoptopenjdk8
```