{"id":13639959,"url":"https://github.com/TimmHess/UnrealImageCapture","last_synced_at":"2025-04-20T02:32:19.447Z","repository":{"id":37235580,"uuid":"228018594","full_name":"TimmHess/UnrealImageCapture","owner":"TimmHess","description":"A small tutorial repository on capturing images with semantic annotation from UnrealEngine to disk.","archived":false,"fork":false,"pushed_at":"2024-01-11T12:41:59.000Z","size":36224,"stargazers_count":216,"open_issues_count":16,"forks_count":49,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-08-03T01:15:34.181Z","etag":null,"topics":["cameracapture","tutorial","ue4","unrealengine4","virtualworlds"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TimmHess.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-14T12:23:01.000Z","updated_at":"2024-07-08T08:56:26.000Z","dependencies_parsed_at":"2023-01-20T19:02:25.367Z","dependency_job_id":null,"html_url":"https://github.com/TimmHess/UnrealImageCapture","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimmHess%2FUnrealImageCapture","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimmHess%2FUnrealImageCapture/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimmHess%2FUnrealImageCapture/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimmHess%2FUnrealImageCapture/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TimmHess","download_url":"https://codeload.github.com/TimmHess/UnrealImageCapture/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223816298,"owners_count":17207833,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cameracapture","tutorial","ue4","unrealengine4","virtualworlds"],"created_at":"2024-08-02T01:01:06.485Z","updated_at":"2025-04-20T02:32:19.439Z","avatar_url":"https://github.com/TimmHess.png","language":"C++","readme":"# Image Capturing With UnrealEngine 5.5 (for deep learning)\n\n\n| Color | Segmentation |\n|---------|---------|\n| ![](https://github.com/TimmHess/UnrealImageCapture/blob/master/gfx/CaptureResult_color.jpeg) | ![](https://github.com/TimmHess/UnrealImageCapture/blob/master/gfx/CaptureResult_segmentation.png) |\n\n# Changelog\n- Updated to UE5.5.4\n- Streamlined capture code to not block the *render thread*, and further reduce the load on the *main game thread*.\n- Fixed alpha channel for pixel annotation in the post-process material. Images are no longer appearing as \"empty\" because of their alpha mask being 0.\n- Removed plugins from repository - reduce the mess of files\n- Slightly updated the tutorial text\n\n# Outline\n- [Introduction](#a-small-introduction)\n- [MAIN: How to Save Images to Disk (without blocking the main threads)](#how-to-save-images-to-disk-in-ue5-without-blocking-the-rendering-or-main-thread)\n- [Capturing Object Pixel-Annotations (segmentation mask)](#capturing-annotations)\n- [Capturing Scene Depth](#setup-a-depth-capture)\n- [Enable Lumen on SceneCapture2D](#enable-lumen-on-scenecapture2d)\n- [Known Issues](#known-issues)\n\n# TLDR\nUse these links to the [FrameCaptureManger.h](https://github.com/TimmHess/UnrealImageCapture/blob/master/CaptureToDisk/Source/CaptureToDisk/Public/FrameCaptureManager.h) and [FrameCaptureManger.cpp](https://github.com/TimmHess/UnrealImageCapture/blob/master/CaptureToDisk/Source/CaptureToDisk/Private/FrameCaptureManager.cpp) file. They are the only source needed.\nPlus, make sure to link the correct unreal-libs to your project - check the [prerequisite](#prerequisite) or [CaptureToDisk.Build.cs](https://github.com/TimmHess/UnrealImageCapture/blob/master/CaptureToDisk/Source/CaptureToDisk/CaptureToDisk.Build.cs).\n\n**Kudos to the UE4 and UE5 community!**\n\n**Special thanks to [Panakotta00](https://github.com/Panakotta00), for pointing to an even better GPU readback!**\n\nUsing the source as is you should get an `AFrameCaptureManager` (`Actor`) in your scene with the following settings.\n![](https://github.com/TimmHess/UnrealImageCapture/blob/master/gfx/FrameCaptureManage_settings.png)\n\nMaybe most important is the `ImageFormat` setting. You want to use `PNG` for lossless compression when storing [object annotation](#capturing-object-segmentation-masks). For color rendered images most likely `JPEG` gives you better file-sizes. `EXR` is the float format - the code automatically stores .exr images when used.\n\n# Known Issues\nCapturing per-pixel annotations is done using the `CustomDepth` feature. The `CustomDepthStencil` is of type `uint8` which allows a range of 0-255, i.e. **we can handle at most 256** different annotations per image!\n\n#\n# A Small Introduction\nIn this repository I condense my findings on how to implement a component to capture images to disk from an arbitrary UE5 (former UE4) scene **from scratch** lowering the bar for UE novices (and potentially bypassing the need for large frameworks that don't fit ones own particular needs). This will include:\n1. Capturing rendered images to disk at high FPS, without blocking the UE rendering thread or the main game thread\n2. Rendering pixel annotations (or other graphics buffers, such as depth) at the same time\n\nUnrealEngine (UE) is a powerful tool to create virtual worlds capable of AAA productions. Generating temporally consistent data with automatic pixel-wise annotations from complex scenes, such as traffic scenarios, is a capability worth leveraging. Especially for training and validation of machine learning- or deep learning applications it has been explored in a variety of projects. Already, there are plugins available that allow rendering images from UE to disk at runtime, such as prominently [Carla](https://carla-ue5.readthedocs.io/en/latest/), [UnrealCV](https://unrealcv.org/), or [AirSim](https://github.com/microsoft/AirSim). This repository aims to be a tutorial that demonstrates such an 'image capturing' mechanism in detail for you to understand its inner workings, and in turn enable you to reuse it in a custom fashions that suit the needs of your project. \n\nWhen I was setting up scenes for my research the plugins mentioned above were just not yet supporting the latest engine versions that I wanted/needed to use. Also, I was missing a place where the knowledge of how to render images to disk was explains for non-advanced graphics-programmers. Of course, there are lots of sources for code available online and also there are community blog-entries scattered across multiple platforms explaining parts of the problem and possible solutions, even though they typically are targeting very particular scenarios.\n\n**Disclaimer: I do not claim to own any of the code. Merely, I condensed the sources already available online for easier use and provide an overview to the general functionality of this particular approach!**\n\n\n\n#\n# How to Save Images to Disk In UE5 (without blocking the rendering or main thread)\nI will go through the main components of the code step-by-step so that hopefully it will be easier to implement each step as you are following along. However, I recommend looking at the source that is merely a single class ([here]()). \n\n*In the explanations I skip certain quality-of-life-like aspects for sakes of readability, for example exposing image resolution settings to the editor instead of hardcoding them. Make sure to check out the sources linked in [TLDR](#tldr)*\n\n\n#\n## Prerequisite\nYou will need a UE5 C++ project. \n\nAlso, you will have to add a few packages to your `'YourProjectName'.Build.cs` file. These are part of UnrealEngine, however, sometimes they are not added automatically resulting in unpleasant (linker) errors. Find the `'YourProjectName'.Build.cs` file in the `Source/'YourProjectName/` directory, and add or extend it to include the modules: `\"ImageWrapper\", \"RenderCore\", \"Renderer\", \"RHI\"`, for example like this:\n\n``` cpp\nPublicDependencyModuleNames.AddRange(new string[] { \"Core\", \"CoreUObject\", \"Engine\", \"InputCore\", \"EnhancedInput\" , \"ImageWrapper\", \"RenderCore\", \"Renderer\", \"RHI\"});\n```\n\n#\n## Setup A FrameCapture Component\nI am using ```SceneCaptureComponent2D``` as the basis for capturing images. Placing one of these into your scene will give you an ```ASceneCaptureComponent``` which is its `Actor` instance. It basically behaves like any other camera component, but its viewport is not restricted by your computer's monitor or main camera viewport. This provides us the possibility to render images of arbitrary resolution independent from the actual screen resolution.\n\n\u003e Add a ```FrameCaptureManager``` class of type `Actor` to your project.\n\nAll functionality to request the capturing of a frame, as well as receiving the rendered image back, and storing the frame to disk will be handled by the `FrameCaptureManager`. \n\nIn the ```FrameCaptureManager.h``` we add the following:\\\n**FrameCaptureManager.h**\n``` cpp\n#pragma once\nclass ASceneCapture2D; // forward declaration\n\n#include ... // the stuff that is already there\n```\nand to our public variables:\n``` cpp\n// Color Capture  Components\nUPROPERTY(EditAnywhere, BlueprintReadWrite, Category=\"Capture\")\nASceneCapture2D* CaptureComponent;\n```\n\nThis enables you to assign a ```CaptureComponent2d``` to your ```FrameCaptureManager``` inside the UE5 Editor.\n\n\u003e Compile and place a ```FrameCaptureManager``` in your scene. \n\nAs it does not have any primitive to render you will only see it in the editor's outline. In the details panel of the placed ```FrameCaptureManager``` you can now see the ```CaptureComponent``` assigned to ```None```. From the drop down menu select the ```CaptureComponent2D``` you already placed in the scene.\n\nBack to code: We will now prepare our yet \"naked\" `CaptureComponent2D` class for capturing images. This includes creating and assigning a `RenderTarget` - which is basically a `Texture` to store our image data to - and setting the camera properties. \n\n*Note: You could also do this in the Editor but if you deal with, i.e. multiple capture components, you may find it handy not to worry about creating and assigning all the components by hand!*\n\n\u003e Create a setup function to put all your setup code for the CaptureComponents in the CaptureManger:\n\n**FrameCaptureManager.h**\n``` cpp\nprotected:\n\n    void SetupCaptureComponent();\n```\n**FrameCaptureManager.cpp**\n``` cpp\n#include ...\n\n// A bunch of includes we need\n#include \"Engine/SceneCapture2D.h\"\n#include \"Components/SceneCaptureComponent2D.h\"\n#include \"ShowFlags.h\"\n\n#include \"Engine/TextureRenderTarget2D.h\"\n#include \"Materials/Material.h\"\n#include \"RHICommandList.h\"\n#include \"IImageWrapper.h\"\n#include \"IImageWrapperModule.h\"\n#include \"ImageUtils.h\"\n\n#include \"Modules/ModuleManager.h\"\n#include \"Misc/FileHelper.h\"\n\n\nvoid AFrameCaptureManager::SetupCaptureComponent(){\n    if(!IsValid(CaptureComponent)){\n        UE_LOG(LogTemp, Error, TEXT(\"SetupCaptureComponent: CaptureComponent is not valid!\"));\n        return;\n    }\n\n    // Create RenderTargets\n    UTextureRenderTarget2D* renderTarget2D = NewObject\u003cUTextureRenderTarget2D\u003e();\n    renderTarget2D-\u003eInitAutoFormat(256, 256); // some random format, got crashing otherwise\n    \n    renderTarget2D-\u003eRenderTargetFormat = ETextureRenderTargetFormat::RTF_RGBA8_SRGB; //ETextureRenderTargetFormat::RTF_RGBA8; //8-bit color format\n    renderTarget2D-\u003eInitCustomFormat(FrameWidth, FrameHeight, PF_R8G8B8A8, true); // PF... disables HDR, which is most important since HDR gives gigantic overhead, and is not needed!\n    renderTarget2D-\u003ebForceLinearGamma = true; // Important for viewport-like color reproduction.\n\n    renderTarget2D-\u003ebGPUSharedFlag = true; // demand buffer on GPU\n\n    // Assign RenderTarget\n    CaptureComponent-\u003eGetCaptureComponent2D()-\u003eTextureTarget = renderTarget2D;\n    // Set Camera Properties\n    CaptureComponent-\u003eGetCaptureComponent2D()-\u003eCaptureSource =  ESceneCaptureSource::SCS_FinalColorLDR;\n    CaptureComponent-\u003eGetCaptureComponent2D()-\u003eTextureTarget-\u003eTargetGamma = GEngine-\u003eGetDisplayGamma();\n    CaptureComponent-\u003eGetCaptureComponent2D()-\u003eShowFlags.SetTemporalAA(true);\n    // lookup additional showflags in documentation\n}\n```\n\u003e Call the code during `BeginPlay` of the `FrameCaptureManager`\n\n**FrameCaptureManager.cpp**\n``` cpp\n// Called when the game starts or when spawned\nvoid AFrameCaptureManager::BeginPlay()\n{\n\tSuper::BeginPlay();\n\n    // Setup CaptureComponent\n    if(CaptureComponent){ // nullptr check\n\t\tSetupCaptureComponent();\n\t} else{\n\t\tUE_LOG(LogTemp, Error, TEXT(\"No CaptureComponent set!\"));\n\t}\n}\n\n```\nNow that because we have a `RenderTarget` applied to our `CaptureComponent` we can read its data and store it to disk.\n\n\n#\n## Organize RenderRequests\nWe do this by basically re-implementing UE's code for taking screenshots. Importantly, with the addition of not flushing our rendering pipeline. This prevents rendering *hiccups* that drop the framerate to 3 - 5 FPS. \n\nThis addition will come at the price of needing to handle 'waiting times' before an image is done and copied from GPU. This is important to prevent reading old or uninitialized buffers (remember that `RenderThread` and `GameThread` are asynchronous). We do this by keeping a queue of ```RenderRequest``` that we can probe for being completed. \n\n\u003e We add the following ```struct``` to our `FrameCaptureManager.h` above the `UCLASS()` definition:\n\n**FrameCaptureManager.h**\n``` cpp\n#include ...\n\n[...]\n\nstruct FRenderRequestStruct{\n    FIntPoint ImageSize;\n    FRHIGPUTextureReadback Readback;\n    FRenderCommandFence RenderFence;\n\n\tvoid* RawData = nullptr;\n\tint64 RawSize = 0;\n\n\tbool bIsComplete = false;\n\n    FRenderRequestStruct(\n        const FIntPoint\u0026 ImageSize,\n        const FRHIGPUTextureReadback\u0026 Readback) :\n            ImageSize(ImageSize),\n            Readback(Readback) {}\n};\n\n[...]\nUCLASS()\nclass ...\n[...]\n```\nThe ```FRHIGPUTextureReadback``` will hold the rendered results, e.g. color or depth values. The ```RenderFence``` is a neat feature of UE, letting you put a 'fence' into the render pipeline that can be checked to notify when it has passed the full rendering-pipeline. This gives a way to determine whether our render request is done and the buffers are safe to read.\n\n\u003e We need to add a ```TQueue``` as a data structure to keep track of our render requests:\n\n**CaptureManger.h**\n``` cpp\nprotected:\n    // RenderRequest Queue\n    TQueue\u003cTSharedPtr\u003cFRenderRequest\u003e\u003e RenderRequestQueue;\n    TQueue\u003cTSharedPtr\u003cFRenderRequestStruct\u003e\u003e InThreadRenderRequestQueue;\n```\n\n#\n## Implement placing render requests: \nThis function will place a render request on the UE rendering pipeline asking the data captured from our `CaptureComponent` to be copied in our image buffer so that we can further process it.\n\n**CaptureManger.h**\n``` cpp\npublic:\n    UFUNCTION(BlueprintCallable, Category = \"ImageCapture\")\n    void CaptureNonBlocking();\n```\n\n**CaptureManger.cpp**\n``` cpp\nvoid AFrameCaptureManager::CaptureNonBlocking(){\n    if(!IsValid(CaptureComponent)){\n        UE_LOG(LogTemp, Error, TEXT(\"CaptureColorNonBlocking: CaptureComponent was not valid!\"));\n        return;\n    }\n    CaptureComponent-\u003eGetCaptureComponent2D()-\u003eTextureTarget-\u003eTargetGamma = GEngine-\u003eGetDisplayGamma();\n\n    // Get RenderConterxt\n    FTextureRenderTargetResource* renderTargetResource = CaptureComponent-\u003eGetCaptureComponent2D()-\u003eTextureTarget-\u003eGameThread_GetRenderTargetResource();\n    \n    TSharedPtr\u003cFRenderRequestStruct\u003e renderRequest = \n        MakeShared\u003cFRenderRequestStruct\u003e(\n            renderTargetResource-\u003eGetSizeXY(), \n            FRHIGPUTextureReadback(TEXT(\"CameraCaptureManagerReadback\")\n        )\n    );\n\n    ENQUEUE_RENDER_COMMAND(SceneDrawCompletion)(\n    [renderRequest, renderTargetResource](FRHICommandListImmediate\u0026 RHICmdList) {\n        FTextureRHIRef Target = renderTargetResource-\u003eGetRenderTargetTexture();\n        renderRequest-\u003eReadback.EnqueueCopy(RHICmdList, Target);\n    });\n\n    // Notifiy new task in RenderQueue\n    RenderRequestQueue.Enqueue(renderRequest);\n\n    // Set RenderCommandFence\n    renderRequest-\u003eRenderFence.BeginFence();\n}\n```\nWith this, the image data will be stored in our queue of requests. Now we can think of storing it to disk. \n\n*Note: UFUNCTION(BlueprintCallable, Category = \"ImageCapture\") exposes this function to blueprint, so that you can easily test it*\n\n\n# \n## Save Image Data to Disk\nIn each tick of the `FrameCaptureManager` we look up the first element of the `RenderQueue`. If it's `RenderFence` is completed and the data is ready to read, we proceed with saving the image to disk.\n\nWe need a procedure to write the data to disk, preferably without blocking our `GameThread`. \nWe implement an [asynchronous](https://wiki.unrealengine.com/Using_AsyncTasks) procedure storing the data to disk.\n\n**FrameCaptureManager.h**\n``` cpp\nUCLASS()\nclass ... {\n[...]\n};\n\n// Below the AFrameCaptureManager class definition\n\nclass AsyncSaveImageToDiskTask : public FNonAbandonableTask\n{\npublic:\n    AsyncSaveImageToDiskTask(\n\t\tTSharedPtr\u003cFRenderRequestStruct\u003e InRenderRequest, \n\t\tFString ImageName, \n\t\tint32 Width, \n\t\tint32 Height, \n\t\tERGBFormat RGBFormat,\n\t\tEImageFormat ImageFormat);\n    ~AsyncSaveImageToDiskTask();\n\n    void DoWork();\n\n    FORCEINLINE TStatId GetStatId() const\n    {\n        RETURN_QUICK_DECLARE_CYCLE_STAT(AsyncSaveImageToDiskTask, STATGROUP_ThreadPoolAsyncTasks);\n    }\n\nprivate:\n    TSharedPtr\u003cFRenderRequestStruct\u003e RenderRequest;  // Hold the shared pointer\n    FString FileName;\n    int32 Width;\n    int32 Height;\n    ERGBFormat RGBFormat;\n\tEImageFormat ImageFormat;\n};\n```\n\n**FrameCaptureManager.cpp**\n``` cpp\n#include ...\n\nAsyncSaveImageToDiskTask::AsyncSaveImageToDiskTask(\n        TSharedPtr\u003cFRenderRequestStruct\u003e InRenderRequest, \n        FString ImageName, \n        int32 Width, \n        int32 Height, \n        ERGBFormat RGBFormat,\n        EImageFormat ImageFormat\n    ):\n    RenderRequest(InRenderRequest),\n    FileName(ImageName),\n    Width(Width),\n    Height(Height),\n    RGBFormat(RGBFormat),\n    ImageFormat(ImageFormat)\n{\n}\n\nAsyncSaveImageToDiskTask::~AsyncSaveImageToDiskTask(){\n    UE_LOG(LogTemp, Warning, TEXT(\"AsyncTaskDone\"));\n}\n\n\nvoid AsyncSaveImageToDiskTask::DoWork(){\n    UE_LOG(LogTemp, Warning, TEXT(\"Starting Work\"));\n\n    // Load the image wrapper module (if not already loaded)\n    IImageWrapperModule\u0026 ImageWrapperModule = FModuleManager::LoadModuleChecked\u003cIImageWrapperModule\u003e(FName(\"ImageWrapper\"));\n\n    // Create an image wrapper\n    TSharedPtr\u003cIImageWrapper\u003e ImageWrapper = ImageWrapperModule.CreateImageWrapper(ImageFormat);\n\n    // Error handling\n    if (!ImageWrapper.IsValid()) {\n        UE_LOG(LogTemp, Error, TEXT(\"Failed to create IImageWrapper!\"));\n        return;\n    }\n\n    // Set the raw data \n    int32 PixelDepth = 8;\n    ImageWrapper-\u003eSetRaw(RenderRequest-\u003eRawData, RenderRequest-\u003eRawSize, Width, Height, RGBFormat, PixelDepth);\n\n    // Compress the image \n    const TArray64\u003cuint8\u003e\u0026 CompressedData = ImageWrapper-\u003eGetCompressed(100);\n\n    // Save the compressed image to disk\n    FFileHelper::SaveArrayToFile(CompressedData, *FileName);\n\n    //unlock the readback after the processing is done\n    RenderRequest-\u003eReadback.Unlock();\n\n    // Indicate that the processing is complete (using a flag)\n    RenderRequest-\u003ebIsComplete = true;\n}\n```\nWe offload the entire image processing into `DoWork()`, from applying compression encoding to finally storing it to disk, to the asynchronous thread. \n\n*Note that this requires our ``RenderRequest`, more precisely its `RawData` image-buffer to stay available while the data is being stored to disk. Otherwise we encounter segmentation faults and the engine will crash!*\n\n\n#\n## Override the `Tick` function of the `FrameCaptureManager`:\nFinally, we put everything together in the `Tick` function of our `FrameCaptureManager`. We wait for a `RenderRequest` to become available, we hand it over to an asynchronous thread for being stored to disk, and we monitor the progress of the asynchronous thread to finally release the `RenderRequest`'s buffers to garbage collection.\n\n\n**FrameCaptureManager.h**\n``` cpp\npublic:\t\n\n\t// Called every frame\n\tvirtual void Tick(float DeltaTime) override;\n```\n**FrameCaptureManager.cpp**\n``` cpp\n// Called every frame\nvoid AFrameCaptureManager::Tick(float DeltaTime)\n{\n\tSuper::Tick(DeltaTime);\n\n    if(!RenderRequestQueue.IsEmpty()){\n        // Peek the next RenderRequest from queue\n        TSharedPtr\u003cFRenderRequestStruct\u003e nextRenderRequest = *RenderRequestQueue.Peek();\n\n        if(nextRenderRequest){ //nullptr check\n            if(nextRenderRequest-\u003eRenderFence.IsFenceComplete() \u0026\u0026 nextRenderRequest-\u003eReadback.IsReady()){\n                // Load the image wrapper module \n                IImageWrapperModule\u0026 ImageWrapperModule = FModuleManager::LoadModuleChecked\u003cIImageWrapperModule\u003e(FName(\"ImageWrapper\"));\n\n                // Get Data from Readback\n                nextRenderRequest-\u003eRawSize = nextRenderRequest-\u003eImageSize.X * nextRenderRequest-\u003eImageSize.Y * sizeof(FColor);\n                \n                int32 RowPitchInPixels;\n                nextRenderRequest-\u003eRawData = nextRenderRequest-\u003eReadback.Lock(RowPitchInPixels, nullptr); // Pass RowPitchInPixels and no buffer size\n                \n                // Generate image name\n                FString fileName = \"\";\n                fileName = FPaths::ProjectSavedDir() + SubDirectoryName + \"/img\" + \"_\" + ToStringWithLeadingZeros(ImgCounter, NumDigits);\n                fileName += GetFileEnding(ImageFormat);\n\n                // Pass the raw data, filename, width, height, and format to the async task\n                RunAsyncImageSaveTask(nextRenderRequest, fileName, 1920, 1080, ERGBFormat::RGBA, EImageFormat::PNG);\n\n                // Security check                    \n                if(VerboseLogging \u0026\u0026 !fileName.IsEmpty()){\n                    UE_LOG(LogTemp, Warning, TEXT(\"%s\"), *fileName);\n                }\n                \n                // Increase ImgCounter for file names\n                ImgCounter += 1;\n\n                // Release RenderRequest form the queue\n                RenderRequestQueue.Pop(); // Delete the first element from RenderQueue\n                // Put it into the queue of outsourced threads\n                InThreadRenderRequestQueue.Enqueue(nextRenderRequest); //Push to InThreadRenderRequestQueue\n            }\n        }\n    }\n    if(!InThreadRenderRequestQueue.IsEmpty()){\n        UE_LOG(LogTemp, Log, TEXT(\"InThreadRenderRequestQueue not empty.\"));\n\n        // Get next element of the InThreadRenderRequestQueue\n        TSharedPtr\u003cFRenderRequestStruct\u003e nextRenderRequest = *InThreadRenderRequestQueue.Peek();\n\n        if(nextRenderRequest){ //nullptr check\n            if(nextRenderRequest-\u003ebIsComplete){ //check if complete\n                InThreadRenderRequestQueue.Pop(); // Remove from queue\n            }\n        }\n    }\n}\n```\n\nFor testing purposes we can call the `CaptureColorBlocking()` from the `LevelBlueprint` by attaching it to a button pressed event.\n\n![](https://github.com/TimmHess/UnrealImageCapture/blob/master/gfx/Debug_level_blueprint_capture.png)\n\nThe captured images will now be saved into your project's `Saved` directory.\n\n\n#\n# Capturing Annotations\nTo be able to render color and segmentation at the same time, we need one additional `SceneCapture2D` component in our scene for each *type* of capture you want. Attach it to the `SceneCapture2D` already placed in your scene (from earlier in this tutorial). Make sure both have exactly the same location, rotation, and perspective settings. Otherwise your annotation will not match.\n\n# Capturing Object Segmentation Masks\nTo get labels for our images we will add a second `CaptureComponent` equipped with a `PostProcessMaterial` that visualizes `CustomDepth`. The `CustomDepthStencil` is settable for each actor in the scene, effectively letting us label and visualize categories of, as well as individual, actors.\n\n## 1. Enable and Set-Up Custom Depth Stencils\nFind the **ProjectSettings** in your editor and search for *stencil* which will bring up `Custom Depth-Stencil Pass`. Switch this option from `Enabled` to `Enabled with Stencil`. \n\nYou can set the custom depth in editor or from code. For simplicity I chose the editor. Place an arbitrary object(MeshActor) into the scene, and search for `custom depth` in its details panel. Under `Rendering` enable `Render CustomDepth Pass`, and set `CustomDepth Stencil Value` to whatever you like. For illustration purposes set it to 200.\n\n*Note: Make sure you have custom depth enabled with stencils in your project settings.*\n![](https://github.com/TimmHess/UnrealImageCapture/blob/master/gfx/Enable_custom_depth_in_project.png)\n\n![](https://github.com/TimmHess/UnrealImageCapture/blob/master/gfx/Apply_custom_depth_stencil.png)\n\n\n## 2. Setting Up The PostProcess Material\nAdd a new `Material` to your project content. (I will call it `PP_Segmentation`)\n\nClick on the material's output node and switch `MaterialDomain` from `Surface` to `PostProcess`.\n\nIn the same panel search for \"alpha\" and activate `Output Alpha`. Set this value to `1.0` in the Material node.\n\nRight-click to open the node search and type `SceneTexture`, select the node from `Texture`-Category.\n\nIn the details of this node, select `CustomStencil` as `SceneTextureId`.\n\nAdd a `Division` node and connect the `SceneTexture`'s `Color` output to the division node. Set the division to be by 255. \n\n*Note: This is needed because the image buffer seems to be float valued, leading to values \u003e 1 having no meaning, as image information ranges from 0.0 to 1.0.*\n\nApply and save the material.\n\n![](https://github.com/TimmHess/UnrealImageCapture/blob/master/gfx/pp_segmentation.png)\n\n\n**FrameCaptureManager.h**\n``` cpp\npublic:\n    // PostProcessMaterial used for segmentation\n    UPROPERTY(EditAnywhere, Category=\"Segmentation Setup\")\n    UMaterial* PostProcessMaterial = nullptr;\n```\n**FrameCaptureManager.cpp**\n``` cpp\nvoid AFrameCaptureManager::SetupCaptureComponent(){\n    [...] // previous function code\n\n    // Assign PostProcess Material if assigned\n    if(PostProcessMaterial){ // check nullptr\n        CaptureComponent-\u003eGetCaptureComponent2D()-\u003eAddOrUpdateBlendable(PostProcessMaterial);\n    } else {\n        UE_LOG(LogTemp, Log, TEXT(\"No PostProcessMaterial is assigend\"));\n    }\n}\n```\n\nYou can now reference the ``PostProcessMaterial` in the details panel of the `FrameCaptureManager` in the editor just like before the `SceneCapture2D`.\n\n\n#\n# Setup a Depth Capture\nCapturing `SceneDepth` information has one important caveat - it requires storing float images (.exr).\nEvery thing else follows like [Object Segmentation](#capturing-object-segmentation-masks). We create a `PostProcessMaterial` to access the respectie GPU buffer. \n\n![](https://github.com/TimmHess/UnrealImageCapture/blob/master/gfx/pp_depth.png)\n\nLuckily UnrealEngine is perfectly capable of storing float images and even provides the .exr file format in its `IImageWrapper`. However, there are three places in our code where we need to take care to handle the float format correctly. Missing any of those will result in segmentation faults of the engine.\n\n## 1. Accounting for the RawSize correctly\nWe adjust the code that allocates the RawSize of our captured images to accomodate the float values using 16 bits. \n\n*Note that I decide for the RawSize to used based on the ImageFormat. This I do not explain in the tutorial - please check the source.*\n\n**FrameCaptureManager.cpp**\n``` cpp\nvoid AFrameCaptureManager::SetupCaptureComponent(){\n    [...] \n\n    // Get Data from Readback\n    nextRenderRequest-\u003eRawSize = nextRenderRequest-\u003eImageSize.X * nextRenderRequest-\u003eImageSize.Y * sizeof(FColor);\n    if(ImageFormat == ECustomImageFormat::EXR){ // handle float case\n        nextRenderRequest-\u003eRawSize = nextRenderRequest-\u003eImageSize.X * nextRenderRequest-\u003eImageSize.Y * sizeof(FFloat16Color); //FLOAT\n    }\n\n    [...] \n}\n```\n\n## 2. Adjust the correct RenderTarget format\n**FrameCaptureManager.cpp**\n``` cpp\nvoid AFrameCaptureManager::SetupCaptureComponent(){\n    [...] \n\n    // Create RenderTargets\n    UTextureRenderTarget2D* renderTarget2D = NewObject\u003cUTextureRenderTarget2D\u003e();\n    renderTarget2D-\u003eInitAutoFormat(256, 256); // some random format, got crashing otherwise\n\n    // Float Capture\n    if(ImageFormat == ECustomImageFormat::EXR){ // handle float case\n        renderTarget2D-\u003eRenderTargetFormat = ETextureRenderTargetFormat::RTF_RGBA32f;\n        renderTarget2D-\u003eInitCustomFormat(FrameWidth, FrameHeight, PF_FloatRGBA, true); // PF_B8G8R8A8 disables HDR which will boost storing to disk due to less image information\n    }\n    // Color Capture\n    else{\n        renderTarget2D-\u003eRenderTargetFormat = ETextureRenderTargetFormat::RTF_RGBA8_SRGB; //ETextureRenderTargetFormat::RTF_RGBA8; //8-bit color format\n        renderTarget2D-\u003eInitCustomFormat(FrameWidth, FrameHeight, PF_R8G8B8A8, true); // PF_R8G8B8A8 //PF_B8G8R8A8 // PF... disables HDR, which is most important since HDR gives gigantic overhead, and is not needed!\n        renderTarget2D-\u003ebForceLinearGamma = true; // Important for viewport-like color reproduction.\n    }\n    renderTarget2D-\u003ebGPUSharedFlag = true; // demand buffer on GPU\n\n    [...] \n}\n```\n\n## 3. Adjust the PixelDepth when storing\n**FrameCaptureManager.cpp**\n``` cpp\nvoid AsyncSaveImageToDiskTask::DoWork(){\n    [...] \n\n    // Set the raw data \n    int32 PixelDepth = 8;\n    if(ImageFormat == EImageFormat::EXR){ // Adjust pixel depth for EXR (float) data  // Has to be EImageFormat because already converted..\n        PixelDepth = 16;\n    }\n    UE_LOG(LogTemp, Warning, TEXT(\"PixelDepth: %d\"), PixelDepth);\n    ImageWrapper-\u003eSetRaw(RenderRequest-\u003eRawData, RenderRequest-\u003eRawSize, Width, Height, RGBFormat, PixelDepth);\n\n    [...] \n}\n```\n\n\n#\n# Enable Lumen on SceneCapture2D\nIn my tests using UE5.5.3+ the `SceneCapture2D` was fully capable of rendering scenes with Lumen. However you need might need to actiate it in the `SceneCapture2D` itself as it was not listening to the `PostProcessVolume` in my scene.\n\n![](https://github.com/TimmHess/UnrealImageCapture/blob/master/gfx/Activate_Lumen.png)\n\n\n","funding_links":[],"categories":["Game Engine"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTimmHess%2FUnrealImageCapture","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTimmHess%2FUnrealImageCapture","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTimmHess%2FUnrealImageCapture/lists"}