{"id":15693833,"url":"https://github.com/finger563/unreal-camera-capture","last_synced_at":"2026-03-09T23:02:07.144Z","repository":{"id":174770939,"uuid":"651523611","full_name":"finger563/unreal-camera-capture","owner":"finger563","description":"Plugin for capturing camera (RGB + depth + motion vectors) data from multiple cameras in UE 5","archived":false,"fork":false,"pushed_at":"2026-02-28T21:08:57.000Z","size":105,"stargazers_count":9,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-01T00:23:09.708Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/finger563.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-06-09T12:32:20.000Z","updated_at":"2026-02-28T21:08:59.000Z","dependencies_parsed_at":"2023-06-30T00:00:53.289Z","dependency_job_id":null,"html_url":"https://github.com/finger563/unreal-camera-capture","commit_stats":null,"previous_names":["finger563/unreal-camera-capture"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/finger563/unreal-camera-capture","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finger563%2Funreal-camera-capture","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finger563%2Funreal-camera-capture/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finger563%2Funreal-camera-capture/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finger563%2Funreal-camera-capture/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/finger563","download_url":"https://codeload.github.com/finger563/unreal-camera-capture/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finger563%2Funreal-camera-capture/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30315965,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T20:05:46.299Z","status":"ssl_error","status_checked_at":"2026-03-09T19:57:04.425Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T18:49:26.252Z","updated_at":"2026-03-09T23:02:07.136Z","avatar_url":"https://github.com/finger563.png","language":"C++","readme":"# Camera Capture Plugin for UE 5\n\nPlugin for capturing camera (RGB + depth + motion vectors) data from multiple\ncameras in UE 5\n\nProvides:\n- An actor component `CaptureComponent` which can be added to any actor. It will\n  automatically find all `UIntrinsicSceneCaptureComponent2D` components that are\n  attached to the parent actor. For each camera it finds, it will configure that\n  camera to output depth and motion vectors (by applying the `M_DmvCapture` post\n  process material to it) and will configure it to output to a render texture.\n  It will then create a copy of each camera which will render the color data to\n  a texture as well. These textures can be captured (for other uses in the\n  engine), and optionally serialized / saved to disk as EXR files. The component\n  can be configured to capture every frame of the camera or to capture frames at\n  a specific interval of time.\n- `ACameraCaptureManager` - A manager actor that provides centralized control\n  over multiple cameras using the `CameraCaptureSubsystem`. Automatically\n  discovers and registers `UIntrinsicSceneCaptureComponent2D` cameras in the\n  level, and provides Blueprint/C++ API for controlling capture.\n- `UCameraCaptureSubsystem` - Game instance subsystem that handles the actual\n  capture logic, file writing, and camera registration. Both `CaptureComponent`\n  and `CameraCaptureManager` can be used - they produce identical output formats.\n- `UIntrinsicSceneCaptureComponent2D` - A subclass of `USceneCaptureComponent2D`\n  that supports custom camera intrinsics for precise camera calibration. Each\n  camera can have different intrinsics defined either inline or via reusable\n  data assets. Also supports frustum visualization in the editor. Use this for\n  capturing/rendering from non-player cameras.\n- `UIntrinsicCameraComponent` - A subclass of `UCameraComponent` that supports\n  custom camera intrinsics for player cameras. Allows you to set precise focal\n  lengths and principal points for first-person/third-person cameras, matching\n  real-world camera specifications. Includes frustum visualization.\n- `FCameraIntrinsics` - Data asset for storing camera intrinsic parameters\n  (focal length, principal point, image dimensions) that can be shared across\n  multiple cameras (e.g., \"RealSense D435\" preset).\n- A `M_DmvCapture` material for saving depth + motion vectors to a single\n  texture (used by both capture systems)\n\n\u003c!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc --\u003e\n**Table of Contents**\n\n- [Camera Capture Plugin for UE 5](#camera-capture-plugin-for-ue-5)\n  - [Usage](#usage)\n  - [Camera Intrinsics](#camera-intrinsics)\n  - [Data Collected](#data-collected)\n  - [Developing](#developing)\n    - [Code style](#code-style)\n  - [References](#references)\n\n\u003c!-- markdown-toc end --\u003e\n\n\n## Usage\n\n### Option 1: Using CaptureComponent (Per-Actor)\n\n1. **Add cameras to your actor**: Attach `UIntrinsicSceneCaptureComponent2D` components\n   to your actor (not the base `USceneCaptureComponent2D`).\n\n2. **Configure camera intrinsics** (optional):\n   - **Option A - Create reusable preset**: Right-click in Content Browser → \n     Miscellaneous → Data Asset → Select `CameraIntrinsicsAsset`. Configure\n     the intrinsics and save (e.g., \"DA_RealSense_D435\").\n   - **Option B - Inline configuration**: Set intrinsics directly on each camera\n     component.\n   - On the camera component, enable `bUseCustomIntrinsics`, choose between\n     `bUseIntrinsicsAsset` or inline parameters.\n\n3. **Add CaptureComponent**: Add the `CaptureComponent` to your actor. It will\n   automatically find and configure all `UIntrinsicSceneCaptureComponent2D` cameras.\n\n4. **Configure capture settings**: Set `TimerPeriod`, `SaveLocation`, and other\n   options on the `CaptureComponent`.\n\n### Option 2: Using CameraCaptureManager (Global)\n\n1. **Add cameras to the level**: Place actors with `UIntrinsicSceneCaptureComponent2D`\n   components in your level.\n\n2. **Add CameraCaptureManager**: Place a `CameraCaptureManager` actor in your level.\n\n3. **Configure manager settings**:\n   - Set `OutputDirectory` for where to save captures\n   - Set `CaptureEveryNFrames` for capture rate\n   - Enable/disable RGB, depth, and motion vector capture\n   - Enable `bAutoStartOnBeginPlay` to start capturing automatically\n\n4. **Control via Blueprint/C++**: Use `StartCapture()`, `StopCapture()`, and other\n   functions to control when capturing happens.\n\n**Note**: Each camera now defines its own image resolution through its intrinsics\nsettings. The previous global `ImageWidth`/`ImageHeight` parameters have been\nremoved.\n\n### Option 3: Using IntrinsicCameraComponent (Player Cameras)\n\nUse `UIntrinsicCameraComponent` instead of the base `UCameraComponent` for player cameras (first-person, third-person, etc.) when you need precise camera calibration.\n\n1. **Replace UCameraComponent**: In your Pawn or Character Blueprint/C++, replace\n   the standard `CameraComponent` with `IntrinsicCameraComponent`.\n\n2. **Configure intrinsics** (optional):\n   - Enable `bUseCustomIntrinsics`\n   - Either reference a `CameraIntrinsicsAsset` or configure inline parameters\n   - Set focal lengths, principal point, and image dimensions\n\n3. **Use as normal camera**: The component works exactly like `UCameraComponent` but\n   with the added ability to use custom projection matrices for precise calibration.\n\n**Use Cases**:\n- Matching a real-world camera's field of view precisely\n- Simulating specific camera hardware (e.g., GoPro, webcam)\n- Research applications requiring exact camera parameters\n- VR/AR applications with specific lens characteristics\n\n## Camera Intrinsics\n\nThe plugin supports two modes for camera projection (available on both `UIntrinsicSceneCaptureComponent2D` and `UIntrinsicCameraComponent`):\n\n1. **Custom Intrinsics Mode** (`bUseCustomIntrinsics = true`):\n   - Builds a custom projection matrix from pixel-based camera parameters\n   - Parameters: `FocalLengthX`, `FocalLengthY`, `PrincipalPointX`, `PrincipalPointY`\n   - Use this to precisely match real-world camera sensors\n\n2. **Maintain Y-Axis Mode** (`bMaintainYAxis = true`):\n   - Adjusts horizontal FOV to maintain vertical FOV as aspect ratio changes\n   - Matches gameplay camera behavior\n   - Use this for consistent framing across different resolutions\n\nSee the\n[unreal-camera-capture-example](https://github.com/finger563/unreal-camera-capture-example)\nrepo for a simple example project using this plugin.\n\nSee also the\n[unreal-python-tools](https://github.com/finger563/unreal-python-tools) repo for\nsome example code which provides processing and display of the data produced by\nthe component in this repo.\n\n## Data Output Format\n\nBoth `CaptureComponent` and `CameraCaptureManager` produce identical output in EXR format with JSON metadata.\n\n### Directory Structure\n\nBoth systems organize output by **actor**, then by **camera**:\n\n```\nSaveLocation/\n├── ActorName1/                        # e.g., \"Robot_BP_C_0\"\n│   ├── camera_config.csv              # Legacy config file (CaptureComponent only)\n│   ├── transformations.csv            # Legacy transform CSV (CaptureComponent only)\n│   ├── CameraName1/                   # e.g., \"HeadCamera\"\n│   │   ├── frame_0000000.exr          # RGB + Depth (in alpha channel)\n│   │   ├── frame_0000000_motion.exr   # Motion vectors (X in R, Y in G)\n│   │   ├── frame_0000000.json         # Metadata\n│   │   ├── frame_0000001.exr\n│   │   ├── frame_0000001_motion.exr\n│   │   └── frame_0000001.json\n│   └── CameraName2/                   # e.g., \"ChestCamera\"\n│       └── ...\n└── ActorName2/                        # e.g., \"Robot_BP_C_1\"\n    └── ...\n```\n\n**Note**: The actor-based folder structure makes it easy to organize data from multiple robots or actors in the same capture session.\n\n### EXR Files\n\n- **`frame_NNNNNNN.exr`**: \n  - **RGB channels**: Linear color data (FLinearColor)\n  - **Alpha channel**: Depth data (in cm, raw from depth buffer)\n  - Format: EXR with 32-bit float precision\n  \n- **`frame_NNNNNNN_motion.exr`**:\n  - **R channel**: Motion vector X (horizontal pixel motion)\n  - **G channel**: Motion vector Y (vertical pixel motion)\n  - **B/A channels**: Unused (0.0)\n  - Format: EXR with 32-bit float precision\n\n### JSON Metadata\n\nEach frame has an accompanying JSON file (`frame_NNNNNNN.json`) with complete camera and transform information:\n\n```json\n{\n  \"frame_number\": 0,\n  \"timestamp\": 1.234,\n  \"camera_id\": \"RGBCamera\",\n  \"world_transform\": {\n    \"location\": [x, y, z],           // In meters (converted from UE cm)\n    \"rotation\": [pitch, yaw, roll],  // In degrees\n    \"quaternion\": [w, x, y, z],      // Normalized quaternion\n    \"scale\": [x, y, z]               // Component scale\n  },\n  \"intrinsics\": {\n    \"focal_length_x\": 320.0,         // Pixels\n    \"focal_length_y\": 320.0,         // Pixels\n    \"principal_point_x\": 320.0,      // Pixels\n    \"principal_point_y\": 240.0,      // Pixels\n    \"image_width\": 640,              // Pixels\n    \"image_height\": 480,             // Pixels\n    \"maintain_y_axis\": false         // Y-axis FOV mode\n  },\n  \"actor_path\": \"/Game/...\",         // Full UE object path\n  \"level_name\": \"MyLevel\"            // Level name\n}\n```\n\n### Legacy CSV Files (CaptureComponent only)\n\nFor backward compatibility, `CaptureComponent` also generates:\n\n- **`camera_config.csv`**: Static camera configuration\n  - Columns: `name`, `width`, `height`, `focalLength`, `fov`, `nearClipPlane`, `farClipPlane`, \n    `tx`, `ty`, `tz`, `qw`, `qx`, `qy`, `qz` (relative transform)\n\n- **`transformations.csv`**: Per-frame actor transforms\n  - Columns: `i` (frame index), `time`, `tx`, `ty`, `tz`, `qw`, `qx`, `qy`, `qz` (world position in meters + quaternion)\n\n**Note**: The JSON metadata format is recommended for new projects as it includes more complete information.\n\n## Reading EXR Files\n\nEXR files can be read using standard libraries:\n\n**Python (OpenEXR):**\n```python\nimport OpenEXR\nimport Imath\nimport numpy as np\n\n# Read RGB + Depth\nexr_file = OpenEXR.InputFile(\"frame_0000000.exr\")\nheader = exr_file.header()\ndw = header['dataWindow']\nwidth = dw.max.x - dw.min.x + 1\nheight = dw.max.y - dw.min.y + 1\n\n# Read channels\npt = Imath.PixelType(Imath.PixelType.FLOAT)\nrgb_str = [exr_file.channel(c, pt) for c in ['R', 'G', 'B']]\ndepth_str = exr_file.channel('A', pt)\n\n# Convert to numpy arrays\nrgb = np.array([np.frombuffer(c, dtype=np.float32).reshape(height, width) for c in rgb_str])\nrgb = np.transpose(rgb, (1, 2, 0))  # HWC format\ndepth = np.frombuffer(depth_str, dtype=np.float32).reshape(height, width)\n```\n\n**Python (imageio):**\n```python\nimport imageio\nimport numpy as np\n\n# Simpler API using imageio\nimg = imageio.imread(\"frame_0000000.exr\")\nrgb = img[:, :, :3]\ndepth = img[:, :, 3]\n```\n\n## Visualization\n\nAn example visualization of captured data:\n\n![example](https://github.com/finger563/unreal-camera-capture/assets/213467/dc91ba5a-21bc-47c5-92cd-5b09414c6420)\n\n## Developing\n\nIf you want to contribute or customize the plugin, you can clone the repository\nand set it up in your Unreal project. If you want to make changes to the code,\nplease follow the code style guidelines below.\n\n### Code style\n\n1. Ensure `clang-format` is installed\n2. Ensure [pre-commit](https://pre-commit.com) is installed\n3. Set up `pre-commit` for this repository:\n\n  ``` console\n  pre-commit install\n  ```\n\nThis helps ensure that consistent code formatting is applied.\n\n## References\n\n- [post process velocity lookup in\n  MaterialTemplate.ush](https://github.com/EpicGames/UnrealEngine/blob/407acc04a93f09ecb42c07c98b74fd00cc967100/Engine/Shaders/Private/MaterialTemplate.ush#L2370)\n- [Velocity vector encoding in shaders](https://github.com/EpicGames/UnrealEngine/blob/407acc04a93f09ecb42c07c98b74fd00cc967100/Engine/Shaders/Private/Common.ush#L1728)\n- [Unreal visualization of velocity vectors](https://github.com/EpicGames/UnrealEngine/blob/407acc04a93f09ecb42c07c98b74fd00cc967100/Engine/Shaders/Private/MotionBlur/VisualizeMotionVectors.usf#L63)\n- [PR 6933 for fixing camera motion velocity](https://github.com/EpicGames/UnrealEngine/pull/6933)\n- [Coordinate space terminology in UE](https://docs.unrealengine.com/5.2/en-US/coordinate-space-terminology-in-unreal-engine/)\n- [UE 5 FViewUniformShaderParameters](https://docs.unrealengine.com/5.0/en-US/API/Runtime/Engine/FViewUniformShaderParameters/)\n- [SceneView.cpp setting the View Uniform Shader Parameters](https://github.com/EpicGames/UnrealEngine/blob/407acc04a93f09ecb42c07c98b74fd00cc967100/Engine/Source/Runtime/Engine/Private/SceneView.cpp#L2670)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffinger563%2Funreal-camera-capture","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffinger563%2Funreal-camera-capture","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffinger563%2Funreal-camera-capture/lists"}