https://github.com/shinyquagsire23/videotoolbox-uncompressed-textures-bug
Sample code to trigger and not trigger uncompressed render targets and overheat your GPU, present from visionOS 2.0/iOS 18 and onwards
https://github.com/shinyquagsire23/videotoolbox-uncompressed-textures-bug
Last synced: 4 months ago
JSON representation
Sample code to trigger and not trigger uncompressed render targets and overheat your GPU, present from visionOS 2.0/iOS 18 and onwards
- Host: GitHub
- URL: https://github.com/shinyquagsire23/videotoolbox-uncompressed-textures-bug
- Owner: shinyquagsire23
- Created: 2024-12-24T19:08:12.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2025-02-06T18:32:21.000Z (4 months ago)
- Last Synced: 2025-02-06T19:40:10.276Z (4 months ago)
- Language: Swift
- Size: 52.7 MB
- Stars: 5
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# VideoToolbox Uncompressed Textures Bug
VideoToolbox/CoreVideo currently has a bug, originating on visionOS 2.0/iOS 18, which causes intermediate and final render targets on color conversions to be uncompressed, resulting in a 10x-ing of GPU bandwidth which can overheat chips on streams as small as 4K, especially with multiple streams at high FPS.
# CLARIFICATION
I've realized this is partly PEBKAC and partly legitimate. The behavior is indeed inconsistent across different hardware, but any format not starting with `kCVPixelFormatType_Lossless` or `kCVPixelFormatType_Lossy` will incur a performance penalty on newer hardware. Why, I have no clue.
## Variant A
10-bit (HDR) HEVC requires the use of private `MTLPixelFormat`s in order to decode without hitting this bug, because any format conversion will trigger the uncompressed render targets. In the sample, this format conversion is to `kCVPixelFormatType_420YpCbCr10BiPlanarFullRange`/`kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange`. There are compressed `kCVPixelFormatType`s that can only be decoded by private `MTLPixelFormat`s used in Webkit.
## Variant B
**Any** identity format conversion (format conversion to the same format that the video stream is already in) will also result in a redudant conversion to an uncompressed render target, 10x-ing the GPU bandwidth for no appearent reason. In the sample this is done with `kCVPixelFormatType_420YpCbCr10PackedBiPlanarFullRange`/`kCVPixelFormatType_420YpCbCr10PackedBiPlanarVideoRange`
## Hardware variance
This bug does NOT trigger on the following hardware (not a conclusive list)
- visionOS simulator on M1 Max on macOS versions: 15.2
This bug continues to trigger on the following hardware
- M2 Apple Vision Pro on visionOS versions: 2.0, 2.1, 2.2