Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/finnvoor/transcoding
Simple video transcoding in Swift
https://github.com/finnvoor/transcoding
h264 h265 hevc transcoding video
Last synced: 25 days ago
JSON representation
Simple video transcoding in Swift
- Host: GitHub
- URL: https://github.com/finnvoor/transcoding
- Owner: finnvoor
- License: cc0-1.0
- Created: 2024-02-08T19:40:52.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-03-03T10:40:34.000Z (10 months ago)
- Last Synced: 2024-11-24T19:55:49.061Z (28 days ago)
- Topics: h264, h265, hevc, transcoding, video
- Language: Swift
- Homepage:
- Size: 47.9 KB
- Stars: 55
- Watchers: 3
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Transcoding
[](https://github.com/finnvoor/Transcoding/actions/workflows/CI.yml) [](https://swiftpackageindex.com/finnvoor/Transcoding) [](https://swiftpackageindex.com/finnvoor/Transcoding)
A simple Swift package for video encoding and decoding with Annex-B adaptors optimized for transferring video over a network.
`Transcoding` is used for video encoding and decoding in [Castaway](https://finnvoorhees.com/castaway), an app that streams HDMI capture devices from an iPad or Mac to a nearby Vision Pro.
## Usage
### VideoEncoder
`VideoEncoder` is an object that takes `CMSampleBuffer`s containing `CVPixelBuffer`s and outputs a stream of `CMSampleBuffer`s containing `CMBlockBuffer`s containing compressed H264/HEVC data. `VideoEncoder` is initialized with a `Config`, with presets for live capture, active transcoding, background transcoding, and ultra-low latency following Apple's recommendations.
#### Usage
```swift
let videoEncoder = VideoEncoder(config: .ultraLowLatency)
encoderStreamTask = Task {
for await encodedSampleBuffer in videoEncoder.encodedSampleBuffers {
// encodedSampleBuffer: CMSampleBuffer > CMBlockBuffer
}
}
videoEncoder.encode(sampleBuffer)
```
### VideoDecoder
`VideoDecoder` is an object that takes `CMSampleBuffer`s containing `CMBlockBuffers`s containing compressed H264/HEVC data and outputs a stream of `CMSampleBuffer`s containing `CVPixelBuffer`s. `VideoDecoder` is initialized with a `Config` containing various optional decompression settings.
#### Usage
```swift
let videoDecoder = VideoDecoder(config: .init(realTime: true))
decoderStreamTask = Task {
for await decodedSampleBuffer in videoDecoder.decodedSampleBuffers {
// decodedSampleBuffer: CMSampleBuffer > CVPixelBuffer
}
}
videoDecoder.decode(sampleBuffer)
```
### Annex B
`VideoEncoderAnnexBAdaptor` and `VideoDecoderAnnexBAdaptor` can be used to convert compressed `CMSampleBuffer`s to and from an Annex B (ITU-T-REC-H.265) byte stream. This is ideal for sending compressed video data over a network.
### Example Pipeline
In this example, video frames from a capture device are encoded and decoded as an Annex-B data stream optimized for low latency.
```swift
let videoEncoder = VideoEncoder(config: .ultraLowLatency)
let videoEncoderAnnexBAdaptor = VideoEncoderAnnexBAdaptor(
videoEncoder: videoEncoder
)
let videoDecoder = VideoDecoder(config: .init(realTime: true))
let videoDecoderAnnexBAdaptor = VideoDecoderAnnexBAdaptor(
videoDecoder: videoDecoder,
codec: .hevc
)
videoEncoderTask = Task {
for await data in videoEncoderAnnexBAdaptor.annexBData {
// send data over network or whatever
}
}
videoDecoderTask = Task {
for await decodedSampleBuffer in videoDecoder.decodedSampleBuffers {
// here you have a received decoded sample buffer with image buffer
}
}
receivedMessageTask = Task {
// Replace `realtimeStreaming.receivedMessages` with however you receive encoded data packets
for await (data, _) in realtimeStreaming.receivedMessages {
videoDecoderAnnexBAdaptor.decode(data)
}
}
captureSessionTask = Task {
// Replace `captureSession.pixelBuffers` with your video data source
for await pixelBuffer in captureSession.pixelBuffers {
videoEncoder.encode(pixelBuffer)
}
}
```
### Important Notes
- Currently `VideoDecoderAnnexBAdaptor` only supports decoding full NALU's, meaning if you are passing data over the network or some stream you must ensure you receive full video frame packets, not just an arbitrarily sized data stream. When using Network.framework, for example, you would use a custom `NWProtocolFramerImplementation` to receive individual messages.
- There are a number of instances where either the encoder or decoder needs to be reset during an application lifecycle. The encoder and decoder automatically handle resetting after an iOS app has been backgrounded, but you may need to handle other cases by calling `encoder/decoder.invalidate()`. For example, if you maintain one encoder and disconnect then reconnect from a peer/decoder, the encoder will need to be invalidated to ensure it sends over the H264/HEVC parameter sets again. Otherwise the decoder will not be able to decode frames, as the encoder is optimized to only send SPS/PPS/VPS when necessary.