https://github.com/vade/ffmpegmediaextension
Use macOS Sequia's new Media Extensions to allow playback of FFMPEG supported media via AVFoundation
https://github.com/vade/ffmpegmediaextension
Last synced: over 1 year ago
JSON representation
Use macOS Sequia's new Media Extensions to allow playback of FFMPEG supported media via AVFoundation
- Host: GitHub
- URL: https://github.com/vade/ffmpegmediaextension
- Owner: vade
- Created: 2024-07-25T22:02:33.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-08-09T17:24:26.000Z (almost 2 years ago)
- Last Synced: 2024-10-18T20:31:02.700Z (over 1 year ago)
- Language: Objective-C
- Size: 1.32 MB
- Stars: 32
- Watchers: 3
- Forks: 1
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# FFMPEGMediaExtension
This is an attempt to build a macOS 15.0 MediaExtension for libAV* which in theory would allow for native AVFoundation container parsing and decode of FFMPEG enabled Container formats (.mkv, .wmv, .avi etc) and Codecs (ie VP9, VP8, DIVX, XVID etc).
You can think of Media Extensions as a modern macOS replacement for old macOS 7, 8, 9 and early macOS X Quicktime Extensions.
You can think of this project as an attempt to make a one size fits all Media Extension like the old Perian software.
Since Media Extensions are shipped in a host app, this has also provides a Host app.
We have basic frame decoding working!
But there is a ton left to do!
- [ ] Asset / Track Metadata
- [ ] GOP Support
- [ ] Additional Codecs
## Goal
Provide additional media format playback support for any macOS app which leverages AVFoundation / Core Media as its underlying playback / decode engine
* Finder
* Quicklook
* Quicktime Player
* Any other app which opts in by calling:
* `MTRegisterProfessionalVideoWorkflowFormatReaders()` for format parsing support
* `VTRegisterProfessionalVideoWorkflowVideoDecoders()` for additional decoder support
## Requirements:
* MacOS 15 Beta 4 +
* XCode 16 Beta 4 +
## Building:
If you want to build FFMPEG libAVFormat and libAVCodec:
`brew install automake nasm pkg-config`
FFMPEG is installed as a submodule - so ensure you git checkout the submodule and init
`cd ffmpeg`
```
./configure --prefix=/usr/local --enable-static --disable-shared --disable-programs --disable-doc --disable-avdevice --disable-avfilter --disable-postproc --disable-swscale --disable-swresample --arch=arm64 --disable-avfoundation
```
`make -j$(sysctl -n hw.ncpu)`
This will build you the statically linked libAVFormat.a and libAVCodec.a libries
## Design
MEFormatReader (LibAVFOrmatReader)
* [x] Plugin instantiation via `LibAVFormatReaderFactory`
* [x] Register a limited number of supported container formats via plist / exported content types
* [x] MKV
* Adding new Containers requires editing multiple annoying Plist entries
* [x] Basic Container parsing via libavformat (`LibAVFormatReader`)
* [ ] Container metadata to AVMetadataItems
* [x] Track Parsing (Audio and Video only right now) (`LibAVTrackReader`)
* [ ] Track specific metadata to AVMEtadataItems
* [x] CMFormatDescriptions for our Tracks
* [x] Seek / Stepping via `LibAVSampleCursor`
* [ ] Provide additional sample metadata that some formats require
* [ ] Sample Sync info
* [ ] Sample Dependency info
* [ ] Post Decode Processing Data
MEVideoDecoder
* Not yet started
* This means today we only add container support, not yet additional codecs
* Stay tuned
## Todo:
* [x] libavformat / libavcodec, libavutil compiled as a .a
* [x] ffmpeg headers inlcuded
* [x] Swift Modulemap + Obj-C libav*
* [ ] Implement necessary Media Extension protocols
* [x] MEFormatReaderExtension
* [x] MEFormatReader (partial)
* [x] METrackReader
* [x] METrackInfo
* [x] MESampleCursor (partial)
* [ ] MEVideoDecoderExtension
* [ ] MEVideoDecoder
* [ ] MEVideoDecoderPixelBufferManager
* [x] impelment Extension requirements (plists, entitlements, etc)
* [ ] Memory Management pass in ObjC / C
## Strategy
We Eventually require 2 Bundle targets that are Media Extensions.
* A Media Extention that implements `MEFormatReader`, which will allow container parsing via `libavformat` and provide container metadata, track metadata, and compressed sample metadata.
* A Media Extension that impelments `MEVideoDecoder` - a Video Decompressor which will allow for codec decompression via `libavcodec` and provide pixel buffers from appropriate samples.
FFMPEGs library more or less follows similar design patterns to AVFoundation. The following isnt totally accurate, but more or less gets the point across:
* `CMTime` <~> `AVRational`
* `CMMediaType` <~> `AVMediaType`
* `CMFormatDescription` <~> `AVCodecParameters`
* `CMSampleBuffer` <~> `AVPacket`
* `CMBlockBuffer` <~> `AVBuffer`
* `CVPixelBuffer` <-> `AVFrame` (decoded video)
* `AVAsset` <~> `AVFormatContext`
* `AVAssetTrack` <~> `AVStream`
* `AVSampleCursor` <~> `AVIOContext`
Leverage AVIOContext to allow a MEByteSource to driven by a AVFormatContext to get us container parsing and metadata
## Running Notes:
* Weirdly MediaExtension headers imply macOS 14 supported?
* entitlements for Media Extensions