{"id":1523,"url":"https://github.com/MetalPetal/MetalPetal","last_synced_at":"2025-08-02T04:31:34.227Z","repository":{"id":37470567,"uuid":"95336438","full_name":"MetalPetal/MetalPetal","owner":"MetalPetal","description":"A GPU accelerated image and video processing framework built on Metal.","archived":false,"fork":false,"pushed_at":"2024-04-10T13:30:17.000Z","size":17231,"stargazers_count":2022,"open_issues_count":37,"forks_count":254,"subscribers_count":51,"default_branch":"master","last_synced_at":"2025-05-29T03:12:27.296Z","etag":null,"topics":["apple-silicon","filter","gpgpu","gpu","image","image-processing","ios","maccatalyst","macos","metal","multimedia","opengl","real-time","rendering","tvos","video","video-processing"],"latest_commit_sha":null,"homepage":"https://github.com/MetalPetal/MetalPetal","language":"Objective-C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MetalPetal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-25T03:34:26.000Z","updated_at":"2025-05-28T17:02:19.000Z","dependencies_parsed_at":"2022-07-12T16:18:10.470Z","dependency_job_id":"a3b9a16e-132c-4004-8c9a-a7136965b2e3","html_url":"https://github.com/MetalPetal/MetalPetal","commit_stats":{"total_commits":985,"total_committers":19,"mean_commits":51.8421052631579,"dds":"0.15126903553299498","last_synced_commit":"f9b78897bd4214bb097f352a1bde0a4f4a1e2ddb"},"previous_names":[],"tags_count":116,"template":false,"template_full_name":null,"purl":"pkg:github/MetalPetal/MetalPetal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MetalPetal%2FMetalPetal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MetalPetal%2FMetalPetal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MetalPetal%2FMetalPetal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MetalPetal%2FMetalPetal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MetalPetal","download_url":"https://codeload.github.com/MetalPetal/MetalPetal/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MetalPetal%2FMetalPetal/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268334612,"owners_count":24233793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apple-silicon","filter","gpgpu","gpu","image","image-processing","ios","maccatalyst","macos","metal","multimedia","opengl","real-time","rendering","tvos","video","video-processing"],"created_at":"2024-01-05T20:15:48.466Z","updated_at":"2025-08-02T04:31:33.680Z","avatar_url":"https://github.com/MetalPetal.png","language":"Objective-C","funding_links":[],"categories":["Media","HarmonyOS","Objective-C","Metal Tools, Libraries, and Frameworks","OOM-Leaks-Crash","图像处理的框架"],"sub_categories":["Image","Windows Manager","Other free courses","VS Code Extensions for Developer Productivity","Graphs Processing And Rendering"],"readme":"# MetalPetal\n\n[![Swift](https://github.com/MetalPetal/MetalPetal/workflows/Swift/badge.svg)](https://github.com/MetalPetal/MetalPetal/actions?query=workflow%3ASwift)\n\u003cbr/\u003e\n[![Platforms](https://img.shields.io/badge/Platforms-iOS%2011%2B%20%7C%20tvOS%2013%2B%20%7C%20macOS%2010.13%2B-blue.svg)](#)\n[![Version](https://img.shields.io/github/v/release/MetalPetal/MetalPetal?label=Release)](https://github.com/MetalPetal/MetalPetal/releases)\n\u003cbr/\u003e\n[![Apple Silicon](https://img.shields.io/badge/Apple%20Silicon-%E2%80%8B%20%E2%9C%94-eee)](#)\n[![Mac Catalyst](https://img.shields.io/badge/Mac%20Catalyst-%E2%80%8B%20%E2%9C%94-eee)](#)\n[![Simulator](https://img.shields.io/badge/Simulator-%E2%80%8B%20%E2%9C%94-eee)](#)\n\u003cbr/\u003e\n[![CocoaPods](https://img.shields.io/static/v1?label=CocoaPods\u0026message=%E2%80%8B%20%E2%9C%94\u0026color=eee\u0026logo=CocoaPods\u0026logoColor=white)](#cocoapods)\n[![Swift PM](https://img.shields.io/static/v1?label=Swift%20PM\u0026message=%E2%80%8B%20%E2%9C%94\u0026color=eee\u0026logo=Swift\u0026logoColor=white)](#swift-package-manager)\n\nAn image processing framework based on Metal.\n\n\u003c!-- TOC depthFrom:2 --\u003e\n\n- [Design Overview](#design-overview)\n    - [Goals](#goals)\n    - [Core Components](#core-components)\n        - [MTIContext](#mticontext)\n        - [MTIImage](#mtiimage)\n        - [MTIFilter](#mtifilter)\n        - [MTIKernel](#mtikernel)\n    - [Optimizations](#optimizations)\n    - [Concurrency Considerations](#concurrency-considerations)\n    - [Advantages over Core Image](#advantages-over-core-image)\n- [Builtin Filters](#builtin-filters)\n- [Example Code](#example-code)\n    - [Create a `MTIImage`](#create-a-mtiimage)\n    - [Apply a Filter](#apply-a-filter)\n    - [Render a `MTIImage`](#render-a-mtiimage)\n    - [Display a `MTIImage`](#display-a-mtiimage)\n    - [Connect Filters (Swift)](#connect-filters-swift)\n    - [Process Video Files](#process-video-files)\n    - [Process Live Video (with VideoIO)](#process-live-video-with-videoio)\n- [Best Practices](#best-practices)\n- [Build Custom Filter](#build-custom-filter)\n    - [Shader Function Arguments Encoding](#shader-function-arguments-encoding)\n    - [Simple Single Input / Output Filters](#simple-single-input--output-filters)\n    - [Fully Custom Filters](#fully-custom-filters)\n    - [Multiple Draw Calls in One Render Pass](#multiple-draw-calls-in-one-render-pass)\n    - [Custom Vertex Data](#custom-vertex-data)\n    - [Custom Processing Module](#custom-processing-module)\n- [Alpha Types](#alpha-types)\n    - [Alpha Handling of Built-in Filters](#alpha-handling-of-built-in-filters)\n- [Color Spaces](#color-spaces)\n    - [Color Spaces for Inputs](#color-spaces-for-inputs)\n    - [Color Spaces for Outputs](#color-spaces-for-outputs)\n    - [Color Spaces for `CVPixelBuffer`](#color-spaces-for-cvpixelbuffer)\n    - [Color Space Conversions](#color-space-conversions)\n- [Extensions](#extensions)\n    - [Working with SceneKit](#working-with-scenekit)\n    - [Working with SpriteKit](#working-with-spritekit)\n    - [Working with Core Image](#working-with-core-image)\n    - [Working with JavaScript](#working-with-javascript)\n    - [Texture Loader](#texture-loader)\n- [Install](#install)\n    - [CocoaPods](#cocoapods)\n        - [Sub-pod `Swift`](#sub-pod-swift)\n        - [Sub-pod `AppleSilicon`](#sub-pod-applesilicon)\n    - [Swift Package Manager](#swift-package-manager)\n- [iOS Simulator Support](#ios-simulator-support)\n- [Quick Look Debug Support](#quick-look-debug-support)\n- [Trivia](#trivia)\n- [Contribute](#contribute)\n- [License](#license)\n\n\u003c!-- /TOC --\u003e\n\n## Design Overview\n\nMetalPetal is an image processing framework based on [Metal](https://developer.apple.com/metal/) designed to provide real-time processing for still image and video with easy to use programming interfaces.\n\nThis chapter covers the key concepts of MetalPetal, and will help you to get a better understanding of its design, implementation, performance implications and best practices.\n\n### Goals\n\nMetalPetal is designed with the following goals in mind.\n\n- Easy to use API\n\n    Provides convenience APIs and avoids common pitfalls.\n\n- Performance\n\n    Use CPU, GPU and memory efficiently.\n\n- Extensibility\n\n    Easy to create custom filters as well as plugin your custom image processing unit.\n\n- Swifty\n\n    Provides a fluid experience for Swift programmers.\n\n### Core Components\n\nSome of the core concepts of MetalPetal are very similar to those in Apple's Core Image framework.\n\n#### MTIContext\n\nProvides an evaluation context for rendering `MTIImage`s. It also stores a lot of caches and state information, so it's more efficient to reuse a context whenever possible.\n\n#### MTIImage\n\n A `MTIImage` object is a representation of an image to be processed or produced. It does directly represent image bitmap data instead it has all the information necessary to produce an image or more precisely a `MTLTexture`. It consists of two parts, a recipe of how to produce the texture (`MTIImagePromise`) and other information such as how a context caches the image (`cachePolicy`), and how the texture should be sampled (`samplerDescriptor`).\n\n#### MTIFilter\n\nA `MTIFilter` represents an image processing effect and any parameters that control that effect. It produces a `MTIImage` object as output. To use a filter, you create a filter object, set its input images and parameters, and then access its output image. Typically, a filter class owns a static kernel (`MTIKernel`), when you access its `outputImage` property, it asks the kernel with the input images and parameters to produce an output `MTIImage`. \n\n#### MTIKernel\n\nA `MTIKernel` represents an image processing routine. `MTIKernel` is responsible for creating the corresponding render or compute pipeline state for the filter, as well as building the `MTIImagePromise` for a `MTIImage`.\n\n### Optimizations\n\nMetalPetal does a lot of optimizations for you under the hood.\n\nIt automatically caches functions, kernel states, sampler states, etc.\n\nIt utilizes Metal features like programmable blending, memoryless render targets, resource heaps and metal performance shaders to make the render fast and efficient. On macOS, MetalPetal can also take advantage of the TBDR architecture of Apple silicon.\n\nBefore rendering, MetalPetal can look into your image render graph and figure out the minimal number of intermediate textures needed to do the rendering, saving memory, energy and time.\n\nIt can also re-organize the image render graph if multiple “recipes” can be concatenated to eliminate redundant render passes. (`MTIContext.isRenderGraphOptimizationEnabled`)\n\n### Concurrency Considerations\n\n`MTIImage` objects are immutable, which means they can be shared safely among threads.\n\nHowever, `MTIFilter` objects are mutable and thus cannot be shared safely among threads.\n\nA `MTIContext` contains a lot of states and caches. There's a thread-safe mechanism for `MTIContext` objects, making it safe to share a `MTIContext` object among threads.\n\n### Advantages over Core Image\n\n- Fully customizable vertex and fragment functions.\n\n- MRT (Multiple Render Targets) support.\n\n- Generally better performance. (Detailed benchmark data needed)\n\n## Builtin Filters\n\n- Color Matrix\n\n- Color Lookup\n\n    Uses an color lookup table to remap the colors in an image.\n\n- Opacity\n\n- Exposure\n\n- Saturation\n\n- Brightness\n\n- Contrast\n\n- Color Invert\n\n- Vibrance\n\n    Adjusts the saturation of an image while keeping pleasing skin tones.\n\n- RGB Tone Curve\n\n- Blend Modes\n\n    - Normal\n    - Multiply\n    - Overlay\n    - Screen\n    - Hard Light\n    - Soft Light\n    - Darken\n    - Lighten\n    - Color Dodge\n    - Add (Linear Dodge)\n    - Color Burn\n    - Linear Burn\n    - Lighter Color\n    - Darker Color\n    - Vivid Light\n    - Linear Light\n    - Pin Light\n    - Hard Mix\n    - Difference\n    - Exclusion\n    - Subtract\n    - Divide\n    - Hue\n    - Saturation\n    - Color\n    - Luminosity\n    - ColorLookup512x512\n    - [Custom Blend Mode](https://github.com/MetalPetal/MetalPetal/issues/70#issuecomment-792430483)\n\n- Blend with Mask\n\n- Transform\n\n- Crop\n\n- Pixellate\n\n- Multilayer Composite\n\n- MPS Convolution\n\n- MPS Gaussian Blur\n\n- MPS Definition\n\n- MPS Sobel\n\n- MPS Unsharp Mask\n\n- MPS Box Blur\n\n- [High Pass Skin Smoothing](https://github.com/YuAo/YUCIHighPassSkinSmoothing)\n\n- [CLAHE (Contrast-Limited Adaptive Histogram Equalization)](https://github.com/YuAo/Accelerated-CLAHE)\n\n- [Lens Blur (Hexagonal Bokeh Blur)](https://github.com/YuAo/HexagonalBokehBlur)\n\n- [Surface Blur](https://github.com/MetalPetal/SurfaceBlur)\n\n- Bulge Distortion\n\n- Chroma Key Blend\n\n- Color Halftone\n\n- Dot Screen\n\n- Round Corner (Circular/Continuous Curve)\n\n- [All Core Image Filters](#working-with-core-image)\n\n## Example Code\n\n### Create a `MTIImage`\n\nYou can create a `MTIImage` object from nearly any source of image data, including:\n\n- `URL`s referencing image files to be loaded\n- Metal textures\n- CoreVideo image or pixel buffers (`CVImageBufferRef` or `CVPixelBufferRef`)\n- Image bitmap data in memory\n- Texture data from a given texture or image asset name\n- Core Image `CIImage` objects\n- `MDLTexture` objects\n- SceneKit and SpriteKit scenes\n\n```Swift\nlet imageFromCGImage = MTIImage(cgImage: cgImage, isOpaque: true)\n\nlet imageFromCIImage = MTIImage(ciImage: ciImage)\n\nlet imageFromCoreVideoPixelBuffer = MTIImage(cvPixelBuffer: pixelBuffer, alphaType: .alphaIsOne)\n\nlet imageFromContentsOfURL = MTIImage(contentsOf: url)\n\n// unpremultiply alpha if needed\nlet unpremultipliedAlphaImage = image.unpremultiplyingAlpha()\n```\n\n### Apply a Filter\n\n```Swift\nlet inputImage = ...\n\nlet filter = MTISaturationFilter()\nfilter.saturation = 0\nfilter.inputImage = inputImage\n\nlet outputImage = filter.outputImage\n```\n\n### Render a `MTIImage`\n\n```Swift\nlet options = MTIContextOptions()\n\nguard let device = MTLCreateSystemDefaultDevice(), let context = try? MTIContext(device: device, options: options) else {\n    return\n}\n\nlet image: MTIImage = ...\n\ndo {\n    try context.render(image, to: pixelBuffer) \n    \n    //context.makeCIImage(from: image)\n    \n    //context.makeCGImage(from: image)\n} catch {\n    print(error)\n}\n```\n\n### Display a `MTIImage`\n\n```Swift\nlet imageView = MTIImageView(frame: self.view.bounds)\n\n// You can optionally assign a `MTIContext` to the image view. If no context is assigned and `automaticallyCreatesContext` is set to `true` (the default value), a `MTIContext` is created automatically when the image view renders its content.\nimageView.context = ...\n\nimageView.image = image\n```\n\nIf you'd like to move the GPU command encoding process out of the main thread, you can use a `MTIThreadSafeImageView`. You may assign a `MTIImage` to a `MTIThreadSafeImageView` in any thread.\n\n### Connect Filters (Swift)\n\nMetalPetal has a type-safe Swift API for connecting filters. You can use `=\u003e` operator in `FilterGraph.makeImage` function to connect filters and get the output image.\n\nHere are some examples:\n\n```Swift\nlet image = try? FilterGraph.makeImage { output in\n    inputImage =\u003e saturationFilter =\u003e exposureFilter =\u003e output\n}\n```\n\n```Swift\nlet image = try? FilterGraph.makeImage { output in\n    inputImage =\u003e saturationFilter =\u003e exposureFilter =\u003e contrastFilter =\u003e blendFilter.inputPorts.inputImage\n    exposureFilter =\u003e blendFilter.inputPorts.inputBackgroundImage\n    blendFilter =\u003e output\n}\n```\n\n- You can connect unary filters (`MTIUnaryFilter`) directly using `=\u003e`.\n\n- For a filter with multiple inputs, you need to connect to one of its `inputPorts`.\n\n- `=\u003e` operator only works in `FilterGraph.makeImage` method.\n\n- One and only one filter's output can be connected to `output`.\n\n### Process Video Files\n\nWorking with `AVPlayer`:\n\n```Swift\nlet context = try MTIContext(device: device)\nlet asset = AVAsset(url: videoURL)\nlet composition = MTIVideoComposition(asset: asset, context: context, queue: DispatchQueue.main, filter: { request in\n    return FilterGraph.makeImage { output in\n        request.anySourceImage! =\u003e filterA =\u003e filterB =\u003e output\n    }!\n}\n\nlet playerItem = AVPlayerItem(asset: asset)\nplayerItem.videoComposition = composition.makeAVVideoComposition()\nplayer.replaceCurrentItem(with: playerItem)\nplayer.play()\n```\n\nExport a video:\n\n_[VideoIO](https://github.com/MetalPetal/VideoIO) is required for the following examples._\n\n```Swift\nimport VideoIO\n\nvar configuration = AssetExportSession.Configuration(fileType: .mp4, videoSettings: .h264(videoSize: composition.renderSize), audioSettings: .aac(channels: 2, sampleRate: 44100, bitRate: 128 * 1000))\nconfiguration.videoComposition = composition.makeAVVideoComposition()\nself.exporter = try! AssetExportSession(asset: asset, outputURL: outputURL, configuration: configuration)\nexporter.export(progress: { progress in\n    \n}, completion: { error in\n    \n})\n```\n\n### Process Live Video (with VideoIO)\n\n_[VideoIO](https://github.com/MetalPetal/VideoIO) is required for this example._ \n\n```Swift\nimport VideoIO\n\n// Setup Image View\nlet imageView = MTIImageView(frame: self.view.bounds)\n...\n\n// Setup Camera\nlet camera = Camera(captureSessionPreset: .hd1920x1080, configurator: .portraitFrontMirroredVideoOutput)\ntry camera.enableVideoDataOutput(on: DispatchQueue.main, delegate: self)\ncamera.videoDataOutput?.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange]\n\n...\n\n// AVCaptureVideoDataOutputSampleBufferDelegate\n\nlet filter = MTIColorInvertFilter()\n\nfunc captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {\n    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {\n        return\n    }\n    let inputImage = MTIImage(cvPixelBuffer: pixelBuffer, alphaType: .alphaIsOne)\n    filter.inputImage = inputImage\n    self.imageView.image = filter.outputImage\n}\n\n```\n\nPlease refer to the `CameraFilterView.swift` in the example project for more about previewing and recording filtered live video.\n\n## Best Practices\n\n- Reuse a `MTIContext` whenever possible.\n\n    Contexts are heavyweight objects, so if you do create one, do so as early as possible, and reuse it each time you need to render an image.\n\n- Use `MTIImage.cachePolicy` wisely.\n    \n    Use `MTIImageCachePolicyTransient` when you do not want to preserve the render result of an image, for example when the image is just an intermediate result in a filter chain, so the underlying texture of the render result can be reused. It is the most memory efficient option. However, when you ask the context to render a previously rendered image, it may re-render that image since its underlying texture has been reused.\n    \n    By default, a filter's output image has the `transient` policy.\n\n    Use `MTIImageCachePolicyPersistent` when you want to prevent the underlying texture from being reused.\n    \n    By default, images created from external sources have the `persistent` policy.\n\n- Understand that `MTIFilter.outputImage` is a compute property.\n\n    Each time you ask a filter for its output image, the filter may give you a new output image object even if the inputs are identical with the previous call. So reuse output images whenever possible.\n    \n    For example,\n\n     ```Swift\n    //          ╭→ filterB\n    // filterA ─┤\n    //          ╰→ filterC\n    // \n    // filterB and filterC use filterA's output as their input.\n    ```\n    In this situation, the following solution:\n    \n    ```Swift\n    let filterOutputImage = filterA.outputImage\n    filterB.inputImage = filterOutputImage\n    filterC.inputImage = filterOutputImage\n    ```\n    \n    is better than:\n\n    ```Swift\n    filterB.inputImage = filterA.outputImage\n    filterC.inputImage = filterA.outputImage\n    ```\n\n## Build Custom Filter\n\nIf you want to include the `MTIShaderLib.h` in your `.metal` file, you need to add the path of `MTIShaderLib.h` file to the `Metal Compiler - Header Search Paths` (`MTL_HEADER_SEARCH_PATHS`) setting.\n\nFor example, if you use CocoaPods you can set the `MTL_HEADER_SEARCH_PATHS` to  `${PODS_CONFIGURATION_BUILD_DIR}/MetalPetal/MetalPetal.framework/Headers` or `${PODS_ROOT}/MetalPetal/Frameworks/MetalPetal/Shaders`. If you use Swift Package Manager, set the `MTL_HEADER_SEARCH_PATHS` to `$(HEADER_SEARCH_PATHS)`\n\n### Shader Function Arguments Encoding\n\nMetalPetal has a built-in mechanism to encode shader function arguments for you. You can pass the shader function arguments as `name: value` dictionaries to the `MTIRenderPipelineKernel.apply(toInputImages:parameters:outputDescriptors:)`, `MTIRenderCommand(kernel:geometry:images:parameters:)`, etc.\n\nFor example, the parameter dictionary for the metal function `vibranceAdjust` can be:\n\n```Swift\n// Swift\nlet amount: Float = 1.0\nlet vibranceVector = float4(1, 1, 1, 1)\nlet parameters = [\"amount\": amount,\n                  \"vibranceVector\": MTIVector(value: vibranceVector),\n                  \"avoidsSaturatingSkinTones\": true,\n                  \"grayColorTransform\": MTIVector(value: float3(0,0,0))]\n```\n\n```Metal\n// vibranceAdjust metal function\nfragment float4 vibranceAdjust(...,\n                constant float \u0026 amount [[ buffer(0) ]],\n                constant float4 \u0026 vibranceVector [[ buffer(1) ]],\n                constant bool \u0026 avoidsSaturatingSkinTones [[ buffer(2) ]],\n                constant float3 \u0026 grayColorTransform [[ buffer(3) ]])\n{\n    ...\n}\n\n```\n\nThe shader function argument types and the corresponding types to use in a parameter dictionary is listed below.\n\n| Shader Function Argument Type | Swift | Objective-C | \n| :--- | :--- | :--- |\n| float | Float | float |\n| int | Int32 | int |\n| uint | UInt32 | uint |\n| bool | Bool | bool |\n| simd (float2,float4,float4x4,int4, etc.) | simd (with `MetalPetal/Swift`) / MTIVector | MTIVector |\n| struct | Data / MTIDataBuffer | NSData / MTIDataBuffer |\n| other (float *, struct *, etc.) immutable | Data / MTIDataBuffer | NSData / MTIDataBuffer |\n| other (float *, struct *, etc.) mutable | MTIDataBuffer | MTIDataBuffer |\n\n### Simple Single Input / Output Filters\n\nTo build a custom unary filter, you can subclass `MTIUnaryImageRenderingFilter` and override the methods in the `SubclassingHooks` category. Examples: `MTIPixellateFilter`, `MTIVibranceFilter`, `MTIUnpremultiplyAlphaFilter`, `MTIPremultiplyAlphaFilter`, etc.\n\n```ObjectiveC\n//Objective-C\n\n@interface MTIPixellateFilter : MTIUnaryImageRenderingFilter\n\n@property (nonatomic) float fractionalWidthOfAPixel;\n\n@end\n\n@implementation MTIPixellateFilter\n\n- (instancetype)init {\n    if (self = [super init]) {\n        _fractionalWidthOfAPixel = 0.05;\n    }\n    return self;\n}\n\n+ (MTIFunctionDescriptor *)fragmentFunctionDescriptor {\n    return [[MTIFunctionDescriptor alloc] initWithName:@\"pixellateEffect\" libraryURL:[bundle URLForResource:@\"default\" withExtension:@\"metallib\"]];\n}\n\n- (NSDictionary\u003cNSString *,id\u003e *)parameters {\n    return @{@\"fractionalWidthOfAPixel\": @(self.fractionalWidthOfAPixel)};\n}\n\n@end\n```\n\n```Swift\n//Swift\n\nclass MTIPixellateFilter: MTIUnaryImageRenderingFilter {\n    \n    var fractionalWidthOfAPixel: Float = 0.05\n\n    override var parameters: [String : Any] {\n        return [\"fractionalWidthOfAPixel\": fractionalWidthOfAPixel]\n    }\n    \n    override class func fragmentFunctionDescriptor() -\u003e MTIFunctionDescriptor {\n        return MTIFunctionDescriptor(name: \"pixellateEffect\", libraryURL: MTIDefaultLibraryURLForBundle(Bundle.main))\n    }\n}\n```\n\n### Fully Custom Filters\n\nTo build more complex filters, all you need to do is create a kernel (`MTIRenderPipelineKernel`/`MTIComputePipelineKernel`/`MTIMPSKernel`), then apply the kernel to the input image(s). Examples: `MTIChromaKeyBlendFilter`, `MTIBlendWithMaskFilter`, `MTIColorLookupFilter`, etc.\n\n```ObjectiveC\n\n@interface MTIChromaKeyBlendFilter : NSObject \u003cMTIFilter\u003e\n\n@property (nonatomic, strong, nullable) MTIImage *inputImage;\n\n@property (nonatomic, strong, nullable) MTIImage *inputBackgroundImage;\n\n@property (nonatomic) float thresholdSensitivity;\n\n@property (nonatomic) float smoothing;\n\n@property (nonatomic) MTIColor color;\n\n@end\n\n@implementation MTIChromaKeyBlendFilter\n\n@synthesize outputPixelFormat = _outputPixelFormat;\n\n+ (MTIRenderPipelineKernel *)kernel {\n    static MTIRenderPipelineKernel *kernel;\n    static dispatch_once_t onceToken;\n    dispatch_once(\u0026onceToken, ^{\n        kernel = [[MTIRenderPipelineKernel alloc] initWithVertexFunctionDescriptor:[[MTIFunctionDescriptor alloc] initWithName:MTIFilterPassthroughVertexFunctionName] fragmentFunctionDescriptor:[[MTIFunctionDescriptor alloc] initWithName:@\"chromaKeyBlend\"]];\n    });\n    return kernel;\n}\n\n- (instancetype)init {\n    if (self = [super init]) {\n        _thresholdSensitivity = 0.4;\n        _smoothing = 0.1;\n        _color = MTIColorMake(0.0, 1.0, 0.0, 1.0);\n    }\n    return self;\n}\n\n- (MTIImage *)outputImage {\n    if (!self.inputImage || !self.inputBackgroundImage) {\n        return nil;\n    }\n    return [self.class.kernel applyToInputImages:@[self.inputImage, self.inputBackgroundImage]\n                                      parameters:@{@\"color\": [MTIVector vectorWithFloat4:(simd_float4){self.color.red, self.color.green, self.color.blue,self.color.alpha}],\n                                    @\"thresholdSensitivity\": @(self.thresholdSensitivity),\n                                               @\"smoothing\": @(self.smoothing)}\n                         outputTextureDimensions:MTITextureDimensionsMake2DFromCGSize(self.inputImage.size)\n                               outputPixelFormat:self.outputPixelFormat];\n}\n\n@end\n```\n\n### Multiple Draw Calls in One Render Pass\n\nYou can use `MTIRenderCommand` to issue multiple draw calls in one render pass.\n\n```Swift\n// Create a draw call with kernelA, geometryA, and imageA.\nlet renderCommandA = MTIRenderCommand(kernel: self.kernelA, geometry: self.geometryA, images: [imageA], parameters: [:])\n\n// Create a draw call with kernelB, geometryB, and imageB.\nlet renderCommandB = MTIRenderCommand(kernel: self.kernelB, geometry: self.geometryB, images: [imageB], parameters: [:])\n\n// Create an output descriptor\nlet outputDescriptor = MTIRenderPassOutputDescriptor(dimensions: MTITextureDimensions(width: outputWidth, height: outputHeight, depth: 1), pixelFormat: .bgra8Unorm, loadAction: .clear, storeAction: .store)\n\n// Get the output images, the output image count is equal to the output descriptor count.\nlet images = MTIRenderCommand.images(byPerforming: [renderCommandA, renderCommandB], outputDescriptors: [outputDescriptor])\n```\n\nYou can also create multiple output descriptors to output multiple images in one render pass (MRT, See https://en.wikipedia.org/wiki/Multiple_Render_Targets).\n\n### Custom Vertex Data\n\nWhen `MTIVertex` cannot fit your needs, you can implement the `MTIGeometry` protocol to provide your custom vertex data to the command encoder.\n\nUse the `MTIRenderCommand` API to issue draw calls and pass your custom `MTIGeometry`.\n\n### Custom Processing Module\n\nIn rare scenarios, you may want to access the underlying texture directly, use multiple MPS kernels in one render pass, do 3D rendering, or encode the render commands yourself.\n\n`MTIImagePromise` protocol provides direct access to the underlying texture and the render context for a step in MetalPetal.\n\nYou can create new input sources or fully custom processing units by implementing the `MTIImagePromise` protocol. You will need to import an additional module to do so. \n\nObjective-C\n\n```\n@import MetalPetal.Extension;\n```\n\nSwift\n\n```\n// CocoaPods\nimport MetalPetal.Extension\n\n// Swift Package Manager\nimport MetalPetalObjectiveC.Extension\n```\n\nSee the implementation of `MTIComputePipelineKernel`, `MTICLAHELUTRecipe` or `MTIImage` for example.\n\n## Alpha Types\n\nIf an alpha channel is used in an image, there are two common representations that are available: unpremultiplied (straight/unassociated) alpha, and premultiplied (associated) alpha.\n\nWith unpremultiplied alpha, the RGB components represent the color of the pixel, disregarding its opacity.\n\nWith premultiplied alpha, the RGB components represent the color of the pixel, adjusted for its opacity by multiplication.\n\nMetalPetal handles alpha type explicitly. You are responsible for providing the correct alpha type during image creation.\n\nThere are three alpha types in MetalPetal.\n\n`MTIAlphaType.nonPremultiplied`: the alpha value in the image is not premultiplied.\n\n`MTIAlphaType.premultiplied`: the alpha value in the image is premultiplied.\n\n`MTIAlphaType.alphaIsOne`: there's no alpha channel in the image or the image is opaque.\n\nTypically, `CGImage`, `CVPixelBuffer` and `CIImage` objects have premultiplied alpha channels. `MTIAlphaType.alphaIsOne` is strongly recommended if the image is opaque, e.g. a `CVPixelBuffer` from camera feed, or a `CGImage` loaded from a `jpg` file.\n\nYou can call `unpremultiplyingAlpha()` or `premultiplyingAlpha()` on a `MTIImage` to convert the alpha type of the image.\n\nFor performance reasons, alpha type validation only happens in debug build.\n\n### Alpha Handling of Built-in Filters\n\n- Most of the filters in MetalPetal accept unpremultiplied alpha and opaque images and output unpremultiplied alpha images.\n\n- Filters with `outputAlphaType` property accept inputs of all alpha types. And you can use `outputAlphaType` to specify the alpha type of the output image.\n    \n    e.g. `MTIBlendFilter`, `MTIMultilayerCompositingFilter`, `MTICoreImageUnaryFilter`, `MTIRGBColorSpaceConversionFilter`\n    \n- Filters that do not actually modify colors have passthrough alpha handling rule, that means the alpha types of the output images are the same with the input images.\n\n    e.g. `MTITransformFilter`, `MTICropFilter`, `MTIPixellateFilter`, `MTIBulgeDistortionFilter`\n\nFor more about alpha types and alpha compositing, please refer to [this amazing interactive article](https://ciechanow.ski/alpha-compositing/) by Bartosz Ciechanowski.\n\n## Color Spaces\n\nColor spaces are vital for image processing. The numeric values of the red, green, and blue components have no meaning without a color space.\n\nBefore continuing on how MetalPetal handles color spaces, you may want to know what a color space is and how it affects the representation of color values. There are many articles on the web explaining color spaces, to get started, the suggestion is [Color Spaces - by Bartosz Ciechanowski](https://ciechanow.ski/color-spaces/).\n\nDifferent softwares and frameworks have different ways of handling color spaces. For example, Photoshop has a default sRGB IEC61966-2.1 working color space, while Core Image, by default, uses linear sRGB working color space.\n\nMetal textures do not store any color space information with them. Most of the color space handling in MetalPetal happens during the input (`MTIImage(...)`) and the output (`MTIContext.render...`) of image data.\n\n### Color Spaces for Inputs\n\nSpecifying a color space for an input means that MetalPetal should convert the source color values to the specified color space during the creation of the texture.\n\n- When loading from `URL` or `CGImage`, you can specify which color space you'd like the texture data to be in, using `MTICGImageLoadingOptions`. If you do not specify any options when loading an image, the device RGB color space is used (`MTICGImageLoadingOptions.default`). A `nil` color space disables color matching, this is the equivalent of using the color space of the input image to create `MTICGImageLoadingOptions`. If the model of the specified color space is not RGB, the device RGB color space is used as a fallback.\n\n- When loading from `CIImage`, you can specify which color space you'd like the texture data to be in, using `MTICIImageRenderingOptions`. If you do not specify any options when loading a `CIImage`, the device RGB color space is used (`MTICIImageRenderingOptions.default`). A `nil` color space disables color matching, color values are loaded in the working color space of the `CIContext`.\n\n### Color Spaces for Outputs\n\nWhen specifying a color space for an output, the color space serves more like a tag which is used to communicate with the rest of the system on how to represent the color values in the output. There is no actual color space conversion performed. \n\n- You can specify the color space of an output `CGImage` using `MTIContext.makeCGImage...` or `MTIContext.startTaskTo...` methods with a `colorSpace` parameter.\n\n- You can specify the color space of an output `CIImage` using `MTICIImageCreationOptions`.\n\nMetalPetal assumes that the output color values are in device RGB color space when no output color space is specified.\n\n### Color Spaces for `CVPixelBuffer`\n\nMetalPetal uses `CVMetalTextureCache` and `IOSurface` to directly map `CVPixelBuffer`s to Metal textures. So you cannot specify a color space for loading from or rendering to a `CVPixelBuffer`. However you can specify whether to use a texture with a sRGB pixel format for the mapping.\n\nIn Metal, if the pixel format name has the `_sRGB` suffix, then sRGB gamma compression and decompression are applied during the reading and writing of color values in the pixel. That means a texture with the `_sRGB` pixel format assumes the color values it stores are sRGB gamma corrected, when the color values are read in a shader, sRGB to linear RGB conversions are performed. When the color values are written in a shader, linear RGB to sRGB conversions are performed.\n\n### Color Space Conversions\n\nYou can use `MTIRGBColorSpaceConversionFilter` to perform color space conversions. Color space conversion functions are also available in `MTIShaderLib.h`.\n\n- `metalpetal::sRGBToLinear` (sRGB IEC61966-2.1 to linear sRGB)\n- `metalpetal::linearToSRGB` (linear sRGB to sRGB IEC61966-2.1)\n- `metalpetal::linearToITUR709` (linear sRGB to ITU-R 709)\n- `metalpetal::ITUR709ToLinear` (ITU-R 709 to linear sRGB)\n\n## Extensions\n\n### Working with SceneKit\n\nYou can use `MTISCNSceneRenderer` to generate `MTIImage`s from a `SCNScene`. You may want to handle the SceneKit renderer's linear RGB color space, see issue [#76 The image from SceneKit is darker than normal](https://github.com/MetalPetal/MetalPetal/issues/76).\n\n### Working with SpriteKit\n\nYou can use `MTISKSceneRenderer` to generate `MTIImage`s from a `SKScene`.\n\n### Working with Core Image\n\nYou can create `MTIImage`s from `CIImage`s.\n\nYou can render a `MTIImage` to a `CIImage` using a `MTIContext`.\n\nYou can use a `CIFilter` directly with `MTICoreImageKernel` or the `MTICoreImageUnaryFilter` class. (Swift Only)\n\n### Working with JavaScript\n\nSee [MetalPetalJS](https://github.com/MetalPetal/MetalPetalJS)\n\nWith MetalPetalJS you can create render pipelines and filters using JavaScript, making it possible to download your filters/renderers from \"the cloud\".\n\n### Texture Loader\n\nIt is recommended that you use APIs that accept `MTICGImageLoadingOptions` to load `CGImage`s and images from `URL`, instead of using APIs that accept `MTKTextureLoaderOption`.\n\nWhen you use APIs that accept `MTKTextureLoaderOption`, MetalPetal, by default, uses `MTIDefaultTextureLoader` to load `CGImage`s, images from `URL`, and named images. `MTIDefaultTextureLoader` uses `MTKTextureLoader` internally and has some workarounds for `MTKTextureLoader`'s inconsistencies and bugs at a small performance cost. You can also create your own texture loader by implementing the `MTITextureLoader` protocol. Then assign your texture loader class to `MTIContextOptions.textureLoaderClass` when creating a `MTIContext`.\n\n## Install\n\n### CocoaPods\n\nYou can use [CocoaPods](https://cocoapods.org/) to install the latest version.\n\n```\nuse_frameworks!\n\npod 'MetalPetal'\n\n# Required if you are using Swift.\npod 'MetalPetal/Swift'\n\n# Recommended if you'd like to run MetalPetal on Apple silicon Macs.\npod 'MetalPetal/AppleSilicon'\n\n```\n\n#### Sub-pod `Swift`\n\nProvides Swift-specific additions and modifications to the Objective-C APIs to improve their mapping into Swift. Highly recommended if you are using Swift.\n\n#### Sub-pod `AppleSilicon`\n\nProvides the default shader library compiled in Metal Shading Language v2.3 which is required for enabling programmable blending support on Apple silicon Macs.\n\n### Swift Package Manager\n\n[Adding Package Dependencies to Your App](https://developer.apple.com/documentation/xcode/adding_package_dependencies_to_your_app)\n\n## iOS Simulator Support\n\nMetalPetal can run on Simulator with Xcode 11+ and macOS 10.15+.\n\n`MetalPerformanceShaders.framework` is not available on Simulator, so filters that rely on `MetalPerformanceShaders`, such as `MTIMPSGaussianBlurFilter`, `MTICLAHEFilter`, do not work.\n\nSimulator supports fewer features or different implementation limits than an actual Apple GPU. See [Developing Metal Apps that Run in Simulator](https://developer.apple.com/documentation/metal/developing_metal_apps_that_run_in_simulator) for detail.\n\n## Quick Look Debug Support\n\nIf you do a Quick Look on a `MTIImage`, it'll show you the image graph that you constructed to produce that image.\n\n![Quick Look Debug Preview](https://user-images.githubusercontent.com/1234944/116965587-c6a0a280-ace0-11eb-8918-2f36d1d6114c.jpg)\n\n## Trivia\n\n[Why Objective-C?](https://github.com/MetalPetal/MetalPetal/issues/52)\n\n## Contribute\n\nThank you for considering contributing to MetalPetal. Please read our [Contributing Guidelines](CONTRIBUTING.md).\n\n## License\n\nMetalPetal is MIT-licensed. [LICENSE](LICENSE)\n\nThe files in the `/MetalPetalExamples` directory are licensed under a separate license. [LICENSE.md](MetalPetalExamples/LICENSE.md)\n\nDocumentation is licensed CC-BY-4.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMetalPetal%2FMetalPetal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMetalPetal%2FMetalPetal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMetalPetal%2FMetalPetal/lists"}