Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/heestand-xyz/PixelKit

Live Graphics in Swift & Metal
https://github.com/heestand-xyz/PixelKit

graphics ios live macos metal swift

Last synced: about 1 month ago
JSON representation

Live Graphics in Swift & Metal

Awesome Lists containing this project

README

        

*Check out [AsyncGraphics](https://github.com/heestand-xyz/AsyncGraphics).
Inspired by PixelKit & RenderKit.*

# PixelKit

Live Graphics for iOS, macOS and tvOS

Runs on [RenderKit](https://github.com/heestand-xyz/RenderKit), powered by Metal

PixelKit combines custom shaders, metal performance shaders, core image filters and vision to create tools for real-time rendering.

Examples:
[Camera Effects](#example-camera-effects) -
[Green Screen](#example-green-screen)


Info:
[Coordinate Space](#coordinate-space) -
[Blend Operators](#blend-operators) -
[Effect Convenience Funcs](#effect-convenience-funcs) -
[High Bit Mode](#high-bit-mode)

| | | | | | | |
| --- | --- | --- | --- | --- | --- | --- |
| CameraPIX | DepthCameraPIX | ImagePIX | VideoPIX | ScreenCapturePIX | StreamInPIX | SlopePIX |

| | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ColorPIX | CirclePIX | RectanglePIX | PolygonPIX | ArcPIX | LinePIX | GradientPIX | StackPIX |

| | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- |
| NoisePIX | TextPIX | MetalPIX | TwirlPIX | FeedbackPIX | DelayPIX | SharpenPIX | StreamOutPIX |

| | | | | | | |
| --- | --- | --- | --- | --- | --- | --- |
| LevelsPIX | BlurPIX | EdgePIX | ThresholdPIX | QuantizePIX | TransformPIX | KaleidoscopePIX |

| | | | | | | |
| --- | --- | --- | --- | --- | --- | --- |
| ChannelMixPIX | ChromaKeyPIX | CornerPinPIX | ColorShiftPIX | FlipFlopPIX | RangePIX | StarPIX |

| | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SepiaPIX | ConvertPIX | ReducePIX | ClampPIX | FreezePIX | FlarePIX | AirPlayPIX | RecordPIX |

| | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- |
| BlendPIX | CrossPIX | LookupPIX | DisplacePIX | RemapPIX | ReorderPIX | ResolutionPIX | CropPIX |

| | | | | | |
| --- | --- | --- | --- | --- | --- |
| BlendsPIX | LumaLevelsPIX | LumaBlurPIX | LumaTransformPIX | TimeMachinePIX | ArrayPIX |

---

## Install

### Swift Package

~~~~swift
.package(url: "https://github.com/heestand-xyz/PixelKit", from: "3.0.0")
~~~~

## Setup

### SwiftUI

~~~~swift
import SwiftUI
import PixelKit

struct ContentView: View {

@StateObject var circlePix = CirclePIX()
@StateObject var blurPix = BlurPIX()

var body: some View {
PixelView(pix: blurPix)
.onAppear {
blurPix.input = circlePix
blurPix.radius = 0.25
}
}
}
~~~~

### UIKit

~~~~swift
import UIKit
import PixelKit

class ViewController: UIViewController {

override func viewDidLoad() {
super.viewDidLoad()

let circlePix = CirclePIX()

let blurPix = BlurPIX()
blurPix.input = circlePix
blurPix.radius = 0.25

let finalPix: PIX = blurPix
finalPix.view.frame = view.bounds
view.addSubview(finalPix.view)
}
}
~~~~

## Resolution

In PixelKit all PIXs have a resolution. Some PIXs have defined resolutions (default to `.auto`) and some PIXs have derived resolutions.

The `.auto` resolution will fill up the view and get the correct resolution based on the view size. If a view is 100x100 points, the resolution will be 200x200 pixels on macOS and 300x300 pixels on iPhone.

Import the resolution package to work with resolutions:

```swift
import Resolution
```

You can multiply and divide resolutions with a `CGFloat` or `Int`.

There are predefined resolutions like `._1080p` & `._4K`.

## Rendered Image

~~~~swift
.renderedImage // UIImage or NSImage
.renderedTexture // MTLTexture
~~~~

### Example: Camera Effects

| | | | | |
| --- | --- | --- | --- | --- |

```swift
import SwiftUI
import PixelKit

class ViewModel: ObservableObject {

let camera: CameraPIX
let levels: LevelsPIX
let colorShift: ColorShiftPIX
let blur: BlurPIX
let circle: CirclePIX

let finalPix: PIX

init() {

camera = CameraPIX()
camera.cameraResolution = ._1080p

levels = LevelsPIX()
levels.input = camera
levels.brightness = 1.5
levels.gamma = 0.5

colorShift = ColorShiftPIX()
colorShift.input = levels
colorShift.saturation = 0.5

blur = BlurPIX()
blur.input = colorShift
blur.radius = 0.25

circle = CirclePIX(at: .square(1080))
circle.radius = 0.45
circle.backgroundColor = .clear

finalPix = blur & (camera * circle)
}
}

struct ContentView: View {

@StateObject var viewModel = ViewModel()

var body: some View {
PixelView(pix: viewModel.finalPix)
}
}
```

This can also be done with [Effect Convenience Funcs](#effect-convenience-funcs):

```swift
let pix = CameraPIX().pixBrightness(1.5).pixGamma(0.5).pixSaturation(0.5).pixBlur(0.25)
```

Remeber to add `NSCameraUsageDescription` to your *Info.plist*

### Example: Green Screen

| | | | |
| --- | --- | --- | --- |

`import RenderKit
import PixelKit`

~~~~swift
let cityImage = ImagePIX()
cityImage.image = UIImage(named: "city")

let supermanVideo = VideoPIX()
supermanVideo.load(fileNamed: "superman", withExtension: "mov")

let supermanKeyed = ChromaKeyPIX()
supermanKeyed.input = supermanVideo
supermanKeyed.keyColor = .green

let blendPix = BlendPIX()
blendPix.blendingMode = .over
blendPix.inputA = cityImage
blendPix.inputB = supermanKeyed

let finalPix: PIX = blendPix
finalPix.view.frame = view.bounds
view.addSubview(finalPix.view)
~~~~

This can also be done with [Blend Operators](#blend-operators) and [Effect Convenience Funcs](#effect-convenience-funcs):

```swift
let pix = cityImage & supermanVideo.pixChromaKey(.green)
```

### Example: Depth Camera

`
import RenderKit
import PixelKit`

~~~~swift
let cameraPix = CameraPIX()
cameraPix.camera = .front

let depthCameraPix = DepthCameraPIX.setup(with: cameraPix)

let levelsPix = LevelsPIX()
levelsPix.input = depthCameraPix
levelsPix.inverted = true

let lumaBlurPix = cameraPix.pixLumaBlur(pix: levelsPix, radius: 0.1)

let finalPix: PIX = lumaBlurPix
finalPix.view.frame = view.bounds
view.addSubview(finalPix.view)
~~~~

The `DepthCameraPIX` was added in PixelKit `v0.8.4` and requires an iPhone X or newer.

Note to use the `setup(with:filter:)` method of `DepthCameraPIX`.

It will take care of orientation, color and enable depth on the `CameraPIX`.

To gain access to depth values ouside of the 0.0 and 1.0 bounds,

enable `16 bit` mode like this: `PixelKit.main.render.bits = ._16`

### Example: Multi Camera

~~~~swift
let cameraPix = CameraPIX()
cameraPix.camera = .back

let multiCameraPix = MultiCameraPIX.setup(with: cameraPix, camera: .front)

let movedMultiCameraPix = multiCameraPix.pixScale(by: 0.25).pixTranslate(x: 0.375 * (9 / 16), y: 0.375)

let finalPix: PIX = camearPix & movedMultiCameraPix
finalPix.view.frame = view.bounds
view.addSubview(finalPix.view)
~~~~

Note `MultiCameraPIX` requires iOS 13.

## Coordinate Space

The PixelKit coordinate space is normailzed to the vertical axis (1.0 in height) with the origin (0.0, 0.0) in the center.

Note that compared to native UIKit and SwiftUI views the vertical axis is flipped and origin is moved, this is more convinent when working with graphics in PixelKit.
A full rotation is defined by 1.0

Center: CGPoint(x: 0, y: 0)

Bottom Left: CGPoint(x: -0.5 * aspectRatio, y: -0.5)

Top Right: CGPoint(x: 0.5 * aspectRatio, y: 0.5)

Tip: `Resolution` has an `.aspect` property:

`let aspectRatio: CGFloat = Resolution._1080p.aspect`

## Blend Operators

A quick and convenient way to blend PIXs

These are the supported `BlendingMode` operators:

| `&` | `!&` | `+` | `-` | `*` | `**` | `!**` | `%` | `~` | `°` |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| .over | .under | .add | .subtract | .multiply | .power | .gamma | .difference | .average | cosine |

| `<>` | `><` | `++` | `--` | `<->` | `>-<` | `+-+` |
| --- | --- | --- | --- | --- | --- | --- |
| .minimum | .maximum | .addWithAlpha | .subtractWithAlpha | inside | outside | exclusiveOr |

```swift
let blendPix = (CameraPIX() !** NoisePIX(at: .fullHD(.portrait))) * CirclePIX(at: .fullHD(.portrait))
```

The default global blend operator fill mode is `.fit`, change it like this:

`PIX.blendOperators.globalPlacement = .fill`

## Effect Convenience Funcs

- .pixScaleResolution(to: ._1080p * 0.5) -> ResolutionPIX
- .pixScaleResolution(by: 0.5) -> ResolutionPIX
- .pixBrightness(0.5) -> LevelsPIX
- .pixDarkness(0.5) -> LevelsPIX
- .pixContrast(0.5) -> LevelsPIX
- .pixGamma(0.5) -> LevelsPIX
- .pixInvert() -> LevelsPIX
- .pixOpacity(0.5) -> LevelsPIX
- .pixBlur(0.5) -> BlurPIX
- .pixEdge() -> EdgePIX
- .pixThreshold(at: 0.5) -> ThresholdPIX
- .pixQuantize(by: 0.5) -> QuantizePIX
- .pixPosition(at: CGPoint(x: 0.5, y: 0.5)) -> TransformPIX
- .pixRotatate(by: 0.5) -> TransformPIX
- .pixRotatate(byRadians: .pi) -> TransformPIX
- .pixRotatate(byDegrees: 180) -> TransformPIX
- .pixScale(by: 0.5) -> TransformPIX
- .pixKaleidoscope() -> KaleidoscopePIX
- .pixTwirl(0.5) -> TwirlPIX
- .pixSwap(.red, .blue) -> ChannelMixPIX
- .pixChromaKey(.green) -> ChromaKeyPIX
- .pixHue(0.5) -> ColorShiftPIX
- .pixSaturation(0.5) -> ColorShiftPIX
- .pixCrop(CGRect(x: 0.25, y 0.25, width: 0.5, height: 0.5)) -> CropPIX
- .pixFlipX() -> FlipFlopPIX
- .pixFlipY() -> FlipFlopPIX
- .pixFlopLeft() -> FlipFlopPIX
- .pixFlopRight() -> FlipFlopPIX
- .pixRange(inLow: 0.0, inHigh: 0.5, outLow: 0.5, outHigh: 1.0) -> RangePIX
- .pixRange(inLow: .clear, inHigh: .gray, outLow: .gray, outHigh: .white) -> RangePIX
- .pixSharpen() -> SharpenPIX
- .pixSlope() - > SlopePIX
- .pixVignetting(radius: 0.5, inset: 0.25, gamma: 0.5) -> LumaLevelsPIX
- .pixLookup(pix: pixB, axis: .x) -> LookupPIX
- .pixLumaBlur(pix: pixB, radius: 0.5) -> LumaBlurPIX
- .pixLumaLevels(pix: pixB, brightness: 2.0) -> LumaLevelsPIX
- .pixDisplace(pix: pixB, distance: 0.5) -> DisplacePIX
- .pixRemap(pix: pixB) -> RemapPIX

Keep in mind that these funcs will create new PIXs.

Be careful of overloading GPU memory, some funcs create several PIXs.

## High Bit Mode

Some effects like DisplacePIX and SlopePIX can benefit from a higher bit depth.

The default is 8 bits. Change it like this:
`PixelKit.main.render.bits = ._16`

Enable high bit mode before you create any PIXs.

Note resources do not support higher bits yet.

There is currently there is some gamma offset with resources.

## MetalPIXs

~~~~swift
let metalPix = MetalPIX(at: ._1080p, code:
"""
pix = float4(u, v, 0.0, 1.0);
"""
)
~~~~

~~~~swift
let metalEffectPix = MetalEffectPIX(code:
"""
float gamma = 0.25;
pix = pow(input, 1.0 / gamma);
"""
)
metalEffectPix.input = CameraPIX()
~~~~

~~~~swift
let metalMergerEffectPix = MetalMergerEffectPIX(code:
"""
pix = pow(inputA, 1.0 / inputB);
"""
)
metalMergerEffectPix.inputA = CameraPIX()
metalMergerEffectPix.inputB = ImagePIX("img_name")
~~~~

~~~~swift
let metalMultiEffectPix = MetalMultiEffectPIX(code:
"""
float4 inPixA = inTexs.sample(s, uv, 0);
float4 inPixB = inTexs.sample(s, uv, 1);
float4 inPixC = inTexs.sample(s, uv, 2);
pix = inPixA + inPixB + inPixC;
"""
)
metalMultiEffectPix.inputs = [ImagePIX("img_a"), ImagePIX("img_b"), ImagePIX("img_c")]
~~~~

### Uniforms:

~~~~swift
var lumUniform = MetalUniform(name: "lum")
let metalPix = MetalPIX(at: ._1080p, code:
"""
pix = float4(in.lum, in.lum, in.lum, 1.0);
""",
uniforms: [lumUniform]
)
lumUniform.value = 0.5
~~~~

### Notes:

- To gain camera access, on macOS, check Camera in the App Sandbox in your Xcode project settings under Capabilities.

---

inspired by [TouchDesigner](https://derivative.ca)
created by Anton [Heestand XYZ](http://heestand.xyz/)