Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/timboudreau/audio_unit_rust_demo

Demo of integrating DSP in Rust into an Apple Audio Unit, and of building properly signed installers
https://github.com/timboudreau/audio_unit_rust_demo

Last synced: 10 months ago
JSON representation

Demo of integrating DSP in Rust into an Apple Audio Unit, and of building properly signed installers

Awesome Lists containing this project

README 

          
Rust Audio Unit Demo

====================



This is a *minimal* demo of doing DSP processing in Rust for an Apple Audio Unit.  It builds

a tiny Rust library that implements a Gain/Pan plugin, and uses it from two Audio Unit

implementations.  And it serves as a demo of how to (after configuring various certificates

and ids and whatnot), generate usable signed Mac OS installers in a generic way that, with

some effort, can be used for other projects.



The installers generated by the `audiounits-package` script are for independent distribution,

not the Apple App Store.  It ought to be relatively simple to change the flavor of certificate used

to target the App Store instead if desired.



The installer-generation script requires configuring several app-ids, certificates and provisioning

profiles [in your Apple account](https://developer.apple.com/account/resources/profiles/list) and

using Xcode to set the projects to use them - all of the steps are described below.



Overview

========



This project was created to



 * Tease out issues with automating builds of AudioUnit plugins

 * Sort out the rearranging of sources and projects required to make an AudioUnit able to run

   in-process (Apple's template is not set up to do this, and the documentation makes a cursory

   reference to moving the code into a Framework and having the application extension point to

   it, but nowhere does there exist an open source example of how you actually *do* this, and it

   is non-obvious and non-trivial - there are two plugin projects here - `RustDspPlainDemo` just

   uses Apple's template as-is, while `RustDspInProcessDemo` is that project, rearranged to move

   all of the code into a Framework project)

   * Side note - after several days of work debugging this and finally getting it to work, I

     discovered that Logic Pro will not load AUv3 audio-units in-process anyway.  But perhaps someday

     it will

  * Sort out the vagaries of getting an installer to properly get an AudioUnit registered - all of

    the documentation says to place them in `/Library/Audio/Plug-Ins/Components`, which does not

    appear to work, while simply bunding the semi-useless demo app which is part of the AUv3 template

    causes it to magically be registered - I'd guess due to some filesystem watching black magic,

    but there is no documentation of how it happens

  * Create a fully automated, **command-line** build of multiple audio units working soup-to-nuts, from

    code to signed, notarized, valid, usable installers or dmgs



I have, at present, 13 AudioUnit projects with the DSP done in Rust (most of the audio-unit C++, Swift

and Objective C code I generate from attributes in the Rust code) - releasing all of these by pressing

buttons in an IDE and hoping I don't forget a step, get things out-of-order or mix up the myriad

certificates, app-ids and other fussy bits is a non-starter.  Apple doesn't make it *easy* to do this

stuff outside of Xcode, but it's not entirely impossible given a little one-time set-up in Xcode.



What's Where

------------



### Rust DSP Code



Is in `/crates/mock_dsp_lib` - it is just a dirt-simple plugin that does stereo-only gain and pan, and

is *decidedly* not written with safety or sanity checking in-mind, but it's only here to have a thing

to build a plugin around.



The C-ABI bindings are generated by the `crates/cbindgen-bindgen` binary project, which is just a wrapper

around [cbindgen](https://github.com/mozilla/cbindgen) that knows what to do and where to put things.



The rust-build heavy-lifting is done by the script in the checkout root `build-xcframeworks` which



 * Builds the rust code

 * Builds `cbindgen-bindgen` if it is not present in `$CHECKOUT_ROOT/bin` (it is very slow to build

   and unlikely to change) and puts it there

 * Generates C bindings into the build `target/` dir

 * Builds the rust DSP library once for `aarch64` and once for `x86_64`

 * Merges the results into a single universal binary with `lipo`

 * Creates an XCFramework for the resulting static library and places it in `macos/xcframeworks` where

   the xcode projects can import it from)

 * If the environment variable `CODE_SIGN_ID` is set, signs the XCFramework with it - to find the right

   value for you, run `security find-identity -v -p codesigning` and set the variable to the hash printed

   on the left sign of the line for the appropriate certificate

   * This step is likely unnecessary - the framework is a statically linked library, and there is no need

   to bundle it, but it is here for completeness' sake, and because you may have XCFrameworks you *want*

   to ship.



### XCode Audio Unit Projects



So, there are two Audio Unit plug-in projects in the `macos/` directory.  The first, `RustDspPlainDemo`,

is simply the simplest possible implementation of an Audio Unit that calls out to Rust for DSP.  The second,

`RustDspInProcessDemo` makes the following changes based on [these instructions](https://developer.apple.com/documentation/avfaudio/audio_engine/creating_custom_audio_effects#4302110):

  * A Framework target is added

  * All sources and copied resources are moved from the application extension target's Compile Sources phase to that target (they are in the same locations relative to the project root, just compiled by a different project target)

  * The various C and C++ headers are included in a copy task (as they are in Apple's AudioUnit template - however

    nonituitive it is, this seems to be the way you use such headers in a project without the Objective C build

    tripping over them, trying to parse them as Objective C headers and failing - why the C++ compiler finds

    them *at all* is a mystery to me, but xcodebuild is a foot-deep layer of lipstick on a 1990s pig, so why not?)

  * A single dummy Swift source is added to the Framework target, per the instructions, so there is *something* to compile,

    because reasons.

  * The `Info.plist` of the app extension has the entries described in the link above added to it



Prerequisites

-------------



 * Mac OS 13 or newer + an SDK for Mac OS 13 (or perhaps later if you tweak `build-xcframeworks`)

 * Xcode 14 or newer + command-line tools (`xcodebuild` and `lipo` are used)

 * A Rust toolchain including `cargo` - built with Rust 1.80.1 - whatever `rustup` gives you that supports 2021

   edition or newer should be fine, with target architectures for `aarch64-apple-darwin` and `x86_64-apple-darwin`

   installed (use `rustup target add ...` for whichever you're not running on) - the build scripts

   cross-compile for both architectures



Quickstart

----------



 1. Build the rust libraries by running `build-xcframeworks` (if you don't, the audio unit plugins will be

    missing a library).

   * This builds the Rust library for Intel and Apple Silicon, combines them into a universal binary and

     builds an XCFramework around them which can be loaded by the plugins

 2. Open the Xcode projects in `macos/` and build and run:

   * `RustDspPlainDemo` is the as-unmodified-as-possible Apple Audio Unit Template modified to call

     the Rust library built by `build-xcframeworks` to do the DSP heavy-lifting

   * `RustDspInProcessDemo` is the a revision of the above to move all of the code into a Framework and leave

     the Audio Unit app extension as an empty shell that delegates to the Framework, so that it can

     be (but isn't currently - the problem I'm trying to solve) loaded in-process by the host

 3. The scripts expect Xcode to be configured to build into `$PROJECT/DerivedData`, not the random

    location in your home directory it defaults to. This seems to be a system-wide setting.



Note that you will need to set these projects to be signed with *your* credentials to actually build anything

(IMHO signing is completely orthagonal to developing and building software, but I guess I see the convenience

of shoving it into IDE projects, given that the command-line tooling to do this same is excruciating to work

with).



The audio unit should be visible to audio applications after the Xcode project is built once.  Installers

(see notes about setting up certs and ids below) are output to `macos/packages`.  Interim products

that are useful to examine in isolation are written to `macos/headers` and `macos/xcframeworks`.



The first run of `build-xcframeworks` will be slower because it needs to build `cbindgen` which is used

to generate C headers from Rust code.  It is cached in `bin/cbindgen-bindgen` thereafter.



Details

-------



To deviate as little as possible from the Apple Audio Unit template, the following are the modifications

made to the default code:



 * `RustDspPlainDemoExtensionAudioUnit` configures itself only to support stereo, overrides `canProcessInPlace`,

    and contains a spinlock/busywait to avoid a race condition on deinit in the original Apple code

 * `ObservableAUParameter` exposes the parameter address so the UI can switch on it conveniently, to support

    a small tweak in `ParameterSlider` to convert values to decibels for gain

 * `RustDspPlainDemoExtensionDSPKernel` calls the Rust code, and uses some Rust code to do its parameter reads

    and writes atomically



Signing And Notarizing

----------------------



While it's hard to imagine anyone wanting to endure the steps below, this project is designed as a working

template for how to do this stuff.  So the best approach to begin using it with your own code is to get

a checkout of this project building usable installers, and once you have something you know works, then

start applying it to your own code.



It is not necessary or important that what you're doing involves Rust for DSP - simply both that and

signing, notarizing etc. audio plugins are both ferociously fussy to get working and have non-obvious

failure modes.



### Pitfalls



A few issues encountered in the week+ it took to get this working:



 * While the aim of this project is to create fully automatable command-line builds, some aspects of

   signing simply *must* be configured once in Xcode.

 * If there is any chance you have more than one *Developer ID Application* certificate installed on

   your machine, you **must** explicitly specify *the same specific one* to all of the targets for

   signing.  The "Provisioning Profile" combo-box on the Signing & Capabilities page of Xcode's project settings

   will not let you do this.  Signing > Code Signing Identity in Xcode's Build Settings will let you

   specify exactly the certificate to use.

   * Failure to do this will result in inscrutable error messages at runtime such as

   *external subsystem [NSViewService_PKSubsystem] not present*, or *Unable to find NSExtensionPrincipalClass*

   which have nothing to do with the actual problem.

 * The *provisioning profiles* you set up on Apple's web site and use for signing **must** use exactly

   the same certificate - a mismatch here will also result in failures that give no indication of the

   actual problem

   

### Prerequisites



Some resources need to be created in your Apple account for this to work.  I do not know if it is

necessary to change the bundle-ids of the projects avoid collisions - I would assume not, but I do

not have a second developer account to test with.



You need to set up some SSL certificates:



 * An SSL certificate of the *Developer ID Application* flavor, which you create [here](https://developer.apple.com/account/resources/certificates/list)

 and then need to download, install in your keychain, and possibly tell Xcode about it in Xcode >

 Settings > Accounts > Manage Cetrificates.

 * An SSL certificate of the *Developer ID Installer* flavor, created in the same way

 

and two "app identifiers" which can be created [here](https://developer.apple.com/account/resources/identifiers/list).

The app-id contains the bundle-id of a binary.  You need



 * One for the app ID `com.mastfrog.RustDspInProcessDemo` - for the installer-generation script to

 work, this *must* be given the simple-name `RustDspInProcessDemo`, as the installer-generation script

 assumes a convention that for each bundle-ID it encounters, there is a corresponding app-id named with

 the tailing `.`-delimited element

 

 * One for the app ID `com.mastfrog.RustDspInProcessDemo.RustDspInProcessDemoExtension` which *must*

 be named `RustDspInProcessDemoExtension` for the installer to work.

 

The *framework* target gets signed by `RustDspInProcessDemoExtension` so it does not need its own

app-id, but it **must** be set to use *exactly* the same team and signing certificate as the other

targets, or cryptic errors will haunt you at runtime.



Two *Provisioning Profiles* which you create [here](https://developer.apple.com/account/resources/profiles/list).

As best I can tell, a provisioning profile simply ties together an app-id and a certificate.  These are:



 * A profile named `RustDspInProcessDemo` which ties the app-ID named `RustDspInProcessDemo` to the

 *Developer ID Application* certificate you will sign with (make sure it's the same one as used in the

 build!)

 * A profile named `RustDspInProcessDemoExtension` which ties the app-ID named `RustDspInProcessDemoExtension`

 to the same certificate

 

Download the profiles, and on the Signing & Capabilities page for the app and application-extension projects,

click Provisioning Profile (after disabling *Automatically manage signining* if it is checked) and choose

*Import Profile* to pull it in, once for each project (the same profile cannot be used for both).



### Naming Conventions



If you are reusing these build scripts, the installer and signing phases rely on several hard-and-fast

conventions that allow ids and names and identifiers to be derived from the project name and its bundle

ID.  If you deviate from these, the packaging script must be modified to know what to do.



 * A project consists of three targets that will be nested in each other - if the main project is

   named, say, SidewaysSound

   1. The project folder is named SidewaysSound

   2. The name of the app-extension target is SidewaysSoundExtension

   3. The framework containing the DSP code is named SidewaysSoundFramework

     * Any other miscellaneous frameworks (such as the Rust XCFramework we build here) will be statically

       linked into the final binary, and they do not need to be signed or included

   4. The bundle ID of the app ends with "SidewaysSound"

   5. The bundle ID of the extension ends with "SidewaysSound.SidewaysSoundExtension", and the leading elements

      (e.g. `com.foo.dsp`) are the same for both the app and the extension

   6. The bundle ID of the ends with "SidewaysSoundFramework"

   7. There are App-Ids registered with Apple for **both** the app and the extension, using the above

      described bundle IDs.  Each one's simple name (which you set in the UI on Apple's web site) is

      the simple name of the target / trailing .-delimited element of the bundle-id, e.g. SidewaysSound

      and SidewaysSoundExtension

   8. There are app-specific password credentials already stored in your keychain named "notarytool"

   

### Environment Variables



Certificate hashes, team IDs and similar, used in the build to specify credentials to use when signing

are set.  The required ones are:



 * `APP_SIGNING_ID` - the hash of a developer-id certificate such as "EE3E05AAE04537BA0813CFCB76F35AE832432EBB"

 * `DISTRIBUTION_SIGNING_ID` - this will be a hash of a certificate, such as "EE3E05AAE04537BA0813CFCB76F35AE832432EBB",

 findable using the `security` cli app, and *will not be the same as the `APP_SIGNING_ID` certificate* - Apple

 requires separate and separately registered certificates for applications versus installers.

 * `CODE_SIGN_ID` - The hash of a certificate, such as "3E2A53EF135C0D57EEA876F980A8986A7B0B372E"

 * `NOTARIZATION_TEAM_ID` - a team-id such as "QR2HR8VQP7"

 

 



### Signing Process



In `RustDspInProcessDemo`, signing the three (app, app extension, library) targets is configured as follows - if

you reconfigure the projects to use your certificates, these are the things to change



 * The `RustDspInProcessDemo` (app) and `RustDspInProcessDemoExtension` (.appex - no code) targets are both

   configured to use *the same* **Developer ID Certificate** (we are targeting generating installers for

   non-app store distribution - if we were targeting the app store it would need a different flavor of

   certificate)

 * Both of those targets need separate App IDs registered

 * The `RustDspInProcessDemoFramework` target is set *not to be signed*

 * The `RustDspInProcessDemoExtension` is configured with **Other Signing Flags** set to **`--deep`** - after

   much trial and error, no combination of signing certificates worked to get a usable product when directly

   signing the binary - but, while `--deep` is often discouraged, in this case, signing the library with exactly

   the same credentials as the application extension is exactly what we want

   * If you build and get a cryptic error about app-id entitlements, you probably have `--deep` set somewhere

   it shouldn't be - your entitlements file gets mucked with by xcodebuild, and its contents are *not* what

   actually gets used at signing time.  The invisible entitlement in question seems to be a munged form of the

   the app-id which is dot-prefixed with your team-id which is generated by the build.

 * Building an app and installer that will not be rejected by Mac OS when downloaded involves *notarizing*

   the app and *stapling* the resulting ticket to the result

   * Notarytool comes with its own huge headaches with regards to generating app-specific credentials (a

     password just for Notarytool you generate on the Apple web site).  It's too much to go into, but

     [this article](https://scriptingosx.com/2021/07/notarize-a-command-line-tool-with-notarytool/) is a useful

     guide containing steps that are up-to-date and don't require non-existent commands to be issued.

     The build script expects a keychain-profile named "notarytool" to provide the credentials to use when

     notarizing an app or installer.

 * Building for installers requires two app-ids registered on the Apple website:

   * A profile named RustDspInProcessDemo for an app id `com.mastfrog.dsp.RustDspInProcessDemo` (or whatever

       you changed it to)

   * A profile named `RustDspInProcessDemoExtension` for an app `com.mastfrog.dsp.RustDspInProcessDemo.RustDspInProcessDemoExtension` (or whatever you changed that to)

     



Writing DSP for Audio Units In Rust

===================================



Below are some observations based on building plugins in Rust that may or may not be useful, but

might safe some false starts:



 * The Audio Unit API *relies* on passing pointers around and foreign code on random threads updating

 them in real time while processing happens.  That is pretty antithetical to Rust's strict ownership

 and borrowing rules.  You simply need to embrace that - these are the situations *unsafe* Rust is

 for.  In my case - and you can see a little of that in the atomic-related exported functions in

 `crates/mock_dsp_lib/lib.rs` - the solution was to use atomics and create a small library for setting

 and reading pointers of various types (you can create an atomic `f64` or `f32` using `to_bits()` before

 setting it on an `AtomicU64` or `AtomicU32` and converting back on read with zero runtime cost) 

 that is used by both the C++ code and the Rust code for consistency. If you eventually implement 

 *ramping* - gradually incrementing parameters toward a target as samples are

 processed - then you will have code that updates these parameters on both sides of the language divide.

 * Where practical, capture rather than reread those atomic pointers - for example, if you have a compressor

 with a release time, set a variable with the threshold or ratio from the pointer *at the sample that

 triggers it* and then use that value until the release time has expired - yes, you want those things

 to be dynamic, but it is likely not useful for them to be *that* dynamic, and you get a minor but

 real performance boost at the price of 4 bytes of memory

 * Where possible, keep allocation happening on the C++ side of things.  The one case you can't do that

 for is allocating the thing that does the DSP processing on the rust side.  A simple and workable solution

 is simply to `Box::leak` the allocated DSP processor, and provide a C-ABI call that accepts a pointer and

 deallocates it (as `crates/mock_dsp_lib/lib.rs` does).  It's going to be some code's responsibility to drop

 DSP processors, and only the C++ code has enough information to know when it's safe to do it.

 * Learn to love Rust's [const-generics](https://practice.course.rs/generics-traits/const-generics.html). Const

   generics are powerful tools for bug-prevention and performance.  For example, if you have a chain of

   instances of a trait `SampleProcessor`, and an implementation that chains them

   that is, say, `struct BiSampleProcessor, B: SampleProcessor> {...}`,

   then it is **impossible** to accidentally create a chain with elements expecting a different number of

   channels.  Moreover, since Rust monomorphizes generics, generating a version of the code for each constant

   value the compiler sees, that shifts the switching costs of *all decisions about which code path to

   take **to compile time***.  Zero cost abstractions FTW.  They are also useful for passing the band

   being processed in multiband effects, which channel is being processed, and even buffer sizes correlated

   to sample rates for delay and lookahead buffers. All of this trades small amount of code-complexity (a

   const generic is still an argument, just one passed at compile time) for reduced runtime branching,

   which is usually a worthwhile trade. 

 * Learn to live with ferocious generics signatures. Yes, you could write a DSP chain as a chain of

   `Box` where that's a trait.  But that adds a vtable lookup to every

   invocation, and erases type information you could use to make all sorts of easy-to-have bugs at runtime

   into compile-time errors and impossibilities at runtime.  And that is what Rust is **for**.  Type aliases

   are helpful to write a large generic signature out once, and then use it via the alias in the rest of your

   code, and also gives you a single place to refactor if you change it.

 * Apple's Audio Unit template contains a latent race-condition, where the audio unit can be deallocated

 while the render thread is still inside its render block. This only *really* starts to show up when

 you support presets and try to change the active preset during audio playback. Logic Pro will tear down

 and recreate the plugin while the render thread plugs merrily along.  `RustDspPlainDemoExtensionAudioUnit.mm`

 shows one (not foolproof, but adequate - grabbing a mutex in the render thread is not okay) 

 way to work around this, by adding an atomic spinlock that blocks deinitialization if a thread is inside

 the render block (just an atomic byte and a busywait in `deallocateRenderResources()`)

 * The `os_log` crate may be tempting to use Apple's for unified logging from the Rust side of things.  Don't. It

 assumes logged strings will not be dropped before they are synchronously logged, but - this may be specific

 to real-time threads - the logging may not be synchronous. I have submitted one patch that has been thus

 far been ignored, but it only solves the most common case, not all of the cases. It's pretty but not usable

 in production code.

 * If you have more than a trivial number of plugins, embrace code-generation and generate as much of the

 Objective C and C++ - particularly the DSP kernel and audio unit implementation - as possible.  What I did

 for this, while unpretty and too domain-specific to be useful to open-source, was the following:

   * Used the [`linkme`](https://crates.io/crates/linkme) crate, which can store arbitrary metadata you

   define in a custom linker section of your Rust binary - sort of *roll your own RTTI*, since Rust has none

   * Wrote some custom Rust macros that allow a "create the DSP processor" function to be annotated with

   things like min/default/max parameter values, names of mono-process/stereo-process/destroy/reset functions

   * Generate metadata into that custom linker section - *only when compiled with a feature-flag* (so it doesn't

   end up in the binary I ship at all).  Things that are useful to capture - I simply used a naming convention

   such that `gain_default` defines a default value for `gain` *and also for* `gain_whatever` unless there's

   a `gain_whatever_default` (this requires some care to avoid prefix collisons, but I'm not going for Turing

   completeness here and it's easy enough to avoid):

     * Default/Min/Max value for each parameter

     * Display name templates for parameters

     * The *global namespace* C-ABI name for the destroy/process-mono/process-stereo and reset state functions

     * If you have more than one, the type name of a C `struct` you return from processing functions (in my

       case, I have 4, 8 and 16 band processing result types because they embed input and output level

       metadata used by the UI to display levels on screen)

     * Adhoc flags for things like if a parameter should not be modified in real-time because it will cause

       audible artifacts, or one which is used by-value on Rust DSP processor creation and requires it to be

       torn down and recreated (for example, lookahead buffers which use heap-allocated arrays for performance)

  * Write a code generation app which depends on every plugin *with that flag set*, which can read it and

  generate the appropriate code in the various languages

  * [`uniffi`](https://docs.rs/uniffi/latest/uniffi/) can be tempting for easily generating Swift bindings from

  Rust code, and it *does* work well **but** in performance-critical code, the marshalling it does (and the

  logging it does, though I submitted a patch to gate that on a feature flag) creates unacceptable overhead. This

  is really only a problem if you're using it in places like a render listener called on the realtime audio thread

  (which I needed to do for level indicators updated after each buffer), but that was sufficient for me to switch

  to C-ABI calls and a little more boilerplate.  For infrequently called code, it will probably be fine.

  * The [`tinytemplate`](https://crates.io/crates/tinytemplate/) crate was useful for generating the Audio Unit `.mm`,

  which is largely deciding chunks of boilerplate to include, though you can't beat the granularity of

  old-fashioned programmatic code-generation 

  (which you can actually make [lovely to write](https://github.com/timboudreau/annotation-tools/blob/master/java-vogon/README.md))

  for generating code that is heavily configuration-dependent in fussy ways - I went that route for generating

  the C++ DSP kernel code, which varies widely based on the kinds of I/O stats collected for multiband processing,

  the parameters, etc. which is the sort of thing that becomes nearly unreadable in any templating language.

 * It appears that Apple's template Audio Units just black-hole any render observers passed to

 `AudioUnit.tokenByAddingRenderObserver()` - they are never called - probably the fault of the `AUProcessHelper`

 that the template provides.  You will need to override it, do your own bookkeeping for them, capture a block

 pointer to a dictionary or similar of them in the render block, and call them manually

 * Apple's template sets your plugin up to support eight channels of audio.  Unless you intend to support that,

 you need to edit the generated `WhateverAudioUnit.mm` to report what it really does in several places - the

 `channels` argument to the initial format and `_outputBus.maximumChannelCount` in `setupAudioBuses()`, and

 as pairs of integers in the value returned from `channelCapabilities`. Failure to set all of these consistently can

 result in a plugin validation warning or failure.  The code here is an example of setting a plugin to do stereo-only.

 * There does exist a crate with Rust bindings that could potentially let you create pure-Rust audio-units.

 Looking at it, it appears to have missed a bunch of fundamentals - for example, it hardcodes the plugin

 vendor to be Apple where a cursory read of the docs will tell you that the code the author could only

 find one example of is the plugin identifier - those three blocks of ascii characters.  It does not

 appear to be usable without extensive patching, if at all, delightful as it would be to kick Xcode to the curb.

  

The fun part of writing Audio Units in Rust is that it is pretty ideally suited to DSP - both with easy use of SIMD

and that the type system's features can be leveraged to improve performance, factor code cleanly into separate

concerns with zero runtime cost, and turn several entire categories of easy-to-have bug into compile-time errors.



The not-fun part of writing Audio Units, period, is Apple's antique kludge of a build system and the

layers of cruft that bolt things like code-signing and notarizing onto it.  As someone who spent a lot

of my career in developer tools, I continually resist the temptation to examine what the build is actually

up to and write a tool that recreates that in a sane, modern way.  I suggest you resist that temptation too.



How To Split Apple's Template Code Into A Framework + Application Extension

===========================================================================



Apple's template gives you a project with two targets:  An application (needed, I guess, to give AUv3 users

a way to uninstall?), and an Application Extension which contains the plugin code.



The result of this cannot be loaded in-process by host applications - an application extension's Mach-O type

is *executable*, which cannot be treated as a library by foreign code, but must instead be launched and

communicated with by some form of IPC.  For things that may be called thousands of times per-second to process

real-time audio, that is decidedly not ideal, bordering on unacceptable.



The documentation of how to actually carve up such a project to make it possible to load a plugin in-process

amounts to a [few minimal lines of instructions](https://developer.apple.com/documentation/avfaudio/audio_engine/creating_custom_audio_effects?language=objc) to



 * Move all the code to a new framework target

 * Create a dummy source file and leave that as the only compiled source for the Application Extension (the

   build system appears to need this to have something to build)

 * Modify the appex's `Info.plist` to point to the framework

 * Override a couple of constructors in the main UI



Here are the steps I have used that work to achieve this, starting from a project as Xcode's template creates it:



 1. Create a new Framework target.  Name it - if your effect is named "Whatever" - `WhateverFramework`

 2. On the Build Phases tabs, switching between the old extension target and the new framework target,

 add every source being built by the extension into the Compile Sources phase of the new framework target

 3. In the same place, for the Headers phase, add any `.h` files in the extension project (but not `.hpp` files) to

 the *Public* headers

 4. For any (Rust, or whatever) libraries the application extension was using, you need to

   1. Create links in the project to the header files so Xcode knows about them - even though they exist

   in the xcframeworks you're importing, they won't be found otherwise

   2. Add each of these to the *Private* headers in the Headers build phase

 5. Now we come to the woefully underdocumented parts:  When those libraries were used from the application

   extension, all we needed was to set the build setting *Allow Non Modular Includes* to `YES` and things

   would work.  A framework is a different beast, and you cannot just include stuff.  While we *could*

   have our xcframeworks include a `module.modulemap` file, this runs aground as soon as you have *more

   than one* - it *must* be named `module.modulemap` and the xcodebuild will try to clobber them with

   each other and fail (I spent many days attempting a workaround for this. There is none.).  So 

   what we need to do is create the sum of the module maps we would have put in each library inside the 

   framework project, and tell the build that is the private module map of the project; and create a public

   module map that maps the headers a caller needs to find to use our plugin.  Steps:

    1. Create a file `public.modulemap` in the framework project (example below). It should pull in the

      generated `WhateverFramework.h` file created when you created the framework target as an umbrella header, 

      and at a minimum, the headers for the Audio Unit which were generated by the Apple template - e.g. 

      `WhateverExtensionAudioUnit.h`

    2. Find the build setting for the public modulemap and set it to the relative path using Apple's

      build variable dereferencing syntax, e.g. `$(SRCROOT)/WhateverFramework/public.modulemap`

    3. Create a file `private.modulemap` mapping any headers from libraries (example below)

    4. Find the build setting for the private module map and do likewise for it.  Note the naming in the

      module map files is strict - i.e. `WhateverFramework` and `WhateverFramework_Private` in the first

      line that names the module. This accomplishes the same thing as the *Objective C Bridging Header*

      did in the extension project, allowing library code to be visible to Swift

    5. This step is optional, but helpful if you have projects that share sources (I shouldn't, but I do):

      Alias the private module so the libraries can be imported using a common name across projects -

      in the build setting *Other Swift Flags* add, e.g. `-module-alias PluginLibs=WhateverFramework_Private`.

      Most of my plugins have a several of the same xcframeworks plus one that implements that specific

      effect, and the Swift and Cocoa UI code is shared across them, so this makes those sources build

      unmodified in multiple projects that define `PluginLibs` just by adding `import PluginLibs`.

    6. For whatever reason, the Swift compiler may begin complaining about some esoterica that needs

      to be addressed - in this setup, code that uses `ObservableObject` and similar needs an import of

      `Combine` where it did not before, and you may get some warnings about closed enums defined in the

      same source file possibly growing new members and switches over their members not having a default

      branch. I do not know the reason the Swift compiler behaves differently, but these are all easy

      enough to fix.

    7. When importing from a module-map, the compiler will now complain if the header files for your

      libraries use quotes instead of angle-brackets for `include` statements.  If you are using the

      `cbindgen-bindgen` code from this project, simply add an empty file named `hack_cbindgen_includes`

      in the root of each crate triggering such a warning; if you are doing something else you may need to

      edit your header files or whatever generates them.

  6. Once all of that is done, delete all sources and headers from the extension target's *Headers* and

      *Compile Sources* phases - they now belong to the framework, not the extension, which is an empty shell

     that points to the framework

  7. As described in the docs linked above, create an empty Swift source file with a single empty function

     in it, in the extension project's sources. If you're feeling saucy, name it something like

    `xcodebuild_is_a_bag_of_steve_jobs_ossified_turds()`.

  8. Add that one source file to the Compile Sources phase of the application extension

  9. Update the `Info.plist` for the extension project adding a key in `NSExtensionAttributes` named

     `AudioComponentBundle` with the string value of the *framework* project's bundle-id

  10. Also in the same `Info.plist`, change the two properties that point to `$(PRODUCT_MODULE_NAME).AudioUnitViewController`,

    replacing `$(PRODUCT_MODULE_NAME)` with the *simple* name of your framework, so it reads something

    like `WhateverFramework.AudioUnitViewController`.

  11. Move the dependencies on any libraries (Rust or other) from the extension project to the framework,

    and set the framework as a dependency from the extension project.  Set the dependency to *Embed Without

    Signing* as the framework will be signed when it builds

  12. Make sure any (Rust or other) libraries in the framework project are set to *Do not Embed* (through

    black magic and auto-linking, the code you call will wind up statically linked into its binary)

  13. Set the following settings in the *framework* project's project settings:

     * Build Libraries for Distribution - `YES`

     * Allow Non-modular Includes In Framework Modules - `YES`

     * Link Frameworks Automatically - `YES`

     * Skip Install - `YES`

  14. Set the *Copyright (Human Readable)* build setting for *each* target to something sane and non-empty,

    and the *Application Category* to `Music` (if unset, installer creation will fail)

  15. Set the *entitlements* for the application extension *and* framework to request *Audio Input*,

    *Outgoing Network Connections* (may be needed for IPC) and *Apple Events* (this is questionable,

    but I have seen complaints logged in Console and failure to load when this is not set).  This may

    mean using the *+Capability* button on the *Signing and Capabilities* project tab and adding *App Sandbox*

  16. Shut down Xcode, delete the build directory, reopen it, build it and see what's still broken, and

     iterate until it isn't :-)



#### Example public module map

   

```c

framework module ZenWidenerFramework {

    umbrella header "ZenWidenerFramework.h"

    // All of these must be added to the Build Phases > Headers > Public

    // section on the settings tabs of the Framework project

    header "ZenWidenerPresets.h"

    header "ZenWidenerAddresses.h"

    header "ZenWidenerExtensionAudioUnit.h"

    export *

}

```

   

#### Example private module map

   

```c

framework module ZenWidenerFramework_Private {

    // all of these must be added to Build Phases > Headers > Private

    // on the settings tabs of the framework project

    header "curve.h"

    header "atomics.h"

    header "mute_solo_bypass.h"

    header "widener_plugin.h"

    export *

}

```



### Cleanup



A few bits of settings cleanup can be done on the *application extension* project, which appear to be

harmless if left behind, but no longer needed:



 * Remove the *Objective C Bridging Header* build setting

 * Delete the generated bridging header file, which should be in, 

   e.g. `WhateverProject/WhateverProjectExtension/WhateverProjectExtensionBridgingHeader`. The

   module map files we created take the place of that in the new framework



### Optimization



Once things are working properly, you may want to set the *Optimization Level* setting for the framework

path to `-Ofast` and *Link Time Optimization* to `Monolithic` for release build, and enable `Dead Code Stripping`

if you share sources or libraries across multiple projects.



### Never Trust a Dirty Build; Xcode's Clean ... Doesn't Always



Note that Xcode caches module information in the `DerivedData` dir, and *does not delete it when you clean

the build directory*.  If you fiddle with the contents of module maps or who owns them even a little, exit

Xcode and `rm -Rf` the entire `DerivedData` build dir, or you can spend hours chasing problems that don't

actually exist, because Xcode doesn't invalidate its cache.



Every IDE author's ten commandments should include *Thou shalt not create caches thou

canst not invalidate when they need it 100% of the time*, but in Xcode's case, it was omitted.



If something weird is happening, be sure Xcode doesn't have something stuck in its craw.  This also

goes for error annotations in sources that were fixed many builds ago, but reappear because they are

from the first build that ran after Xcode started.



### Things That Go Wrong



 * *Provisioning profile "WhateverExtension" doesn't match the entitlements file's value for the 

   com.apple.application-identifier entitlement.* - what this cryptic message actually means (and there

   seem to be no links in any Google search permutation where this is explained):  The application

   project has `--deep` among its *Other Signing Flags* - signing generates a synthetic entry in

   the entitlements file for the project, and if `--deep` is set, that will also show up in the

   extension's entitlements, where it will point to the wrong provisioning profile.

 * Application launches but cannot load the UI - did you override the constructors in AudioUnitViewController?

 * *Principal object (null) must derive from AUViewController but doesn't.* - most likely

   you forgot to add the `AudioComponentBundle` entry to the extension's `Info.plist`

   or the `factoryFunction` or `NSExtensionPrincipalClass` were not updated correctly

   (they should be identical).

 * *The project named "Whatever" does not contain a scheme named "Whatever"* - the

   installer script assumes that there is a scheme with the same name as the

   project/application target for it to build. Ensure there is one - just go

   to Product > Scheme > New Scheme and it will be the default in the dialog that

   pops up - just press Enter.

   



Misc Audio Unit Build, Run and Registration Quirks

--------------------------------------------------



A few magic strings are created within the Audio Unit template wizard in Audio Unit which

can cause misbehavior if they deviate from un- or barely-documented patterns:



 * That 4-letter ascii *Manufacturer Code* **really** must contain a capital letter or

   bad things will happen.  It, plus the plugin id get passed around in the Audio Unit

   API as a *number* and should be unique.

 * The result of the manufacturer name and display name entered in the wizard are concatenated

   with a `:` to generate the `name` field generated into the extension project's `Info.plist`.

   You can edit it, but when your plugin shows up in a host application like Logic Pro, the

   text *before* the `:` is used for a sub-menu, and the text after it is the title of the

   menu item for your plugin.  The colon must be there, and there must be non-whitespace

   characters on both sides of it.

 * You can test that hosts will accept your plugin using the utility `auval` and the codes you

   entered in the wizard.  Run `auval -l | grep `.  Then run

   it with `-v` (means validate, not verbose), e.g. `auval -v aufx rdid Msfg`

    * If you have a very complex SwiftUI that is slow to load, validation can fail due to

      timing-out.  Find ways to optimize it (fixed sizes make a big difference), or make

      the view load lazily on a timer (wreaks havoc with dynamic sizing), or do what I did -

      rewrite the parts that do the heavy lifting in Cocoa with `CALayer`s and get finer

      grained control of what renders when and how