Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/fiberplane/fp-bindgen
Bindings generator for full-stack WASM plugins
https://github.com/fiberplane/fp-bindgen
Last synced: 7 days ago
JSON representation
Bindings generator for full-stack WASM plugins
- Host: GitHub
- URL: https://github.com/fiberplane/fp-bindgen
- Owner: fiberplane
- License: apache-2.0
- Created: 2021-08-03T14:52:11.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-03-27T15:36:03.000Z (8 months ago)
- Last Synced: 2024-04-20T09:08:20.445Z (7 months ago)
- Language: Rust
- Homepage: https://fiberplane.dev
- Size: 712 KB
- Stars: 467
- Watchers: 13
- Forks: 18
- Open Issues: 20
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE-APACHE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# fp-bindgen
[](https://crates.io/crates/fp-bindgen)
[](https://discord.gg/MJr7pYzZQ4)
Bindings generator for full-stack WASM plugins.
## Comparison to other "bindgen" tools
`fp-bindgen` is not the only tool for generating Wasm bindings. The most well-known tool for this
is probably `wasm-bindgen`, though it is limited to Rust modules running inside browser
environments. A more generic alternative, based on the Wasm
[interface types proposal](https://github.com/WebAssembly/interface-types/blob/main/proposals/interface-types/Explainer.md),
is `wit-bindgen`. We do believe interface types to be the future of Wasm bindings, but for the
short-term, `fp-bindgen` provides bindings that work with a stable serialization format, which helps
us to avoid versioning issues and opens up compatibility with tools such as Serde.
It is worth mentioning that, though we have a [specification](#specification) for our communication
primitives that allows generators for other languages to be contributed, `fp-bindgen` is opinionated
towards Rust. It uses Rust data structures and function signatures as its "protocol format",
enabling tight integration with existing crates from the Rust ecosystem.
The following table is intended to highlight the major differences between the different tools:
| Feature | `fp-bindgen` | `wasm-bindgen` | `wit-bindgen` |
| --------------------------------------------------------- | :-------------------------: | :------------: | :-----------------------------: |
| Host environments | Rust (Wasmer), TypeScript\* | JS/TS | Rust/Python (Wasmtime), JS/TS\* |
| Guest languages | Rust\* | Rust | Rust, C\* |
| Protocol format | Rust (using macros) | N/A | .wit |
| Serialization format | MessagePack | JSON | Custom |
| [Can use existing Rust types](#using-existing-rust-types) | ✅ | ❌ | ❌ |
\*) These are only the _currently supported_ options. More may be added in the future.
## Quickstart
* Check out the repository, using `git clone`. We use symlinks in the repo, so on Windows use
`git clone -c core.symlinks=true` instead.
* To quickly build an example protocol and plugin and run all available tests use:
`cargo xtask test`
## Usage
Using `fp-bindgen` is a three-step process:
- First you [define a protocol](#defining-a-protocol) that specifies the functions and data
structures available for communication across the Wasm bridge.
- Then you [generate the bindings](#generating-bindings) for the hosts and plugin language that are
relevant to you.
- Finally, you can start [implementing plugins and runtimes](#using-the-bindings) using the
generated bindings.
## Defining a protocol
Before you can generate bindings using this library, you first define a protocol of functions that
can be called by the _runtime_ (the Wasm host) and functions that can be called by the _plugin_ (the
Wasm guest module). The protocol specifies the function declarations, which are placed inside two
macros: `fp_import!` and `fp_export!`. These macros specify which functions can be imported and
which can be exported, _from the perspective of the plugin_. In other words, `fp_import!` functions
can be called by the plugin and must be implemented by the runtime, while `fp_export!` functions can
be called by the runtime and _may_ be implemented by the plugin.
**Example:**
```rust
fp_bindgen::prelude::fp_import! {
fn my_imported_function(a: u32, b: u32) -> u32;
}
fp_bindgen::prelude::fp_export! {
fn my_exported_function(a: u32, b: u32) -> u32;
}
```
**Important caveat:** There must be exactly one `fp_import!` block and one `fp_export!` block in the
same module as where you invoke `fp_bindgen!()`. If you only have imports, or only have exports, you
should create an empty block for the other.
### Data structures
Besides primitives, functions can pass Rust `struct`s and `enum`s as their arguments and return
value, but only by value (passing a reference across the Wasm bridge is currently not supported) and
only for types that implement `Serializable`.
**Example:**
```rust
#[derive(fp_bindgen::prelude::Serializable)]
pub struct MyStruct {
pub foo: i32,
pub bar: String,
}
fp_bindgen::prelude::fp_import! {
fn my_function(data: MyStruct) -> MyStruct;
}
```
Note that `Serializable` is implemented by default for some common standard types, such as
`Option`, `Vec`, and other container types.
### Async functions
Functions can also be `async`, which works as you would expect:
**Example:**
```rust
fp_bindgen::prelude::fp_import! {
async fn my_async_function(data: MyStruct) -> Result;
}
```
### Using existing Rust types
Sometimes you may wish to use Rust types for your protocol that you also want to use directly in the
generated runtime or plugin implementation. In such a case, generation of the data types might force
you to perform unnecessary copies, so we allow explicit annotations to import the existing
definition instead of generating a new one:
**Example:**
```rust
use fp_bindgen::prelude::Serializable;
#[derive(Serializable)]
#[fp(rust_module = "my_crate::prelude")]
pub struct MyStruct {
pub foo: i32,
pub bar_qux: String,
}
```
In this example, `MyStruct` has a double function: it acts both as a type definition for the
protocol (through `fp-bindgen`'s `Serializable` trait), which can still be used for generating a
TypeScript type definition, for instance. _And_ it acts as a type that can be directly used by the
Rust Wasmer runtime, under the assumption the runtime can import it from `my_crate::prelude`.
Please note that in this case, you do have a bigger responsibility to make sure the definition
fulfills the requirements of the code generator, hence why Serde's trait derives and annotations
have to be added manually here, in accordance with how the generator would otherwise generate them.
For now, this feature is limited to the Rust generators through the
`rust_module` annotation. For us, this makes sense given the
protocol itself is specified using Rust syntax as well. If desired, we could extend this to the
TypeScript generator as well, though that would imply an even bigger responsibility for the user to
keep their TypeScript types in sync with the protocol.
### Cargo features
The `fp-bindgen` crate supports optional Cargo features for compatibility with some common types
from the crate ecosystem:
- `bytes-compat`: Enables compatibility with the `bytes::Bytes` type.
- `http-compat`: Enables compatibility with various types from the `http` crate.
- `rmpv-compat`: Enables compatibility with the `rmpv::Value` type.
- `serde-bytes-compat`: Enables compatibility with the `serde_bytes::ByteBuf` type (the `Bytes` type
is a reference type, which `fp-bindgen` doesn't support in general).
- `serde-json-compat`: Enables compatibility with `serde_json::Map` and `serde_json::Value` types.
- `time-compat`: Enables compatibility with `time`'s `PrimitiveDateTime` and `OffsetDateTime` types.
## Generating bindings
To generate bindings based on your protocol, you first need to create a function that will generate
them for you. Creating this function is easy, because its implementation can be created for you
using the `fp_bindgen` macro:
```rust
let bindings_type = fp_bindgen::BindingsType::RustWasmerRuntime;
fp_bindgen::prelude::fp_bindgen!(fp_bindgen::BindingConfig {
bindings_type,
path: &format!("bindings/{}", bindings_type)
});
```
Currently, we support the following binding types:
- `BindingsType::RustPlugin`: Generates bindings for a Rust plugin.
- `BindingsType::RustWasmerRuntime`: Generates runtime bindings for use with Wasmer.
- `BindingsType::TsRuntimeWithExtendedConfig`: Generates bindings for a TypeScript runtime.
Note that some binding types take an additional config argument.
## Using the bindings
How to use the generated bindings differs between the various types.
### Using the Rust plugin bindings
The generator for our Rust plugin bindings generates a complete crate that allows to be linked
against by plugins. The plugin can import all the functions from the `fp_import!` block from it,
and call them like any other functions.
In order to export the functions that are defined in the `fp_export!` block, it can use the exported
`fp_export_impl` macro, like so:
```rust
#[fp_bindgen_macros::fp_export_impl(bindings_crate_path)]
fn my_exported_function(a: u32, b: u32) -> u32 {
/* ... */
}
```
`bindings_crate_path` is expected to match with the module path from which the bindings crate
itself is imported. The function signature must match exactly with one of the `fp_export!`
functions.
When compiling a plugin, don't forget to compile against the "wasm32-unknown-unknown" target, or you
will receive linker errors.
See the `example-plugin/` directory for an example of a plugin that uses bindings generated from
our `example-protocol/` (do note this plugin only builds after you've run `cargo run` inside the
`example-protocol/` directory).
### Using the Rust Wasmer runtime bindings
The generator for our Rust Wasmer runtime works a bit differently. Instead of generating a crate,
it generates two files: `bindings.rs` and `types.rs`. These can be placed in a module of your
choosing (we chose a module named `spec` in the `example-rust-runtime/`).
As the implementor of the runtime, it is then your responsibility to implement the `fp_import!`
functions within the same module as you've placed the generated files. You can see an example of
this in `example-rust-runtime/spec/mod.rs` (do note the example runtime only builds after you've run
`cargo run` inside the `example-protocol/` directory).
Finally, the `bindings.rs` file contains a constructor (`Runtime::new()`) that you can use to
instantiate Wasmer runtimes with the Wasm module provided as a blob. The `fp_export!` functions are
provided on the `Runtime` instance as methods. Please be aware that implementation of the
`fp_export!` functions is always at the discretion of the plugin, and an attempt to invoke a missing
implementation can fail with an `InvocationError::FunctionNotExported` error.
### Using the TypeScript runtime bindings
The TypeScript runtime generator can work with browsers, Node.js and Deno.
It works similarly to that for the Wasmer runtime, but it generates an `index.ts` and a `types.ts`.
`types.ts` contains the type definitions for all the data structures, while the `index.ts` exports a
`createRuntime()` function that you can use for instantiating the runtime. Upon instantiation, you
are expected to provide implementations for all the `fp_import!` functions, while the returned
`Promise` will give you an object with all the `fp_export!` functions the provided plugin has
implemented.
## Examples
Please have a look at [`examples/README.md`](examples/README.md) for various examples on how to use
`fp-bindgen`.
## Specification
We have written down a specification that describes the primitives used by our bindings. This is
aimed primarily at those that want to understand how the bindings work under the hood, and may be
valuable if you want to implement bindings for your own favorite language.
If that is you, please have a look at [`docs/SPEC.md`](docs/SPEC.md).
## Known Limitations
- Data types may only contain value types. References are currently unsupported.
- Referencing types using their full module path is prone to cause mismatches during type
discovery. Please import types using a `use` statement and refer to them by their name only.
- TypeScript bindings handle 64-bit integers somewhat inconsistently. When passed as primitives (as
plain function arguments or return values) they will be encoded using the `BigInt` type. But when
they're part of a MessagePack-encoded data type, they will be encoded using `number`, which
effectively limits them to a maximum size of `2^53 - 1`. For more information, see:
## FAQ
### I added a `Serializable` derive to my type, why don't I see it included in the bindings?
Are you using the type in one of the `fp_import!` or `fp_export!` functions? Deriving `Serializable`
makes it possible to use the type as part of your protocol, but it won't become part of the
generated bindings until it is actually referenced. Note that types can be either referenced
directly by one of the `fp_import!` or `fp_export!` functions, or indirectly by another type that is
already in use.
If a type is not referenced either directly or indirectly by any of the functions that are part of
your protocol, you can force inclusion by adding a `use` statement referencing the type to either
the `fp_import!` or `fp_export!` section:
```rust
fp_bindgen::prelude::fp_import! {
use MyType;
}
```
Are you referencing the type and it is still not included in your bindings? Please
[file an issue](https://github.com/fiberplane/fp-bindgen/issues).
### Can I use aliases?
Yes, but because aliases cannot have a derive macro, please repeat the alias in either the
`fp_import!` or `fp_export!` section:
```rust
fp_bindgen::prelude::fp_import! {
type MyType = SomeOtherType;
}
```
### What about versioning?
Generally, versioning is considered out-of-scope for this project. This means it is your own
responsibility to verify a plugin you execute was compiled against a compatible version of the
protocol your runtime provides.
If your protocol ever needs to introduce breaking changes, we advise to include a `version() -> u32`
export function in the protocol itself that you can call before invoking any other functions.
As for what constitutes a breaking change, we offer the following guidelines:
- All plugin exports are always optional. Because of this, new exports can always be added without
breaking existing plugins, unless your runtime performs an explicit check that mandates an
export's existence.
- Adding new imports is always safe, as they will simply be ignored by existing plugins.
- Adding fields to `struct`s is always safe, unless your runtime mandates the existence of such
fields in arguments or return values coming from the plugin.
- Adding new types is always safe.
- **Anything else should be considered a breaking change.**
Note that, because of the above guidelines, you should never need to define a versioning function in
your first iteration. Because plugin exports are optional, the absense of a versioning function can
simply be interpreted as meaning the plugin is at version 1.
## Getting Help
Please see
[COMMUNITY.md](https://github.com/fiberplane/fiberplane-rs/blob/main/COMMUNITY.md)
for ways to reach out to us.
## Contributing
Please follow our [Contributing Guidelines](CONTRIBUTING.md) to learn how best
to contribute to this project.
## Code of Conduct
See
[CODE_OF_CONDUCT.md](https://github.com/fiberplane/fiberplane-rs/blob/main/CODE_OF_CONDUCT.md).
## License
This project is distributed under the terms of both the MIT license and the
Apache License (Version 2.0).
See [LICENSE-APACHE](LICENSE-APACHE) and [LICENSE-MIT](LICENSE-MIT).