Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/jamesgober/pack-io

Compact binary wire format with schema evolution and zero-copy deserialization for Rust.
https://github.com/jamesgober/pack-io

binary codec reps rust schema-evolution serialization zero-copy

Last synced: about 16 hours ago
JSON representation

Compact binary wire format with schema evolution and zero-copy deserialization for Rust.

Awesome Lists containing this project

README 

          


    Rust logo

    


    pack-io

    


    

        COMPACT BINARY WIRE FORMAT

    






    Crates.io

    Crates.io Downloads

    docs.rs

    GitHub CI

    MSRV










    


        pack-io is a compact binary wire format for Rust built around three properties existing serialization crates split across three separate libraries: speed, schema evolution, and zero-copy deserialization. It is engineered as the serialization substrate underneath network-protocol, wire-codec, and Hive DB - so it has to be fast, deterministic, schema-aware, and safe under untrusted input from day one.

    


    


        Existing crates each cover part of the problem. bincode is fast but has no schema evolution. rkyv gives zero-copy but requires unsafe alignment discipline at the use site. postcard is embedded-focused and lean. None of them own all three properties as a single coherent contract. pack-io does, behind a small, predictable wire format you can read with a spec instead of with the source code.

    


    


        The common-case API is one line - encode(&value) and decode::<T>(&bytes) - and that path is the fast path. Schema versions, evolution helpers, and the zero-copy view API live behind feature flags so the default build stays small.

    


    


    


    


        MSRV is 1.85+ (Rust 2024 edition). no_std-capable. Deterministic encoding. No unsafe on the safe-decoding path.

    


    


        Status: pre-1.0, in active development. v0.1.0 is the scaffold release - structure, tooling, and quality gates only; codec logic lands across the 0.x series. The wire format is being designed and frozen across the 0.x line; 1.0.0 is the wire-format freeze. See CHANGELOG.md for detail.

    











## Why pack-io



Existing crates each cover a slice of the problem; none of them own all three properties together.



| Crate       | Speed | Schema evolution | Zero-copy decode | Wire-format spec |

|-------------|:-----:|:----------------:|:----------------:|:----------------:|

| `bincode`   |   ✓   |        —         |        —         |        —         |

| `rkyv`      |   ✓   |        —         |        ✓ *       |        —         |

| `postcard`  |   ✓   |        —         |        —         |        ✓         |

| **`pack-io`** | **✓** |       **✓**      |       **✓**      |       **✓**      |



* `rkyv` requires alignment discipline at every use site; the cost is paid by the caller, not the codec.



The 1.0 contract is the same wire format on every supported platform, the same bytes for the same value every time, and no panic / no unbounded allocation on any input.












## What it does



- **Compact binary encoding** of Rust values into a small, predictable wire format

- **Schema-versioned messages** - producer and consumer can be at different revisions and still interoperate

- **Zero-copy deserialization** for `&[u8]` / `&str` / length-prefixed slices when the input lives long enough

- **Deterministic output** - the same value always produces the same bytes (canonical encoding)

- **Safe under untrusted input** - bounded allocation, length-prefix validation, no panics on malformed bytes

- **Runtime-agnostic** - synchronous codec, usable from any context







## Features



- **Compact** — small, fixed-overhead encoding; varint integers; length-prefixed byte slices

- **Schema evolution** — additive field changes, optional fields, version negotiation

- **Zero-copy decode** — view types that borrow from the input buffer where possible

- **Deterministic** — canonical encoding for hashing, signing, content-addressing

- **Safe defaults** — bounded allocation, validated lengths, no panics on bad input

- **`no_std`-capable** — embedded and constrained environments

- **Derive macro** — `#[derive(Serialize, Deserialize)]` for any struct or enum (feature `derive`)

- **Optional `serde` interop** — read and write `serde` types via a thin adapter (feature `serde`)









## Roadmap snapshot



| Version | Scope | Status |

|---------|-------|:------:|

| `0.1.0` | Scaffold: structure, CI, lints, quality gates | ✅ shipped |

| `0.2.0` | Foundation: `encode` / `decode`, primitive types, `Serialize` / `Deserialize`, round-trip + determinism + adversarial-decode proptests | ✅ shipped |

| `0.3.0` | Wire-format freeze, collections, streaming over `Read` / `Write`, `docs/WIRE_FORMAT.md` | _next_ |

| `0.4.0` | `View` zero-copy decode + `derive` macro | planned |

| `0.5.0` | Schema evolution attributes + version negotiation | planned |

| `0.6.0` | Optimization pass + comparative benchmarks | planned |

| `0.7.0` | Hardening, fuzz, API freeze | planned |

| `0.8.x` → `0.9.x` | Alpha → Beta → RC | planned |

| `1.0.0` | Wire-format + API freeze | planned |



The roadmap is followed strictly; phases are not skipped. Per-phase exit criteria are tracked internally and surfaced in each release note.









## Installation



```toml

[dependencies]

pack-io = "0.2"



# With derive macro (planned for 0.4+):

pack-io = { version = "0.2", features = ["derive"] }



# no_std build:

pack-io = { version = "0.2", default-features = false }

```







## API surface (v0.2.0)



The foundation surface — Tier-1 free functions, the Tier-2 in-memory `Encoder` / `Decoder`, and the Tier-3 `Serialize` / `Deserialize` traits — is live. See [`docs/API.md`](./docs/API.md) for the full reference.



### Tier 1 — the lazy path



The headline. One function each direction, no setup, no type parameters the caller has to name beyond the target type.



```rust

use pack_io::{encode, decode};



let bytes: Vec = encode(&(7_u64, true, String::from("hello"))).unwrap();

let back: (u64, bool, String) = decode(&bytes).unwrap();

assert_eq!(back, (7, true, String::from("hello")));

```



### Tier 2 — the in-memory `Encoder` / `Decoder`



Re-use a single `Vec` across many encodes; read several values from one buffer. Configuration (`Config::max_alloc`) is validated at construction time, not on every operation. Streaming over `std::io::Read` / `Write` arrives in `0.3`.



```rust

use pack_io::{Encoder, Decoder, Config};



let mut enc = Encoder::new();

enc.write(&7_u64).unwrap();

enc.write(&"hello").unwrap();

let bytes = enc.into_inner();



let cfg = Config::new().with_max_alloc(16 * 1024); // refuse > 16 KiB allocs

let mut dec = Decoder::with_config(&bytes, cfg).unwrap();

let n: u64 = dec.read().unwrap();

let s: String = dec.read().unwrap();

assert_eq!((n, s.as_str()), (7, "hello"));

```



### Tier 3 — implement [`Serialize`] / [`Deserialize`] on your own types



Today: write the impls by hand. The derive macro arrives in `0.4`.



```rust

use pack_io::{Serialize, Deserialize, Encoder, Decoder, SerialError};



struct Point { x: i32, y: i32 }



impl Serialize for Point {

    fn serialize(&self, enc: &mut Encoder) -> Result<(), SerialError> {

        self.x.serialize(enc)?;

        self.y.serialize(enc)

    }

}



impl Deserialize for Point {

    fn deserialize(dec: &mut Decoder<'_>) -> Result {

        Ok(Point {

            x: i32::deserialize(dec)?,

            y: i32::deserialize(dec)?,

        })

    }

}

```



### Primitive types supported in v0.2.0



`u8` … `u128`, `i8` … `i128`, `usize` / `isize`, `bool`, `f32`, `f64`, `String` / `&str`, `Vec` / `&[u8]`, fixed-size arrays `[T; N]`, tuples of arity 1 … 12, `Option`, `Result`, and `()`.



Collections (`Vec`, `HashMap`, …), schema evolution, and the zero-copy `View` surface arrive in later 0.x phases.







## Invariants (held from v0.1.0)



- **Round-trip integrity** — `decode(encode(v)) == v` for every supported type, under any input.

- **Determinism** — the same value always produces the same bytes; no map-iteration-order leaks, no time-dependence, no platform-dependence.

- **Safe decode** — no panic, no unbounded allocation, no read past input, on any byte sequence.

- **Wire-format stability** — frozen at `1.0`; any `1.x` decoder reads any `1.x`-or-earlier encoding.



These invariants hold for every release in the `0.x` series. As of `0.2.0` they are enforced by `proptest` round-trip + determinism harnesses for every primitive (46 properties) and an adversarial-decode harness that fuzzes the public decode surface with random bytes, truncations, and hostile length prefixes (20 properties). A `cargo-fuzz` harness lands in `0.7`.







## Testing



```bash

# Stable + MSRV (1.85) on Linux / macOS / Windows, full feature matrix

cargo fmt --all -- --check

cargo clippy --all-targets --all-features -- -D warnings

cargo test --all-features

cargo test --no-default-features

RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features



# Supply chain

cargo audit

cargo deny check



# Concurrency model checking (decoders are stateless; loom coverage is light-touch)

RUSTFLAGS="--cfg loom" cargo test --test loom_codec



# Microbenchmarks

cargo bench --bench codec_bench

```







## Examples



Each example is self-contained and runs against the published API of the version it was added in.



```bash

cargo run --example basic_roundtrip --release   # Tier-1 encode/decode of a tuple

cargo run --example primitive_tour --release    # one encoded value per primitive type

cargo run --example reuse_buffer --release      # Tier-2 Encoder + multi-value Decoder

```









## Cross-Platform Support



**Tier 1 Support:**

- ✅ Linux (x86_64, aarch64)

- ✅ macOS (x86_64, Apple Silicon)

- ✅ Windows (x86_64)



Encoding is byte-deterministic across all three; the CI matrix runs every target on stable and MSRV. Platform-specific behaviour is forbidden in the codec — there is no `#[cfg(target_os = …)]` branch on the encode or decode path.







## Where It Fits



`pack-io` is the serialization substrate under [`network-protocol`](https://github.com/jamesgober/network-protocol), [`wire-codec`](https://github.com/jamesgober/wire-codec), and Hive DB. It is consumed by [`raft-io`](https://github.com/jamesgober/raft-io) for log entries and by `event-stream` (when it lands) for message framing. It stays foreign-compatible: it works on its own without any other crate in the family.







## Contributing



Before opening a PR, the full local checklist must pass:



```bash

cargo fmt --all -- --check

cargo clippy --all-targets --all-features -- -D warnings

cargo test --all-features

RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features

cargo deny check

cargo audit

```



Any change touching the wire format requires a `proptest` round-trip and a determinism test in the same commit. Wire-format-breaking changes are not accepted after `0.3` without an accompanying migration note in [`CHANGELOG.md`](./CHANGELOG.md).















    
License


    
Licensed under either of


    

    
at your option. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.







  


  COPYRIGHT © 2026 JAMES GOBER.