https://github.com/inkandswitch/subduction
🏔️ Sync protocol for hash-linked data
https://github.com/inkandswitch/subduction
hashlink protocol synchronization
Last synced: 7 days ago
JSON representation
🏔️ Sync protocol for hash-linked data
- Host: GitHub
- URL: https://github.com/inkandswitch/subduction
- Owner: inkandswitch
- License: apache-2.0
- Created: 2025-07-18T16:11:12.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2026-05-06T18:42:55.000Z (28 days ago)
- Last Synced: 2026-05-06T19:17:46.665Z (28 days ago)
- Topics: hashlink, protocol, synchronization
- Language: Rust
- Homepage:
- Size: 9.04 MB
- Stars: 55
- Watchers: 2
- Forks: 6
- Open Issues: 11
-
Metadata Files:
- Readme: README.md
- License: LICENSE-APACHE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# Subduction
[](https://github.com/inkandswitch/subduction/actions/workflows/build.yml)
[](https://github.com/inkandswitch/subduction/actions/workflows/test-host.yml)
[](https://github.com/inkandswitch/subduction/actions/workflows/test-wasm.yml)
[](LICENSE-APACHE)
> [!CAUTION]
> This is an early release preview. It has a very unstable API. No guarantees are given. DO NOT use for production use cases at this time. USE AT YOUR OWN RISK.
**Subduction** is a peer-to-peer synchronization protocol and implementation for [CRDTs]. It enables efficient synchronization of encrypted, partitioned data between peers without requiring a central server.
## Features
- **Efficient Sync Protocol**: Uses [Sedimentree] for history sharding, diffing, and efficient incremental synchronization
- **Encryption-Friendly**: Designed to work with encrypted data partitions without requiring decryption during sync
- **No Central Server**: True peer-to-peer synchronization via pluggable transports (WebSocket, HTTP long-poll, Iroh/QUIC)
- **Multi-Platform**: Runs on native Rust, WebAssembly (browser & Node.js), and provides a CLI tool
- **Automerge Integration**: While not the only data that can be synced via Subduction, [Automerge] was the original target.
## Documentation
The [`design/`](./design/) directory contains protocol design documents including message formats, threat model, and so on.
## Architecture
Subduction is organized as a Rust workspace with multiple crates:
```mermaid
graph TD
subgraph Core
sedimentree_core
subduction_crypto
subduction_core
end
subgraph Storage
sedimentree_fs_storage
end
subgraph Transport
subduction_http_longpoll
subduction_iroh
subduction_websocket
end
subgraph Integrations
automerge_sedimentree
subduction_keyhive
subduction_keyhive_policy
end
subgraph Wasm
sedimentree_wasm
subduction_wasm
automerge_sedimentree_wasm
automerge_subduction_wasm
end
subduction_cli
sedimentree_core --> subduction_crypto
subduction_crypto --> subduction_core
sedimentree_core --> automerge_sedimentree
subduction_core --> sedimentree_fs_storage
subduction_core --> subduction_http_longpoll
subduction_core --> subduction_iroh
subduction_core --> subduction_websocket
subduction_core --> subduction_keyhive_policy
subduction_core --> sedimentree_wasm
subduction_core --> subduction_wasm
sedimentree_wasm --> subduction_wasm
subduction_wasm --> automerge_sedimentree_wasm
sedimentree_wasm --> automerge_sedimentree_wasm
automerge_sedimentree_wasm --> automerge_subduction_wasm
subduction_wasm --> automerge_subduction_wasm
subduction_http_longpoll --> subduction_cli
subduction_iroh --> subduction_cli
subduction_websocket --> subduction_cli
sedimentree_fs_storage --> subduction_cli
```
| Crate | Description |
|----------------------------|----------------------------------------------------------------------------------------|
| `sedimentree_core` | The core data partitioning scheme that enables efficient metadata-based synchronization |
| `sedimentree_fs_storage` | Filesystem-based persistent storage for Sedimentree |
| `subduction_crypto` | Cryptographic types: signed payloads and verification witnesses |
| `subduction_core` | The main synchronization protocol implementation |
| `subduction_http_longpoll` | HTTP long-poll transport for restricted network environments |
| `subduction_iroh` | Iroh (QUIC) transport for direct peer-to-peer connections |
| `subduction_websocket` | WebSocket transport layer for peer-to-peer connections |
### Platform Bindings
| Crate | Description |
|------------------------------|------------------------------------------------------------|
| `sedimentree_wasm` | WebAssembly bindings for Sedimentree |
| `subduction_wasm` | WebAssembly bindings for browser and Node.js environments |
| `automerge_sedimentree` | Sedimentree adapter for Automerge documents |
| `automerge_sedimentree_wasm` | Wasm bindings for Automerge + Sedimentree |
| `automerge_subduction_wasm` | Wasm bindings for Automerge + Subduction (full sync stack) |
### Integrations
| Crate | Description |
|-----------------------------|--------------------------------------------------------|
| `subduction_keyhive` | Keyhive integration types |
| `subduction_keyhive_policy` | Keyhive-based authorization policy for connections |
### Tools
| Crate | Description |
|------------------|------------------------------------------------------------------|
| `subduction_cli` | Command-line tool for running Subduction servers and purging data |
## Sedimentree
[Sedimentree][sedimentree-design] is a novel data structure for organizing encrypted data into hierarchical layers (strata). Each layer contains metadata (hashes) that represent fragments of a larger file or log. This enables:
1. **Efficient Diffing**: Compare metadata to determine which fragments need synchronization
2. **Privacy**: Sync without exposing plaintext data
3. **Incremental Updates**: Only transfer changed fragments
Sedimentree uses a depth-based partitioning scheme where fragments are organized by the number of leading zero bytes in their hashes, creating natural hierarchical layers.
## Quick Start
### Using Nix ❄️
If you use Nix, the Subduction server is wrapped in a flake:
```bash
# Run the server directly
nix run github:inkandswitch/subduction -- server --socket 0.0.0.0:8080
# Install to your profile
nix profile install github:inkandswitch/subduction
```
The flake also provides NixOS and Home Manager modules for running Subduction as a system service:
```nix
{
inputs.subduction.url = "github:inkandswitch/subduction";
outputs = { nixpkgs, subduction, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
subduction.nixosModules.default
{
services.subduction.server.enable = true;
}
];
};
};
}
```
See the [CLI README] for full configuration options including Home Manager and reverse proxy setup.
### Prerequisites
- Rust 1.90 or later
- For Wasm development: [wasm-pack]
- For browser testing: Node.js 22+ and pnpm
### Building
Build all workspace crates:
```bash
cargo build --release
```
Build Wasm packages:
```bash
cd subduction_wasm
pnpm install
pnpm build
```
### Running Tests
Run all Rust tests:
```bash
cargo test
```
Run Wasm browser tests:
```bash
cd subduction_wasm
pnpm install
npx playwright install
npx playwright test
```
### Using the CLI
Start a WebSocket server:
```bash
cargo run --release -p subduction_cli -- server --socket 127.0.0.1:8080
```
## Usage Examples
### Rust
```rust
use subduction_core::Subduction;
use subduction_core::storage::MemoryStorage;
// Create a Subduction instance with in-memory storage
let storage = MemoryStorage::new();
let subduction = Subduction::new(storage);
// Connect to peers, sync data...
```
### WebAssembly (Browser)
```typescript
import * as subduction from '@automerge/subduction';
// Create a Subduction instance
const storage = new subduction.MemoryStorage();
const syncer = new subduction.Subduction(storage);
// Connect to a WebSocket server
const ws = new WebSocket('ws://localhost:8080');
const peerId = new subduction.PeerId(new Uint8Array(32)); // Your peer ID
const subductionWs = await subduction.SubductionWebSocket.connect(
new URL('ws://localhost:8080'),
peerId,
5000
);
// Register the connection
await syncer.register(subductionWs);
// Start syncing...
```
## Development
### Project Structure
```
subduction/
├── sedimentree_core/ # Core Sedimentree data structure
├── sedimentree_fs_storage/ # Filesystem storage for Sedimentree
├── sedimentree_wasm/ # Wasm bindings for Sedimentree
├── subduction_crypto/ # Signed payloads and verification witnesses
├── subduction_core/ # Sync protocol implementation
├── subduction_http_longpoll/ # HTTP long-poll transport
├── subduction_iroh/ # Iroh (QUIC) transport
├── subduction_websocket/ # WebSocket transport
├── subduction_wasm/ # Wasm bindings for Subduction
├── subduction_cli/ # CLI server and data management
├── subduction_keyhive/ # Keyhive integration types
├── subduction_keyhive_policy/ # Keyhive authorization policy
├── automerge_sedimentree/ # Automerge integration
├── automerge_sedimentree_wasm/ # Wasm wrapper for automerge_sedimentree
└── automerge_subduction_wasm/ # Wasm wrapper for automerge_sedimentree + subduction
```
### Testing
The project uses several testing strategies:
| Strategy | Description |
|----------------------|-----------------------------------------------------------------|
| Unit Tests | Standard `cargo test` for Rust code |
| Property-based Tests | [`bolero`] for fuzz testing |
| E2E Tests | Playwright tests for Wasm bindings (see `subduction_wasm/e2e/`) |
| Integration Tests | Transport connection tests (WebSocket, HTTP long-poll, Iroh) |
[CLI README]: ./subduction_cli/README.md#running-as-a-system-service
[Sedimentree]: ./sedimentree_core
[Automerge]: https://automerge.org/
[`bolero`]: https://github.com/camshaft/bolero
[CRDTs]: https://crdt.tech/
[Ink & Switch]: https://www.inkandswitch.com/
[sedimentree-design]: https://github.com/inkandswitch/keyhive/blob/main/design/sedimentree.md
[wasm-pack]: https://rustwasm.github.io/wasm-pack/