{"id":33926101,"url":"https://github.com/inkandswitch/subduction","last_synced_at":"2026-06-06T08:01:13.026Z","repository":{"id":316274445,"uuid":"1022198504","full_name":"inkandswitch/subduction","owner":"inkandswitch","description":"🏔️ Sync protocol for hash-linked data","archived":false,"fork":false,"pushed_at":"2026-06-04T20:54:39.000Z","size":10617,"stargazers_count":82,"open_issues_count":36,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-06-04T21:18:54.350Z","etag":null,"topics":["hashlink","protocol","synchronization"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/inkandswitch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-18T16:11:12.000Z","updated_at":"2026-06-04T20:40:20.000Z","dependencies_parsed_at":"2025-10-20T23:23:14.456Z","dependency_job_id":"6e86096d-bc6d-4923-b38c-90cb6d293b45","html_url":"https://github.com/inkandswitch/subduction","commit_stats":null,"previous_names":["inkandswitch/subduction"],"tags_count":67,"template":false,"template_full_name":null,"purl":"pkg:github/inkandswitch/subduction","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inkandswitch%2Fsubduction","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inkandswitch%2Fsubduction/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inkandswitch%2Fsubduction/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inkandswitch%2Fsubduction/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/inkandswitch","download_url":"https://codeload.github.com/inkandswitch/subduction/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inkandswitch%2Fsubduction/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33973868,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-06T02:00:07.033Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hashlink","protocol","synchronization"],"created_at":"2025-12-12T10:04:30.612Z","updated_at":"2026-06-06T08:01:13.013Z","avatar_url":"https://github.com/inkandswitch.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Subduction\n\n[![Build](https://github.com/inkandswitch/subduction/actions/workflows/build.yml/badge.svg)](https://github.com/inkandswitch/subduction/actions/workflows/build.yml)\n[![Tests](https://github.com/inkandswitch/subduction/actions/workflows/test-host.yml/badge.svg)](https://github.com/inkandswitch/subduction/actions/workflows/test-host.yml)\n[![Wasm Tests](https://github.com/inkandswitch/subduction/actions/workflows/test-wasm.yml/badge.svg)](https://github.com/inkandswitch/subduction/actions/workflows/test-wasm.yml)\n[![License](https://img.shields.io/badge/License-MIT_OR_Apache_2.0-blue.svg)](LICENSE-APACHE)\n\n\u003e [!CAUTION]\n\u003e This is an early release preview. It has a very unstable API. No guarantees are given. DO NOT use for production use cases at this time. USE AT YOUR OWN RISK.\n\n**Subduction** is a peer-to-peer synchronization protocol and implementation for [CRDTs]. It enables efficient synchronization of encrypted, partitioned data between peers without requiring a central server.\n\n## Features\n\n- **Efficient Sync Protocol**: Uses [Sedimentree] for history sharding, diffing, and efficient incremental synchronization\n- **Encryption-Friendly**: Designed to work with encrypted data partitions without requiring decryption during sync\n- **No Central Server**: True peer-to-peer synchronization via pluggable transports (WebSocket, HTTP long-poll, Iroh/QUIC)\n- **Multi-Platform**: Runs on native Rust, WebAssembly (browser \u0026 Node.js), and provides a CLI tool\n- **Automerge Integration**: While not the only data that can be synced via Subduction, [Automerge] was the original target.\n\n## Documentation\n\nThe [`design/`](./design/) directory contains protocol design documents including message formats, threat model, and so on.\n\n## Architecture\n\nSubduction is organized as a Rust workspace with multiple crates:\n\n```mermaid\ngraph TD\n    subgraph Core\n        sedimentree_core\n        subduction_crypto\n        subduction_core\n    end\n\n    subgraph Storage\n        sedimentree_fs_storage\n    end\n\n    subgraph Transport\n        subduction_http_longpoll\n        subduction_iroh\n        subduction_websocket\n    end\n\n    subgraph Integrations\n        subduction_keyhive\n        subduction_keyhive_policy\n    end\n\n    subgraph Wasm\n        sedimentree_wasm\n        subduction_wasm\n        automerge_subduction_wasm\n    end\n\n    subduction_cli\n\n    sedimentree_core --\u003e subduction_crypto\n    subduction_crypto --\u003e subduction_core\n\n    subduction_core --\u003e sedimentree_fs_storage\n    subduction_core --\u003e subduction_http_longpoll\n    subduction_core --\u003e subduction_iroh\n    subduction_core --\u003e subduction_websocket\n    subduction_core --\u003e subduction_keyhive_policy\n\n    subduction_core --\u003e sedimentree_wasm\n    subduction_core --\u003e subduction_wasm\n    sedimentree_wasm --\u003e subduction_wasm\n\n    subduction_wasm --\u003e automerge_subduction_wasm\n    sedimentree_wasm --\u003e automerge_subduction_wasm\n\n    subduction_http_longpoll --\u003e subduction_cli\n    subduction_iroh --\u003e subduction_cli\n    subduction_websocket --\u003e subduction_cli\n    sedimentree_fs_storage --\u003e subduction_cli\n```\n\n| Crate                      | Description                                                                            |\n|----------------------------|----------------------------------------------------------------------------------------|\n| `sedimentree_core`         | The core data partitioning scheme that enables efficient metadata-based synchronization |\n| `sedimentree_fs_storage`   | Filesystem-based persistent storage for Sedimentree                                    |\n| `subduction_crypto`        | Cryptographic types: signed payloads and verification witnesses                        |\n| `subduction_core`          | The main synchronization protocol implementation                                       |\n| `subduction_http_longpoll` | HTTP long-poll transport for restricted network environments                           |\n| `subduction_iroh`          | Iroh (QUIC) transport for direct peer-to-peer connections                              |\n| `subduction_websocket`     | WebSocket transport layer for peer-to-peer connections                                 |\n\n### Platform Bindings\n\n| Crate                       | Description                                                |\n|-----------------------------|------------------------------------------------------------|\n| `sedimentree_wasm`          | WebAssembly bindings for Sedimentree                       |\n| `subduction_wasm`           | WebAssembly bindings for browser and Node.js environments  |\n| `automerge_subduction_wasm` | Wasm bindings for Automerge + Subduction (full sync stack) |\n\n### Integrations\n\n| Crate                       | Description                                            |\n|-----------------------------|--------------------------------------------------------|\n| `subduction_keyhive`        | Keyhive integration types                              |\n| `subduction_keyhive_policy` | Keyhive-based authorization policy for connections     |\n\n### Tools\n\n| Crate            | Description                                                      |\n|------------------|------------------------------------------------------------------|\n| `subduction_cli` | Command-line tool for running Subduction servers and purging data |\n\n## Sedimentree\n\n[Sedimentree][sedimentree-design] is a novel data structure for organizing encrypted data into hierarchical layers (strata). Each layer contains metadata (hashes) that represent fragments of a larger file or log. This enables:\n\n1. **Efficient Diffing**: Compare metadata to determine which fragments need synchronization\n2. **Privacy**: Sync without exposing plaintext data\n3. **Incremental Updates**: Only transfer changed fragments\n\nSedimentree uses a depth-based partitioning scheme where fragments are organized by the number of leading zero bytes in their hashes, creating natural hierarchical layers.\n\n## Quick Start\n\n### Using Nix ❄️ \n\nIf you use Nix, the Subduction server is wrapped in a flake:\n\n```bash\n# Run the server directly\nnix run github:inkandswitch/subduction -- server --socket 0.0.0.0:8080\n\n# Install to your profile\nnix profile install github:inkandswitch/subduction\n```\n\nThe flake also provides NixOS and Home Manager modules for running Subduction as a system service:\n\n```nix\n{\n  inputs.subduction.url = \"github:inkandswitch/subduction\";\n\n  outputs = { nixpkgs, subduction, ... }: {\n    nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {\n      modules = [\n        subduction.nixosModules.default\n        {\n          services.subduction.server.enable = true;\n        }\n      ];\n    };\n  };\n}\n```\n\nSee the [CLI README] for full configuration options including Home Manager and reverse proxy setup.\n\n### Prerequisites\n\n- Rust 1.90 or later\n- For Wasm development: [wasm-pack]\n- For browser testing: Node.js 22+ and pnpm\n\n### Building\n\nBuild all workspace crates:\n\n```bash\ncargo build --release\n```\n\nBuild Wasm packages:\n\n```bash\ncd subduction_wasm\npnpm install\npnpm build\n```\n\n### Running Tests\n\nRun all Rust tests:\n\n```bash\ncargo test\n```\n\nRun Wasm browser tests:\n\n```bash\ncd subduction_wasm\npnpm install\nnpx playwright install\nnpx playwright test\n```\n\n### Using the CLI\n\nStart a WebSocket server:\n\n```bash\ncargo run --release -p subduction_cli -- server --socket 127.0.0.1:8080\n```\n\n## Usage Examples\n\n### Rust\n\n```rust\nuse subduction_core::Subduction;\nuse subduction_core::storage::MemoryStorage;\n\n// Create a Subduction instance with in-memory storage\nlet storage = MemoryStorage::new();\nlet subduction = Subduction::new(storage);\n\n// Connect to peers, sync data...\n```\n\n### WebAssembly (Browser)\n\n```typescript\nimport * as subduction from '@automerge/subduction';\n\n// Create a Subduction instance\nconst storage = new subduction.MemoryStorage();\nconst syncer = new subduction.Subduction(storage);\n\n// Connect to a WebSocket server\nconst ws = new WebSocket('ws://localhost:8080');\nconst peerId = new subduction.PeerId(new Uint8Array(32)); // Your peer ID\nconst subductionWs = await subduction.SubductionWebSocket.connect(\n  new URL('ws://localhost:8080'),\n  peerId,\n  5000\n);\n\n// Register the connection\nawait syncer.register(subductionWs);\n\n// Start syncing...\n```\n\n## Development\n\n### Project Structure\n\n```\nsubduction/\n├── sedimentree_core/           # Core Sedimentree data structure\n├── sedimentree_fs_storage/     # Filesystem storage for Sedimentree\n├── sedimentree_wasm/           # Wasm bindings for Sedimentree\n├── subduction_crypto/          # Signed payloads and verification witnesses\n├── subduction_core/            # Sync protocol implementation\n├── subduction_http_longpoll/   # HTTP long-poll transport\n├── subduction_iroh/            # Iroh (QUIC) transport\n├── subduction_websocket/       # WebSocket transport\n├── subduction_wasm/            # Wasm bindings for Subduction\n├── subduction_cli/             # CLI server and data management\n├── subduction_keyhive/         # Keyhive integration types\n├── subduction_keyhive_policy/  # Keyhive authorization policy\n├── automerge_subduction_ingest/ # CLI to ingest Automerge documents into Subduction\n└── automerge_subduction_wasm/  # Wasm wrapper for the full Automerge + Subduction stack\n```\n\n### Testing\n\nThe project uses several testing strategies:\n\n| Strategy             | Description                                                     |\n|----------------------|-----------------------------------------------------------------|\n| Unit Tests           | Standard `cargo test` for Rust code                             |\n| Property-based Tests | [`bolero`] for fuzz testing                                     |\n| E2E Tests            | Playwright tests for Wasm bindings (see `subduction_wasm/e2e/`) |\n| Integration Tests    | Transport connection tests (WebSocket, HTTP long-poll, Iroh)   |\n\n\u003c!-- Internal Links --\u003e\n\n[CLI README]: ./subduction_cli/README.md#running-as-a-system-service\n[Sedimentree]: ./sedimentree_core\n\n\u003c!-- External Links --\u003e\n\n[Automerge]: https://automerge.org/\n[`bolero`]: https://github.com/camshaft/bolero\n[CRDTs]: https://crdt.tech/\n[Ink \u0026 Switch]: https://www.inkandswitch.com/\n[sedimentree-design]: https://github.com/inkandswitch/keyhive/blob/main/design/sedimentree.md\n[wasm-pack]: https://rustwasm.github.io/wasm-pack/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finkandswitch%2Fsubduction","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finkandswitch%2Fsubduction","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finkandswitch%2Fsubduction/lists"}