{"id":51024190,"url":"https://github.com/nativebpm/wasman","last_synced_at":"2026-06-21T18:02:12.034Z","repository":{"id":362125195,"uuid":"1256567476","full_name":"nativebpm/wasman","owner":"nativebpm","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-03T20:02:12.000Z","size":242,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-03T22:04:16.948Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nativebpm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-01T22:42:54.000Z","updated_at":"2026-06-03T20:02:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/nativebpm/wasman","commit_stats":null,"previous_names":["nativebpm/durable-wasm","nativebpm/wasman"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/nativebpm/wasman","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nativebpm%2Fwasman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nativebpm%2Fwasman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nativebpm%2Fwasman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nativebpm%2Fwasman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nativebpm","download_url":"https://codeload.github.com/nativebpm/wasman/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nativebpm%2Fwasman/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34620358,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-21T02:00:05.568Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-21T18:02:10.935Z","updated_at":"2026-06-21T18:02:12.022Z","avatar_url":"https://github.com/nativebpm.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wasman: Durable WebAssembly (WASM) Execution Engine\n\nA robust, highly-reusable, and lightweight **Durable Execution Engine** built on Go, WebAssembly (WASM), and the `wazero` runtime. It provides fault-tolerant, stateful execution of custom business logic inside a secure sandbox with automatic memory snapshotting, failure recovery, and memory-efficient streaming, completely free of CGO and glibc dependencies.\n\n---\n\n## The Durable Execution Philosophy\n\nModern distributed architectures often require executing long-running or multi-step business logic (such as workflows, integrations, and orchestrations) that must survive infrastructure failures. Traditional approaches rely on complex database state machines or heavy external orchestrators.\n\n**Wasman** addresses this by leveraging WebAssembly's sandboxed linear memory:\n1. **Host-Guest Isolation**: The guest business logic is compiled into a `.wasm` module (compiled via TinyGo/Go) and executed inside the pure-Go `wazero` runtime.\n2. **Stateless Host, Stateful Storage**: The execution host runs the virtual machine sandboxes. It remains stateless. All state (linear memory snapshots, execution logs) is persisted in S3 or local file snapshot stores.\n3. **Black Box API**: Developers using the platform interact only with high-level client APIs (generated via Protobuf/GRPC or client SDKs). The underlying complexity of WebAssembly, snapshotting, and transaction control is entirely hidden.\n\n```\n       [ Client / API Request ] (StartProcess / CompleteTask)\n                 │\n                 ▼\n       ┌──────────────────┐\n       │  WorkflowEngine  │ (Host Orchestrator)\n       └─────────┬────────┘\n                 │\n      ┌──────────┴──────────┐\n      ▼                     ▼\n┌───────────┐         ┌───────────┐\n│ bpmn_vm   │ (WASM)  │  worker   │ (WASM Business Logic)\n│ Interpreter         │  Executor │\n└─────┬─────┘         └─────┬─────┘\n      │                     │\n      └──────────┬──────────┘\n                 │ (State \u0026 Memory Checkpoints)\n                 ▼\n       ┌──────────────────┐\n       │  Snapshot Store  │ (Gzip-compressed Snapshots \u0026 Deltas)\n       └─────────┬────────┘\n                 │\n        ┌────────┴────────┐\n        ▼                 ▼\n   [ S3 Storage ]   [ File Storage ]\n```\n\n---\n\n## Technical Features \u0026 Architectural Design\n\n### 1. Strict Storage Compression\nCheckpointing large WebAssembly modules generates snapshots of their linear memory (typically multiples of 64KB pages). To prevent S3/disk space bloat under high throughput:\n- **Gzip Compression**: Snapshots, page deltas, and oplogs are transparently compressed using the standard gzip format.\n- **Strict Format Enforcement**: All reads enforce the presence of gzip compression. Raw uncompressed snapshots are not supported, ensuring consistent storage compression benefits across all process states.\n\n\n### 2. $O(1)$ RAM Stream-first I/O\nFor high-performance data processing (e.g., streaming files, large JSON/CSV payloads):\n- Data is transferred directly to/from WASM linear memory in chunks using stream buffers.\n- This guarantees constant memory footprint ($O(1)$ RAM) regardless of payload size, avoiding heap exhaustion and high GC pause times.\n- All communications are executed fully in-memory via user-provided download/upload stream handlers, entirely avoiding network loopbacks and TCP port exposures.\n\n### 3. Page-Level Delta Snapshots\nInstead of writing a full multi-megabyte memory snapshot on every single checkpoint:\n- **Hashing**: Wasman uses FNV-64a to hash individual 64KB memory pages.\n- **Deltas**: On subsequent checkpoints, it only writes pages that have actually been modified (dirty pages), drastically reducing I/O latency.\n\n### 4. Optimistic Concurrency Control (OCC)\nIn high-concurrency environments where multiple orchestrator nodes might receive step execution triggers for the same process instance:\n- **S3 ETag Headers**: The S3 storage client uses native HTTP `If-Match` headers.\n- **State Integrity**: If another node has updated the snapshot in the meantime, the write fails with an OCC conflict, preventing state corruption.\n\n---\n\n## Defeated Corner Cases (Failure \u0026 Crash Recovery)\n\nWasman guarantees durable execution by checkpointing and restoring state across node crashes:\n\n### Scenario: Server Crash During Execution\n1. **Before Step**: The VM starts executing a process. It hits a checkpoint (e.g., before an external API call or a User Task wait state).\n2. **Checkpointing**:\n   - The engine halts execution.\n   - It captures the current state, writing a `Full Snapshot` or `Delta Snapshot` to S3.\n   - It logs the expected step transition.\n3. **Crash**: The host server crashes (e.g., hardware failure, OOM, or manual redeployment).\n4. **Resumption**:\n   - Another node receives the request to resume.\n   - It reads the metadata, loads the compiled WASM binary, and pulls the compressed snapshot.\n   - It restores the linear memory of the WASM VM to the exact page-level state of the last checkpoint.\n   - It replays the execution logs (Oplog) to restore transient state and resumes execution seamlessly.\n\n---\n\n## Directory Structure\n\n- [wasman.go](wasman.go): WASM compilation, runtime setup, and engine execution loops.\n- [compress.go](compress.go): Transparent Gzip compression utilities.\n- [fs_store.go](fs_store.go): Local file-system snapshot store with optional compression.\n- [s3_store.go](s3_store.go): S3-compatible object snapshot store with OCC.\n- [types.go](types.go): Common structures, interfaces, configurations, and error mappings.\n- [examples/](examples/):\n  - [process-csv/](examples/process-csv/): High-throughput CSV mapping with simulated crash recovery and $O(1)$ RAM usage.\n  - [camunda/](examples/camunda/): Integration with Camunda 7 External Tasks.\n  - [temporal/](examples/temporal/): CRM/Math activities in a simulated Temporal environment.\n  - [gotenberg-telegram/](examples/gotenberg-telegram/): Streaming PDF generation bot integration.\n  - [s3-store/](examples/s3-store/): Direct S3/MinIO snapshotting baseline demonstration.\n  - [in-memory-channel/](examples/in-memory-channel/): Purely in-memory host-guest stream data exchange bypassing TCP loopbacks entirely.\n  - [safe-task/](examples/safe-task/): Execution of sandboxed tasks utilizing the safe, high-level RunTask runner utility.\n  - [wasm-inspector/](examples/wasm-inspector/): Low-level WebAssembly inspect utility executing guest WASM binaries under customized WASI settings.\n\n\n---\n\n## Getting Started\n\n### Running Tests\nTo run unit and integration tests for the core engine:\n```bash\ngo test -v .\n```\n\n### Running the CSV Crash Demonstration\nThe `process-csv` example demonstrates a complete crash-and-restore cycle:\n1. Compile the WASM worker:\n   ```bash\n   make build-worker\n   ```\n2. Run the CSV pipeline:\n   ```bash\n   make run-csv-example\n   ```\nThis will:\n- Start a mock HTTP server.\n- Initiate execution of the CSV pipeline.\n- Simulate a host crash on the first checkpoint.\n- Verify the compressed snapshot is written to disk.\n- Restore the memory from the snapshot and complete the execution successfully.\n\n---\n\n## API Usage Example\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"github.com/nativebpm/wasman\"\n)\n\nfunc main() {\n\t// 1. Initialize snapshot store with compression enabled\n\tstore := \u0026wasman.FileSnapshotStore{\n\t\tDir:         \"snapshots\",\n\t\tCompression: true,\n\t}\n\n\t// 2. Define stream handlers\n\tdownloadHandler := func() ([]byte, error) {\n\t\treturn []byte(\"my input data stream\"), nil\n\t}\n\tuploadHandler := func(payload []byte) error {\n\t\tfmt.Printf(\"Received output payload: %s\\n\", string(payload))\n\t\treturn nil\n\t}\n\n\t// 3. Execute session using the high-level Fluent Runner API.\n\t// If a snapshot exists under this session ID, memory is restored automatically.\n\tcrashed, err := wasman.NewRunner().\n\t\tWithWasmPath(\"worker.wasm\").\n\t\tWithStore(store).\n\t\tWithSessionID(\"my-session-id\").\n\t\tWithEntrypoint(\"run\").\n\t\tWithDownloadHandler(downloadHandler).\n\t\tWithUploadHandler(uploadHandler).\n\t\tRun()\n\n\tif err != nil {\n\t\tif crashed {\n\t\t\tfmt.Println(\"Execution suspended at checkpoint.\")\n\t\t} else {\n\t\t\tfmt.Printf(\"Execution failed: %v\\n\", err)\n\t\t}\n\t} else {\n\t\tfmt.Println(\"Execution completed successfully!\")\n\t}\n}\n```\n\n## Performance \u0026 Benchmarks\n\nDetailed CPU and memory benchmark profiles (focusing on the optimized warm resume JIT execution performance of ~38 µs) are available in the [Benchmarks \u0026 Profiling Profile](docs/benchmarks.md) document.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnativebpm%2Fwasman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnativebpm%2Fwasman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnativebpm%2Fwasman/lists"}