{"id":51465404,"url":"https://github.com/ray-project/distributed-zkml","last_synced_at":"2026-07-06T10:30:39.821Z","repository":{"id":331392459,"uuid":"1118574151","full_name":"ray-project/distributed-zkml","owner":"ray-project","description":"Distributed Proofs with ZMKL and Ray","archived":false,"fork":false,"pushed_at":"2026-01-05T03:57:04.000Z","size":14141,"stargazers_count":2,"open_issues_count":4,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-07T12:19:28.182Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ray-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-18T01:04:46.000Z","updated_at":"2026-01-05T03:57:08.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ray-project/distributed-zkml","commit_stats":null,"previous_names":["ray-project/distributed-zkml"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ray-project/distributed-zkml","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fdistributed-zkml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fdistributed-zkml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fdistributed-zkml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fdistributed-zkml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ray-project","download_url":"https://codeload.github.com/ray-project/distributed-zkml/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fdistributed-zkml/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35187624,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-06T02:00:07.184Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-06T10:30:39.095Z","updated_at":"2026-07-06T10:30:39.813Z","avatar_url":"https://github.com/ray-project.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# distributed-zkml\n\nExtension of [zkml](https://github.com/uiuc-kang-lab/zkml) for distributed proving using Ray, layer-wise partitioning, and Merkle trees.\n\n\u003e **⚠️ Status Note:** This is an experimental research project. Also consider [zk-torch](https://github.com/uiuc-kang-lab/zk-torch).\n\n## Completed Milestones\n\n1. ~~**Make Merkle root public**: Add root to public values so next chunk can verify it~~ Done\n2. ~~**Complete proof generation**: Connect chunk execution to actual proof generation ([#8](https://github.com/ray-project/distributed-zkml/issues/8))~~ Done\n3. ~~**Ray-Rust integration**: Connect Python Ray workers to Rust proof generation ([#9](https://github.com/ray-project/distributed-zkml/issues/9))~~ Done\n4. ~~**GPU acceleration**: ICICLE GPU backend for MSM operations ([#10](https://github.com/ray-project/distributed-zkml/issues/10))~~ Done - see [GPU Acceleration](#gpu-acceleration)\n\n---\n\n## Table of Contents\n\n- [Status and Limitations](#status-and-limitations)\n- [Overview](#overview)\n- [Implementation](#implementation)\n  - [How Distributed Proving Works](#how-distributed-proving-works)\n  - [Security Model and Trust Boundaries](#security-model-and-trust-boundaries)\n  - [Structure](#structure)\n- [Requirements](#requirements)\n- [Quick Start](#quick-start)\n- [GPU Acceleration](#gpu-acceleration)\n- [Testing](#testing)\n- [References](#references)\n\n---\n\n## Status and Limitations\n\n### Project Status\n\nThis project implements a **Ray-based distributed proving approach** for zkml. It is experimental research code and should be considered useful for studying alternative approaches to zkML parallelization. The current status lacks formal security analysis and proof composition.\n\n### Known Limitations\n\n**Proof Composition**: This implementation generates separate proofs per chunk. It does not implement recursive proof composition or aggregation. Verifiers must check O(n) proofs rather than O(1), limiting succinctness.\n\n**Trust Domain**: \n- **Merkle trees provide privacy for proof readers, not compute providers**: The prover must know all weights and activations to generate a valid ZK proof. Merkle trees hide intermediate values from people *reading the published proof*, not from the compute provider *during execution*.\n- **Multi-party security requires different trust domains**: Security only applies when chunks are distributed across different trust domains (e.g., your servers + AWS), not just different AWS regions.\n- **Comparison to TEE/FHE/MPC**: Trusted Execution Environments (TEEs), Fully Homomorphic Encryption (FHE), or Multi-Party Computation (MPC) provide stronger privacy guarantees but at significant costs that are beyond the threshold of scalable AI applications.\n\n### When to Use This\n\n**Consider this project if:**\n- Researching alternative zkml parallelization approaches\n- Need examples of Ray integration for cryptographic workloads\n- Studying Merkle-based privacy for intermediate computations\n- Building distributed halo2 proving (not zkML-specific)\n- **Use case**: You trust compute providers but want to limit public proof exposure, or model is partitioned across multiple non-colluding organizations\n\n**Use alternatives if:**\n- Need to hide data from compute providers themselves → Requires TEEs/FHE/MPC \n- Need single aggregated proof → Consider [zk-torch](https://github.com/uiuc-kang-lab/zk-torch)\n\n---\n\n## Overview\n\nThis repository extends zkml (see [ZKML paper](https://ddkang.github.io/papers/2024/zkml-eurosys.pdf)) with distributed proving capabilities. zkml provides an optimizing compiler from TensorFlow to halo2 ZK-SNARK circuits.\n\ndistributed-zkml adds:\n- **Layer-wise partitioning**: Split ML models into chunks for parallel proving across GPUs via Ray\n- **Merkle tree commitments**: Hash intermediate activations with Poseidon; only publish root in proof\n- **ICICLE GPU acceleration**: Hardware-accelerated MSM operations\n\n### Comparison to zkml\n\n| Feature | zkml | distributed-zkml |\n|---------|------|------------------|\n| Architecture | Single-machine | Distributed across GPUs |\n| Scalability | Single GPU memory | Horizontal scaling |\n| Privacy | Outputs public | Intermediate values hidden from proof readers via Merkle trees |\n\n## Implementation\n\n### How Distributed Proving Works\n\n1. **Model Partitioning**: Split model into chunks at layer boundaries\n2. **Parallel Execution**: Each chunk runs on a separate GPU via Ray\n3. **Merkle Commitments**: Hash intermediate outputs with Poseidon, only root is public\n4. **On-Chain**: Publish only the Merkle root (O(1) public values vs O(n) without)\n\n**Note**: Each chunk produces a separate proof. This implementation does not aggregate proofs into a single succinct proof. Verifiers must check all chunk proofs individually (O(n) verification time). For single-proof aggregation, see [zk-torch](https://github.com/uiuc-kang-lab/zk-torch)'s accumulation-based approach.\n\n```\nModel: 9 layers -\u003e 3 chunks\n  Chunk 1: Layers 0-2 -\u003e GPU 1 -\u003e Hash A\n  Chunk 2: Layers 3-5 -\u003e GPU 2 -\u003e Hash B\n  Chunk 3: Layers 6-8 -\u003e GPU 3 -\u003e Hash C\n\nMerkle Tree:\n        Root (public)\n       /    \\\n    Hash(AB) Hash C\n    /    \\\n Hash A  Hash B\n```\n\n### Trust Boundaries\n\n#### What Merkle Trees Provide\n\n| Scenario | Hidden? | Explanation |\n|----------|---------|-------------|\n| Proof readers reconstructing weights via model inversion | Yes | Intermediate activations are hashed, not exposed in proof |\n| Compute provider seeing weights during execution | No | Provider must have weights to generate ZK proof |\n| Compute provider seeing intermediate activations during execution | No | Provider computes them |\n\n**Key insight:** Merkle trees hide intermediate values from people *reading the published proof*, not from the compute provider *during execution*. The prover must know all values to generate a valid ZK proof.\n\n#### Multi-Party Proving and Trust Domains\n\nSecurity depends on **trust domains**, not physical location:\n\n| Setup | Trust Domains | What's Private |\n|-------|---------------|----------------|\n| Single AWS account (any region) | 1 | Nothing from AWS — they control all regions |\n| Your servers + AWS | 2 | Your portion's weights never sent to AWS |\n| AWS + Google + Azure | 3 | Each provider sees only their chunk (assuming non-collusion) |\n\n**Multi-party benefit:** If model is partitioned across different trust domains (e.g., your servers + AWS), no single party has the full model. Combined with Merkle trees, this provides layered privacy:\n- **Partitioning** → limits what any single provider can access\n- **Merkle trees** → limits what proof readers can observe\n\n#### Comparison with ZKTorch\n\n| Aspect | distributed-zkml | ZKTorch |\n|--------|------------------|---------|\n| Scaling strategy | Horizontal (more machines via Ray) | Vertical (proof compression via Mira) |\n| Final output | N separate proofs | 1 accumulated proof |\n| Verification cost | O(N) proofs to verify | O(1) single proof |\n| Intermediate privacy | Merkle trees hide from proof readers | Exposed in proof |\n| Base system | halo2 (~30M param limit) | Custom pairing-based (6B params tested) |\n\n**These approaches are orthogonal** — could theoretically combine Ray parallelism with Mira accumulation.\n\n### Structure\n\n```\ndistributed-zkml/\n├── python/                 # Python wrappers for Rust prover\n├── tests/                  # Distributed proving tests\n└── zkml/                   # zkml (modified for Merkle + chunking)\n    ├── src/bin/prove_chunk.rs\n    └── testing/\n```\n\n## Requirements\n\n### Docker (Recommended)\n\nJust Docker and Docker Compose. Everything else is in the container.\n\n### Native Build\n\n| Dependency | Notes |\n|------------|-------|\n| Rust (nightly) | Install via [rustup](https://rustup.rs/) |\n| Python \u003e=3.10 | |\n| pip | `pip install -e .` |\n| Build tools | Linux: `build-essential pkg-config libssl-dev`; macOS: Xcode CLI |\n\n**Python deps** (installed via `pip install -e .`):\n- `ray[default]\u003e=2.31.0`\n- `msgpack`, `numpy`\n\n**Optional**: NVIDIA GPU + CUDA 12.x + ICICLE backend for GPU acceleration\n\n---\n\n## Quick Start\n\n### Docker\n\n```bash\ndocker compose build dev\ndocker compose run --rm dev\n# Inside container:\ncd zkml \u0026\u0026 cargo test --test merkle_tree_test -- --nocapture\n```\n\n### Native\n\n```bash\n# Install Rust\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n\n# Build\ncd zkml \u0026\u0026 rustup override set nightly \u0026\u0026 cargo build --release \u0026\u0026 cd ..\n\n# Python deps\npip install -e .\n```\n\n---\n\n## GPU Acceleration\n\nUses [ICICLE](https://github.com/ingonyama-zk/icicle) for GPU-accelerated MSM (Multi-Scalar Multiplication).\n\n### Requirements\n\n- NVIDIA GPU (tested on A10G/T4, compatible with A100/H100)\n- CUDA 12.x\n- Ubuntu 20.04+\n\n### Setup\n\n```bash\n# 1. Download ICICLE backend (Ubuntu 22.04 - use ubuntu20 for 20.04)\ncurl -L -o /tmp/icicle.tar.gz \\\n  https://github.com/ingonyama-zk/icicle/releases/download/v3.1.0/icicle_3_1_0-ubuntu22-cuda122.tar.gz\n\n# 2. Install\nmkdir -p ~/.icicle \u0026\u0026 tar -xzf /tmp/icicle.tar.gz -C /tmp \u0026\u0026 cp -r /tmp/icicle/lib/backend ~/.icicle/\n\n# 3. Set env var (add to ~/.bashrc)\nexport ICICLE_BACKEND_INSTALL_DIR=~/.icicle/backend\n\n# 4. Build with GPU\ncd zkml \u0026\u0026 cargo build --release --features gpu\n\n# 5. Verify\ncargo test --test gpu_benchmark_test --release --features gpu -- --nocapture\n```\n\nExpected output:\n```\nRegistered devices: [\"CUDA\", \"CPU\"]\nSuccessfully set CUDA device 0\n```\n\n### Benchmarks (T4)\n\n| Size | GPU MSM Time | Throughput |\n|------|--------------|------------|\n| 2^14 (16K) | 6.5ms | 2.5M pts/sec |\n| 2^16 (65K) | 7.9ms | 8.3M pts/sec |\n| 2^18 (262K) | 13ms | 19.5M pts/sec |\n\n### FFT/NTT Notes\n\n- **Measure FFT time**: `HALO2_FFT_STATS=1`\n- **GPU NTT (experimental)**: `HALO2_USE_GPU_NTT=1` - currently slower due to conversion overhead\n\n---\n\n## Testing\n\n### Distributed Proving\n\n```bash\n# Simulation (fast)\npython tests/simple_distributed.py \\\n    --model zkml/examples/mnist/model.msgpack \\\n    --input zkml/examples/mnist/inp.msgpack \\\n    --layers 4 --workers 2\n\n# Real proofs\npython tests/simple_distributed.py ... --real\n```\n\n### Rust Tests\n\n```bash\ncd zkml\ncargo test --test merkle_tree_test --test chunk_execution_test -- --nocapture\n```\n\n### CI\n\nRuns on PRs to `main`/`dev`: builds zkml, runs tests (~3-4 min). GPU tests excluded to save costs.\n\n---\n\n## References\n\n- [ZKML Paper](https://ddkang.github.io/papers/2024/zkml-eurosys.pdf) (EuroSys '24) - Original zkml framework\n- [zkml Repository](https://github.com/uiuc-kang-lab/zkml) - Base framework this project extends\n- [zk-torch](https://github.com/uiuc-kang-lab/zk-torch) - Alternative approach using proof accumulation/folding.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fray-project%2Fdistributed-zkml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fray-project%2Fdistributed-zkml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fray-project%2Fdistributed-zkml/lists"}