{"id":19155005,"url":"https://github.com/succinctlabs/zkvm-perf","last_synced_at":"2025-04-19T17:31:15.835Z","repository":{"id":238901616,"uuid":"797654641","full_name":"succinctlabs/zkvm-perf","owner":"succinctlabs","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-16T22:28:08.000Z","size":17875,"stargazers_count":9,"open_issues_count":5,"forks_count":10,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-10-18T22:38:23.126Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/succinctlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-08T09:02:07.000Z","updated_at":"2024-10-17T12:50:00.000Z","dependencies_parsed_at":null,"dependency_job_id":"32e9218e-8c50-4c3d-a4e7-a7b87ae2c7fc","html_url":"https://github.com/succinctlabs/zkvm-perf","commit_stats":null,"previous_names":["succinctlabs/zkvm-perf"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/succinctlabs%2Fzkvm-perf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/succinctlabs%2Fzkvm-perf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/succinctlabs%2Fzkvm-perf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/succinctlabs%2Fzkvm-perf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/succinctlabs","download_url":"https://codeload.github.com/succinctlabs/zkvm-perf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223804543,"owners_count":17205742,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T08:29:07.715Z","updated_at":"2025-04-19T17:31:15.829Z","avatar_url":"https://github.com/succinctlabs.png","language":"Rust","funding_links":[],"categories":["bench"],"sub_categories":[],"readme":"# ZKVM-Perf And You 🫵🏻\n\nWelcome to ZKVM-Perf, a powerful benchmarking tool for ZKVM implementations. This guide covers both automated workflow-based benchmarking and manual benchmarking processes.\n\n## Automated Benchmarking\n\n### Quick Start Guide\n\n1. Go to the [Actions tab](https://github.com/succinctlabs/zkvm-perf/actions) in the ZKVM-Perf repository.\n2. Click on the \"Execute ZKVM-Perf (Matrix)\" workflow.\n3. Click the \"Run workflow\" button.\n4. (Optional) Customize parameters or use defaults.\n5. Click the green \"Run workflow\" button at the bottom.\n6. Wait for all matrix jobs to complete.\n7. Check individual job results and download CSV artifacts for detailed analysis.\n\n### Workflow Details\n\nThe benchmarking process is split into two main workflows:\n\n1. `adhoc-matrix.yml`: Orchestrates the overall benchmarking process.\n2. `run-on-runner.yml`: Executes the actual benchmarks on EC2 instances.\n\n#### adhoc-matrix.yml\n\nThis workflow sets up the benchmarking environment and triggers individual benchmark runs.\n\n##### Inputs\n\n- `provers`: Provers to use (comma-separated, default: 'sp1')\n- `programs`: Programs to benchmark (comma-separated, default: 'loop10k, loop100k, loop1m, loop3m, loop10m, loop30m, loop100m, fibonacci,tendermint,reth1,reth2')\n- `filename`: Filename for the benchmark (default: 'benchmark')\n- `trials`: Number of trials to run (default: '1')\n- `sp1_ref`: SP1 reference (commit hash or branch name, default: 'dev')\n- `additional_params`: Additional parameters as JSON (default: '{\"hashfns\":\"poseidon\",\"shard_sizes\":\"22\"}')\n\n##### Matrix Strategy\n\nThe workflow runs benchmarks on two types of EC2 instances:\n\n- GPU: g6.16xlarge\n- CPU: r7i.16xlarge\n\n#### run-on-runner.yml\n\nThis workflow is triggered by `adhoc-matrix.yml` and runs the actual benchmarks on the specified EC2 instance.\n\n##### Key Steps\n\n1. Sets up the Docker environment.\n2. Builds the Docker image with the specified SP1 reference.\n3. Runs the benchmark using the `sweep.py` script.\n4. Uploads the benchmark results as artifacts.\n\n### Running Automated Benchmarks\n\n1. **Navigate to the Actions Tab**\n   Go to the [Actions tab](https://github.com/succinctlabs/zkvm-perf/actions) in the repository.\n\n2. **Select the Workflow**\n   Click on \"Execute ZKVM-Perf (Matrix)\".\n\n3. **Configure the Run**\n\n   - You can use the default settings for a quick start.\n   - Customize inputs as needed.\n\n4. **Start the Benchmark**\n   Click \"Run workflow\".\n\n5. **Monitor Progress**\n\n   - The workflow will start two jobs: one for GPU and one for CPU.\n   - Each job will trigger a separate `run-on-runner` workflow.\n\n6. **Access Results**\n   - Once complete, each job will upload its results as an artifact.\n   - Download the artifacts to analyze the benchmark data.\n   - A combined results file will also be available.\n\n## Manual Benchmarking\n\n### Setup\n\n1. Install Rust:\n\n   ```sh\n   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n   rustup install nightly\n   ```\n\n2. Install the [SP1 toolchain](https://docs.succinct.xyz/getting-started/install.html):\n\n   ```sh\n   curl -L https://sp1.succinct.xyz | bash\n   source ~/.bashrc\n   sp1up\n   cargo prove --version\n   ```\n\n3. Install the [Risc0 toolchain](https://dev.risczero.com/api/zkvm/install):\n\n   ```sh\n   curl -L https://risczero.com/install | bash\n   source ~/.bashrc\n   rzup install\n   cargo risczero --version\n   ```\n\n4. Install [Docker](https://docs.docker.com/engine/install/ubuntu/).\n\n5. If using NVIDIA GPUs, install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).\n\n**Note:** Run one round of a small program (e.g., Fibonacci) to download the R0 docker image before benchmarking to avoid affecting benchmark times.\n\n**Note:** On Ubuntu 22.04, you might need to install libssl1.0 for the Risc0 toolchain. Follow these [instructions](https://stackoverflow.com/questions/72133316/libssl-so-1-1-cannot-open-shared-object-file-no-such-file-or-directory/73604364#73604364).\n\n### Running a Manual Sweep\n\nTo conduct a sweep of the benchmarks:\n\n```sh\npython3 sweep.py [options]\n```\n\nAvailable options:\n\n- `--filename`: Filename for the benchmark (default: \"benchmark\")\n- `--trials`: Number of trials to run (default: 1)\n- `--programs`: List of programs to benchmark (choices: loop10k, loop100k, loop1m, loop3m, loop10m, loop30m, loop100m, fibonacci, tendermint, reth1, reth2)\n- `--provers`: List of provers to use (choices: sp1, risc0)\n- `--hashfns`: List of hash functions to use (currently only poseidon is supported)\n- `--shard-sizes`: List of shard sizes to use\n- `--block-1`: Block number for reth1 (default: \"17106222\")\n- `--block-2`: Block number for reth2 (default: \"19409768\")\n\n### Running a Single Benchmark\n\nTo run a single benchmark:\n\n```sh\n./eval.sh \u003cprogram\u003e \u003cprover\u003e \u003chashfn\u003e \u003cshard_size\u003e \u003cfilename\u003e [block_number]\n```\n\nExamples:\n\n```\n./eval.sh fibonacci sp1 poseidon 22 benchmark\n./eval.sh fibonacci jolt-zkvm poseidon 22 benchmark\n./eval.sh fibonacci risc0 poseidon 22 benchmark\n./eval.sh reth sp1 poseidon 22 benchmark 19409768\n```\n\n## Analyzing Results\n\n- Each benchmark run produces a CSV file with detailed performance metrics.\n- The CSV includes the instance type, allowing for easy comparison between GPU and CPU performance.\n- Use the combined results file for a comprehensive view of all benchmarks.\n\n## Troubleshooting\n\nIf you encounter issues:\n\n1. Check the logs of both the matrix job and the individual runner jobs.\n2. Ensure your AWS credentials and permissions are correctly set up.\n3. Verify that the SP1 reference is valid and accessible.\n4. For GPU jobs, confirm that GPU support is properly configured in the EC2 instance.\n\n### Common Issues\n\nFor C++ compiler and library issues:\n\nUbuntu/Debian:\n\n```sh\nsudo apt update \u0026\u0026 sudo apt upgrade\nsudo apt install build-essential libc6\n```\n\nCentOS/RHEL:\n\n```\nsudo yum update\nsudo yum groupinstall \"Development Tools\"\nsudo yum install -y gcc-c++\n```\n\nFedora:\n\n```sh\nsudo dnf update\nsudo dnf groupinstall \"Development Tools\"\n```\n\nSetting up NVIDIA:\n\n```sh\nsudo nvidia-ctk runtime configure --runtime=docker\nsudo systemctl restart docker\n```\n\n## Contributing\n\nWe welcome contributions to improve ZKVM-Perf! If you encounter issues or have suggestions:\n\n1. Check existing issues in the repository.\n2. If your issue is new, create a detailed bug report or feature request.\n3. For code contributions, please submit a pull request with a clear description of your changes.\n\nHappy benchmarking! 🚀\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuccinctlabs%2Fzkvm-perf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuccinctlabs%2Fzkvm-perf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuccinctlabs%2Fzkvm-perf/lists"}