Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/delendum-xyz/zk-benchmarking
comparing the performance of different zero-knowledge proof libraries
https://github.com/delendum-xyz/zk-benchmarking
Last synced: about 1 month ago
JSON representation
comparing the performance of different zero-knowledge proof libraries
- Host: GitHub
- URL: https://github.com/delendum-xyz/zk-benchmarking
- Owner: delendum-xyz
- Created: 2022-11-07T17:00:00.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-10-15T05:56:12.000Z (about 1 year ago)
- Last Synced: 2024-04-27T01:34:27.662Z (8 months ago)
- Language: Rust
- Homepage:
- Size: 13.5 MB
- Stars: 124
- Watchers: 9
- Forks: 15
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-risc0 - zk-benchmarking
- awesome-risc0 - zk-benchmarking
README
# ZK System Benchmarking
zk-benchmarking is a suite of benchmarks designed to compare different zero-knowledge proof libraries. Zero-knowledge proof libraries are used to enable privacy-preserving transactions and other cryptographic protocols, and it's important to be able to compare the performance of different implementations to choose the best one for a given use case.
With zk-benchmarking, you can run a suite of standardized benchmarks against different zero-knowledge proof libraries and see how they perform in terms of speed, memory usage, and other metrics. This can help you make informed decisions about which library is the best choice for your needs.
Features:
- A collection of standard benchmarks for comparing the performance of different zero-knowledge proof libraries.
- The ability to run the benchmarks against multiple libraries and compare the results.
- Outputs results in an easy-to-read format, including graphs and tables.
## ZK systems
Currently, the following ZK systems are benchmarked.
| System | ZKP System | Default Security Level |
| ----------------------------------------------------------- | :--------: | :-------------------------------------------------------------------------------------------------------------------: |
| [Polygon Miden](https://github.com/0xPolygonMiden/miden-vm) | STARK | [96 bits](https://github.com/maticnetwork/miden/blob/e941cf8dc6397a830d9073c8730389248e82f8e1/air/src/options.rs#L29) |
| [RISC Zero](https://github.com/risc0/risc0/) | STARK | [100 bits](https://github.com/risc0/risc0/#security) |
## Principles
### Relevant
Our benchmarks measure the performance of realistic, relevant tasks and use cases. This allows third-party engineers to estimate how each proof system would perform in their application.
### Neutral
We do not favor or disfavor any particular proof system. We seek to allow each proof system to show its best features in a variety of realistic test cases.
### Idiomatic
We allow each proof system to interpret each benchmark in whatever way makes the most sense for that system. This allows each proof system to showcase its performance when being used idiomatically.
### Reproducible
Our measurements are automated and reproducible.
## Factors
What are the relevant factors when picking a ZK system?
### Performance
We measure execution time and memory requirements in various standard hardware environments, thereby allowing each proof system to showcase its real-world behavior when running on commonly available systems.
For each benchmark, we measure the performance of these tasks:
- Proof generating
- Verifying a valid proof
- Rejecting an invalid proof
### Security
- What is the security model?
- How many "bits of security" does the system offer?
- Is it post-quantum?
- What hash functions does it support?
### Ease of building new apps
- How hard is it to write new apps for the platform?
- Does it require custom circuits?
- Does it support custom circuits?
- Are there libraries and developer tools? How mature are they?
### Upgradability
- Is the VM tightly coupled to its cryptographic core? Or is there an abstraction layer between them?
If a breakthrough in ZKPs took place tomorrow, would the VM be able to incorporate the new advances without breaking existing apps?
## Benchmarks
We start with smaller computations and will eventually move on to larger end-to-end scenarios (e.g., the integrity of modified images).
### Iterated hashing
(Scenario type: building block)
Iterated hashing is an essential building block for Merkle tree structures and whenever one needs to succinctly commit larger amounts of data.
To benchmark iterative hashing we compute a _hash chain_ as `H(H(H(...H(x))))`, where `H()` is a cryptographic hash function, for some input `x`. As input `x`, we chose a 32-bytes input `[0_u8; 32]` and the number of invocations of `H()` defines the length of the hash chain.
#### Prover performance
The table below shows the time it takes to generate a proof for a hash chain of a given length using a given hash function. This time includes the time needed to generate the witness for the computation. The time shown is in **seconds**.
Prover time (sec)
SHA256
BLAKE3
RP64_256
10
100
10
100
100
1000
Apple M2 (4P + 4E cores), 24GB RAM
Miden VM
1.86
17.14
0.92
8.26
0.05
0.25
RISC Zero
1.29
5.30
AWS Graviton 3 (64 cores), 128 GB RAM
Miden VM
0.49
3.99
0.33
2.06
0.05
0.13
RISC Zero
0.40
1.59
Dell XPS 15 9500 (i7-10875H CPU @ 2.30GHz × 16), 32 GB RAM
Miden VM
3.85
37.79
1.59
17.37
0.07
0.53
RISC Zero
1.83
7.26
A few notes:
- For RISC Zero the native hash function is SHA256, while for Miden VM it is Rescue Prime.
- On Apple-based systems, RISC Zero prover can take advantage of GPU resources.
#### Verifier performance
The table below shows the time it takes to verify a proof of correctly computing a hash chain of a given length and a given hash function. The time shown is in **milliseconds**.
Verifier time (ms)
SHA256
BLAKE3
RP64_256
10
100
10
100
100
1000
Apple M2 (4P + 4E cores), 24GB RAM
Miden VM
2.42
3.73
2.56
2.52
2.28
2.42
RISC Zero
1.92
2.44
AWS Graviton 3 (64 cores), 128 GB RAM
Miden VM
3.26
3.54
3.24
3.47
2.81
3.04
RISC Zero
3.03
4.05
Dell XPS 15 9500 (i7-10875H CPU @ 2.30GHz × 16), 32 GB RAM
Miden VM
3.65
3.18
2.96
3.78
2.87
3.13
RISC Zero
3.81
6.39
#### Proof size
The table below shows the size of a generated proof in **kilobytes**. Proof sizes do not depend on the platform used to generate proofs.
Proof size (KB)
SHA256
BLAKE3
RP64_256
10
100
10
100
100
1000
Miden VM
87.7
105.0
81.3
98.4
56.2
71.0
RISC Zero
183.4
205.1
---
### Merkle inclusion
(Scenario type: building block)
Proving the inclusion of a leaf to a Merkle tree is an essential building block for real world applications. For the scenario we construct a Merkle path of depth 32 and verify the inclusion of as many leafs as the current `job_size`. So, for job_size=10 we verify 10 Merkle paths of depth 32.
#### Prover performance
The table below shows the time it takes to generate a proof for a given number of merkle path inclusion verifications of depth 32 using a given hash function. This time includes the time needed to generate the witness for the computation. The time shown is in **seconds**.
Prover time (sec)
RP64_256
1
10
100
1000
Apple M2 (4P + 4E cores), 24GB RAM
Miden VM
0.034
0.081
0.495
8.46
AWS Graviton 3 (64 cores), 128 GB RAM
Miden VM
1
1
1
1
### Recursion
_Coming soon!_
## Running the benchmarks
### Requirements
- [`cargo`](https://doc.rust-lang.org/stable/cargo/)
### Running all of the benchmarks
```console
$ ./all.sh
```
## Contributing
If you would like to contribute to zk-benchmarking, please fork the repository and submit a pull request with your changes. All contributions are welcome, including new benchmarks and improvements to existing ones.