zkcollective/zk-harness

Benchmarking framework for general purpose zero-knowledge proofs languages and libraries

zk-harness's Issues

Improve benchmarking for Circom

Currently, we just use a bash script to perform the benchmarks. We should probably create a JavaScript runner and use a benchmarking framework

Update Input format

Update the JSON structure of inputs to contain more details.
Currently, the input JSONs contain only the input values for the circuits.
We might want to add a description field to describe the input, so we can improve the UI.

Old format:

{
    "PreImage": "68656c6c6f20776f726c64",
    "Hash":     "b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"
}

Proposed format:

{
    "Description": {
       "name": "X Bytes"
       "order": 0
    }
    "PreImage": "68656c6c6f20776f726c64",
    "Hash":     "b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"
}

@tumberger wdyt?

Bellman prepare vk in proving step

https://github.com/zkCollective/zk-Harness/blob/26b003ca06bf491ffcf15aa6b5dc0cf22d370718/frameworks/bellman_circuits/bellman_utils/src/lib.rs#L106C5-L106C58

Support gnark-crypto v0.10.0

Currently, gnark-crypto is version v0.9.1.
We should update to v0.10.0, e.g. SHA-256 only works for Groth16 with gnark-crypto v0.9.1.
For evaluating e.g. SHA-256 performance for PLONK and PLONK-FRI the following works (as set in the develop branch of gnark):
gnark v0.7.2
gnark-crypto v0.10.1

generalize Rust bench runner

The current Rust-based bench runner introduced in #9 serves as a good start, but is not generalized just yet. This should be refactored to be able to plug-and-play different frameworks for benchmarks.

starky verify / memory

currently, memory measurement for starky is not performed.

Reason - no implementation for serialization, the therefore fails with independent binaries and interrmediate serialization.

Time benchmark logic is sound.

Remove MaxRam and parse from txt memory file

https://github.com/zkCollective/zk-Harness/blob/26b003ca06bf491ffcf15aa6b5dc0cf22d370718/frameworks/gnark/cmd/groth16.go#L76C29-L76C29

`time` command: what version works with these options?

I cannot reproduce the circom benchmarks on my end because /usr/bin/time is not behaving as expected for me; it does not recognize the options -h -l -o. What version of time works with the circom benchmarks script? @StefanosChaliasos

introduce memory usage benchmarks for Rust-based frameworks

In private conversations with @tumberger we discussed 2 ways of measuring memory usage:

1. in-language memory benchmarks: this would require most likely require usage of jemalloc, which means we need to depend on libc directly.

2. external tools: we could also use /usr/bin/time or something like valgrind.

If we decide on 2), the logic between the test runner<>benchmarks would have to be separated, since in the current runner we are doing other tasks like writing to CSV, etc. which means we may not get the most accurate results.

Provide Nix build code

Hi! I want Nix build code so that I can more easily build this project and it can more easily be reproduced on other people's machines. I am interested in doing this issue as part of the Berkeley ZK hackathon.

Save multiple benchmark runs

To save multiple benchmark runs, we need to update the structure of the benchmarks directory to include a date or an incremental number.

extend preliminary jellyfish benchmarks.

#19 introduces a basic Rust program that runs a toy circuit with jellyfish, but right it only covers 1 circuit and 1 kind of operation (compile).

This should be extended to have the same coverage as the circom and gnark benches, in terms of the 1) types of benchmarks and 2) the operations.

Add qualitative table

Add a table with qualitative information for frameworks and curves supported by zk-Harness.

Add SHA-256 starky

Currently, SHA-256 in starky is not included in the benchmarks.

There is a fork of plonky2 that includes sha256 in the starky src directory (based on this previous fork that has been used in the benchmarks as published by celer network). However, there is no concrete test for correctness in this implementation.

Adding SHA-256 with a test for correctness for starky in the current set of benchmarks would be a great addition.

Merge zkalc backend

Merge with the zkalc backend for arithmetic / curve operations

• arkworks benchmarks
• blstrs benchmarks
• curve25519-dalek benchmarks
• pasta curves (halo2, zcash)
• zkcrypto
• gnark - the zkalc way (maintenance of benchmarks on gnark side)
• config specification
• compliance with log format + memory benchmarks (parser of criterion output?)

Circom/snarkjs TODOs

• Currently, we are benchmarking circom/snarkjs, using the time command. This way, the comparison is not equal to libraries as we have the nodejs overhead. However, this will most probably be negligible for large benchmarks.
• Add PLONK and FFLONK support
• Add support for C++ witness generation. In this case, should we also include the time for complying with the cpp code in the logs?
• Cores/Thread: Node uses one single thread in one core. Nevertheless, snarkjs (and the underlying library ffjavascript) use workers to perform operations; hence it isn't actually single-threaded. We could instrument snarkjs so it uses only a single worker. -- We decided yo use best available options
• Add proof size
• Use rapidsnark

Question on Benchmarking ZK-Circuits in Circom Integration?

In this paper Benchmarking ZK-Circuits in Circom:

We integrated the following signature schemes and hash functions into ZK-Harness: Poseidon, Pedersen, MiMC, SHA-256, ECDSA, EdDSA, Sparse Merkle Tree, Keccak-256, and Schnorr.

Is this already integrated or is it being integrated? Where can I find relevant content?

Recursion Specificiation

Add specification for benchmarking recursive SNARKs

• config format
• log format
• gnark - add Groth16 verifier BLS12-377/BW6-761
• gnark - add Groth16 verifier BLS24-315/BW6-762
• frontend - display 2-chain benchmarks

Input Format Synchronization

Create a document with the expected inputs, parameters, and other specifics for each circuit:

I'm working on the zk-benchmarking track. For some circuits, MIMC specifically, they have parameters for the number of rounds that the circuit does a computation. See this Gnark implementation of MIMC for example:

Gnark has various versions of MIMC, while Circomlib has only 1 which is that the number of rounds must be 91.

Some circuits like EDDSA and SMT also have different variants because you can build them with different hash functions (pedersen, mimc, poseidon, blake2b) and it's unclear right now which variant to choose.

Having a document that lists the expected inputs and parameters for each circuit would help circuit builders create their circuits to conform to the standard.

Circuit results CSV filenames should be consistent

Currently, there is no naming convention for filenames:

├── bellman
│   ├── bellman_bls12_381_exponentiate.csv
│   ├── bellman_bls12_381_sha256.csv
│   ├── jsons
│   └── memory
├── circom
│   ├── circom_exponentiate.csv
│   └── circom_sha256.csv
├── gnark
│   ├── groth16_exponentiate.csv
│   ├── groth16_sha256.csv
│   ├── plonk_exponentiate.csv
│   └── plonk_sha256.csv
├── halo2_pse
│   ├── halo2_pse_bn256_exponentiate.csv
│   ├── halo2_pse_bn256_sha256.csv
│   ├── jsons
│   └── memory
└── starky
    ├── jsons
    ├── memory
    └── starky_goldilocks_exponentiate.csv