New-generation Marlin with SonicKZG10 as PC

[weikengchen]

Motivation

This is a tracking issue for the work-in-progress of a new generation of Marlin. It is expected to take about three days (or more).

The new generation of Marlin consists of two updates:

• a different AHP is introduced, which has fewer polynomials and opens fewer commitments, see Pratyush's work upstream (Change arithmetization to improve proof and vk size arkworks-rs/marlin#80)
• a different PC is selected, which is almost better than MarlinKZG10 in any way

Progress as of 7/28:

• a new concept, "universal verification parameters", was introduced to SNARK and SNARKGadgets, which allows us to split the verifying key into a part that is universal, and another part that is circuit-specific. As of 8/1, this concept is withdrawn, as it is too complicated, and we may just use alloc_constant for PC VK and skip it during serialization.
• added all the data structures for SonicKZG10 gadgets.
• initial works in refactoring KZG10 to sample only the needed parameters.

Progress as of 7/29:

• KZG10 now only samples the needed parameters

Progress as of 7/30:

• debug

Progress as of 7/31:

• fixed a bug from Zexe about how G2 prepared gadget being incorrectly converted (Fix BLS12 G2PreparedGadget's AllocVar with a divisive twist arkworks-rs/r1cs-std#77)

Progress as of 8/1:

• added initial works in SonicKZG10 gadget.
• initial tests show that SonicKZG10 works like a charm!
• end-to-end test with old Marlin + SonicKZG10
• replaced old Marlin native with new Marlin native, which passes the native tests

Progress as of 8/2:

• added the new Marlin gadgets
• PR review and cleanup

Test Plan

Plans to use the same checks that we have today for this new generation.

[weikengchen]

This commit is ready for review.

The improvement on gadgets is not the major part, but still impressive. Outer circuit is now 878967 => 787899 constraints.

The improvement on native is the major part. Proving will be much faster. Proofs are shorter.

[ljedrz]

this method is very long and complex - it would be great if it could be split into smaller chunks

[weikengchen]

I may do it in a future PR. The upstream arkworks-rs/poly-commit is doing some similar refactoring aiming to simplify the arguments passing to the function. And we may follow up with those changes later.

[weikengchen]

@ljedrz comments have been mostly solved.

[weikengchen]

And can confirm that the remaining tests should pass upon parameter resampling.

[Pratyush]

The outer circuit is smaller by 100k constraints? That's great. Are there benchmarks on concrete circuits for the faster Marlin proving? I'm interested in how the new AHP affects prover time and memory consumption.

[weikengchen]

I can try to get a number by running the inner circuit instead of our benchmark circuit (which is, how to say, too sparse), and see how much is the improvement.

[howardwu]

Leaving a quick note that I'm halfway through this PR, and reviewing it piece-wise.

This PR looks great, most notably, the nice things I see are:

• We have unified the various instantiations of Marlin, which was much needed tech debt reduction, many thanks for doing this.
• The changes to arithmetization, balancing matrices, and nonzero are incredibly nice, for developers these are abstract as lower-level details that are either now unnecessary or black boxed.
• A handful of gadget traits have been simplified or unified

[weikengchen]

And in case necessary, I can add some explanations on certain parts. Just circle out those that confuses you.

Let me resolve the merge conflict.

[weikengchen]

Below is a summary of a discussion with Pratyush. We analyze the improvement. The experiment is done on inner circuit for testnet2, although the inner circuit is indeed proved in Groth16.

Improvement

Shorter proof. The proof for succinct work reduces to 771 bytes from 972 bytes.

Fewer constraints. The outer circuit, which verifies the program proof that is in Marlin, reduces to 787k constraints from 878k.

Faster proving. We expect the proving time to go down by 20%. This is due to the change of PC scheme to Sonic KZG10.

oldmarlin+marlinkzg10 takes 102s.
oldmarlin+sonickzg10 takes 85s.

newmarlin+marlinkzg10 takes 107s.
newmarlin+sonickzg10 takes 83s.

Faster verification. The verification time goes down slightly (so it is not that important actually) because the new Marlin AHP has fewer commitments to play with. The time for Sonic KZG10 is slightly higher, but we believe it can be optimized further (without changing the design), which we describe in arkworks-rs/poly-commit#85.

oldmarlin+marlinkzg10 takes 45ms.
oldmarlin+sonickzg10 takes 49ms.

newmarlin+marlinkzg10 takes 42ms.
newmarlin+sonickzg10 takes 46ms.

Ability to prove. For an R1CS instance consisting of matrices A, B, and C, the old Marlin requires the SRS to have length at least 3 * max(|A|, |B|, |C|) where |M| is the number of non-zero entries in that matrix. The new Marlin requires (|A| + |B| + |C|), which has a hope to be much smaller than the previous one.

For the inner circuit that we experiment on, the new Marlin will require a smaller SRS. To say it differently, if we have a setup of a fixed size, the new Marlin can prove more complicated statements.

oldmarlin is for a max-degree of 3145725
newmarlin is for a max-degree of 2097152

Other changes

SRS is slightly longer. Now the SRS also contains the negative powers of h in G2, but they are only a few.

Verifier key is longer, or shorter. Now the PC verifier key in all Marlin verifier keys are the same, regardless of the circuit, as long as they are in the same setup. If you transmit the PC verifier key all the time, then the key is longer. But you can choose not to transmit the PC verifier key, but only transfer the part that is circuit-dependent, then the verifier key is shorter.

There is a possibility to reduce the verifier key size further, as three index comms (for vanishing polynomials) are indeed universal to the setup and is not circuit-dependent, and can be moved out of the verifier key. This is slightly messy and would be done in a future PR, with which the program may retrieve the PC verifier key from the parameters system and generate the commitments to vanishing polynomials deterministically.