Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kaustinen with shapella #259

Closed
wants to merge 57 commits into from

Conversation

gballet
Copy link
Owner

@gballet gballet commented Aug 29, 2023

Move kaustinent changes from #176 and update it for the proof to be generated inside FinalizeAndAssemble so that proof generation is available from all places.

In order for the verkle test to work, I had to re-implement a more functional chain reader than the one that is currently available withfakeChainReader (that only returns nil).

  • This effort is currently blocked by the fact that GetProofItems does not resolve HashedNodes. -> merged relevant PR
  • The post-state is also not provided, which is a blocker for the next instantiation of kaustinen. -> fixed with use go-verkle's post-state API to verify proofs #262

@gballet gballet marked this pull request as ready for review September 5, 2023 15:56
k verkle.StateDiff
keys = state.Witness().Keys()
)
if chain.Config().IsVerkle(header.Number, header.Time) {
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the extra part: if we are past the fork, generate the proof to go with it

Comment on lines +393 to +411
parent := chain.GetHeaderByNumber(header.Number.Uint64() - 1)
if parent == nil {
return nil, fmt.Errorf("nil parent header for block %d", header.Number)
}

preTrie, err := state.Database().OpenTrie(parent.Root)
if err != nil {
return nil, fmt.Errorf("error opening pre-state tree root: %w", err)
}
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one very inefficient part: because at this point we only have the post tree, we need to reopen the pre-tree.

There is a better approach, that I tried once but failed to get to work: capture the initial value in the witness so that we don't have to do this. I will have to try again, since I should have been catching this in GetCommittedState instead of all over the codebase. Now that your AccessWitness rewrite simplified a lot of things, it might be easier. Nonetheless, this is for a different PR. Just pointing out the huge optimization that is possible here.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've a PR candidate trying to optimize this thing about loading the pre-state tree in the access witness #255

We still haven't discussed much that PR... but that's exactly the idea. Build the pre-state tree while we're merging witnesses, so we have all we need to call prove and serialize.

}

vtrpre, okpre := preTrie.(*trie.VerkleTrie)
vtrpost, okpost := state.GetTrie().(*trie.VerkleTrie)
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

post trie is the current trie

@@ -558,3 +523,59 @@ func (cr *fakeChainReader) GetHeaderByHash(hash common.Hash) *types.Header
func (cr *fakeChainReader) GetHeader(hash common.Hash, number uint64) *types.Header { return nil }
func (cr *fakeChainReader) GetBlock(hash common.Hash, number uint64) *types.Block { return nil }
func (cr *fakeChainReader) GetTd(hash common.Hash, number uint64) *big.Int { return nil }

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to make my own fake chain reader, I will work on a geth PR to enhance the default one to reduce the footprint of this PR. Disregard.

@@ -365,8 +365,9 @@ func (db *cachingDB) openStorageMPTrie(stateRoot common.Hash, address common.Add

// OpenStorageTrie opens the storage trie of an account
func (db *cachingDB) OpenStorageTrie(stateRoot common.Hash, address common.Address, root common.Hash, self Trie) (Trie, error) {
// TODO this should only return a verkle tree
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will disappear with the rebase.

}

if len(keys) > 0 {
p, k, err = trie.ProveAndSerialize(vtrpre, vtrpost, keys, vtrpre.FlatdbNodeResolver)
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to export (*VerkleTrie).FlatdbNodeResolver so that it can be used here, because in the case of a stem proving the absence of another stem, the proving code will need to resolve that former stem when it hasn't been read yet.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have expected all the necessary stems for the proof be already loaded in the L410 loop.
If one of the keys is absent, wouldn't that indirectly load the stem that prove it's absence?

In any case, you can ignore my comment. Prob we can continue this chat when we evaluate #255, and maybe we realize that can't work for this reason.

Comment on lines 410 to 438
for _, key := range keys {
_, err := vtrpre.GetWithHashedKey(key)
if err != nil {
panic(err)
}
}
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have to resolve all keys from the witness here, because otherwise they would be hashed nodes. That precedes the Prover taking a resolving function as a parameter, so it might no longer be necessary.

Of course, it's even less necessary if we can get these values from the access witness, but I see now that we won't be able to completely remove the need to open the pre-trie, because of that need to get the proof-of-absence stem...

}

tree, err := verkle.TreeFromProof(proof, rootC)
pretree, err := verkle.PreStateTreeFromProof(proof, rootC)
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First step of the proof reconstruction: rebuild the pre tree from the proof.

trie/verkle.go Outdated
Comment on lines 364 to 384
return fmt.Errorf("error rebuilding the pre-tree from proof: %w", err)
}
for _, stemdiff := range statediff {
for _, suffixdiff := range stemdiff.SuffixDiffs {
var key [32]byte
copy(key[:31], stemdiff.Stem[:])
key[31] = suffixdiff.Suffix

val, err := tree.Get(key[:], nil)
val, err := pretree.Get(key[:], nil)
if err != nil {
return nil, nil, nil, nil, fmt.Errorf("could not find key %x in tree rebuilt from proof: %w", key, err)
return fmt.Errorf("could not find key %x in tree rebuilt from proof: %w", key, err)
}
if len(val) > 0 {
if !bytes.Equal(val, suffixdiff.CurrentValue[:]) {
return nil, nil, nil, nil, fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue)
return fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue)
}
} else {
if suffixdiff.CurrentValue != nil && len(suffixdiff.CurrentValue) != 0 {
return nil, nil, nil, nil, fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue)
return fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue)
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same thing as line 410 & following in consensus/beacon/consensus.go, except it might not be necessary. I still need to check that.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory, the tree was reconstructed using statediff.. So technically, this is only for defensive programming regarding implementation bugs and not malicious input, right?

Saying it another way: if there're no bugs in DeserializeProof(vp ,statediff) then these checks can never fail.

Mentioning this to double check the intention is catching implementation bugs, and not "malicious" provers.

(Is that what you refer to "might not be necessary"?)

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah it's code that has changed a lot in the past, I would keep it for now in order to make sure that nothing breaks. I will add a reminder though.

trie/verkle.go Outdated
// no need to resolve as the tree has been reconstructed from the proof
// and must not contain any unresolved nodes.
pe, _, _, err := tree.GetProofItems(proof.Keys)
posttree, err := verkle.PostStateTreeFromProof(pretree, statediff)
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2nd part of making the proof: reconstruct the post tree from the pre-tree by copying the pre-tree and inserting all the NewValues from the proof into that post trie.

trie/verkle.go Outdated
}

return proof, pe.Cis, pe.Zis, pe.Yis, err
return verkle.VerifyVerkleProofWithPreAndPostTrie(proof, pretree, posttree, nil /* no resolver needed TODO: remove it from the interface */)
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point Guillaume, remove it from the interface!

Copy link
Collaborator

@jsign jsign left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments for your consideration.

Comment on lines +321 to +327
if posttrie != nil {
postroot = posttrie.root
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we safely assume that posttrie was posttrie.Commit()ed?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, because it happens after the block finalization code has been called.

trie/verkle.go Outdated

return nil
}
rootC.SetBytesUnsafe(root)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this correct? (compared to using the safe version that does security check)

If we receive the root from a block that is untrusted, I don't think so.

trie/verkle.go Outdated
@@ -338,69 +342,55 @@ func addKey(s set, key []byte) {

func DeserializeAndVerifyVerkleProof(vp *verkle.VerkleProof, root []byte, statediff verkle.StateDiff) error {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side-question: is there a reason StateDIff isn't part of verkle.VerkleProof? Looks like only having verkle.VerkleProof is somewhat incomplete to do anything relevant?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess StateDiff is considered part of the witness, so not part of the proof per se.

trie/verkle.go Outdated
Comment on lines 358 to 360
if len(proof.Keys) != len(proof.PreValues) {
return fmt.Errorf("keys and values are of different length %d != %d", len(proof.Keys), len(proof.PreValues))
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be useful to move this check to PreStateTreeFromProof.

Comment on lines +393 to +411
parent := chain.GetHeaderByNumber(header.Number.Uint64() - 1)
if parent == nil {
return nil, fmt.Errorf("nil parent header for block %d", header.Number)
}

preTrie, err := state.Database().OpenTrie(parent.Root)
if err != nil {
return nil, fmt.Errorf("error opening pre-state tree root: %w", err)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've a PR candidate trying to optimize this thing about loading the pre-state tree in the access witness #255

We still haven't discussed much that PR... but that's exactly the idea. Build the pre-state tree while we're merging witnesses, so we have all we need to call prove and serialize.


copy(ret.VerkleProof.DepthExtensionPresent, ew.VerkleProof.DepthExtensionPresent)
for i := range ew.VerkleProof.CommitmentsByPath {
copy(ret.VerkleProof.CommitmentsByPath[i][:], ew.VerkleProof.CommitmentsByPath[i][:])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Could be simplified as: ret.VerkleProof.CommitmentsByPath[i] = ew.VerkleProof.CommitmentsByPath[i] since arrays are values in Go. So avoid copy and [:] stuff.

The code is still correct, to up to you.

for i := range ew.VerkleProof.CommitmentsByPath {
copy(ret.VerkleProof.CommitmentsByPath[i][:], ew.VerkleProof.CommitmentsByPath[i][:])
}
copy(ret.VerkleProof.D[:], ew.VerkleProof.D[:])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Comment on lines 99 to 105
if ew.VerkleProof.IPAProof != nil {
for i := range ew.VerkleProof.IPAProof.CL {
copy(ret.VerkleProof.IPAProof.CL[i][:], ew.VerkleProof.IPAProof.CL[i][:])
copy(ret.VerkleProof.IPAProof.CR[i][:], ew.VerkleProof.IPAProof.CR[i][:])
}
copy(ret.VerkleProof.IPAProof.FinalEvaluation[:], ew.VerkleProof.IPAProof.FinalEvaluation[:])
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto. In this case we can get rid of the for loop.
ret.VerkleProof.IPAProof.CL = ew.VerkleProof.IPAProof.CL (and same with the other).

Proof in playground.

So actually you can do ew.VerkleProof.IPAProof = ret.VerkleProof.IPAProof and all L100-L104 are gone. And it's a copy, not a reference.

trie/verkle.go Outdated
Comment on lines 364 to 384
return fmt.Errorf("error rebuilding the pre-tree from proof: %w", err)
}
for _, stemdiff := range statediff {
for _, suffixdiff := range stemdiff.SuffixDiffs {
var key [32]byte
copy(key[:31], stemdiff.Stem[:])
key[31] = suffixdiff.Suffix

val, err := tree.Get(key[:], nil)
val, err := pretree.Get(key[:], nil)
if err != nil {
return nil, nil, nil, nil, fmt.Errorf("could not find key %x in tree rebuilt from proof: %w", key, err)
return fmt.Errorf("could not find key %x in tree rebuilt from proof: %w", key, err)
}
if len(val) > 0 {
if !bytes.Equal(val, suffixdiff.CurrentValue[:]) {
return nil, nil, nil, nil, fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue)
return fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue)
}
} else {
if suffixdiff.CurrentValue != nil && len(suffixdiff.CurrentValue) != 0 {
return nil, nil, nil, nil, fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue)
return fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory, the tree was reconstructed using statediff.. So technically, this is only for defensive programming regarding implementation bugs and not malicious input, right?

Saying it another way: if there're no bugs in DeserializeProof(vp ,statediff) then these checks can never fail.

Mentioning this to double check the intention is catching implementation bugs, and not "malicious" provers.

(Is that what you refer to "might not be necessary"?)

}
if len(val) > 0 {
if !bytes.Equal(val, suffixdiff.CurrentValue[:]) {
return nil, nil, nil, nil, fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue)
return fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue)
}
} else {
if suffixdiff.CurrentValue != nil && len(suffixdiff.CurrentValue) != 0 {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code (else case) won't return an error in the following cases:

  • val = nil and suffixdiff.CurrentValue = []byte{}, and
  • val = []byte{} and suffixdiff.CurrentValue = nil

Is that fine?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, nil and []byte{} both represent an absence

@gballet gballet requested a review from holiman as a code owner September 6, 2023 09:22
@gballet
Copy link
Owner Author

gballet commented Sep 9, 2023

This branch, once all bugs have been solved, should be merged. There's little point having 3 branches (Beverly hills, kautinen and replays) when we can only have 2: kaustinen and replays. Beverly hills is nice for other clients to check they support the basic tree structures without the proofs, but this could be controlled by a simple command line switch.

@gballet
Copy link
Owner Author

gballet commented Sep 10, 2023

left to do before merge:

  • rebase
  • fix test dependency cycle
  • ensure that TestProcessVerkle still runs and see if it can use the "new" merge technique
  • add a similar test that does the transition at the 2nd block
  • act on review
  • add cli switch to disable proofs in blocks, so that this branch can work on beverly hills as well
  • move all replay-specific code to the replay branch, in order to have only 2 branches to maintain

@gballet gballet force-pushed the kaustinen-with-shapella branch from 71186ff to 0420c57 Compare September 22, 2023 18:32
activate proof generation on fork  + remove code dups

use go-verkle's post-state API to verify proofs (#262)

use prague as the verkle activation fork (#263)

upgrade to latest go-ipa

activate verkle transition in "miner" (#265)

fix: do not force cancunTime upon verkle activation

workaround: do not use root translation in replay

workaround: deactivate overlay transition for now

fixes from trying to get the devnet to work (#267)

this line was left out from the previous commit

upgrade to go-verkle with fixed newvalue serialization

fix: ensure point cache isn't nil in copy (#268)

fix: dependency cycle in tests (#269)

upgrade to latest go-verkle

fix: write trie preimage data to db (#274)

fix: zero-root in produced block + sync (#275)

upgrade go-ipa

fix build

fix typo

include review feedback

add switch to add proofs to blocks (#278)

add fee recipient to witness (#279)

touch all fields in withdrawal account header (#277)
@gballet gballet force-pushed the kaustinen-with-shapella branch from 53de425 to 160f735 Compare September 25, 2023 15:30
gballet and others added 28 commits January 26, 2024 11:32
* implement eip 2935

* add touched historical contract slot to the witness
* fix: support a verkle pre-tree at the conversion block

* make linter happy
* fix: decide if genesis is verkle using genesis timestamp

* fix linter message
* set verkle mode from genesis with override

* make linter happy

* fix more linter snafu
* fix: CREATE-time PUSHn adds non-existent entries to witness

* this also affects CODECOPY

* fix: add code returned by CREATE* to the witness

* fix gas costs
* eip2935 redesign: only read target block hash

* review fixes

* fix: t8n tool
* instructions: add access witness recording for EXTCODEHASH

* add test for EXTCODEHASH witness recording

* add test for access witness EXTCODEHASH

Signed-off-by: Ignacio Hagopian <[email protected]>

* do not touch version

Signed-off-by: Ignacio Hagopian <[email protected]>

---------

Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
* record access witness balance in BALANCE opcode

Signed-off-by: Ignacio Hagopian <[email protected]>

* add tests for BALANCE opcode

Signed-off-by: Ignacio Hagopian <[email protected]>

* add version touching

Signed-off-by: Ignacio Hagopian <[email protected]>

* do not touch version

Signed-off-by: Ignacio Hagopian <[email protected]>

---------

Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
…nce in witness (#378)

* fix SELFDESTRUCT witness recording

Signed-off-by: Ignacio Hagopian <[email protected]>

* add selfdestruct tests

Signed-off-by: Ignacio Hagopian <[email protected]>

* solve lint nit

Signed-off-by: Ignacio Hagopian <[email protected]>

---------

Signed-off-by: Ignacio Hagopian <[email protected]>
@gballet
Copy link
Owner Author

gballet commented Mar 2, 2024

kaustinen-with-shapella is the new "master" branch. Closing this PR as the base branch won't be me merged in geth's master in any case.

@gballet gballet closed this Mar 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants