-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kaustinen with shapella #259
Conversation
consensus/beacon/consensus.go
Outdated
k verkle.StateDiff | ||
keys = state.Witness().Keys() | ||
) | ||
if chain.Config().IsVerkle(header.Number, header.Time) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the extra part: if we are past the fork, generate the proof to go with it
parent := chain.GetHeaderByNumber(header.Number.Uint64() - 1) | ||
if parent == nil { | ||
return nil, fmt.Errorf("nil parent header for block %d", header.Number) | ||
} | ||
|
||
preTrie, err := state.Database().OpenTrie(parent.Root) | ||
if err != nil { | ||
return nil, fmt.Errorf("error opening pre-state tree root: %w", err) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is one very inefficient part: because at this point we only have the post tree, we need to reopen the pre-tree.
There is a better approach, that I tried once but failed to get to work: capture the initial value in the witness so that we don't have to do this. I will have to try again, since I should have been catching this in GetCommittedState
instead of all over the codebase. Now that your AccessWitness
rewrite simplified a lot of things, it might be easier. Nonetheless, this is for a different PR. Just pointing out the huge optimization that is possible here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've a PR candidate trying to optimize this thing about loading the pre-state tree in the access witness #255
We still haven't discussed much that PR... but that's exactly the idea. Build the pre-state tree while we're merging witnesses, so we have all we need to call prove and serialize.
consensus/beacon/consensus.go
Outdated
} | ||
|
||
vtrpre, okpre := preTrie.(*trie.VerkleTrie) | ||
vtrpost, okpost := state.GetTrie().(*trie.VerkleTrie) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
post trie is the current trie
@@ -558,3 +523,59 @@ func (cr *fakeChainReader) GetHeaderByHash(hash common.Hash) *types.Header | |||
func (cr *fakeChainReader) GetHeader(hash common.Hash, number uint64) *types.Header { return nil } | |||
func (cr *fakeChainReader) GetBlock(hash common.Hash, number uint64) *types.Block { return nil } | |||
func (cr *fakeChainReader) GetTd(hash common.Hash, number uint64) *big.Int { return nil } | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had to make my own fake chain reader, I will work on a geth PR to enhance the default one to reduce the footprint of this PR. Disregard.
@@ -365,8 +365,9 @@ func (db *cachingDB) openStorageMPTrie(stateRoot common.Hash, address common.Add | |||
|
|||
// OpenStorageTrie opens the storage trie of an account | |||
func (db *cachingDB) OpenStorageTrie(stateRoot common.Hash, address common.Address, root common.Hash, self Trie) (Trie, error) { | |||
// TODO this should only return a verkle tree |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will disappear with the rebase.
} | ||
|
||
if len(keys) > 0 { | ||
p, k, err = trie.ProveAndSerialize(vtrpre, vtrpost, keys, vtrpre.FlatdbNodeResolver) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had to export (*VerkleTrie).FlatdbNodeResolver
so that it can be used here, because in the case of a stem proving the absence of another stem, the proving code will need to resolve that former stem when it hasn't been read yet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would have expected all the necessary stems for the proof be already loaded in the L410 loop.
If one of the keys
is absent, wouldn't that indirectly load the stem that prove it's absence?
In any case, you can ignore my comment. Prob we can continue this chat when we evaluate #255, and maybe we realize that can't work for this reason.
consensus/beacon/consensus.go
Outdated
for _, key := range keys { | ||
_, err := vtrpre.GetWithHashedKey(key) | ||
if err != nil { | ||
panic(err) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have to resolve all keys from the witness here, because otherwise they would be hashed nodes. That precedes the Prover taking a resolving function as a parameter, so it might no longer be necessary.
Of course, it's even less necessary if we can get these values from the access witness, but I see now that we won't be able to completely remove the need to open the pre-trie, because of that need to get the proof-of-absence stem...
} | ||
|
||
tree, err := verkle.TreeFromProof(proof, rootC) | ||
pretree, err := verkle.PreStateTreeFromProof(proof, rootC) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First step of the proof reconstruction: rebuild the pre tree from the proof.
trie/verkle.go
Outdated
return fmt.Errorf("error rebuilding the pre-tree from proof: %w", err) | ||
} | ||
for _, stemdiff := range statediff { | ||
for _, suffixdiff := range stemdiff.SuffixDiffs { | ||
var key [32]byte | ||
copy(key[:31], stemdiff.Stem[:]) | ||
key[31] = suffixdiff.Suffix | ||
|
||
val, err := tree.Get(key[:], nil) | ||
val, err := pretree.Get(key[:], nil) | ||
if err != nil { | ||
return nil, nil, nil, nil, fmt.Errorf("could not find key %x in tree rebuilt from proof: %w", key, err) | ||
return fmt.Errorf("could not find key %x in tree rebuilt from proof: %w", key, err) | ||
} | ||
if len(val) > 0 { | ||
if !bytes.Equal(val, suffixdiff.CurrentValue[:]) { | ||
return nil, nil, nil, nil, fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue) | ||
return fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue) | ||
} | ||
} else { | ||
if suffixdiff.CurrentValue != nil && len(suffixdiff.CurrentValue) != 0 { | ||
return nil, nil, nil, nil, fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue) | ||
return fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the same thing as line 410 & following in consensus/beacon/consensus.go
, except it might not be necessary. I still need to check that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In theory, the tree was reconstructed using statediff
.. So technically, this is only for defensive programming regarding implementation bugs and not malicious input, right?
Saying it another way: if there're no bugs in DeserializeProof(vp ,statediff)
then these checks can never fail.
Mentioning this to double check the intention is catching implementation bugs, and not "malicious" provers.
(Is that what you refer to "might not be necessary"?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah it's code that has changed a lot in the past, I would keep it for now in order to make sure that nothing breaks. I will add a reminder though.
trie/verkle.go
Outdated
// no need to resolve as the tree has been reconstructed from the proof | ||
// and must not contain any unresolved nodes. | ||
pe, _, _, err := tree.GetProofItems(proof.Keys) | ||
posttree, err := verkle.PostStateTreeFromProof(pretree, statediff) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
2nd part of making the proof: reconstruct the post tree from the pre-tree by copying the pre-tree and inserting all the NewValues
from the proof into that post trie.
trie/verkle.go
Outdated
} | ||
|
||
return proof, pe.Cis, pe.Zis, pe.Yis, err | ||
return verkle.VerifyVerkleProofWithPreAndPostTrie(proof, pretree, posttree, nil /* no resolver needed TODO: remove it from the interface */) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point Guillaume, remove it from the interface!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left some comments for your consideration.
if posttrie != nil { | ||
postroot = posttrie.root | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we safely assume that posttrie
was posttrie.Commit()
ed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, because it happens after the block finalization code has been called.
trie/verkle.go
Outdated
|
||
return nil | ||
} | ||
rootC.SetBytesUnsafe(root) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this correct? (compared to using the safe version that does security check)
If we receive the root from a block that is untrusted, I don't think so.
trie/verkle.go
Outdated
@@ -338,69 +342,55 @@ func addKey(s set, key []byte) { | |||
|
|||
func DeserializeAndVerifyVerkleProof(vp *verkle.VerkleProof, root []byte, statediff verkle.StateDiff) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Side-question: is there a reason StateDIff isn't part of verkle.VerkleProof
? Looks like only having verkle.VerkleProof
is somewhat incomplete to do anything relevant?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess StateDiff
is considered part of the witness, so not part of the proof per se.
trie/verkle.go
Outdated
if len(proof.Keys) != len(proof.PreValues) { | ||
return fmt.Errorf("keys and values are of different length %d != %d", len(proof.Keys), len(proof.PreValues)) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be useful to move this check to PreStateTreeFromProof
.
parent := chain.GetHeaderByNumber(header.Number.Uint64() - 1) | ||
if parent == nil { | ||
return nil, fmt.Errorf("nil parent header for block %d", header.Number) | ||
} | ||
|
||
preTrie, err := state.Database().OpenTrie(parent.Root) | ||
if err != nil { | ||
return nil, fmt.Errorf("error opening pre-state tree root: %w", err) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've a PR candidate trying to optimize this thing about loading the pre-state tree in the access witness #255
We still haven't discussed much that PR... but that's exactly the idea. Build the pre-state tree while we're merging witnesses, so we have all we need to call prove and serialize.
core/types/block.go
Outdated
|
||
copy(ret.VerkleProof.DepthExtensionPresent, ew.VerkleProof.DepthExtensionPresent) | ||
for i := range ew.VerkleProof.CommitmentsByPath { | ||
copy(ret.VerkleProof.CommitmentsByPath[i][:], ew.VerkleProof.CommitmentsByPath[i][:]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Could be simplified as: ret.VerkleProof.CommitmentsByPath[i] = ew.VerkleProof.CommitmentsByPath[i]
since arrays are values in Go. So avoid copy
and [:]
stuff.
The code is still correct, to up to you.
core/types/block.go
Outdated
for i := range ew.VerkleProof.CommitmentsByPath { | ||
copy(ret.VerkleProof.CommitmentsByPath[i][:], ew.VerkleProof.CommitmentsByPath[i][:]) | ||
} | ||
copy(ret.VerkleProof.D[:], ew.VerkleProof.D[:]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto.
core/types/block.go
Outdated
if ew.VerkleProof.IPAProof != nil { | ||
for i := range ew.VerkleProof.IPAProof.CL { | ||
copy(ret.VerkleProof.IPAProof.CL[i][:], ew.VerkleProof.IPAProof.CL[i][:]) | ||
copy(ret.VerkleProof.IPAProof.CR[i][:], ew.VerkleProof.IPAProof.CR[i][:]) | ||
} | ||
copy(ret.VerkleProof.IPAProof.FinalEvaluation[:], ew.VerkleProof.IPAProof.FinalEvaluation[:]) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto. In this case we can get rid of the for loop.
ret.VerkleProof.IPAProof.CL = ew.VerkleProof.IPAProof.CL
(and same with the other).
So actually you can do ew.VerkleProof.IPAProof = ret.VerkleProof.IPAProof
and all L100-L104 are gone. And it's a copy, not a reference.
trie/verkle.go
Outdated
return fmt.Errorf("error rebuilding the pre-tree from proof: %w", err) | ||
} | ||
for _, stemdiff := range statediff { | ||
for _, suffixdiff := range stemdiff.SuffixDiffs { | ||
var key [32]byte | ||
copy(key[:31], stemdiff.Stem[:]) | ||
key[31] = suffixdiff.Suffix | ||
|
||
val, err := tree.Get(key[:], nil) | ||
val, err := pretree.Get(key[:], nil) | ||
if err != nil { | ||
return nil, nil, nil, nil, fmt.Errorf("could not find key %x in tree rebuilt from proof: %w", key, err) | ||
return fmt.Errorf("could not find key %x in tree rebuilt from proof: %w", key, err) | ||
} | ||
if len(val) > 0 { | ||
if !bytes.Equal(val, suffixdiff.CurrentValue[:]) { | ||
return nil, nil, nil, nil, fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue) | ||
return fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue) | ||
} | ||
} else { | ||
if suffixdiff.CurrentValue != nil && len(suffixdiff.CurrentValue) != 0 { | ||
return nil, nil, nil, nil, fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue) | ||
return fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In theory, the tree was reconstructed using statediff
.. So technically, this is only for defensive programming regarding implementation bugs and not malicious input, right?
Saying it another way: if there're no bugs in DeserializeProof(vp ,statediff)
then these checks can never fail.
Mentioning this to double check the intention is catching implementation bugs, and not "malicious" provers.
(Is that what you refer to "might not be necessary"?)
} | ||
if len(val) > 0 { | ||
if !bytes.Equal(val, suffixdiff.CurrentValue[:]) { | ||
return nil, nil, nil, nil, fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue) | ||
return fmt.Errorf("could not find correct value at %x in tree rebuilt from proof: %x != %x", key, val, *suffixdiff.CurrentValue) | ||
} | ||
} else { | ||
if suffixdiff.CurrentValue != nil && len(suffixdiff.CurrentValue) != 0 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This code (else case) won't return an error in the following cases:
val = nil
andsuffixdiff.CurrentValue = []byte{}
, andval = []byte{}
andsuffixdiff.CurrentValue = nil
Is that fine?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, nil
and []byte{}
both represent an absence
This branch, once all bugs have been solved, should be merged. There's little point having 3 branches (Beverly hills, kautinen and replays) when we can only have 2: kaustinen and replays. Beverly hills is nice for other clients to check they support the basic tree structures without the proofs, but this could be controlled by a simple command line switch. |
left to do before merge:
|
71186ff
to
0420c57
Compare
activate proof generation on fork + remove code dups use go-verkle's post-state API to verify proofs (#262) use prague as the verkle activation fork (#263) upgrade to latest go-ipa activate verkle transition in "miner" (#265) fix: do not force cancunTime upon verkle activation workaround: do not use root translation in replay workaround: deactivate overlay transition for now fixes from trying to get the devnet to work (#267) this line was left out from the previous commit upgrade to go-verkle with fixed newvalue serialization fix: ensure point cache isn't nil in copy (#268) fix: dependency cycle in tests (#269) upgrade to latest go-verkle fix: write trie preimage data to db (#274) fix: zero-root in produced block + sync (#275) upgrade go-ipa fix build fix typo include review feedback add switch to add proofs to blocks (#278) add fee recipient to witness (#279) touch all fields in withdrawal account header (#277)
53de425
to
160f735
Compare
* implement eip 2935 * add touched historical contract slot to the witness
* fix: support a verkle pre-tree at the conversion block * make linter happy
* fix: decide if genesis is verkle using genesis timestamp * fix linter message
* set verkle mode from genesis with override * make linter happy * fix more linter snafu
Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
* fix: CREATE-time PUSHn adds non-existent entries to witness * this also affects CODECOPY * fix: add code returned by CREATE* to the witness * fix gas costs
* eip2935 redesign: only read target block hash * review fixes * fix: t8n tool
* instructions: add access witness recording for EXTCODEHASH * add test for EXTCODEHASH witness recording * add test for access witness EXTCODEHASH Signed-off-by: Ignacio Hagopian <[email protected]> * do not touch version Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
* record access witness balance in BALANCE opcode Signed-off-by: Ignacio Hagopian <[email protected]> * add tests for BALANCE opcode Signed-off-by: Ignacio Hagopian <[email protected]> * add version touching Signed-off-by: Ignacio Hagopian <[email protected]> * do not touch version Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
…nce in witness (#378) * fix SELFDESTRUCT witness recording Signed-off-by: Ignacio Hagopian <[email protected]> * add selfdestruct tests Signed-off-by: Ignacio Hagopian <[email protected]> * solve lint nit Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
…istory contract (#390) Co-authored-by: Ignacio Hagopian <[email protected]>
|
Move kaustinent changes from #176 and update it for the proof to be generated inside
FinalizeAndAssemble
so that proof generation is available from all places.In order for the verkle test to work, I had to re-implement a more functional chain reader than the one that is currently available with
fakeChainReader
(that only returnsnil
).This effort is currently blocked by the fact that-> merged relevant PRGetProofItems
does not resolveHashedNodes
.The post-state is also not provided, which is a blocker for the next instantiation of kaustinen.-> fixed with use go-verkle's post-state API to verify proofs #262