Kaustinen with shapella rebase pathdb #437
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* all: implement path-based state scheme * all: edits from review * core/rawdb, trie/triedb/pathdb: review changes * core, light, trie, eth, tests: reimplement pbss history * core, trie/triedb/pathdb: track block number in state history * trie/triedb/pathdb: add history documentation * core, trie/triedb/pathdb: address comments from Peter's review Important changes to list: - Cache trie nodes by path in clean cache - Remove root->id mappings when history is truncated * trie/triedb/pathdb: fallback to disk if unexpect node in clean cache * core/rawdb: fix tests * trie/triedb/pathdb: rename metrics, change clean cache key * trie/triedb: manage the clean cache inside of disk layer * trie/triedb/pathdb: move journal function * trie/triedb/path: fix tests * trie/triedb/pathdb: fix journal * trie/triedb/pathdb: fix history * trie/triedb/pathdb: try to fix tests on windows * core, trie: address comments * trie/triedb/pathdb: fix test issues --------- Co-authored-by: Felix Lange <[email protected]> Co-authored-by: Martin Holst Swende <[email protected]>
Squash the main verkle PR ahead of rebase don't call Bytes() in GetTreeKey (#137) trie: avoid endianness conversion in GetTreeKey (#140) * trie/utils: add concrete expected value in trie key generation test Signed-off-by: Ignacio Hagopian <[email protected]> * mod: update to latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: avoid endianness conversions Signed-off-by: Ignacio Hagopian <[email protected]> * apply review changes & update to official go-verkle version Signed-off-by: Ignacio Hagopian <[email protected]> Signed-off-by: Ignacio Hagopian <[email protected]> upgrade go-verkle to CoW version and get TestProcessVerkle to build (#138) updating ci to use self-hosted machine (#143) fix: storage offset in non-header group + reuse of value buffer (#145) dedup call to ChunkifyCode, same as replay branch (#156) * dedup call to ChunkifyCode, same as replay branch * fix some linter issues fix code offset in tree update (#157) fix REVERT in state processor test execution (#158) * fix code offset in tree update * fix REVERT in test execution save on key hashing: lump code size update with first code chunk group (#159) fix code chunk key calculation and storage key calculation (#161) * fix codeKey calculation * Remove * fix storageOffset * fix the fix to the fix to the offset fix * Remove copy/pasted, unused code in test * fix linter --------- Co-authored-by: Guillaume Ballet <[email protected]> fix: infinite loop when calling extcodecopy on empty code (#151) upgrade to latest go-verkle fix: only update code in the tree if it's dirty (#174) fix: read-touch the code size and Keccak of the origin (#175) List of changes for converting a sepolia database (#182) * naive conversion rebased on top of beverly hills * changes for the sepolia shadow fork conversion * fixes to please the linter * fixes to please the linter Unified point cache (#180) * Unified point cache * Use cache for Try*Account * alter Trie interface to use caching for slots (#181) * alter Trie interface to use caching for slots * fix: use a lock to protect the point cache (#185) * use fastest non-master go-verkle version & pull trie/Verkle.go changes to use new api (#184) * mod: update to fastest go-verkle version today Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: use new batch serialization api Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: TryDelete signature in unit tests --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> trie/utils: fix potential overflow (#191) * trie/utils: fix potential overflow Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: receive storage key as a byte slice Signed-off-by: Ignacio Hagopian <[email protected]> * revert formatter changes Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: fix mod 256 Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> trie/utils: fix incorrect bigint assignment (#193) Signed-off-by: Ignacio Hagopian <[email protected]> upgrade precomp link to fix CI fix: add missing code size&keccak leaves in empty accounts (#192) fixes to use the latest go-verkle@master (#197) * fixes to use the latest go-verkle@master * linter fixes * linter fixes for tests * fix: use jsign's go-verkle fix refactor: remove unused (*StateDB).GetXLittleEndian methods (#204) fix gas accounting issue in state_processor_test.go (#207) update go-verkle not to use StatelessNode anymore (#206) * update go-verkle not to use StatelessNode anymore * update go-verkle to latest refactor: move verkle gas accounting to its own block in TransitionDB (#208) fix a panic in deserializeVerkleProof if GetProofItems returns a nil ProofElements use the cachingDB instead of a custom VerkleDB (#209) * use the cachingDB instead of a custom VerkleDB * fix stack trace in LES remove holiman from CODEOWNERS as he gets too many emails read from tree in state object if the snapshot is nil (#205) add missing error checks for the root node type (#214) implement OpenStorageTrie for verkle trees (#210) * implement OpenStorageTrie for verkle trees * add a few comments for future maintenance * fix linter issue fix: copy balance leaf to new buffer in TryGetAccount (#217) implement some heretofore unimplemented iterator methods (#219) params: move verkle params to their own file (#228) fix: proper number of chunk evals (#215) overlay transition (#244) * overlay transition Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB * fix: mark the verkle transition as started if it's ended without being started * fix the verkle state processing test * fix linter errors * Add a function to clear verkle params for replay * fix: handle TransitionTrie in OpenStorageTrie * fix linter issue * fix the deleted account error (#247) * code cleanup (#248) * fix: don't error on a missing conversion.txt (#249) * Overlay Tree preimages exporting and usage (#246) * export overlay preimages tool Signed-off-by: Ignacio Hagopian <[email protected]> * use preimages flat file in overlay tree migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * cmd/geth: add --roothash to overlay tree preimage exporting command Signed-off-by: Ignacio Hagopian <[email protected]> * cleanup Signed-off-by: Ignacio Hagopian <[email protected]> * review feedback Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * fix: reduce the PR footprint (#250) * fix: don't fail when preimages.bin is missing (#251) * fix: don't fail when preimages.bin is missing * fix: don't open the preimages file when outside of transition --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> review changes remove replay-specific code
Use plain addresses/slot numbers instead of hashing them in the witness
* trie/utils: add helper to calculate code tree indices * core/state: rewrite optimized version of keyValueMigrator Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: remove uint256 allocs (#257) Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>
* add post-pbss rebase branches to CI * fix go version in CI * fix linter issues * upgrade go version to 1.21.1 to avoid github deploy error
activate proof generation on fork + remove code dups use go-verkle's post-state API to verify proofs (#262) use prague as the verkle activation fork (#263) upgrade to latest go-ipa activate verkle transition in "miner" (#265) fix: do not force cancunTime upon verkle activation workaround: do not use root translation in replay workaround: deactivate overlay transition for now fixes from trying to get the devnet to work (#267) this line was left out from the previous commit upgrade to go-verkle with fixed newvalue serialization fix: ensure point cache isn't nil in copy (#268) fix: dependency cycle in tests (#269) upgrade to latest go-verkle fix: write trie preimage data to db (#274) fix: zero-root in produced block + sync (#275) upgrade go-ipa fix build fix typo include review feedback add switch to add proofs to blocks (#278) add fee recipient to witness (#279) touch all fields in withdrawal account header (#277)
* instructions: add access witness recording for EXTCODEHASH * add test for EXTCODEHASH witness recording * add test for access witness EXTCODEHASH Signed-off-by: Ignacio Hagopian <[email protected]> * do not touch version Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
* record access witness balance in BALANCE opcode Signed-off-by: Ignacio Hagopian <[email protected]> * add tests for BALANCE opcode Signed-off-by: Ignacio Hagopian <[email protected]> * add version touching Signed-off-by: Ignacio Hagopian <[email protected]> * do not touch version Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
…nce in witness (#378) * fix SELFDESTRUCT witness recording Signed-off-by: Ignacio Hagopian <[email protected]> * add selfdestruct tests Signed-off-by: Ignacio Hagopian <[email protected]> * solve lint nit Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
…istory contract (#390) Co-authored-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
* move self-destruct witness logic Signed-off-by: Ignacio Hagopian <[email protected]> * fix test Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>
* eip2935: use ring buffer Signed-off-by: Ignacio Hagopian <[email protected]> * limit resolving scope Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>
…m#27901) (#403) Co-authored-by: imulmat4 <[email protected]>
* trie/verkle: change tree key generation Signed-off-by: Ignacio Hagopian <[email protected]> * fix tests Signed-off-by: Ignacio Hagopian <[email protected]> * update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>
* core: fix TestProcessVerkleInvalidContractCreation Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix TestProcessVerkleContractWithEmptyCode Signed-off-by: Ignacio Hagopian <[email protected]> * add separate action for testprocessverkle tests Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
* simplified gas accounting layer * integrate some review feedback * Apply suggestions from code review Co-authored-by: Ignacio Hagopian <[email protected]> * more suggestions from code review * don't charge creation gas + charge code chunks in create * A couple more fixes * make linter happy * fix create init gas consumption issue * fix: in gas funcs, use tx witness instead of global witness * fix linter issue * Apply suggestions from code review Co-authored-by: Ignacio Hagopian <[email protected]> * fix: EXTCODECOPY gas consumption * fix warm gas costs * fix the order gas is charged in during contract creation epilogue * fix selfdestruct * fix #365 in eip rewrite (#407) * fix: OOG type in code creation OOG (#408) * core/vm: charge BLOCKHASH witness cost (#409) * core/vm: charge BLOCKHASH witness cost Signed-off-by: Ignacio Hagopian <[email protected]> * remove gas optimization for now Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * remove redundant logic for contract creation (#413) Signed-off-by: Ignacio Hagopian <[email protected]> * fix precompile address check for charging witness costs & fix missing value-bearing rule (#412) Signed-off-by: Ignacio Hagopian <[email protected]> * core/vm: fix wrong check (#416) Signed-off-by: Ignacio Hagopian <[email protected]> * charge for account creation if selfdestruct creates a new account (#417) * add key comparison test (#418) * core/vm: charge contract init before execution logic (#419) * core/vm: charge contract init before execution logic Signed-off-by: Ignacio Hagopian <[email protected]> * fix CREATE2 as well --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * quell linter --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]>
Signed-off-by: Ignacio Hagopian <[email protected]>
Support post-genesis transition Reactivate a post-genesis transition for a kaustinen-like testnet that goes through the conversion. It adds the ability: * To undergo the verkle transition "verge" past the genesis block, which so far was impossible with kaustinen-with-shapella. * To perform a shadow fork of holesky in the special kurtosis setup defined by the devops team. It is also serving as the basis for multiple other branches, to be merged at a later time: * The branch to replay historical blocks in verkle * The branch to execute transition tests Co-authored-by: Ignacio Hagopian <[email protected]>
* add quick conversion test * add debug traces * activate debug rpc * where I discover I have an outdated version of geth on my desktop
* various verkle iterator fixes * remove unused nodeToDBKey
|
Not running CI, closing |
|
Not going to pursue this, will strive to merge more stuff into master first. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.