t8n: verkle genesis

[jsign]

This PR adds support for verkle-genesis test for t8n.

[jsign]

Each run uploads the generated fixtures as artifacts. For example, see the bottom of this run.

As we chatted in TG, after everything is stabilized we should probably:

• Make these filling CI run once per day or similar, instead of on every PR since it takes a long time to run and that can be annoying.
• The CI to be run on each PR, should be only the "consumption" of fixed (i.e: already generated) fixtures. Running fixtures is quite fast, and also it makes sense to target a stable/tagged set of fixtures. (i.e: filling fixtures on the same branch that consume them always has a high chance of always passing. The main idea is consuming against previous fixtures to catch regressions).

In any case, we can chat when we get a stable set of fixtures and we can change a bit the CI jobs.

[jsign]

With this line we generate all "targetable" tests on a Verkle fork, that is, using a VKT for the tree which includes:

• All the new verkle-genesis test vectors I created.
• All existing tests previously that could be run on a VKT (i.e: tests created from Shanghai or even previous ones).

These means around 458 filled test-vectors using verkle trees.

[jsign]

Here the single state-conversion test vector is filled since we target EIP6800Transition. But also we try to fill all existing tests from Shanghai forward; this isn't strictly useful for "Verkle needs" but we're just making sure filling for previous forks keeps working (and in fact, I had to fix things in t8n to make this work again).

This fills ~605 tests. Most of them aren't entirely useful but only the state-conversion one. The usefulness of consuming those fixtures, is making sure the EL client can keep passing tests for previous forks (e.g: full sync isn't broken).

The more test the better, but probably the other 458 fixtures are much more important to catch Verkle bugs for devnet7.

[jsign]

Consume all fixtures. As I mentioned in other place, I found a problem in consume direct which doesn't return !=0 if any test fails, so the CI doesn't "fail" of some test fails.

That's quite bad since that's the hole point of the CI, but Spencer will work on that to fix it. Until that happens, it might be good to "eyeball" the CI run logs and see all passed.

[jsign]

In the case of a non-verkle (nor state-conversion) fork, we simply re-open the MPT with the mptRoot calculated before. This fixes consuming fixtures for those.

[gballet]

I expect some pushback on the geth side, but we'll see. I'd prefer to be "agnostic" but they might prefer it to be obvious.

[jsign]

This is a new CLI tool to help the testing framework deal with a situation I found when fixing test filling.

The testing-library uses their own code for generating the genesis block. They only have an MPT implementation, so the genesis block was always generated with a MPT root. This is wrong for verkle-genesis tests.

Until they have a pure Python impl of a Verkle Tree and its cryptography, I offered them a new CLI that can calculate this root for them in the case we're in verkle mode. Whenever there's a execution-spec codebase implemented in Python for Verkle, they can copy-paste that implementation and come back generating the genesis vector on their side (and we can remove this CLI helper). For now, this should work and unblocked us from the problem.

[jsign]

This is a required fix to make state-conversion fixture consumption work. For that test (and future state-conversion ones), whenever we generate the genesis block before running test-blocks we need to enable pre-image recording.

This is needed since in the block execution of overlay tree we require the preimages of previous blocks (i.e: genesis block) to iterate the MPT and do the actual conversion.

[jsign]

This is a bug I found when filling the verkle-genesis tests. One of them made geth panic. After digging here was the cause.

The problem was that doing -= clearly can cause an integer underflow. This made geth crash later in the execution since the gas available was garbage.

The fix is simple, use the usual UseGas which does the required checks.

[jsign]

Remember that now we run tests with snapshots enabled? So that's the snaphsotter value, which now is always true.

Before, this if wasn't run since clearly snapshotter was always false.

So the problem here is that if I don't disable this if now, the test consumption fails. And the failure is basically that the re-generated snapshot doesn't match the underlying tree.

Unfortunately, we need to do this since in the state-conversion test the snapshot to be reconstructed actually needs to consider two trees, and not the "main active" one since it's lacking more data (i.e: things that still live in the MPT).

So I wasn't sure if it was worth the effort of trying to attempt fixing this since:

1. I don't know if the Geth team wants to keep doing this check?
2. This check isn't strictly necessary, since the actual test run assertion is done anyway. (i.e: checking that the generated new state root matches with the provided one, gas usage, etc... all that still is done (remember, this if wasn't run when you run without snapshots... it's more like a "are snapshots still coherent" kind of check than a "fixture run verifies" check).

I'd say we can live with this for a while, but eventually it should be fixed or maybe only skipped if the underlying tree is an Overlay Tree (if that helps).

[jsign]

This is how --cache.preimages work under the hood, which is used in L127.

[gballet]

I expect some pushback on the geth side, but we'll see. I'd prefer to be "agnostic" but they might prefer it to be obvious.

[gballet]

Suggested change

	statedb.SetBalance(addr, account.Balance)
	// TODO set back to AddBalance after rebase
	statedb.SetBalance(addr, account.Balance)