Large reorg test

[Tristan-Wilson]

If the expectedCounter field for this test is increased to 10_000 then it will fail with the following error:

WARN [07-16|20:35:55.452] Large chain reorg detected              
 number=2 hash=7f6bea..02fe3e drop=10000 dropfrom=b187f2..c3897a a
dd=0 addfrom=000000..000000                                       
INFO [07-16|20:35:55.648] Chain head was updated                  
 number=2 hash=7f6bea..02fe3e root=080b98..3b3b72 elapsed=422.2294
66ms age=1m46s                                                    
INFO [07-16|20:35:55.648] Trying to resequence messages           
 number=10000                                                     
ERROR[07-16|20:35:55.649] failed to re-sequence old user message r
emoved by reorg err="insert lock taken"                           
ERROR[07-16|20:35:55.669] Failed to journal state snapshot        
 err="snapshot [0x080b98823f4e081af621c256e1448a6f6a636e394f729553
6fb0fa64873b3b72] missing"                   

We are tracking resolution of this issue separately and can increase the reorg size once it's fixed.

[diegoximenes]

The error is due to this insertionMutex being held here.

During the reorg:

1. TransactionStreamer, holding insertionMutex lock, calls ExecutionEngine, which then adds old messages to a channel, so they can be resequenced asynchronously by ExecutionEngine. After that, and before releasing the lock, TransactionStreamer does more computations.
2. Asynchronously, ExecutionEngine reads from this channel and calls TransactionStreamer, which expects that insertionMutex is free in order to succeed.

This issue is more likely to occur with larger reorgs, since step 1 will take more time to execute, which make step 2 more likely to start while step 1 is still executing.

Not sure if this issue is already tracked in Linear, but we can bring it up with the team.

[Tristan-Wilson]

Fixes NIT-2686

[diegoximenes]

This test is failing with path scheme :(

There are two issues that I am aware with it:

• path scheme will only be able to handle reorgs that want to revert to a block before than HEAD-128 if ancients storage is enabled. Our tests don't enable it today, here. We should be using OpenDatabaseWithFreezer for this test instead. Not sure if there is an issue with using freezer in all Nitro's system tests 🤔, if not it will be simpler to adjust this :)
• NewNodeBuilder(ctx).DefaultConfig currently sets gethexec.CachingConfig.StateHistory to 0. We should set to a different value for this test. As an example, if StateHistory is set to 2 then it will be possible to revert to HEAD-130 but not to HEAD-131.

But even if those issues are fixed right now, not sure if we should merge this PR without first solving the issue with re-sequencing old messages during a reorg.
Even TestReorgResequencing, that has a smaller reorg than the one that is being proposed in this PR, is flaky, sometimes failing with the same error that you added to this PR's description.

[CLAassistant]

All committers have signed the CLA.