Archived state rebuilding is slow

[rolfyone]

Archived state retrieval seems to have become very slow when rebuilding states that weren't stored.

It appears to potentially be related to how busy the storage query channel is, but we need to investigate more.

To replicate, its relatively simple.

• mainnet archive node
• can use /teku/v1/beacon/state/finalized/slot/before/{slot} to find the most recent slot stored from a given point (added rest api endpoint to fetch closest available slot #8336)
• select a state beyond what you've got stored, maybe 10-15 slots ahead of it

time curl -X 'GET' \
 'http://localhost:5051/eth/v2/debug/beacon/states/<SLOT>' \
 -H 'accept: application/octet-stream' > /dev/null

Will go through StateTransition.processSlots -> StateTransition.processSlot
Have put a timer around preState.hashTreeRoot, and that's fast, but the processSlot functions often have a long period between calls, so it's possible it's just hung up in other queries.
It's possible that with so heavy use of the storage query channel, that it's now just very congested, but either way we need a lot more investigation on this and to fix whatever the root cause is.

[rolfyone]

will raise a follow up ticket to continue on electra state cache fix.