Execution client very slow to catchup

[ncavedale-xlabs]

Hi berateam!
We are having an issue with the execution client: whenever reth service is down for just a few minutes (like 2 or 3 minutes), it takes several hours to catchup (during this time, reth does not report block height). We're using pruning settings on reth, to keep some history.
This issue is happening with the following version:

• reth: v1.0.6 and v1.1.0
• beacond: v0.2.0-alpha.6 and v0.2.0-alpha.8

During catchup, this is all we see in reth logs (all normal I think):

Oct 17 12:54:25 reth[2168334]: 2024-10-17T12:54:25.286179Z  INFO Received forkchoice updated message when syncing head_block_hash=0xf6b083e91bef89b139ffd48d5f1446282802b26ea8b2169792287f32b5415776 safe_block_hash=0xa677f711b49f24c4d4f3fe42d68c8b6375a2903bcae410cab281221f0555d949 finalized_block_hash=0xa677f711b49f24c4d4f3fe42d68c8b6375a2903bcae410cab281221f0555d949
Oct 17 12:54:26 reth[2168334]: 2024-10-17T12:54:26.185520Z  INFO Finished stage pipeline_stages=9/14 stage=MerkleExecute checkpoint=5690672 target=5690672 stage_progress=100.00%
Oct 17 12:54:26 reth[2168334]: 2024-10-17T12:54:26.630001Z  INFO Status connected_peers=47 latest_block=5690093
Oct 17 12:54:28 reth[2168334]: 2024-10-17T12:54:28.896596Z  INFO Preparing stage pipeline_stages=10/14 stage=TransactionLookup checkpoint=5690093 target=5690672
Oct 17 12:54:28 reth[2168334]: 2024-10-17T12:54:28.896622Z  INFO Executing stage pipeline_stages=10/14 stage=TransactionLookup checkpoint=5690093 target=5690672
Oct 17 12:54:28 reth[2168334]: 2024-10-17T12:54:28.896622Z  INFO Updating transaction lookup tx_range=5690093..=5690672
Oct 17 12:54:28 reth[2168334]: 2024-10-17T12:54:28.898291Z  INFO Calculating transaction hashes tx_range=372315732..372397829
Oct 17 12:54:34 reth[2168334]: 2024-10-17T12:54:34.412474Z  INFO Inserting hashes append_only=false progress=10.00%
Oct 17 12:54:35 reth[2168334]: 2024-10-17T12:54:35.734721Z  INFO Received forkchoice updated message when syncing head_block_hash=0xa82fd262c095731e66e93474781a7ea5aa3441919b0f9f29d3dba4bfb9b8a6c0 safe_block_hash=0xf6b083e91bef89b139ffd48d5f1446282802b26ea8b2169792287f32b5415776 finalized_block_hash=0xf6b083e91bef89b139ffd48d5f1446282802b26ea8b2169792287f32b5415776
Oct 17 12:54:37 reth[2168334]: 2024-10-17T12:54:37.121047Z  INFO Received forkchoice updated message when syncing head_block_hash=0x7e80d467d711181463b7b818e596f70cd1ccf2e0d6a2fe4e247a58de8cabd879 safe_block_hash=0xa82fd262c095731e66e93474781a7ea5aa3441919b0f9f29d3dba4bfb9b8a6c0 finalized_block_hash=0xa82fd262c095731e66e93474781a7ea5aa3441919b0f9f29d3dba4bfb9b8a6c0
Oct 17 12:54:39 reth[2168334]: 2024-10-17T12:54:39.705310Z  INFO Inserting hashes append_only=false progress=20.00%
Oct 17 12:54:44 reth[2168334]: 2024-10-17T12:54:44.359277Z  INFO Received forkchoice updated message when syncing head_block_hash=0x802dea18a9e2753f602773afe14e725adcf3101438e01b125f29a7441bf462dc safe_block_hash=0x7e80d467d711181463b7b818e596f70cd1ccf2e0d6a2fe4e247a58de8cabd879 finalized_block_hash=0x7e80d467d711181463b7b818e596f70cd1ccf2e0d6a2fe4e247a58de8cabd879
Oct 17 12:54:44 reth[2168334]: 2024-10-17T12:54:44.875227Z  INFO Inserting hashes append_only=false progress=30.00%
Oct 17 12:54:45 reth[2168334]: 2024-10-17T12:54:45.643071Z  INFO Received forkchoice updated message when syncing head_block_hash=0x4dd1bedfd1e275e1de642c42200b3e39f68f49215e7b78bc61f52991a1032c7d safe_block_hash=0x802dea18a9e2753f602773afe14e725adcf3101438e01b125f29a7441bf462dc finalized_block_hash=0x802dea18a9e2753f602773afe14e725adcf3101438e01b125f29a7441bf462d

beacond logs:

Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z INFO Starting MConnection service module=p2p [email protected]:26656 impl=MConn{139.59.151.125:26656}
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z INFO Received incoming beacon block service=blockchain state_root=0xe26ffc0e0a7cb460c414e8323c05823c451ed93d256f2adf42d61d348b4614f6 slot=0x5df9bb
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z ERRR Received accepted syncing payload status service=execution-engine payload_block_hash=0x6083366bde3e9727691e1c8e13c6e6df12506c9caaf11d46bf651354ccffc675 parent_hash=0x2f955c029274965b4db1efac157c8121599ba43a855daeedfda049eec92f5c7d is_optimistic=false
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z ERRR Rejecting incoming beacon block ❌  service=blockchain state_root=0xe26ffc0e0a7cb460c414e8323c05823c451ed93d256f2adf42d61d348b4614f6 reason=payload status is SYNCING
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z ERRR failed to process proposal height=6158779 hash=80AEBDBC4625A82E603F1978F9DA7DD71BAFAE37FB74A9A997172F83278891DB err=payload status is SYNCING
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z ERRR Prevote step: state machine rejected a proposed block; this should not happen:the proposer may be misbehaving; prevoting nil module=consensus height=6158779 round=0 err=null
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z INFO Rebuilding payload for rejected block ⏳  service=blockchain
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z WARN Suggested fee recipient is not configured 🔆 service=engine.client fee-recipent=0x0000000000000000000000000000000000000000
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z WARN Received accepted syncing payload status during forkchoice update call service=execution-engine head_block_hash=0x2f955c029274965b4db1efac157c8121599ba43a855daeedfda049eec92f5c7d safe_block_hash=0x419234890a0c86e94cdf0dc12d5d72d3fcffb2e076515361c71ca02f74fd73bd finalized_block_hash=0x419234890a0c86e94cdf0dc12d5d72d3fcffb2e076515361c71ca02f74fd73bd error=payload status is SYNCING
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z INFO Connection is closed @ recvRoutine (likely by the other side) module=p2p [email protected]:26656 conn=MConn{139.59.151.125:26656}
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z INFO Stopping MConnection service module=p2p [email protected]:26656 impl=MConn{139.59.151.125:26656}
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z ERRR Stopping peer for error module=p2p peer=Peer{MConn{139.59.151.125:26656} 9c50cc419880131ea02b6e2b92027cefe17941b9 out} err=EOF
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z INFO Stopping Peer service [email protected]:26656 module=p2p impl=Peer{MConn{139.59.151.125:26656} 9c50cc419880131ea02b6e2b92027cefe17941b9 out}
Oct 17 13:07:03 beacond[2176648]: 2024-10-17T13:07:03Z INFO Reconnecting to peer module=p2p [email protected]:26656

Is there something you could recommend for this?
I'm also going to create an issue in reth repo, but I figured I'd ask here as well since this might be related to beacon in some way.

[andreclaro]

We have also noticed that the safe and finalized blocks are not progressing since the upgrade to v1.1.0:

Latest Block Number: 5691261
Safe Block Number: 5601958
Finalized Block Number: 5601958

[chuck-bear]

Hey do you have the pruning enabled by any chance?

[ncavedale-xlabs]

Now we have replaced our reth node with geth, and it has already caught up.

But yes, we had pruning settings enabled on reth because we need to keep a certain amount of historic data.

[chuck-bear]

My guess is that was probably the pruning. I tested v1.1.0 from scratch no issue, not yet with a snapshot.

Reth is getting stuck sometimes, especially on some of the old versions.

[chuck-bear]

To catch up fast make sure that the beacond has also the last config here -> https://github.com/berachain/beacon-kit/tree/main/testing/networks/80084