"ExecutionPayloadContext is not provided for production of post-merge block" error with Teku

[kristijankuhar]

Hey everyone,

I am using eth2-testnet-genesis to create a shadowfork of Holesky with Besu and Teku.

I am performing the following steps:

1. Start the Holesky node and wait for it to sync
2. Prepare the following inputs:
   • config.json - a copy of the Holesky EL configuration
   • config.yaml - a copy of Holesky CL configuration with modified DEPOSIT_CONTRACT_ADDRESS, MIN_GENESIS_TIME (a minute or so in the past), MIN_GENESIS_ACTIVE_VALIDATOR_COUNT, *_FORK_VERSION, *_FORK_EPOCH.
   • mnemonics.yaml
3. I execute the following command:
   ./eth2-testnet-genesis deneb --config=config/config.yaml --mnemonics=config/mnemonics.yaml --eth1-config=config/config.json --shadow-fork-eth1-rpc=<my synced holesky node>
4. I restart Teku node with --initial-state=genesis.ssz, --network=config.yaml, new --data-path and specified --validator-keys and --validators-proposer-default-fee-recipient. I do restart around MIN_GENESIS_TIME+GENESIS_DELAY

In most cases Teku starts printing out errors like:

Aug 06 13:44:00 my.node teku[1001500]: 13:44:00.001 INFO - Slot Event *** Slot: 3, Block: ... empty, Justified: 0, Finalized: 0, Peers: 0
Aug 06 13:44:00 my.node teku[1001500]: 13:44:00.021 INFO - Validator *** Published attestation Count: 4, Slot: 3, Root: ff3e0721a34e2c0de1a4cb5f746482cac77d8ff7e04b70fc73c246494131b4b2
Aug 06 13:44:04 my.node teku[1001500]: 13:44:04.001 WARN - Validator *** Skipped aggregation at slot 3 because there was nothing to aggregate
Aug 06 13:44:04 my.node teku[1001500]: 13:44:04.006 INFO - Validator *** Published aggregate Count: 1, Slot: 3, Root: ff3e0721a34e2c0de1a4cb5f746482cac77d8ff7e04b70fc73c246494131b4b2
Aug 06 13:44:08 my.node teku[1001500]: 13:44:08.005 ERROR - Validator *** Failed to produce block Slot: 4 Validator: ae03ef7 java.lang.IllegalStateException: ExecutionPayloadContext is not provided for production of post-merge block at slot 4 (See log file for full stack trace)
Aug 06 13:44:12 my.node teku[1001500]: 13:44:12.001 WARN - Validator *** Skipped producing sync_signature while node is syncing Count: 1, Slot: 4

In some rare cases, after 5 hours of trying/experimenting, it successfully creates a shadow fork and everything works afterward.

I repeated this process a few times until now and would like to avoid making endless attempts before it starts working. Would you happen to know what is causing this error and how to prevent it?