fix(Dataworker): Account for total required netSendAmount when executing PoolRebalanceLeaves

[nicholaspai]

Context

The dataworker executor functionality is supposed to detect when to call sync() before executing L1 PoolRebalance and RelayerRefund leaves depending on the liquidReserves value of l1 tokens before executing those leaves.

There are two important areas where we might want to call sync before executing any pool rebalance leaves:

1. First, we check if we should call sync when executing the Ethereum pool rebalance leaf and the Ethereum relayer refund leaves. Pool rebalance leaves subtract netSendAmount from liquidReserves while refund leaves add amountToReturn to liquid reserves (because Ethereum refund leaves are on the same chain as the hub pool, this amountToReturn is transferred atomically). This step is done here.
2. Secondly, we check if we need to call sync before executing the non-Ethereum pool rebalance leaves. These all will subtract their netSendAmounts from the HubPool. This step subtly takes in any updated liquid reserves value from the first step if we executed a sync pre-execution of the Ethereum pool rebalance leaves. It also is aware of any used balance from netSendAmounts in the first step via the BalanceAllocator.

Problem

In the second step where we call _updateExchangeRatesBeforeExecutingNonHubChainLeaves we are not evaluating the total netSendAmount obligation of all non-Ethereum PoolRebalance leaves we want to execute. This means we can submit a transaction to execute a set of pool leaves whose netSendAmounts might sum greater than the liquidReserves (plus any BalanceAllocator adjustments), but individually the netSendAmounts are all less than liquidReserves. This batch transaction would then fail in simulation.

Examples of problems

We saw a bundle of PoolRebalanceLeaves today fail to execute because three of the leaves, one Ethereum leaf and two non-Ethereum leaves, had a total netSendAmount greater than the HubPool's liquidReserves, but each individually had a netSendAmount < the liquidReserves. For example, the three leaves had netSendAmounts of:

• 40
• 90
• 70

While the hubPool's liquidReserves was 180:

• 40 + 90 + 70 = 200 > 180
• 40 < 180
• 90 < 180
• 70 < 180

If you take these numbers and run them through the executePoolRebalanceLeaves code above, you'll see how a PoolRebalanceLeaf execution was able to be submitted but then fail in simulation, without preceding the non-Ethereum pool rebalance leaf executions with a sync transaction.

PR side effects

This PR also refines the logic within the various internal functions called by executePoolRebalanceLeaves:

• _updateExchangeRatesBeforeExecutingHubChainLeaves: enqueues a sync call if necessary to execute any hub chain pool leaves
• _executePoolRebalanceLeaves: submits executeRootBundle calls but also adds any loadEthForL2Calls/transfer calls required to pay for an Arbitrum orbit L1->L2 message
• _updateExchangeRatesBeforeExecutingNonHubChainLeaves: enqueues a sync call if necessary to execute any non-hub chain pool leaves. Useful after _updateExchangeRatesBeforeExecutingHubChainLeaves which might pull Ethereum_SpokePool funds into the HubPool that need to be synced right before executing non-Hubchain leaves
• Various Dataworker logs are refined to be more helpful. For example, the blockRangesAreInvalid warning will now indicate which chain specifically requires a longer lookback.

Unit tests are updated as well

[nicholaspai]

Moving to draft stage as I think we can simplify this logic more #1933 (comment)

I've been thinking for a couple days now that something feels weird about including the balance allocator logic when executing pool leaves. It works when executing spoke leaves because spokes don't have a separate accounting of "liquid reserves"--all ERC20 balances are used to execute leaves, whereas the hub pool has a special accounting method.

[bmzig]

I think this makes sense. I also think the balanceAllocator/liquid reserves tracking in the executor is complicated, but in general, this looks to, at worst, add extra calls and log more info, so I think it's worth trying.