indexer-agent: Warn and wait if subgraph is behind

[fordN]

The data from the network subgraph (and the epoch subgraph) provide vital information for making decisions and sending transactions.
We should add checks to verify that a subgraph is up to the chain head (within a range). If the network subgraph is behind the agent should warn loudly and wait.

Requirements

Beta Give feedback

1. Define a freshness threshold (block distance) to which the Network Subgraph and the Epoch Subgraph are allowed to be away from the chain head for their data to be considered valid.
2. Let that freshness threshold be configured within each network specification file, as different networks have different block speeds.
3. Provide sensible defaults to each supported protocol network so users can safely update their stack without changing their configuration.
4. For single-network setups, add the appropriate command line option, which defers setting its default value after identifying the network the agent is currently operating on. This is required because there's no upfront protocol network identification for single-network setups
5. Implement a check that queries each protocol subgraph (Network, Epoch) for its latest indexed block and compares it to the current chain's head block number. If their difference is higher than the freshness threshold, log a warning and sleep for a predetermined amount of time.
6. Include that check at the start of every protocol subgraph query to prevent outdated subgraph data from entering the rest of the system.
7. Gracefully handle the cases for when that check fails, with a sensible warning/log explaining why the Agent will pause for a few seconds before retrying.
8. The Agent will keep warning, sleeping, and retrying until the subgraph is fresh. Adverse network conditions may trap it in an endless loop, requiring user intervention.

Success Criteria

Beta Give feedback

1. New unit tests
2. Manual test: Run the Agent with an extremely low freshness threshold value (like zero) and observe if it emits the expected warnings and halts its operation.
3. Manual test: Create a new indexing rule and confirm that the Agent manages its deployment (Qma9jonh6cqBMdYuh9arVpc32pRr3LLWFXerweGSfbJSoV)
4. Manual test: Observe the Agent creating, closing and renewing allocations (https://goerli.etherscan.io/tx/0xf0ffd614c22aa4b571423e23aa720e18e7a60a84f687f03849b6895e835316ef#eventlog)

[alex-pakalniskis]

@tilacog what are the freshness thresholds going to be for each network?

[tilacog]

Based on what we observe with the EBO, which has a similar check, 50 blocks for Ethereum and 500 blocks for Arbitrum.

I'm still not sure if those values would be the same for the Agent, but we will obtain an estimate later today when we run it.