[WIP] Scheduled try-state checks and alerting

[liamaharon]

Schedule a Github Action every day to run try-runtime try-state checks on all runtimes.

This is a 0 -> 1 task to get our try-state checks deployed at all. In the future, it would be better for us to move these checks to dedicated infrastructure that can run them more frequently, perhaps using follow-chain.

Task list

• try-state passes on all runtimes
  • Westend: needs fix Broken Staking try-state invariants on Westend #2020
  • Rococo: checked Oct 26 ✅
  • Asset Hub Westend: checked Oct 26 ✅
  • Bridge Hub Rococo: checked Oct 26 ✅
  • Contracts Rococo: checked Oct 26 ✅
• Better logging: Improve try-state developer experience & fix bug #2019
• Use pre-built artefacts from latest release TODO: looking into why workflow isn't triggered on release
• Alerting (POST to a Matrix channel on any failure) https://github.com/paritytech/devops/issues/2917
• See if we can optimise build times in the runner we'll have caching to resolve this soon
• Figure out value passed in; value comes out; qed error from contracts rococo https://github.com/paritytech/polkadot-sdk/actions/runs/6796531510/job/18476812057?pr=1938#step:7:97)
  • Fixed in: Rejig block selection to properly respect --at try-runtime-cli#55

[bkchr]

What is the purpose of running it every hour?

[liamaharon]

What is the purpose of running it every hour?

To notify us asap in the case of any storage invariants breaking.

[bkchr]

What is the purpose of running it every hour?

To notify us asap in the case of any storage invariants breaking.

Shouldn't this be some continuous job?

[ggwpez]

Shouldn't this be some continuous job?

I think follow-chain should check the storage invariants, or?

[liamaharon]

Shouldn't this be some continuous job?

I think follow-chain should check the storage invariants, or?

Ideally, yes we would have dedicated infrastructure that runs follow-chain to check these invariants.

In reality, there're some drawbacks of using follow-chain:

• Requires high up-front and ongoing human and capital cost to maintain and monitor infrastructure for every chain, kept up to date with the latest runtimes.
• Couldn't actually run checks 'continuously' anyway. The checks are too expensive to run every block, so would be executed in round-robin mode, checking a subset of pallets each block.

Internally, follow-chain does the same thing as execute-block. The benefit of using execute-block directly is:

• Can define the job as a Github Action.
  • Trivial for Fellowship or parachain teams to replicate these checks in their own repos for their own runtimes (exponentially less initial and on-going cost compared to spinning up infra and CI/CD for follow-chain)
  • Trivial for all developers to monitor and maintain the job, rather than requiring work from other (infrastructure) team to maintain, monitor, and manage access to infra
• With artifact caching we can run it much more frequently than once an hr, so not like it would take that much longer to be notified of issues compared to follow-chain.

For these reasons, I suggested to Kian and he agreed to opt to go with this approach for checking storage invariants. My backlog of work is pretty large, so this option is nice as it allows us to get something out the door quickly that works and doesn't (as far as we could tell) have significant drawbacks compared with dedicated infra running follow-chain (which does have some significant drawbacks). If we've missed something let us know.

[ggwpez]

nice

[ggwpez]

Can do now?

[liamaharon]

1.3.0 did not include the Github workflow that builds the runtimes so we need to wait 1 more release for that to happen automatically, but I'm inclined to upload the debug builds there manually so we can use it.

[liamaharon]

Now that we're running full try-state checks in the CI I'm questioning how valuable this cron job will be.

Looking into some alternatives that will hopefully provide more value and be more easily accessible to ecosystem chains without them needing to spin up dedicated CI infra.