feat(node): new voting reactor state machine

[cryptoAtwill]

This PR is the start of in total 4 PRs that implements the new topdown flow. The implementation is broken down as follows:

• Implement the vote tally skeleton state machine (This PR). The skeleton state machine only outlines the state transition from Paused -> Active -> Paused. The other two recovery modes are yet to be implemented.
• Implement the vote tally logic: feat(node): migrate vote tally  #1139
• Implement the new syncer: feat(node): refactor parent view syncer #1151
• Integrate certified topdown checkpoint proposal into fendermint execution (prepare, process, execute): feat(node): integrating topdown cert into interpreters #1187

For this PR, the key changes are:

• Using tokio single thread reactor mode
• Use an enum to represent the different states of the state machine
• Introduce operation mode state machine skeleton with Active and Paused operation mode
• Use events emitted from syncer to communicate with the state machine: TopDownSyncEvent

The state machine is as follow:

stateDiagram-v2
  [*] --> Paused : Process start
  Paused --> Active : Synced
  Active --> Recovery : Checkpoints quiet
  Recovery --> Active : Checkpoints observed
  Active --> Paused : Stopping
  Recovery --> Paused : Stopping
	
  state Recovery {
    [*] --> SoftRecovery
    SoftRecovery --> HardRecovery : Still no new topdown checkpoints
    SoftRecovery --> [*] : New checkpoints
    HardRecovery --> [*] : New checkpoints
  }

Loading

[LePremierHomme]

I found the Paused naming to be misleading, because it only relates to the state of the votes publishing (I assume), while other operations are still running. What about Sync/Syncing?

[cryptoAtwill]

Paused could be triggered by many other condition, mostly not just in syncing. Just not implemented yet.

[LePremierHomme]

What I mean is that Paused isn't really idle, so this name is a bit confusing. I didn't refer to what triggers that mode, but to how it's best to describe it (given what it's doing).

[LePremierHomme]

break will only break the batch loop, before re-entering. There's no shutdown / error handling mechanism.

[cryptoAtwill]

if there are error, we just log it or maybe raise an alert, but the loop should continue running. As for shutdown, I think we can add it in VoteReactorRequest, should be pretty straightforward to do.

[LePremierHomme]

The rationale behind the sequential phases within a single advance(), for both modes, is unclear. Can you elaborate more on that?

[cryptoAtwill]

sorry, I dont quite get the question.

[LePremierHomme]

We can continue here: #1136 (comment)

[cryptoAtwill]

updated accordingly.