Signed-off-by: Simon Baatz <[email protected]>

Up to now the sentinel test initialization verified that all sentinels detect each other.
However, detection does not imply connection, which led to intermittent failures in the
coordinated failover tests (no leader elected since disconnected sentinels do not take part in a
vote).

Fix this by waiting until no sentinel reports being "disconnected".

Signed-off-by: Simon Baatz <[email protected]>

Valkey supports the "FAILOVER" command to switch primary and replica
roles in a coordinated fashion. Add a "COORDINATED" option to "SENTINEL
FAILOVER". When given, use "FAILOVER" in the Sentinel forced failover
procedure. Since we force the failover, i.e. we don't elect a leader,
the failover must succeed quickly before other Sentinels will regard
the primary as down.

Signed-off-by: Simon Baatz <[email protected]>

The FAILOVER command may ["encounter some scenarios it can not automatically remediate from and
may get stuck."](https://valkey.io/commands/failover/#failover-abort). The supervision of the
Sentinels is used to handle these situations:

1. "REPLICAOF" is not accepted during a failover. Thus, send a "FAILOVER ABORT" before sending a
   "REPLICAOF" to a node in `sentinelSendReplicaOf()`. (If there is no ongoing failover, the
   resulting error will just be ignored)
2. Sentinel monitors nodes for deviations from the expected state (wrong role or replication).
   Add a check for replicas that are in a failover state for too long and reconfigure them.

In addition, monitoring the failover state is used to find out whether the FAILOVER command
is supported by the Valkey instance. If not, don't use "FAILOVER" or "FAILOVER ABORT".

Signed-off-by: Simon Baatz <[email protected]>

Doing a proper leader election allows to increase the time available
for the actual FAILOVER command, as other Sentinels will not attempt
to initiate another failover during the failover timeout.

Interestingly, Sentinel does not take the reported primary up/down
status into account when counting replies to leader election. Thus,
Sentinel will simply proceed with the failover once we reach the
quorum even if we don't are in O_DOWN state.

Signed-off-by: Simon Baatz <[email protected]>