[FEATURE] Check status of remote istios before pruning stale revisions

[nrfox]

Is this the right place to submit this?

• This is not a question about how to use the sail-operator

Feature Description

If a control plane manages an external cluster but it isn't in use on the control plane cluster, the Sail Operator will consider the control plane revision stale and remove it. This is a problem if there are workloads on the remote cluster still connected to the control plane.

The Sail Operator should not automatically prune revisions that manage external clusters.

Additional Information

No response

[nrfox]

@luksa we previously spoke offline about the operator checking the status of the remote revision before pruning and doing this by using the remote secret that istiod itself uses. An additional idea I had on that same concept is: rather than having the operator in the primary cluster manually reconcile all the resources in the remote cluster to determine if its in use, what if the operator in the remote cluster writes the Istio status somewhere the primary operator can read and the primary operator reads that to determine the status of the remote istio.

e.g.

Remote operator --> reconciles status and writes status to a sail.operator.io annotation on the mutating webhook.
Primary operator --> detects remote secret --> uses remote secret to read the status of remote istiorevision from the mutating webhook. If there's a remote revision that is in use then the primary revision counts as in use as well.

If there's a remote secret but there's no webhook (where the remote status gets written to) then the primary operator can assume there's no remote revisions.

Trying to avoid having a single operator need to reconcile/watch all workload resources in remote clusters. I don't think that will scale very well.