fix(scale): move reschedule_lock to ScaleController & use universal scale_controller

[shanicky]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Closes #15018 .

We need to implement mutual exclusion between the recovery and the auto scaling loop. Previously, the lock was inside the stream manager, which made it unattainable for recovery. By moving it to the scale controller, we can now acquire the lock.

Additionally, we mistakenly used separate scale controllers in both the stream manager and recovery, which resulted in the lock becoming ineffective. This pull request has also addressed and fixed that issue.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

[yezizp2012]

LGTM, could you please share some details about why that problem occurred?

[kwannoel]

Just asking some questions to better understand the problem and solution.

We need to implement mutual exclusion between the recovery and the auto scaling loop. Previously, the lock was inside the stream manager, which made it unattainable for recovery. By moving it to the scale controller, we can now acquire the lock.

During recovery, the stream manager is not online, is that why the lock is not acquirable on recovery?
Why scale controller is available during recovery, but not stream manager?

During recovery why do we need to acquire the reschedule lock? Is it to prevent any scaling from happening?

I suppose the scaling controller controls the scaling process, and so conceptually it also makes sense that it should hold the rw lock.

Additionally, we mistakenly used separate scale controllers in both the stream manager and recovery, which resulted in the lock becoming ineffective. This pull request has also addressed and fixed that issue.

Could you elaborate on how it led to the panic in #15018?

[shanicky]

Just asking some questions to better understand the problem and solution.

We need to implement mutual exclusion between the recovery and the auto scaling loop. Previously, the lock was inside the stream manager, which made it unattainable for recovery. By moving it to the scale controller, we can now acquire the lock.

During recovery, the stream manager is not online, is that why the lock is not acquirable on recovery? Why scale controller is available during recovery, but not stream manager?

During recovery why do we need to acquire the reschedule lock? Is it to prevent any scaling from happening?

I suppose the scaling controller controls the scaling process, and so conceptually it also makes sense that it should hold the rw lock.

Additionally, we mistakenly used separate scale controllers in both the stream manager and recovery, which resulted in the lock becoming ineffective. This pull request has also addressed and fixed that issue.

Could you elaborate on how it led to the panic in #15018?

First off, our barrier control and stream control are currently managed by two distinct managers with a lot of disconnected logic. This will be unified in the future.

During startup, our recovery loop and auto scale loop are initiated at the same time. For a newly added node, this means they will both generate a plan simultaneously, which is clearly incorrect. Previously, because the recovery loop couldn't acquire the lock, the two couldn't interfere with each other. We used to rely on the barrier generated by the auto scale loop failing to prevent issues. However, in some rare circumstances, if the recovery loop is sufficiently fast, it may result in the auto scale loop producing an outdated plan that could still be successfully deployed, leading to problems.

Therefore, we have moved the lock into the scale controller itself, so now both the recovery loop and the auto scale loop can possess the lock concurrently. Moreover, previously the individual scale controllers led to separate locks within each controller, which did not serve the purpose of mutual exclusion effectively.