fix: introduce ddl controller to make all ddl's running in the background

[yezizp2012]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Introduced a ddl controller to run all ddl commands in the background to fix the deterministic recovery test fail in #8086 . The root cause is that when frontend got killed during running ddl, the requests will be cancelled by tonic, then only part of the operations will be applied in meta side.

Checklist For Contributors

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features).
• I have demonstrated that backward compatibility is not broken by breaking changes and created issues to track deprecated features to be removed in the future. (Please refer to the issue)
• All checks passed in ./risedev check (or alias, ./risedev c)

Checklist For Reviewers

• I have requested macro/micro-benchmarks as this PR can affect performance substantially, and the results are shown.

Documentation

• My PR DOES NOT contain user-facing changes.

Click here for Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

• Installation and deployment
• Connector (sources & sinks)
• SQL commands, functions, and operators
• RisingWave cluster configuration changes
• Other (please specify in the release note below)

Release note

Fix #8086
Fix #8131

[codecov]

Codecov Report

Merging #8145 (d71a2a8) into main (b640cbd) will decrease coverage by 0.07%.
The diff coverage is 2.17%.

@@            Coverage Diff             @@
##             main    #8145      +/-   ##
==========================================
- Coverage   71.64%   71.57%   -0.07%     
==========================================
  Files        1133     1134       +1     
  Lines      182249   182402     +153     
==========================================
- Hits       130575   130561      -14     
- Misses      51674    51841     +167     

Flag	Coverage Δ
rust	71.57% <2.17%> (-0.07%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/meta/src/manager/catalog/fragment.rs	30.51% <0.00%> (ø)
src/meta/src/rpc/ddl_controller.rs	0.00% <0.00%> (ø)
src/meta/src/rpc/service/ddl_service.rs	0.00% <0.00%> (ø)
src/meta/src/manager/catalog/mod.rs	32.82% <9.33%> (+0.08%)	⬆️
src/batch/src/executor/group_top_n.rs	68.23% <0.00%> (-6.48%)	⬇️
src/object_store/src/object/mem.rs	86.87% <0.00%> (-0.39%)	⬇️
src/storage/src/hummock/compactor/iterator.rs	96.40% <0.00%> (-0.28%)	⬇️
src/storage/src/hummock/compactor/mod.rs	80.69% <0.00%> (-0.20%)	⬇️
src/storage/src/hummock/sstable_store.rs	64.77% <0.00%> (-0.16%)	⬇️
... and 3 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[BugenZhao]

LGTM!!

[xxchan]

(Can this PR be related with #8051? 🥸

[yezizp2012]

(Can this PR be related with #8051? 🥸

Yes, that issue is partially supported in this PR but to fully support it still requires some persistence of state and metadata.