weekly test: meta panicked during nexmark q4 cleanup

[cyliu0]

Describe the bug

And led to the following queries failure.

Failed Execution List:
- nexmark-q5-many-windows-blackhole-4x-medium-1cn-affinity: 30319, https://buildkite.com/risingwave-test/nexmark-benchmark/builds/3839
- nexmark-q7-blackhole-4x-medium-1cn-affinity: 30324, https://buildkite.com/risingwave-test/nexmark-benchmark/builds/3839
- nexmark-q9-blackhole-4x-medium-1cn-affinity: 30327, https://buildkite.com/risingwave-test/nexmark-benchmark/builds/3839

https://buildkite.com/risingwave-test/nexmark-benchmark/builds/3839#018fd488-bc9b-4353-b0c2-07d7f649e16c

meta log

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20240601

Additional context

https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240602

It failed in the weekly test. And there are not new commits for nightly-20240601 according to the history. We need to focus on the commits in the past week.

[lmatz]

cc @Little-Wallace @Li0k @yezizp2012

Could you take a look to see if it is directly caused by a compaction-group-related issue or if there is some deeper root cause(pure guess: maybe some wrong order of applying metadata operations)?

[zwang28]

related to #16862
@wenym1
The get_compaction_group_levels should be called on old version, but after #16862 is called on new version.

risingwave/src/meta/src/hummock/manager/compaction_group_manager.rs

Line 320 in ac93e24

let max_level = versioning

We should also review other similar cases.