nightly-20240523 backfill create mv latency perf degradation

[cyliu0]

Describe the bug

http://metabase.risingwave-cloud.xyz/dashboard/979-backfill-datagen-100m-rows-medium-arm-all-affinity-history?start_date=2024-04-23

+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| BENCHMARK NAME                                                | EXECUTION ID | STATUS     | KEY METRICS                                        | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS | FLUCTUATION OF LAST EXECUTION |
+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| backfill-datagen-100m-rows-medium-arm-all-affinity            |        29311 | Negative   | backfill-create-mv-latency-per-100m-rows           | 56.82%              | 46.75%                      | 56.82%                        |
| backfill-datagen-100m-rows-medium-arm-all-affinity            |        29311 | Negative   | backfill-create-watermark-mv-latency-per-100m-rows | 52.14%              | 38.62%                      | 52.14%                        |

https://buildkite.com/risingwave-test/backfill/builds/556

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20240523

Additional context

https://github.com/risingwavelabs/rw-commits-history#commit-history
https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240523

[cyliu0]

The re-ran result looks fine. https://buildkite.com/risingwave-test/backfill/builds/557

[cyliu0]

@kwannoel I'd like to reopen this one since the degradation is pretty stable to reproduce in the daily test of last week

[cyliu0]

Fixed by #17222 according to the daily performance test result http://metabase.risingwave-cloud.xyz/dashboard/979-backfill-datagen-100m-rows-medium-arm-all-affinity-history?start_date=2024-05-13