nightly-20240314 backfill performance degradation

[cyliu0]

Describe the bug

https://buildkite.com/risingwave-test/backfill/builds/396

+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| BENCHMARK NAME                                                | EXECUTION ID | STATUS     | KEY METRICS                                        | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS | FLUCTUATION OF LAST EXECUTION |
+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| backfill-datagen-100m-rows-medium-arm-all-affinity            |        22907 | Negative   | backfill-create-mv-latency-per-100m-rows           | 10.96%              | 14.89%                      | 43.97%                        |
| backfill-datagen-100m-rows-medium-arm-all-affinity            |        22907 | Negative   | backfill-create-watermark-mv-latency-per-100m-rows | 48.86%              | 37.87%                      | 26.71%                        |

http://metabase.risingwave-cloud.xyz/question/17042-backfill-datagen-100m-rows-medium-arm-all-affinity-backfill-create-watermark-mv-latency-per-100m-rows-ms-history-tltb-2959?start_date=2024-02-13

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20230314

Additional context

https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240314

[cyliu0]

We have another performance degradation for nightly-20240320.

http://metabase.risingwave-cloud.xyz/dashboard/979-backfill-datagen-100m-rows-medium-arm-all-affinity-history?start_date=2024-02-16

+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| BENCHMARK NAME                                                | EXECUTION ID | STATUS     | KEY METRICS                                        | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS | FLUCTUATION OF LAST EXECUTION |
+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| backfill-datagen-100m-rows-medium-arm-all-affinity            |        23391 | Negative   | backfill-batch-query-latency-per-100m-rows         | 2.89%               | -0.31%                      | 1.51%                         |
| backfill-datagen-100m-rows-medium-arm-all-affinity            |        23391 | Negative   | backfill-create-mv-latency-per-100m-rows           | 123.99%             | 56.97%                      | 49.61%                        |
| backfill-datagen-100m-rows-medium-arm-all-affinity            |        23391 | Negative   | backfill-create-watermark-mv-latency-per-100m-rows | 131.89%             | 50.37%                      | 40.41%                        |
| backfill-datagen-100m-rows-medium-arm-all-affinity            |        23391 | Negative   | backfill-mv-query-latency-per-100m-rows            | 11.50%              | 5.59%                       | 10.18%                        |

[cyliu0]

Another backfill perf degradation for nightly-20240326

+--------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| BENCHMARK NAME                                         | EXECUTION ID | STATUS     | KEY METRICS                                        | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS | FLUCTUATION OF LAST EXECUTION |
+--------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| backfill-datagen-100m-rows-medium-arm-all-affinity     |        23889 | Negative   | backfill-batch-query-latency-per-100m-rows         | 10.41%              | 5.66%                       | 3.14%                         |
| backfill-datagen-100m-rows-medium-arm-all-affinity     |        23889 | Negative   | backfill-create-mv-latency-per-100m-rows           | 73.13%              | 8.33%                       | 42.10%                        |
| backfill-datagen-100m-rows-medium-arm-all-affinity     |        23889 | Negative   | backfill-create-watermark-mv-latency-per-100m-rows | 105.71%             | 18.10%                      | 40.31%                        |
| backfill-datagen-100m-rows-medium-arm-all-affinity     |        23889 | Negative   | backfill-mv-query-latency-per-100m-rows            | 10.74%              | 4.52%                       | 4.71%                         |

[hzxa21]

@Little-Wallace Can you help take a look?

[cyliu0]

According to the test result, it's fixed by #16519
Thanks @shanicky