nightly-20240427 perf degradation

[cyliu0]

Describe the bug

+----------------------------------------------------------+--------------+------------+-----------------------------------+---------------------+-----------------------------+-------------------------------+
| BENCHMARK NAME                                           | EXECUTION ID | STATUS     | KEY METRICS                       | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS | FLUCTUATION OF LAST EXECUTION |
+----------------------------------------------------------+--------------+------------+-----------------------------------+---------------------+-----------------------------+-------------------------------+
| nexmark-q4-blackhole-medium-1cn                          |        26945 | Negative   | avg-source-output-rows-per-second | -27.74%             | -14.42%                     | -13.11%                       |
| nexmark-q7-blackhole-4x-medium-1cn-affinity              |        26962 | Negative   | avg-source-output-rows-per-second | -28.79%             | -10.22%                     | -18.55%                       |
| nexmark-q9-blackhole-medium-1cn                          |        26994 | Negative   | avg-source-output-rows-per-second | -34.15%             | -19.17%                     | -5.36%                        |
| tpch-q20-bs-medium-1cn-affinity                          |        27042 | Negative   | avg-source-output-rows-per-second | -37.25%             | -14.30%                     | -25.02%                       |

http://metabase.risingwave-cloud.xyz/dashboard/241-nexmark-blackhole-1cn-anti-affinity-rw-avg-source-throughput?namespace=daily

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20240427

Additional context

https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240427

[cyliu0]

The last normal execution of nexmark-q4-blackhole-medium-1cn is for nightly-20240424

[fuyufjh]

Suspect fix(storage): update config and remove redundent code #15286

[lmatz]

If it is really because #15286
we should do a grid-search to set the default parameters

[cyliu0]

Looks like it's back to normal now. Close this for now. Feel free to reopen if you have anything to update