potential sysbench insert perf degradation and barriers pile up

[lmatz]

          And the sysbench prepare job failed to check completion in 5 retries with an interval of 60 seconds. So I think there might be insertion performance degradation as well on `nightly-20231211`. Because the prepare finished on time with same configuration when running on `nightly-20231210`

https://buildkite.com/risingwave-test/sysbench/builds/620#018c5aaf-b799-4cd2-9e4e-d2b89cd4b7e0

Originally posted by @cyliu0 in #13931 (comment)

Commit history:
https://github.com/risingwavelabs/rw-commits-history#nightly-20231211

[lmatz]

sysbench-daily-test-20231210:

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=P2453400D1763B4D9&var-namespace=sysbench-daily-test-20231210&var-instance=benchmark-risingwave&var-pod=All&var-component=All&var-table=All&from=1702242323849&to=1702243006982

sysbench-daily-test-20231211:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=P2453400D1763B4D9&var-namespace=sysbench-daily-test-20231211&var-instance=benchmark-risingwave&var-pod=All&var-component=All&var-table=All&from=1702328529992&to=1702330072720

somehow the barriers pile up after insertion or when insertion is close to finish?

[lmatz]

It looks more like a bug, the duration of insertion and the throughput of insertion between 1210 and 1211 are quite close
And from the commit history, it looks like no PR is directly related to insertion.

[cyliu0]

Happened again with nighty-20231212 https://buildkite.com/risingwave-test/sysbench/builds/621#018c5fd6-5b1d-46a8-9339-e86ffcf9525a

[cyliu0]

Still failed with nightly-20231213 https://buildkite.com/risingwave-test/sysbench/builds/622#018c64fc-7290-419b-a2aa-5d07bc32014a

[chenzl25]

nightly-20231213

I couldn't find this day's Grafana metric data. Or how to keep them from being deleted.

[kwannoel]

nightly-20231213

I couldn't find this day's Grafana metric data. Or how to keep them from being deleted.

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1702439537779&orgId=1&to=1702517972779&var-component=All&var-datasource=P2453400D1763B4D9&var-instance=benchmark-risingwave&var-namespace=sysbench-daily-test-20231213&var-pod=All&var-table=All

You can adjust the timestamp range, otherwise namespace will not showup.

[cyliu0]

@chenzl25 We have metrics lost last night. They are investigating the problem.

[chenzl25]

@chenzl25 We have metrics lost last night. They are investigating the problem.

Anyway. We have located the root cause.