2024-02-19 sysbench-select-random-limits-medium-3cn performance regression

[lmatz]

https://buildkite.com/risingwave-test/sysbench/builds/687#

https://risingwave-labs.slack.com/archives/C04R6R5236C/p1708387203780599

[lmatz]

http://metabase.risingwave-cloud.xyz/question/5639-sysbench-select-random-limits-medium-3cn-sysbench-qps-transactions-s-history-thtb-1910?start_date=2024-01-08

somehow this test case fluctuates widely

[lmatz]

http://metabase.risingwave-cloud.xyz/question/5639-sysbench-select-random-limits-medium-3cn-sysbench-qps-transactions-s-history-thtb-1910?start_date=2024-01-08

02-15 is much better: https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1708033097000&orgId=1&to=1708033408000&var-datasource=Prometheus:+test-useast1-eks-a&var-namespace=sysbench-daily-test-20240215

02-19 is much worse: https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1708378710000&to=1708379021000&var-namespace=sysbench-daily-test-20240219

CPU

02-15:

02-19:

somehow 02-19's CPU usage is consistently higher.

Hummock Read

02-15:

02-19:

drastically different, which raises the question if there is any difference in compaction

Compaction

02-15:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1708033097000&orgId=1&to=1708033408000&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=sysbench-daily-test-20240215&editPanel=131

02-19:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1708378710000&to=1708379021000&var-namespace=sysbench-daily-test-20240219&editPanel=131

Is the difference shown here considered very different?

02-15:

02-19:

Is the difference shown here considered very different?

guess not?

[lmatz]

searched the issue history and the fluctuation happened a while ago: #14022

@cyliu0 did the config change again?
or shall we change it again?
The CPU is right now not fully utilized I suppose,
since BENCH_TESTBED="medium-3cn" and medium machines have 8 cpus while currently only 600% is used.

also cc: @chenzl25

[cyliu0]

No... we didn't change it since the above issue. It's still 64 threads. Which config would you like to change?

[lmatz]

Which config would you like to change?

I am curious about the story behind why we changed the thread number in the previous issue: #14022

And also how was it determined that the problem was gone last time?

[cyliu0]

Which config would you like to change?

I am curious about the story behind why we changed the thread number in the previous issue: #14022

Originally from here https://risingwave-labs.slack.com/archives/C03GTKX3C8G/p1702015068556349

And also how was it determined that the problem was gone last time?

+1 @chenzl25

[lmatz]

Originally from here https://risingwave-labs.slack.com/archives/C03GTKX3C8G/p1702015068556349

thanks, I see, but I feel that not reaching full CPU usage is independent of having fluctuation, just cannot think of a possible cause that can relate these two phenomenon at the moment

[chenzl25]

Maybe we can increase the threads number for this test? because its performance is better than before now and our resources are not fully utilized/

[lmatz]

Maybe we can increase the threads number for this test? because its performance is better than before now and our resources are not fully utilized/

increase the sys_threads from 64 to an even larger number, e.g. 128 cc: @cyliu0 , could you run the new setting once, let's see if CPU can be fully utilized under such setting

[cyliu0]

The highest one 10667.46 is the 128 threads result https://buildkite.com/risingwave-test/sysbench/builds/689#018dc9f0-aa60-45d3-94c8-7848b15bc601

[cyliu0]

It's still not stable after we changed the threads to 128 in daily test. http://metabase.risingwave-cloud.xyz/dashboard/278-sysbench-rw-qps

[lmatz]

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1709501871000&to=1709502182000&var-namespace=sysbench-daily-test-20240303

CPU usage seems to be stable, but still fluctuate a lot

http://metabase.risingwave-cloud.xyz/question/5639-sysbench-select-random-limits-medium-3cn-sysbench-qps-transactions-s-history-thtb-1910?start_date=2024-01-08

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.