-
Notifications
You must be signed in to change notification settings - Fork 595
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nightly-20240518 weekly test perf degradation #16817
Comments
No conclusion has been reached regarding the reason for the performance degradation of TPCH Q8.
rerun 0518 (slow): https://buildkite.com/risingwave-test/tpch-benchmark/builds/1075 |
For nexmark q7, the network bandwidth between RW and Kafka is not the same: https://buildkite.com/risingwave-test/nexmark-benchmark/builds/3708
Hmmm, there is a slight chance that Kafka is not enough, or because the machine is not large enough and only an "unstable" "up to 12.5Gbps" bandwidth can be achieved, see #15142 (comment) |
Overall, there has been some fluctuation in the performance of the images on the 15th and 16th, but I believe the main performance drop is due to a change on the |
Unfortunately, it appears randomly... The degradation can happen in 0516's image but not happen in 0517's image... |
cc. @cyliu0 Does this problem recur now? |
It appears that the streaming performance of TPCH-Q8 is quite unstable. Before we focus on optimizing the streaming performance for TPCH, we will not conduct an in-depth investigation |
Just like @st1page said, the tpch q8 fluctuation is quite relatively big. But for weekly test, it's relatively stable. And the low points are the release test for v1.9.2-rt.1 and v1.10.0-rt.1. |
Describe the bug
Perf degrdation in weekly test. Those two SKUs runs only in weekly test now.
https://buildkite.com/risingwave-test/tpch-benchmark/builds/1074
https://buildkite.com/risingwave-test/nexmark-benchmark/builds/3708
http://metabase.risingwave-cloud.xyz/question/4966-tpch-q8-bs-medium-1cn-affinity-avg-source-output-rows-per-second-rows-s-history-thtb-371?start_date=2023-09-24
http://metabase.risingwave-cloud.xyz/question/9112-nexmark-q7-rewrite-blackhole-4x-medium-1cn-affinity-avg-source-output-rows-per-second-rows-s-history-thtb-2808?start_date=2023-12-21
Error message/log
No response
To Reproduce
No response
Expected behavior
No response
How did you deploy RisingWave?
No response
The version of RisingWave
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: