nightly-20240220 nexmark perf degradation

[cyliu0]

Describe the bug

+---------------------------------------------------------------+--------------+------------+---------------------+-----------------------------+
| BENCHMARK NAME                                                | EXECUTION ID | STATUS     | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS |
+---------------------------------------------------------------+--------------+------------+---------------------+-----------------------------+
| nexmark-q17-blackhole-medium-1cn                              |        21038 | Negative   | -28.64%             | -14.31%                     |
| nexmark-q5-rewrite-blackhole-medium-1cn                       |        21047 | Negative   | -31.09%             | -19.14%                     |
| nexmark-q19-blackhole-medium-1cn                              |        21053 | Negative   | -23.29%             | -11.83%                     |
| nexmark-q105-blackhole-medium-1cn                             |        21088 | Negative   | -30.68%             | -12.06%

http://metabase.risingwave-cloud.xyz/dashboard/241-nexmark-blackhole-1cn-anti-affinity-rw-avg-source-throughput?start_date=2024-02-01&namespace=daily

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

nightly-20240220

• 4197ad50c766c8d086d29948ef34e13582e72251 feat(object store): introduce new s3 object store via OpenDAL (#14409)
• 06572cfe221cb127e94aef5b908931aee4be1ba9 fix(cdc): enable cdc transaction for shared source only (#14946)
• b1ffbafcd2d8a27297aebccb603e1c393f3b5c8f refactor: rename TO_TIMESTAMP{,1} in expr.proto for clarity (#15143)
• 6739733daa3de7c24261a94eec134f346e96b7e3 fix(storage): remove unnecessary metadata io during list (#15133)

[lmatz]

I suppose #14409 OpenDAL is not enabled in the daily test yet, see: https://github.com/risingwavelabs/kube-bench/pull/368

Then the only PR (suppose the regression is indeed caused by kernel changes) can only be #15133. Do you think it is possible to lead to such a big regression? @zwang28

The other two, #14946 affects the CDC source only, and #15143 is purely renaming.

might as well check changes in the testing environment

[st1page]

The queries all are CPU intensive(more than 750/800 CPU usage) and I do not find any IO performance difference. We can generate the CPU flame graph later if we can not find the issue on the PR side.

[zwang28]

Then the only PR (suppose the regression is indeed caused by kernel changes) can only be #15133. Do you think it is possible to lead to such a big regression?

No. This PR only affect openDAL.

[st1page]

https://buildkite.com/risingwavelabs/cpu-flamegraph/builds/31
https://buildkite.com/risingwavelabs/cpu-flamegraph/builds/32

[st1page]

https://buildkite.com/risingwavelabs/cpu-flamegraph/builds/31 https://buildkite.com/risingwavelabs/cpu-flamegraph/builds/32

No significant difference between the two flame graph...

[st1page]

check if it can be reproduced

BENCH_NAMESPACE="sts0220"
BENCH_TESTBED="medium-1cn"
CI="true"
ENABLE_BLACKHOLE="true"
LABELS="medium-1cn-anti-affinity"
NEXMARK_QUERY="q5-rewrite,q17,q19,q105"
RW_VERSION="nightly-20240220"

https://buildkite.com/risingwave-test/nexmark-benchmark/builds/3106

[st1page]

check if it can be reproduced

BENCH_NAMESPACE="sts0220"
BENCH_TESTBED="medium-1cn"
CI="true"
ENABLE_BLACKHOLE="true"
LABELS="medium-1cn-anti-affinity"
NEXMARK_QUERY="q5-rewrite,q17,q19,q105"
RW_VERSION="nightly-20240220"

https://buildkite.com/risingwave-test/nexmark-benchmark/builds/3106

The reruned result comes back 😇

[lmatz]

nexmark-q5-rewrite-blackhole-medium-1cn:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1708453098000&to=1708454903000&var-namespace=nexmark-bs-0-14-daily-20240220

task failure

[lmatz]

nexmark-q17-blackhole-medium-1cn:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1708450421000&to=1708452226000&var-namespace=nexmark-bs-15-105-daily-20240220

but this one occurs at the end I suppose should not have so much impact(-14%)?

[lmatz]

nexmark-q19-blackhole-medium-1cn:
https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1708454470000&to=1708456275000&var-namespace=nexmark-bs-15-105-daily-20240220

[Li0k]

q105

Base level compaction always skip.

I found that a task hangs for a long time without being canceled, which is why the task can't be picked.

fix pr : #15194

[lmatz]

I suppose #15194 fixed the problem in q105, but not the compaction task failure shown in q17 and q19, right? @Li0k

if so, I will keep the issue open