nexmark q19 and q20 blackhole performance degradation on affinity testbed between 1130-1203

[cyliu0]

Describe the bug

The daily performance test shows we have performance degradation on nexmark q19 and q20 with affinity testbed. https://buildkite.com/risingwave-test/nexmark-benchmark/builds/2583#018c306a-745d-4244-9b94-8a9234f74a42

http://metabase.risingwave-cloud.xyz/question/7039-nexmark-avg-source-throughput?start_date=2023-11-01&workload=q20-blackhole

+-----------------------------------------------------------------+--------------+------------+---------------------+-----------------------------+
| BENCHMARK NAME                                                  | EXECUTION ID | STATUS     | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS |
+-----------------------------------------------------------------+--------------+------------+---------------------+-----------------------------+
| nexmark-q19-blackhole-medium-1cn-affinity                       |        15919 | Negative   | -18.25%             | -13.15%                     |
| nexmark-q20-blackhole-medium-1cn-affinity                       |        15927 | Negative   | -29.25%             | -22.28%                     |

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20231203

Additional context

No response

[xxchan]

commits history:

## nightly-20231203
- `0b6184d8fce4275e9a81a2d66c171c20d2b7a529` [doc(docker): add comments for minio disk full issue (#13768)](https://github.com/risingwavelabs/risingwave/pull/13768)

## nightly-20231202
- `ab011eb0e58b8637e9d53f70fd7221a49fcd9e75` [feat(jni): pass stream chunk directly without serde (#13430)](https://github.com/risingwavelabs/risingwave/pull/13430)
- `b149c67b46a7b42f253f7f09c1c69d9184fa7797` [refactor(error): simplify all boxed error wrapper definition (#13725)](https://github.com/risingwavelabs/risingwave/pull/13725)
- `7677abcb6fa87ea3f667ae2e111650296fdd8e97` [fix(cdc-backfill): ensure snapshot read starts after source (#13663)](https://github.com/risingwavelabs/risingwave/pull/13663)

## nightly-20231201
- `fce66c0b06fe27c30a89d9809ebab102b2d68e0e` [test: add vector demo to integration tests (#13753)](https://github.com/risingwavelabs/risingwave/pull/13753)
- `61d364cbea2d9e3687ce358c9f630c72bedd4494` [feat: check-in `cargo dylint` with `format_error` lint (#13750)](https://github.com/risingwavelabs/risingwave/pull/13750)
- `26bd9ef630fd360717be23535b39d07949be9570` [feat(storage): optimize data alignment for default compaction group (#13075)](https://github.com/risingwavelabs/risingwave/pull/13075)
- `0bd10c4f67746075b83913011e86a1eb8fb475e7` [fix(udf): fix decimal values (#11839)](https://github.com/risingwavelabs/risingwave/pull/11839)
- `da79ff5ba64362cb2ee07f3114040fa00e8e62c4` [chore: Adjust default large query memory (#13686)](https://github.com/risingwavelabs/risingwave/pull/13686)
- `67c2d70523fd4645c158428a11825e5e4926de7d` [feat(expr): distributed make timestamptz (#13647)](https://github.com/risingwavelabs/risingwave/pull/13647)

## nightly-20231130
- ce0121fc7cf30251b5bc58065d1ed5f09f4512ab ...

[xxchan]

compactor CPU increased, and compute CPU decreased

Is it possible that #13075 caused this? cc @Li0k

[fuyufjh]

Is it possible that #13075 caused this? cc @Li0k

I'm also suspecting this commit. Additionally, the bloom-filter positive rate differs a lot, which may indicate that the compactor cannot catch up with writes.

1130:

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1701371750000&to=1701373554000&var-namespace=nexmark-bs-15-105-affinity-daily-20231130

1203:

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1701630923000&orgId=1&to=1701632727000&var-datasource=Prometheus%3A+test-useast1-eks-a&var-namespace=nexmark-bs-15-105-affinity-daily-20231203

[Li0k]

Yes, I guess it has negative effects, do I need to revert it immediately?

[huangjw806]

Yes, I guess it has negative effects, do I need to revert it immediately?

If you can't fix it quickly, just revert it, but fortunately it's not in this release.

[Li0k]

I found a couple of preliminary conclusions

1. feat(storage): optimize data alignment for default compaction group #13075 wrong sst switch condition, which causes sst to become large

3. fix(compaction): fast compact may not cut sst if not meet point key #13690 stricter fast compact condition, which requires more cpu for compacting

• 1130

• 1203

I'm doing some test and expected to be fixed today

test: https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1701678218000&to=1701680021000&var-namespace=li0k-nexmark-q20-debug-fix-align

[cyliu0]

Is the PR in nightly-20231204? According to the test result, the perf is not totally recovered.

[Li0k]

Is the PR in nightly-20231204? According to the test result, the perf is not totally recovered.

Unfortunately, Should I need to run the nexmark manually?

[Li0k]

Compare affinity

CPU

before

after

throughput

before

after

[cyliu0]

According to the test result, it's fixed by #13790. But in the daily perf test, we found some new perf degradations #13821