Performance degradation for nightly-20240102

[st1page]

Date: 2024-01-01
Scheduled Executions: 41
Passed Executions: 39
Pass Percentage: 95%
Positive Executions: 0
Negative Executions: 0
Reasonable Executions: 42
Failed Executions: 0
Not Run Benchmarks: 2
Not Run Benchmark List:
- sysbench-pg-cdc-100-tables-1m-rows-medium-1cn-affinity
- sysbench-mysql-cdc-100-tables-1m-rows-medium-1cn-affinity
Passed Execution List:
+-----------------------------------------------------------------+--------------+------------+---------------------+-----------------------------+
| BENCHMARK NAME                                                  | EXECUTION ID | STATUS     | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS |
+-----------------------------------------------------------------+--------------+------------+---------------------+-----------------------------+
| nexmark-q0-blackhole-medium-1cn-affinity                        |        17705 | Reasonable | -23.32%             | -1.06%                      |
| nexmark-q0-medium-1cn-affinity                                  |        17706 | Reasonable | -0.36%              | 3.21%                       |
| nexmark-q15-blackhole-medium-1cn-affinity                       |        17707 | Reasonable | -5.18%              | -0.23%                      |
| nexmark-q3-blackhole-medium-1cn-affinity                        |        17708 | Reasonable | -12.30%             | 0.77%                       |
| nexmark-q16-blackhole-medium-1cn-affinity                       |        17709 | Reasonable | -18.22%             | -0.89%                      |
| nexmark-q4-blackhole-medium-1cn-affinity                        |        17710 | Reasonable | -6.42%              | 1.12%                       |
| nexmark-q5-blackhole-watermark-medium-1cn-affinity              |        17711 | Reasonable | -9.67%              | 0.90%                       |
| nexmark-q17-blackhole-medium-1cn-affinity                       |        17712 | Reasonable | -11.85%             | -0.94%                      |
| nexmark-q5-blackhole-medium-1cn-affinity                        |        17713 | Reasonable | -9.09%              | 1.09%                       |
| nexmark-q7-blackhole-watermark-medium-1cn-affinity              |        17714 | Reasonable | -22.24%             | 0.70%                       |
| nexmark-q18-blackhole-medium-1cn-affinity                       |        17715 | Reasonable | -4.58%              | -1.16%                      |
| nexmark-q5-rewrite-blackhole-medium-1cn-affinity                |        17716 | Reasonable | -20.61%             | -7.60%                      |
| nexmark-q8-blackhole-watermark-medium-1cn-affinity              |        17717 | Reasonable | -4.53%              | -0.33%                      |
| nexmark-q19-blackhole-medium-1cn-affinity                       |        17718 | Reasonable | -10.24%             | -4.54%                      |
| nexmark-q7-blackhole-medium-1cn-affinity                        |        17719 | Reasonable | -30.50%             | 0.21%                       |
| nexmark-q9-blackhole-watermark-medium-1cn-affinity              |        17720 | Reasonable | -6.12%              | -0.54%                      |
| nexmark-q7-rewrite-blackhole-medium-1cn-affinity                |        17721 | Reasonable | -16.73%             | -0.16%                      |
| nexmark-q20-blackhole-medium-1cn-affinity                       |        17722 | Reasonable | -4.29%              | -0.70%                      |
| nexmark-q18-blackhole-watermark-medium-1cn-affinity             |        17723 | Reasonable | -3.02%              | -0.43%                      |
| nexmark-q8-blackhole-medium-1cn-affinity                        |        17724 | Reasonable | -8.31%              | -0.00%                      |
| nexmark-q101-blackhole-medium-1cn-affinity                      |        17725 | Reasonable | -3.98%              | -0.36%                      |
| nexmark-q6-group-top1-blackhole-watermark-medium-1cn-affinity   |        17726 | Reasonable | -5.72%              | 0.49%                       |
| nexmark-q9-blackhole-medium-1cn-affinity                        |        17728 | Reasonable | -6.07%              | 0.72%                       |
| nexmark-q102-blackhole-medium-1cn-affinity                      |        17729 | Reasonable | -6.83%              | -1.35%                      |
| nexmark-q12-blackhole-medium-1cn-affinity                       |        17730 | Reasonable | -5.84%              | -0.39%                      |
| nexmark-q5-many-windows-blackhole-watermark-medium-1cn-affinity |        17731 | Reasonable | -11.18%             | 1.94%                       |
| sysbench-oltp-point-select-medium-3cn                           |        17732 | Reasonable | -15.58%             | 0.80%                       |
| nexmark-q103-blackhole-medium-1cn-affinity                      |        17733 | Reasonable | -3.59%              | -0.60%                      |
| sysbench-select-random-points-medium-3cn                        |        17734 | Reasonable | -15.01%             | 3.41%                       |
| nexmark-q13-blackhole-medium-1cn-affinity                       |        17735 | Reasonable | -15.87%             | 0.23%                       |
| sysbench-select-random-ranges-medium-3cn                        |        17736 | Reasonable | -17.66%             | 3.78%                       |
| backfill-datagen-100m-rows-medium-all-affinity                  |        17727 | Reasonable | -7.61%              | -11.44%                     |
| sysbench-oltp-insert-medium-3cn                                 |        17737 | Reasonable | -6.78%              | 1.01%                       |
| sysbench-select-random-limits-medium-3cn                        |        17738 | Reasonable | -40.55%             | 13.81%                      |
| nexmark-q3-no-condition-blackhole-medium-1cn-affinity           |        17740 | Reasonable | -4.40%              | 0.74%                       |
| nexmark-q104-blackhole-medium-1cn-affinity                      |        17741 | Reasonable | -6.10%              | -2.43%                      |
| sysbench-bulk-insert-medium-3cn                                 |        17739 | Reasonable | 0.07%               | 0.28%                       |
| nexmark-q5-many-windows-blackhole-medium-1cn-affinity           |        17742 | Reasonable | -44.13%             | 6.61%                       |
| nexmark-q105-blackhole-medium-1cn-affinity                      |        17743 | Reasonable | -6.57%              | -1.84%                      |
+-----------------------------------------------------------------+--------------+------------+---------------------+-----------------------------+

PLEASE CHECK THE NEGATIVE EXECUTIONS :bangbang:Date: 2024-01-02
Scheduled Executions: 41
Passed Executions: 39
Pass Percentage: 95%
Positive Executions: 0
Negative Executions: 5
Reasonable Executions: 37
Failed Executions: 0
Not Run Benchmarks: 2
Not Run Benchmark List:
- sysbench-pg-cdc-100-tables-1m-rows-medium-1cn-affinity
- sysbench-mysql-cdc-100-tables-1m-rows-medium-1cn-affinity
Passed Execution List:
+-----------------------------------------------------------------+--------------+------------+---------------------+-----------------------------+
| BENCHMARK NAME                                                  | EXECUTION ID | STATUS     | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS |
+-----------------------------------------------------------------+--------------+------------+---------------------+-----------------------------+
| backfill-datagen-100m-rows-medium-all-affinity                  |        17769 | Negative   | 13.67%              | -0.47%                      |
| nexmark-q9-blackhole-medium-1cn-affinity                        |        17770 | Negative   | -19.37%             | -12.06%                     |
| sysbench-bulk-insert-medium-3cn                                 |        17781 | Negative   | -18.81%             | -16.36%                     |
| nexmark-q3-no-condition-blackhole-medium-1cn-affinity           |        17783 | Negative   | -16.59%             | -10.75%                     |
| nexmark-q105-blackhole-medium-1cn-affinity                      |        17784 | Negative   | -15.91%             | -10.35%                     |
| nexmark-q0-blackhole-medium-1cn-affinity                        |        17747 | Reasonable | -19.14%             | 3.78%                       |
| nexmark-q15-blackhole-medium-1cn-affinity                       |        17749 | Reasonable | -3.40%              | 1.43%                       |
| nexmark-q3-blackhole-medium-1cn-affinity                        |        17750 | Reasonable | -12.75%             | 0.22%                       |
| nexmark-q0-medium-1cn-affinity                                  |        17748 | Reasonable | 3.00%               | 5.80%                       |
| nexmark-q16-blackhole-medium-1cn-affinity                       |        17751 | Reasonable | -19.39%             | -2.02%                      |
| nexmark-q4-blackhole-medium-1cn-affinity                        |        17752 | Reasonable | -16.90%             | -9.04%                      |
| nexmark-q5-blackhole-watermark-medium-1cn-affinity              |        17753 | Reasonable | -14.49%             | -3.94%                      |
| nexmark-q17-blackhole-medium-1cn-affinity                       |        17754 | Reasonable | -11.42%             | -0.40%                      |
| nexmark-q5-blackhole-medium-1cn-affinity                        |        17755 | Reasonable | -13.86%             | -3.70%                      |
| nexmark-q7-blackhole-watermark-medium-1cn-affinity              |        17756 | Reasonable | -18.99%             | 4.27%                       |
| nexmark-q18-blackhole-medium-1cn-affinity                       |        17757 | Reasonable | -6.83%              | -3.07%                      |
| nexmark-q5-rewrite-blackhole-medium-1cn-affinity                |        17758 | Reasonable | -20.03%             | -6.12%                      |
| nexmark-q8-blackhole-watermark-medium-1cn-affinity              |        17759 | Reasonable | -8.28%              | -3.74%                      |
| nexmark-q19-blackhole-medium-1cn-affinity                       |        17760 | Reasonable | -4.91%              | 0.99%                       |
| nexmark-q7-blackhole-medium-1cn-affinity                        |        17761 | Reasonable | -32.38%             | -2.19%                      |
| nexmark-q9-blackhole-watermark-medium-1cn-affinity              |        17762 | Reasonable | -11.81%             | -5.79%                      |
| nexmark-q7-rewrite-blackhole-medium-1cn-affinity                |        17763 | Reasonable | -13.11%             | 3.64%                       |
| nexmark-q20-blackhole-medium-1cn-affinity                       |        17764 | Reasonable | -1.94%              | 1.52%                       |
| nexmark-q18-blackhole-watermark-medium-1cn-affinity             |        17765 | Reasonable | -4.30%              | -1.54%                      |
| nexmark-q8-blackhole-medium-1cn-affinity                        |        17766 | Reasonable | -15.51%             | -6.94%                      |
| nexmark-q101-blackhole-medium-1cn-affinity                      |        17767 | Reasonable | -8.24%              | -4.21%                      |
| nexmark-q6-group-top1-blackhole-watermark-medium-1cn-affinity   |        17768 | Reasonable | -6.98%              | -0.75%                      |
| backfill-datagen-100m-rows-medium-all-affinity                  |        17769 | Reasonable | 13.67%              | -0.47%                      |
| nexmark-q102-blackhole-medium-1cn-affinity                      |        17771 | Reasonable | -11.33%             | -5.39%                      |
| nexmark-q12-blackhole-medium-1cn-affinity                       |        17772 | Reasonable | -10.38%             | -4.57%                      |
| nexmark-q5-many-windows-blackhole-watermark-medium-1cn-affinity |        17773 | Reasonable | -13.36%             | -0.49%                      |
| sysbench-oltp-point-select-medium-3cn                           |        17774 | Reasonable | -19.60%             | -3.52%                      |
| sysbench-select-random-points-medium-3cn                        |        17775 | Reasonable | -17.99%             | -0.19%                      |
| nexmark-q103-blackhole-medium-1cn-affinity                      |        17776 | Reasonable | -6.35%              | -3.02%                      |
| sysbench-select-random-ranges-medium-3cn                        |        17777 | Reasonable | -20.19%             | 0.50%                       |
| nexmark-q13-blackhole-medium-1cn-affinity                       |        17778 | Reasonable | -21.10%             | -5.29%                      |
| sysbench-oltp-insert-medium-3cn                                 |        17779 | Reasonable | -10.29%             | -2.41%                      |
| sysbench-select-random-limits-medium-3cn                        |        17780 | Reasonable | -46.00%             | 2.88%                       |
| nexmark-q104-blackhole-medium-1cn-affinity                      |        17782 | Reasonable | -8.93%              | -4.73%                      |
| nexmark-q5-many-windows-blackhole-medium-1cn-affinity           |        17785 | Reasonable | -48.43%             | -1.40%                      |
+-----------------------------------------------------------------+--------------+------------+---------------------+-----------------------------+

nightly-20240102

• 56ce4e2bdf063c7a1cc5f565880c273c74fada7a fix(optimizer): internal tables return no stream key (#14303)
• 72ccc845dac6afaaa6fa1a29c6c94463b8c549ac fix(connector-node): only log events if enabled (#14295)
• f54cb93d9a16b382ab1a4809318fc9c64cd1c29e refactor(sink): unify log write path and distinguish table sink from external sink (#14260)
• 43383aaacbf1896b0430b253f30aea8c75f8d8e0 fix: dummy table fragments are not persist when drop sink with table replacement (#14299)
• 7c84d06ff4f8b705117b862619a1f6766ca2d4c0 chore: make some source log debug instead of info (#14225)
• b38da9cead2dbda82409b3e3926a85537253b309 chore(sink): exclude java iceberg and deltalake sink from connector node (#14277)
• 1f4b882dbf3f5b4571f1814cc88c54b861fa88f8 feat(sql-backend): [PART 2] introduce metadata manager and adapt it in all RPC services, dashboard, metric etc (#14191)
• 0c94093a898e78fa04bb295f08e0552cf334cad9 fix: fix checkpoint_frequency check (#13331)
• ceedd4195f8d5b3084f7b494ca0ab7965634246d fix(pgwire): TruncatedFmt should respect utf8 char boundary (#14292)
• 144e87a375c220f1571a6e49137a1fa9ebf882ba test: disable streaming over window range cache e2e temporarily (#14289)
• 1b691dd93910d7234e90f9ac8ca80f486f8c1628 refactor(source): consolidate duplicate schema loading logic for avro/protobuf/json-schema (#14281)
• 4f5b16707058f52f9221c571b2acedc996ea1377 chore(deps): Bump similar from 2.3.0 to 2.4.0 (#14288)
• 05fae9f6bf0885a50ec93be3aec5616aa5d07b34 chore(deps): Bump async-nats from 0.32.1 to 0.33.0 (#13296)
• 481aaaba820037d0d064cd678063fafeae2f1067 chore(deps): Bump google-cloud-pubsub from 0.21.0 to 0.22.0 (#14266)
• 9161189d54006c25d4d11b5480baf10a7a5976cf chore(deps): Bump golang.org/x/crypto from 0.9.0 to 0.17.0 in /integration_tests/client-library/go (#14278)

[st1page]

#13331 is suspicious

[xxchan]

#13331 is suspicious

Trying to confirm with

https://buildkite.com/risingwave-test/nexmark-benchmark/builds/2838

RW_CONFIG="{'system':{'checkpoint_frequency':2}}"
BENCH_NAMESPACE="xx-freq"
BENCH_TESTBED="medium-1cn-affinity"
LABELS="compute-compactor-affinity"
NEXMARK_QUERY="q9"
RW_VERSION="nightly-20240102"

[st1page]

It works(194.2k).

[st1page]

So because of the bug fixed in #13331, RW's benchmark result before nightly-20240101 is run with checkpoint_frequency=2. As we do not use the special config in real-world cases, maybe we should use the benchmark result after nightly-20240102 as the new baseline?
c.c. @fuyufjh @hzxa21 @cyliu0

[cyliu0]

I can set up the new baseline if that's OK.

[lmatz]

As we do not use the special config in real-world cases, maybe we should use the benchmark result after nightly-20240102 as the new baseline?

Dicussed today with the QA team that we will try three configs, cf = 1,2,10, if the results differ a lot, we keep them as a weekly test, if diff little, we will run the three settings even more infrequently