Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nightly-20240709 huge performance degradation #17637

Closed
cyliu0 opened this issue Jul 10, 2024 · 4 comments
Closed

nightly-20240709 huge performance degradation #17637

cyliu0 opened this issue Jul 10, 2024 · 4 comments
Labels
type/bug Something isn't working type/perf
Milestone

Comments

@cyliu0
Copy link
Collaborator

cyliu0 commented Jul 10, 2024

Describe the bug

http://metabase.risingwave-cloud.xyz/dashboard/241-nexmark-blackhole-1cn-anti-affinity-rw-avg-source-throughput?namespace=daily

+--------------------------------------------------------+--------------+------------+-----------------------------------+---------------------+-----------------------------+-------------------------------+
| BENCHMARK NAME                                         | EXECUTION ID | STATUS     | KEY METRICS                       | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS | FLUCTUATION OF LAST EXECUTION |
+--------------------------------------------------------+--------------+------------+-----------------------------------+---------------------+-----------------------------+-------------------------------+
| nexmark-q0-medium-1cn                                  |        33537 | Negative   | avg-source-output-rows-per-second | -38.97%             | -32.97%                     | -36.55%                       |
| nexmark-q0-blackhole-medium-1cn                        |        33539 | Negative   | avg-source-output-rows-per-second | -58.65%             | -44.20%                     | -46.29%                       |
| nexmark-q7-blackhole-watermark-medium-1cn              |        33550 | Negative   | avg-source-output-rows-per-second | -16.18%             | -10.13%                     | -13.10%                       |
| nexmark-q8-blackhole-watermark-medium-1cn              |        33557 | Negative   | avg-source-output-rows-per-second | -25.36%             | -16.14%                     | -16.86%                       |
| nexmark-q7-blackhole-medium-1cn                        |        33560 | Negative   | avg-source-output-rows-per-second | -19.65%             | -11.90%                     | -17.19%                       |
| nexmark-q7-rewrite-blackhole-medium-1cn                |        33566 | Negative   | avg-source-output-rows-per-second | -55.72%             | -40.52%                     | -44.80%                       |
| nexmark-q8-blackhole-medium-1cn                        |        33572 | Negative   | avg-source-output-rows-per-second | -36.00%             | -12.42%                     | -14.24%                       |
| nexmark-q12-blackhole-medium-1cn                       |        33585 | Negative   | avg-source-output-rows-per-second | -42.04%             | -34.27%                     | -37.06%                       |
| nexmark-q13-blackhole-medium-1cn                       |        33588 | Negative   | avg-source-output-rows-per-second | -49.12%             | -39.07%                     | -42.87%                       |
| nexmark-q105-blackhole-medium-1cn                      |        33590 | Negative   | avg-source-output-rows-per-second | -26.65%             | -12.09%                     | -17.01%                       |
| nexmark-q3-no-condition-blackhole-medium-1cn           |        33591 | Negative   | avg-source-output-rows-per-second | -37.37%             | -24.01%                     | -24.54%                       |

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-202040709

Additional context

https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240709

@cyliu0 cyliu0 added type/bug Something isn't working type/perf labels Jul 10, 2024
@github-actions github-actions bot added this to the release-1.10 milestone Jul 10, 2024
@lmatz
Copy link
Contributor

lmatz commented Jul 10, 2024

q0 is stateless, I suspect it is occasional fluctuation not caused by the kernel but some unknown env issues

SCR-20240710-lyj

SCR-20240710-m1b

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1720542507000&to=1720544405000&var-namespace=nexmark-bs-0-14-daily-20240709&editPanel=18

remember change $__rate_interval to 2m

@fuyufjh fuyufjh modified the milestones: release-1.10, release-1.11 Jul 10, 2024
@cyliu0
Copy link
Collaborator Author

cyliu0 commented Jul 11, 2024

We still have this degradation for nightly-20240710

+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| BENCHMARK NAME                                                | EXECUTION ID | STATUS     | KEY METRICS                                        | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS | FLUCTUATION OF LAST EXECUTION |
+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| nexmark-q0-medium-1cn                                         |        33643 | Negative   | avg-source-output-rows-per-second                  | -39.09%             | -29.19%                     | -0.20%                        |
| nexmark-q0-blackhole-medium-1cn                               |        33644 | Negative   | avg-source-output-rows-per-second                  | -58.98%             | -41.15%                     | -0.80%                        |
| nexmark-q8-blackhole-watermark-medium-1cn                     |        33657 | Negative   | avg-source-output-rows-per-second                  | -26.68%             | -15.35%                     | -1.77%                        |
| nexmark-q7-blackhole-medium-1cn                               |        33658 | Negative   | avg-source-output-rows-per-second                  | -21.31%             | -12.07%                     | -2.07%                        |
| nexmark-q7-rewrite-blackhole-medium-1cn                       |        33665 | Negative   | avg-source-output-rows-per-second                  | -56.03%             | -37.55%                     | -0.71%                        |
| nexmark-q8-blackhole-medium-1cn                               |        33668 | Negative   | avg-source-output-rows-per-second                  | -36.68%             | -11.95%                     | -1.06%                        |
| nexmark-q12-blackhole-medium-1cn                              |        33681 | Negative   | avg-source-output-rows-per-second                  | -41.64%             | -30.30%                     | 0.69%                         |
| nexmark-q13-blackhole-medium-1cn                              |        33686 | Negative   | avg-source-output-rows-per-second                  | -47.39%             | -33.47%                     | 3.39%                         |
| nexmark-q3-no-condition-blackhole-medium-1cn                  |        33689 | Negative   | avg-source-output-rows-per-second                  | -40.87%             | -24.94%                     | -5.59%                        |

@cyliu0
Copy link
Collaborator Author

cyliu0 commented Jul 11, 2024

The active queries processing is not working. It led to this misreporting.

@cyliu0 cyliu0 closed this as completed Jul 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Something isn't working type/perf
Projects
None yet
Development

No branches or pull requests

3 participants