watermark e2e test failed

[zwang28]

Describe the bug

The issue was found by this branch. It only modified metric and batch-availability code, so it shouldn't affect correctness.

failed to run e2e_test/streaming/watermark.slt

  |  
  | Caused by:
  | query result mismatch:
  | [SQL] select * from mv1;
  | [Diff] (-expected|+actual)
  | - 2023-05-06 16:51:00 1
  | - 2023-05-06 16:51:00 2
  | - 2023-05-06 16:51:00 3
  | at e2e_test/streaming/watermark.slt:27

https://buildkite.com/risingwavelabs/pull-request/builds/31234#018a739c-a2d8-45a4-ac9d-dd5758ae21b1

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

[zwang28]

@yuhao-su

[yuhao-su]

The test is flaky as expected. Rerun will be fine. Please let me know if it happens again.

[zwang28]

Is this the same issue?
end-to-end test (parallel, in-memory)

https://buildkite.com/risingwavelabs/main-cron/builds/873#018acdf5-9472-4523-83df-2c23ff9b5df3

e2e_test/streaming/temporal_join/temporal_join_watermark.slt .. [FAILED]
--
  |  
  | failed to run `e2e_test/streaming/temporal_join/temporal_join_watermark.slt`
  |  
  | Caused by:
  | query result mismatch:
  | [SQL] select * from eowc_mv;
  | [Diff] (-expected\|+actual)
  | -   2023-09-14 06:00:00+00:00 1
  | at e2e_test/streaming/temporal_join/temporal_join_watermark.slt:41

[yuhao-su]

Very likely, you can try change the two sleep 5s in temporal_join_watermark.slt into sleep 10s

[zwang28]

e2e_test/streaming/temporal_join/temporal_join_watermark.slt .. [FAILED]

This test case has failed for two consecutive days in main-cron.
https://buildkite.com/risingwavelabs/main-cron/builds/877#018ad843-59e3-4243-9e9d-17f31884f1e1