backfill test, it failed to create watermark mv for 1h timeout on nightly-20231211.

[lmatz]

          For backfill test, it failed to create watermark mv for 1h timeout on `nightly-20231211`. https://buildkite.com/risingwave-test/backfill/builds/72#018c5b1d-515e-49ac-aedf-4da032ee503f

  | 2023-12-12 07:26:13 | create watermark mv for backfill table
  | 2023-12-12 07:26:13 | Timing is on.
  | 2023-12-12 08:26:11 | ❌ backfill test timeout

The backfill test step for creating watermark mv succeed on nightly-20231208 in 2 minutes

  | 2023-12-09 07:25:54 | create watermark mv for backfill table
  | 2023-12-09 07:25:54 | Timing is on.
  | 2023-12-09 07:27:35 | NOTICE:  Your session timezone is UTC. It was used in the interpretation of timestamps and dates in your query. If this is unintended, change your timezone to match that of your data's with `set timezone = [timezone]` or rewrite your query with an explicit timezone conversion, e.g. with `AT TIME ZONE`.
  | 2023-12-09 07:27:35 | CREATE_MATERIALIZED_VIEW
  | 2023-12-09 07:27:35 | Time: 103637.007 ms (01:43.637)

Originally posted by @cyliu0 in #13931 (comment)

[lmatz]

The watermark mv is created since 7:26 and the barriers pile up

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=P2453400D1763B4D9&var-namespace=backfill-20231211-230338-20231211&var-instance=benchmark-risingwave&var-pod=All&var-component=All&var-table=All&from=1702336021000&to=1702337863000

[cyliu0]

Happened again with nightly-20231212 https://buildkite.com/risingwave-test/backfill/builds/75#018c6043-d2c0-4afd-b231-89c0dcebf497