unaligned results of now() between streaming and batch queries when barrier piles up

[lmatz]

https://buildkite.com/risingwavelabs/main-cron/builds/1869#018da85a-078a-4811-a227-906f26b55713

It has passed for a few days now.
Not sure if it was solved a few days ago, or if it's a flaky test.

[BugenZhao]

Found such warnings in the log:

2024-02-14T16:18:41.846288325Z  WARN risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-02-14T16:18:41.854057912Z  WARN risingwave_stream::executor::now: handle multiple barriers at once in now executor: 2
2024-02-14T16:18:42.006623223Z  WARN risingwave_stream::executor::now: handle multiple barriers at once in now executor: 3

So I guess this could be a side effect introduced by #13271. The Now executor now processes the barriers in batch (if possible), skipping the data updates for intermediate epochs. If we coincidentally pin a snapshot on an intermediate epoch, we'll get inconsistent results for NOW() between the streaming and batch queries.

I think this somehow breaks the design goal of RisingWave that streaming queries should always yield exactly the same results as the batch queries. cc @wenym1 @fuyufjh Would you please share your ideas?

[wenym1]

I think we can solve this issue by having a global singleton now executor and now state table. IIUC, currently we may have multiple now executors for different streaming jobs. If so, such inconsistency may exist not only between batch query and streaming query, but also between different streaming queries with different now executor instances.

If we have a global singleton now executor and state table, instead of using the epoch directly as the now timestamp, we can query the global now state table to get a globally unified timestamp, so that the query can be consistent globally.

To temporarily solve this issue, instead of directly using epoch as the timestamp, we can try to query the now state table if there is any one associated to mv in the batch query.

[BugenZhao]

This sounds like a feasible approach. From my perspective, if we don't think it's really necessary to provide such strict guarantee, it's okay to just leave it as it is.

[BugenZhao]

Temporarily disabled in #15157.

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.