Bug(batch): Zombie batch query runs for a long time in the system

[Li0k]

Describe the bug

We have observed an unintended batch task that continues to execute in the system. This task lasts longer than 10h, thus affecting the execution of compaction in storage.

The reason for this is that the batch query will hold a snapshot, which is the basis for the compaction task to clean up the data, and we will determine the safe epoch of the task based on the snapshot.

Since the safe epoch cannot be raised, the compaction task cannot reclaim the expired data, resulting in a large number of tombstones in the system.

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20231123

Additional context

No response

[Li0k]

@liurenjie1024

[liurenjie1024]

Duplicated with #9732