Docker Compose fails to resume from suspend

[kwannoel]

Describe the bug

After timing out, the container fails to resume again, and after starting the stopping the container. There's currently no known way to recover the container, apart from just recreating it from scratch and losing the cluster data.

Error message/log

There's no clear error log. Just a repeat of these error logs again and again, until ~8.24am:

2024-05-29 08:23:22 2024-05-29T12:23:22.026025381Z  WARN rw-main risingwave_stream::task::barrier_manager: control stream reset with error error=gRPC request failed: Internal error: end of stream
2024-05-29 08:23:22 2024-05-29T12:23:22.026041422Z  WARN rw-main risingwave_stream::task::barrier_manager: failed to notify finish of control stream
2024-05-29 08:23:24 2024-05-29T12:23:24.706507882Z  INFO rw-main bootstrap_recovery{prev_epoch=6536473939345408}:recovery_attempt: risingwave_meta::barrier::recovery: recovering mview progress
2024-05-29 08:23:24 2024-05-29T12:23:24.710513173Z  INFO rw-main bootstrap_recovery{prev_epoch=6536473939345408}:recovery_attempt: risingwave_meta::barrier::recovery: recovered mview progress
2024-05-29 08:23:24 2024-05-29T12:23:24.948064757Z  WARN rw-main 

The user has tried to restart the docker service. But it still fails.

The compute node is started at 8.22:

2024-05-29 08:22:42 2024-05-29T12:22:42.743734918Z  INFO rw-main risingwave_compute::server:

To Reproduce

1. docker compose up to start the docker container.
2. Suspend the docker process, e.g. just let your computer sleep.
3. After 8hrs, use ctrl-c and then docker-compose up to restart the container.

(I was unable to reproduce it with these steps, they were provided by the user).

Expected behavior

At least allow the service to resume.

How did you deploy RisingWave?

No response

The version of RisingWave

v1.9.1-rc2

Additional context

docker-compose.yml is the docker compose file used.