bug(storage): read block seems to be stuck

[zwang28]

Describe the bug

In a cluster using Google Cloud Storage, according to await tree,

• fetch_block sometimes gets stuck for 5hours before manually terminating it.
• And it's not uncommon fetch_block takes over 10 seconds to finish.

Error message/log

Will keep the await tree when encountered next time.

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

v1.6.1

Additional context

• Heavy workload. The barrier typically takes over one hour to finish, if the stuck issue doesn't occurs.
• Over 10k actors per compute node.
• 40 CPU and 200GB memory per. compute node.

[hzxa21]

What is value of object_store_read_timeout? By default it is 8mins. Like #15209, It seems that the timeout didn't trigger as well.

[zwang28]

object_store_read_timeout

In v1.6.1, object_store_read_timeout is 1hour. But the specific fetch_block hangs for several hours.

[Xuanwo]

OpenDAL has native await-tree support, please enable it for more information.

[hzxa21]

Potential root cause found:
#15209 (comment)