Bug: Backfill tests broke due to read failed error=Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000)

[kwannoel]

Describe the bug

Error: read failed error=Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000)

Test: test_no_shuffle_backfill_snapshot_and_upstream_runtime

Cluster: ci-3cn-1fe

Error message/log

2024-05-09T16:28:22.487197779Z ERROR risingwave_object_store::object: read failed error=Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000) and read_retry_attempts (current=3) under [storage.object_store.retry] in the config accordingly if needed.
2024-05-09T16:28:22.487250147Z ERROR risingwave_storage::hummock::sstable_store: prefetch meet error when read 3995259..4322738 from sst-158 (12707301)
2024-05-09T16:28:22.487261098Z  WARN risingwave_storage::hummock::sstable::forward_sstable_iterator: failed to recreate stream meet IO error error=ObjectStore failed with IO error: Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000) and read_retry_attempts (current=3) under [storage.object_store.retry] in the config accordingly if needed.
2024-05-09T16:28:23.32170976Z ERROR risingwave_object_store::object: read failed error=Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000) and read_retry_attempts (current=3) under [storage.object_store.retry] in the config accordingly if needed.
2024-05-09T16:28:23.321778907Z ERROR risingwave_storage::hummock::sstable_store: prefetch meet error when read 5631039..5892951 from sst-153 (33575459)
2024-05-09T16:28:23.321790781Z  WARN risingwave_storage::hummock::sstable::forward_sstable_iterator: failed to create stream for prefetch data, fall back to block get error=ObjectStore failed with IO error: Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000) and read_retry_attempts (current=3) under [storage.object_store.retry] in the config accordingly if needed.
2024-05-09T16:28:23.418499742Z ERROR risingwave_object_store::object: read failed error=Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000) and read_retry_attempts (current=3) under [storage.object_store.retry] in the config accordingly if needed.
2024-05-09T16:28:23.418553323Z ERROR risingwave_storage::hummock::sstable_store: prefetch meet error when read 29997221..30324699 from sst-157 (33573285)
2024-05-09T16:28:23.418562879Z  WARN risingwave_storage::hummock::sstable::forward_sstable_iterator: failed to recreate stream meet IO error error=ObjectStore failed with IO error: Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000) and read_retry_attempts (current=3) under [storage.object_store.retry] in the config accordingly if needed.
2024-05-09T16:28:23.531554511Z ERROR risingwave_object_store::object: read failed error=Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000) and read_retry_attempts (current=3) under [storage.object_store.retry] in the config accordingly if needed.
2024-05-09T16:28:23.531613903Z ERROR risingwave_storage::hummock::sstable_store: prefetch meet error when read 12108879..12239888 from sst-244 (15837354)
2024-05-09T16:28:23.531625862Z  WARN risingwave_storage::hummock::sstable::forward_sstable_iterator: failed to create stream for prefetch data, fall back to block get error=ObjectStore failed with IO error: Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000) and read_retry_attempts (current=3) under [storage.object_store.retry] in the config accordingly if needed.
2024-05-09T16:28:23.878632444Z ERROR risingwave_object_store::object: read failed error=Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000) and read_retry_attempts (current=3) under [storage.object_store.retry] in the config accordingly if needed.
2024-05-09T16:28:23.878688697Z ERROR risingwave_storage::hummock::sstable_store: prefetch meet error when read 8252508..8514485 from sst-158 (12707301)
2024-05-09T16:28:23.878698851Z  WARN risingwave_storage::hummock::sstable::forward_sstable_iterator: failed to recreate stream meet IO error error=ObjectStore failed with IO error: Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000) and read_retry_attempts (current=3) under [storage.object_store.retry] in the config accordingly if needed.
2024-05-09T16:28:24.315864219Z ERROR risingwave_object_store::object: read failed error=Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=8000) and read_retry_attempts (current=3) under [storage.object_store.retry] in the config accordingly if needed.
2024-05-09T16:28:24.315889905Z ERROR risingwave_storage::hummock::sstable_store: get_block_response meet error when read 566077..631592 from sst-225, total length: 785415
2024-05-09T16:28:24.316046805Z ERROR risingwave_storage::monitor::monitored_store: Failed in iter error=Hummock error: ObjectStore failed with IO error: Internal error: channel closed
2024-05-09T16:28:24.316110245Z ERROR risingwave_storage::monitor::monitored_store: Failed in iter error=Hummock error: ObjectStore failed with IO error: Internal error: channel closed

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

Likely #16231 @Li0k