gcs: can not access gcs due to credential fail

[wcy-fdu]

Describe the bug

This error occurs in both cloud e2e test and one customer cluster, fail to access gcs when set up meta pod.

Error message/log

{"timestamp":"2024-06-19T16:34:40.814638084Z","level":"ERROR","fields":{"message":"upload failed","error":"PermissionDenied (permanent) at Writer::close, context: { uri: https://storage.googleapis.com/upload/storage/v1/b/test-gcp-usce1-gke-a-gcs/o?uploadType=media&name=test-gcp-usce1-gke-a-rwc-g1i0ol2enkfql8mfl4dh00hkfd-e2e-ci-developer-basic-84938/test-gcp-usce1-gke-a-rwc-g1i0ol2enkfql8mfl4dh00hkfd-e2e-ci-developer-basic-84938/checkpoint/0, response: Parts { status: 403, version: HTTP/1.1, headers: {\"date\": \"Wed, 19 Jun 2024 16:34:40 GMT\", \"vary\": \"Origin\", \"vary\": \"X-Origin\", \"cache-control\": \"no-cache, no-store, max-age=0, must-revalidate\", \"expires\": \"Mon, 01 Jan 1990 00:00:00 GMT\", \"pragma\": \"no-cache\", \"x-guploader-uploadid\": \"ACJd0Nq335pnCYB6YbIUpub7O7xaOWB0c2CHz6osWrmKDXR_cxxlmER4JqLylzZr-nPeDDdaNQ\", \"content-length\": \"616\", \"server\": \"UploadServer\", \"content-type\": \"text/html; charset=UTF-8\"} }, service: gcs, path: test-gcp-usce1-gke-a-rwc-g1i0ol2enkfql8mfl4dh00hkfd-e2e-ci-developer-basic-84938/test-gcp-usce1-gke-a-rwc-g1i0ol2enkfql8mfl4dh00hkfd-e2e-ci-developer-basic-84938/checkpoint/0 } => GcsErrorResponse { error: GcsError { code: 403, message: \"[email protected] does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist).\", errors: [GcsErrorDetail { domain: \"global\", location: \"\", location_type: \"\", message: \"[email protected] does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist).\", reason: \"forbidden\" }] } }"},"target":"risingwave_object_store::object","threadName":"rw-main"}
--

[wjf3121]

Context: this issue was found in cloud dailiy e2e after we switch the default version from v1.8.2 to v1.9.1. It's not consistently reproducible.

We are asking QA team to run longevity on GKE against both v1.8.2 and v1.9.1 to verify whether this is a regression of the new kernel version.

cc @huangjw806

[wjf3121]

Sorry for the false alarm: turns out the timestamp of the error isn't aligned with the e2e failure time. This error happens when the e2e test cleaned up the resources and deleted the tenant so that permission error is expected at that time because we reclaimed the IAM policy.

The actually panic maybe related to this: #17343, which should be fixed already. We will observe the result after the fix is released.