Error sinking to delta lake on gcs using s3-compatible interface.

[erikamundson]

Describe the bug

Using the s3-compatible gcs interface, getting an error sinking data to delta lake.

The sink is created with no error, and the parquet files are being written. The issue appears to occur when the log file is attempted. There is no error message in the sql interface until calling flush, but there is a constant error loop in the compute pod logs.

Error message/log

2024-04-03T23:44:05.087515415Z ERROR risingwave_stream::task::stream_manager: actor exit with error actor_id=51 error=Executor error: Sink error: Internal error: gRPC request failed: The operation was aborted: failed to run coordination

To Reproduce

We created a kafka source and an intermediate materialized view, both of which work fine.

Then created the delta sink with

create sink <sink>
from <mv> 
with (
	connector = 'deltalake',
	type = 'append-only',
	location= 's3a://bucket/folder',
	s3.endpoint = 'https://storage.googleapis.com',
	s3.access.key = 'xxx',
	s3.secret.key = 'xxx'
);

flush;  -- error shows up here

Expected behavior

I expected the delta sink to write the parquet files in the expected partition, then commit the log files.

The parquet files are written correctly, but the sink seems to be unable to create the log file.

How did you deploy RisingWave?

via Kubernetes Operator. We're using the gcs state store with postgresql meta store.

The version of RisingWave

PostgreSQL 9.5.0-RisingWave-1.7.3 (cfefe78)

Additional context

Full log file for one traceback loop here.
error_log.csv

[fuyufjh]

@erikamundson Can you please provide a full copy of logs of meta nodes?

---

cc. @wenym1 The actual error was not properly reported to the caller. PTAL

risingwave/src/meta/src/manager/sink_coordination/coordinator_worker.rs

Lines 310 to 313 in 151fd5d

	if result.is_err() {
	self.send_to_all_sink_writers(|| Err(Status::aborted("failed to run coordination")))
	.await;
	}

[erikamundson]

Hi @fuyufjh , here is the meta node log dump for that error. It appears to be a 411 error.
meta_error_log.csv

[fuyufjh]

Thanks a lot!

RisingWave sink_coordination got 411 Length Required error from Google cloud storage.

72,2024-04-03T23:43:59.364971003Z,""", after 0 retries: HTTP status client error (411 Length Required) for url (https://storage.googleapis.com/**********/00000000000000000001.json)"
73,2024-04-03T23:43:59.364963133Z,  <p>POST requests require a <code>Content-length</code> header.  <ins>That’s all we know.</ins>
74,2024-04-03T23:43:59.364956416Z,  <p><b>411.</b> <ins>That’s an error.</ins>
75,2024-04-03T23:43:59.364950438Z,  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
76,2024-04-03T23:43:59.364942815Z,  </style>
77,2024-04-03T23:43:59.364906Z,"    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}"
78,2024-04-03T23:43:59.364893291Z,  <style>
79,2024-04-03T23:43:59.364883343Z,  <title>Error 411 (Length Required)!!1</title>
80,2024-04-03T23:43:59.364877121Z,"  <meta name=viewport content=""initial-scale=1, minimum-scale=1, width=device-width"">"
81,2024-04-03T23:43:59.364871025Z,  <meta charset=utf-8>
82,2024-04-03T23:43:59.364862227Z,<html lang=en>
83,2024-04-03T23:43:59.364806594Z,"2024-04-03T23:43:59.364563948Z ERROR risingwave_meta::manager::sink_coordination::coordinator_worker: failed to commit metadata of epoch epoch=6222816107560960 error=DeltaLake error: Failed to read delta log object: Generic S3 error: response error ""<!DOCTYPE html>"

Perhaps there is some compatibility issue of DeltaLake sink on GCS. We will investigate it and fix it asap.

[xxhZs]

The issue came from https://github.com/delta-io/delta-rs , a message had been left for them, but this way was not their default recommended way of using gcs, so we supported the default way of using gcs #16182, It can work!