Multipart upload error on GCS

[everpcpc]

Save gcs cache failed: Error: Unexpected (permanent) at Writer::write, context: { uri: https://storage.googleapis.com/upload/storage/v1/b/xxx/o?uploadType=resumable&name=cloud/cache/Linux/go/xxx/xxx/cache.tzst&upload_id=ACJd0No-AuMDWKM0kCVnjavxxx686_MltwDqow-J_vbVrZ0mfhyxxxokY0Oql7cjS4yWlqs4CD-_HjTPmasgjbHcZ, response: Parts { status: 400, version: HTTP/1.1, headers: {"content-type": "text/plain; charset=utf-8", "content-length": "261", "date": "Mon, 24 Jun 2024 02:55:25 GMT", "server": "UploadServer"} }, service: gcs, path: Linux/go/xxx/xxx/cache.tzst, write_buf: 65536 } => Invalid request. The number of bytes uploaded is required to be equal or greater than 262144, except for the final request (it's recommended to be the exact multiple of 262144). The received request contained 65536 bytes, which does not meet this requirement.

code:

const op = new Operator(provider, { endpoint, bucket, root });
const rs = fs.createReadStream(archivePath);
const w = await op.writer(object);
const ws = w.createWriteStream();
await pipeline(rs, ws);
await finished(rs);

[Xuanwo]

Hi, which version are you using? The latest version will set correct chunk for every storage service.

[everpcpc]

https://github.com/everpcpc/actions-cache/blob/main/package.json#L12
0.45.1

[everpcpc]

https://www.npmjs.com/search?q=%40opendal
It seems like @opendal/lib-darwin-x64 release failed

[Xuanwo]

https://www.npmjs.com/search?q=%40opendal

It seems like @opendal/lib-darwin-x64 release failed

Yes, it did fail. Let's address it.

According to this issue, services like gcs require to align with specific size to upload successfully. Users either need to set chunk or meet the requirements by hand.

AFAIK, nodejs doesn't support setting chunk in 0.45.1, so we have to add a buffer for this writer.

[Xuanwo]

Yes, it did fail. Let's address it.

Should be fixed in #4798

It's related to our CI env changes...

[Xuanwo]

Hi, @everpcpc, this should be address in next release.

[Xuanwo]

Sorry, nodejs release failed: https://github.com/apache/opendal/actions/runs/9771241606/job/26973848658

I will get it fixed and start a new release now.

[Xuanwo]

Hi, @everpcpc, this issue should have been fixed. Please give it a try. Welcome to report new issues if it still exists.