bug: opendal state store may not operate atomically

[BugenZhao]

Operations on S3/OSS are always atomic. For example, we'll never get partial objects uploaded to the object store.

However, this is not true for all opendal services by default. We may get the data corrupted if users are using opendal state store like hummock+fs://... This may lead to unexpected results as we also leverage state store for Hummock metadata management.

It turns out that we should manually enable atomicity guarantee by setting the atomic_write_dir config when building services.
https://docs.rs/opendal/latest/opendal/services/struct.Fs.html#method.atomic_write_dir

[hzxa21]

If atmoic_write_dir is not set, is it possible to have a object that is present in metadata (e.g. HummockVersion) but absent in the object store (e.g. HDFS)?

[BugenZhao]

If atmoic_write_dir is not set, is it possible to have a object that is present in metadata (e.g. HummockVersion) but absent in the object store (e.g. HDFS)?

I think it depends. If a future cancellation or node crash occurs during the procedure, and there is also no defensive programming on the next startup, things could get wrong.

The recent issue reported by a user is getting an empty file at the version_checkpoint_path on the file system. Note that an empty slice is also a valid proto3 encoding, we finally panicked at unwrapping a required protobuf field and failed to make any progress. I suppose there was a failure when updating the checkpoint object in write_checkpoint.

risingwave/src/meta/src/hummock/manager/checkpoint.rs

Line 44 in 2593017

version: HummockVersion::from_persisted_protobuf(checkpoint.version.as_ref().unwrap()),

risingwave/src/meta/src/hummock/manager/checkpoint.rs

Lines 90 to 92 in 2593017

	self.object_store
	.upload(&self.version_checkpoint_path, buf.into())
	.await?;

[hzxa21]

If atmoic_write_dir is not set, is it possible to have a object that is present in metadata (e.g. HummockVersion) but absent in the object store (e.g. HDFS)?

I think it depends. If a future cancellation or node crash occurs during the procedure, and there is also no defensive programming on the next startup, things could get wrong.

The recent issue reported by a user is getting an empty file at the version_checkpoint_path on the file system. Note that an empty slice is also a valid proto3 encoding, we finally panicked at unwrapping a required protobuf field and failed to make any progress. I suppose there was a failure when updating the checkpoint object in write_checkpoint.

risingwave/src/meta/src/hummock/manager/checkpoint.rs

Line 44 in 2593017

version: HummockVersion::from_persisted_protobuf(checkpoint.version.as_ref().unwrap()),

risingwave/src/meta/src/hummock/manager/checkpoint.rs

Lines 90 to 92 in 2593017

	self.object_store
	.upload(&self.version_checkpoint_path, buf.into())
	.await?;

I see. I understand that this issue will cause the file content of an object to be empty or incorrect. We are debugging another case reported by a user when there is an object that is present in HummockVersion but absent in HDFS. I guess it is not relevant to this issue?

[BugenZhao]

I see. I understand that this issue will cause the file content of an object to be empty or incorrect. We are debugging another case reported by a user when there is an object that is present in HummockVersion but absent in HDFS. I guess it is not relevant to this issue?

I have no idea. 😕

[wcy-fdu]

fixed in #15155