feat: add optional sidecar files for metadata

[benmcclelland]

This adds the option to store metadata for objects and buckets within a hidden folder alongside the bucket or object:
bucket: .vgw_meta.
object: .vgw_meta.

Example invocation:
./versitygw -a myaccess -s mysecret posix --sidecar /tmp/gw

The attributes are stored by name within the hidden directory.

[howff]

Thank you very much for this PR, it's just what I need! (backing filesystem is beegfs without xattr)

My initial tests of this PR show that it is working.

The example invocation above is wrong, it doesn't understand posix --sidecar DIR, you need posix --metadata sidecar and it might be writing to a hidden .vgw_meta directory.

Is there any chance this PR could be merged?

[howff]

I've merged current main into this branch ben/sidecar_meta by making the necessary changes. I don't think I have permission or know how to add commits to your PR though.

[benmcclelland]

Thank you very much for this PR, it's just what I need! (backing filesystem is beegfs without xattr)

My initial tests of this PR show that it is working.

The example invocation above is wrong, it doesn't understand posix --sidecar DIR, you need posix --metadata sidecar and it might be writing to a hidden .vgw_meta directory.

Is there any chance this PR could be merged?

The main issue with this PR is that it can become out of sync somewhat unexpectedly. For instance if you upload a file through the gateway then change the file directly in the filesystem, the attributes (such as etag/checksum, etc) may be stale without any clear indication of why. And metadata could be orphaned if the file was directly deleted. Maybe this is fine if the use case is to only have access through the gateway. There is a file data version in scoutfs that we can check to see if inode/data_version changed and then throw out the potentially stale attrs. But I'm not aware of a generic way to support this for all filesystems.

I could be convinced that this behavior is acceptable and just needs documentation on the caveats, but I was concerned about getting bug reports that metadata was getting reported for older versions of files in these cases.

[howff]

That's a valid concern. I think it could be addressed in one or more of:

• document it, and issue a warning upon start-up
• make it default to read-only (as per your other branch) and have an option to enable writes
• merge it as-is, and address the consistency issues in additional PRs further down the road

[rkm]

Just commenting to say that I've rebased the changes from this PR so it is up-to-date with the main branch: SMI#1.

The use-case @howff and I have necessitates read-only access to objects, so luckily (for us) we can avoid much of the detail around updates to objects and metadata. I wonder if this is a common enough case for a simpler implementation to be acceptable and easier to maintain? Something like posix --metadata read-only?

If not, we would be happy to contribute clarifications and documentation as @howff suggests that would make the existing changes more palatable and minimise the maintenance burden.

[benmcclelland]

Since there is sufficient interest in this, I can fixup this PR and merge in. And I'll probably update documentation for this with the known caveats. I dont think this needs to be read only as long as the limitations are understood.