test: use EtcdStore in IT cases

[tisonkun]

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

1. Provision an etcd cluster for tests
2. Use it.

Checklist

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

This closes #2357.

[tisonkun]

Dockerhub down causes test to fail. I'm going to retry once it gets recovered.

[codecov]

Codecov Report

Merging #2734 (e98f39b) into develop (f5eede4) will decrease coverage by 0.93%.
Report is 49 commits behind head on develop.
The diff coverage is 87.32%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2734      +/-   ##
===========================================
- Coverage    85.23%   84.31%   -0.93%     
===========================================
  Files          765      730      -35     
  Lines       124205   113711   -10494     
===========================================
- Hits        105864    95870    -9994     
+ Misses       18341    17841     -500     

[tisonkun]

Implement a "chroot" alike mechanism is not easy (#2734 (comment)).

Either we implement it in upstream etcdv3/etcd-client#67 or we chroot/undo_chroot for every keys handled in EtcdStore. For the latter, I'm hacking locally and found some issues.

Converting this PR as draft for now.

[tisonkun]

Implement a "chroot" alike mechanism is not easy (#2734 (comment)).

Either we implement it in upstream etcdv3/etcd-client#67 or we chroot/undo_chroot for every keys handled in EtcdStore. For the latter, I'm hacking locally and found some issues.

Converting this PR as draft for now.

I remember that OpenDAL's root can achieve this feature. But we need to access underneath Etcd client APIs (txns, which is not provided by the OpenDAL API). @Xuanwo may you have a suggestion here how we can implement such chroot or root as in OpenDAL feature?

[Xuanwo]

@Xuanwo may you have a suggestion here how we can implement such chroot or root as in OpenDAL feature?

OpenDAL doesn't support chroot either. We support root by insert the root for every key and strip it while returning to users.

[MichaelScofield]

I think we can do the same thing as OpenDAL does: prepend a common "root" path for every key in a certain "namespace". The "namespace" could be set by Metasrv when bootstrapping, differ in each test case.

[tisonkun]

@MichaelScofield Yep. This is what I'm prototyping locally.

And I found that we should change both the request side and the reply side. The latter means that when returning prev_kvs to users, we should strip the root prefix before returning to client. And this causes a refactor I need for etcdv3/etcd-client#68.

[tisonkun]

Merge #2786 for running test.

[WenyXu]

BTW, We can implement the chroot in the KvBackend level. Then we use it like let kv_backend = ChrootKvBackend::new(kv_backend). For better performance, It's also a good idea to add some fast paths while the chroot is empty.

[fengjiachun]

Hi @tisonkun , would it be better if we support chroot at the KvBackend layer?
If we do it this way, then all KvBackend implementations, including etcd KvBackend, will have the chroot feature.

Another point, if the chroot is empty, can we avoid the copying of those keys?

[tisonkun]

Another point, if the chroot is empty, can we avoid the copying of those keys?

If chroot is empty, no copy will happen. Rust compiler should be smart enough to handle this ownership to ownership in-place moved.

[tisonkun]

would it be better if we support chroot at the KvBackend layer

It can be a follow-up to discuss. It can largely increase the scope of this PR and finally we don't achieve anything.

[tisonkun]

But I can give it a try, or just give up and let you take it over - do not depend on me.

[tisonkun]

If chroot is empty

Anyway I debug_assert(!root.is_empty()) for ChrootKvBackend so lift the responsibility to the caller.

For kvs = kvs.drain(..).xxx().collect() things, we may leverage take_mut, but they're technically similar because only pointers modification happen, no copy.

[tisonkun]

@MichaelScofield Let's do this refactors thing in a follow-up PR? Since this PR is already non-trival and the refactor should not be a blocker.

@fengjiachun Please see #2734 (comment)

[MichaelScofield]

@tisonkun LGTM. However, github CI check complained with the notorious annoying "waiting status to be updated" error. I changed the workflow step name back to "coverage", hope you don't mind.

[tisonkun]

Error response from daemon: Head "https://registry-1.docker.io/v2/bitnami/etcd/manifests/latest": error parsing HTTP 429 response body: invalid character 'T' looking for beginning of value: "Too Many Requests (HAP429).\n"

I may spend a time to use a proxy to overcome the rate limit issue.