Shard selecting load balancing

[Lorak-mmk]

This is a more polished version of #791
The changes from #791 are:

• Totally reworked commit structure, so that each commit actually compiles and passes all tests and checks. Doing this took quite a lot of time. Original commit structure was... a bit chaotic.
• Fixed all the warnings
• Removed some functions that were no longer used.
• Updated a comment that became incorrect

Posting the original description, slightly modified, as it is still relevant:

Motivation

Most of our drivers, being inherited from Cassandra, load balance only over nodes, not specific shards. Multiple ideas have arised that could benefit from having a shard-selecting load balancing. Among them:

• shard-aware batching (Shard aware batching - add Session::shard_for_statement & Batch::enforce_target_node #738);
• tablets support:
  with tablets enabled (ATM experimental in ScyllaDB), target shard is not derived from token (computed from partition key), but rather read from system.tablets. Therefore, a load balancer should be able to decide a target shard on its own, by abstracting over either token ring or tablets being used for cluster topology.
• overloaded shard optimisation:
  some tests have shown that sometimes, when a shard is particularly overloaded, it may be beneficial (performance-wise) to send the request to the proper node, but a wrong shard. That shard would then do part of the work that the overloaded shard would else have to do itself.

Design

• LB policy now is to return a (NodeRef, Shard) pair, enabling finer-grained control over targeted shards.
• regarding tablets support: ReplicaLocator is the place where the abstraction over either token ring or tablets is to be implemented. Ideally, the LB policy does not have to be aware of the actual mechanism (token ring or tablets) being used for a particular query.

What's done

• internal and public load-balancing-related interfaces are changed from NodeRef to (NodeRef, Shard) pair,
• shard selection logic is removed from NodeConnectionPool; a method is added there that returns a connection to a specific shard,
• Session's logic propagates the load balancing policy's target shard down to the connection pool,
• a stub implementation of shard selection is added to ReplicaLocator. At the moment, it simply computes the shard based on the token, the same way as it was done in the connection pool layer before.

Pre-review checklist

• I have split my patch into logically separate commits.
• All commit messages clearly explain what they change and why.
• I added relevant tests for new features and bug fixes.
• All commits compile, pass static checks and pass test.
• PR description sums up the changes and reasons why they should be introduced.
• I have provided docstrings for the public items that I want to introduce.
• I have adjusted the documentation in ./docs/source/.
• I added appropriate Fixes: annotations to PR description.

[wprzytula]

Justification for this name change: the seed is now not only used for shuffling nodes, but also for sampling a random shard in case we don't know the proper one.

[github-actions]

cargo semver-checks found no API-breaking changes in this PR! 🎉🥳
Checked commit: 28ae015

[wprzytula]

cargo semver-checks found no API-breaking changes in this PR! 🎉🥳

Well, that's unfortunately not true, but it's a known limitation of semver-checks...

The key takeaway is to not trust semver-checks fully, as there can be quite a lot of false negatives.

[piodul]

cargo semver-checks found no API-breaking changes in this PR! 🎉🥳

Well, that's unfortunately not true, but it's a known limitation of semver-checks...

The key takeaway is to not trust semver-checks fully, as there can be quite a lot of false negatives.

The message indeed sounds too confident, perhaps we should add some caution about the possible false negatives.

[wprzytula]

Mostly LGTM, some nits only.