-
Notifications
You must be signed in to change notification settings - Fork 110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ReplicationConfigs to SessionConfig for excluding keyspaces from replica precompute #831
base: main
Are you sure you want to change the base?
Conversation
bcd496f
to
d20d1ea
Compare
@piodul @wprzytula @havaker would you mind taking a quick look? Happy to write tests and finish up docs if this looks like it's in the right track. |
9f61a0a
to
2b965fe
Compare
42a1742
to
fa8fbd5
Compare
…replica precompute We add ReplicationOptions which includes a vector of keyspace names to SessionConfig. When instantiating ClusterData, we exclude keyspaces from provided by ReplicationOptions when computing PrecomputedReplicas to conserve memory/compute resources.
fa8fbd5
to
3bd2969
Compare
@Lorak-mmk please review |
You can already choose which keyspaces to precompute (see
So I think this change may be valuable. I'll try to post initial review soon. |
@michaelhly Sorry for so delayed response. |
host_filter: Option<&dyn HostFilter>, | ||
replication_opts: Option<&ReplicationConfigs>, | ||
) -> Self { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's not make it an Option
; ReplicationConfig
should be always passed to where ReplicaLocator
is constructed.
// Exclude these keyspaces from replica set precomputation | ||
let keyspaces_to_exclude = match replication_opts { | ||
Some(opts) => opts.keyspaces_to_exclude().to_vec(), | ||
None => vec![], | ||
}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Better not to allocate here. Let's just use the borrowed Vec
there.
// Exclude these keyspaces from replica set precomputation | |
let keyspaces_to_exclude = match replication_opts { | |
Some(opts) => opts.keyspaces_to_exclude().to_vec(), | |
None => vec![], | |
}; | |
// Exclude these keyspaces from replica set precomputation | |
let keyspaces_to_exclude = replication_config.keyspaces_to_exclude(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even better, let's allow for both Allowlist and Denylist variant:
// Exclude these keyspaces from replica set precomputation | |
let keyspaces_to_exclude = match replication_opts { | |
Some(opts) => opts.keyspaces_to_exclude().to_vec(), | |
None => vec![], | |
}; | |
let replicas_precomputation_config = replication_config.replicas_precomputation_config(); |
let keyspace_strategies = keyspaces | ||
.iter() | ||
.filter(|(k, _)| !keyspaces_to_exclude.contains(k)) | ||
.map(|(_, ks)| &ks.strategy); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let keyspace_strategies = keyspaces | |
.iter() | |
.filter(|(k, _)| !keyspaces_to_exclude.contains(k)) | |
.map(|(_, ks)| &ks.strategy); | |
let keyspace_strategies = keyspaces | |
.iter() | |
.filter(|(k, _)| match replicas_precomputation_config { | |
Denylist(disallowed_keyspaces) => !disallowed_keyspaces.contains(k), | |
AllowList(allowed_keyspaces) => allowed_keyspaces.contains(k), | |
) | |
.map(|(_, ks)| &ks.strategy); |
@@ -21,6 +21,23 @@ use std::{ | |||
}; | |||
use tracing::debug; | |||
|
|||
/// TODO(michael): Docs | |||
#[derive(Debug, Default, Clone)] | |||
pub struct ReplicationConfigs { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pub struct ReplicationConfigs { | |
pub struct ReplicationConfig { |
} | ||
|
||
impl ReplicationConfigs { | ||
pub fn exclude_keyspace(&mut self, keyspace: String) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pub fn exclude_keyspace(&mut self, keyspace: String) { | |
pub fn exclude_keyspace(&mut self, keyspace: impl Into<String>) { |
pub fn keyspaces_to_exclude(&self) -> &Vec<String> { | ||
&self.keyspaces_to_exclude | ||
} | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's create a ReplicasPrecomputationList
enum:
enum ReplicasPrecomputationList {
Allowlist { allowed_keyspaces: Vec<String> },
Denylist { disallowed_keyspaces: Vec<String> },
}
and store it ReplicationConfig
as Option<ReplicaPrecomputationList>
.
|
||
/// TODO(michael): Docs ... | ||
pub replication_configs: ReplicationConfigs, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SessionBuilder
needs the corresponding setters as well, as it's the intended way to construct a Session
.
Thank you for contributing the idea @michaelhly! |
We add ReplicationOptions which includes a vector of keyspace names to SessionConfig. When instantiating ClusterData, we exclude keyspaces from provided by ReplicationOptions when computing PrecomputedReplicas to conserve memory/compute resources.
Fixes: #671
Pre-review checklist
./docs/source/
.Fixes:
annotations to PR description.