Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split batch_aggregation_shard_count configuration in two #3463

Open
divergentdave opened this issue Nov 4, 2024 · 0 comments
Open

Split batch_aggregation_shard_count configuration in two #3463

divergentdave opened this issue Nov 4, 2024 · 0 comments

Comments

@divergentdave
Copy link
Contributor

We've been discussing increasing batch_aggregation_shard_count. I think it would be good to split this into two configuration parameters in order to make deploying such changes safer. We could use one count for writes to random shards, and another for broadcast writes to all shards, which advance the overall batch's state machine. (note that reads scan all available shards, so the batch_aggregation_shard_count configuration parameter does not directly impact the read path) So long as all running processes and all existing batches in the database use a count for broadcast writes that is greater than or equal to the count for randomly sharded writes, then our justification for the batch state machine is sound.

Firstly, this would provide a safe avenue for decreasing the shard count, by decreasing the count for random writes, waiting for all the data to turn over, and then decreasing the count for broadcast writes. Secondly, this would make changes to this parameter when running as the helper aggregator safer. Note that, for the leader, it would be possible to sequence an increase in batch_aggregation_shard_count between the different components to obey the above rule, i.e. collection job driver first. For the helper role, however, we do both random sharded writes and broadcast writes from the same process, so a rolling update of the configuration parameter as-is may introduce a small window for strange race conditions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant