Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply time window to memtables after compaction #5098

Open
evenyag opened this issue Dec 4, 2024 · 0 comments · May be fixed by #5197
Open

Apply time window to memtables after compaction #5098

evenyag opened this issue Dec 4, 2024 · 0 comments · May be fixed by #5197
Assignees
Labels
A-storage Involves code in storage engines C-enhancement Category Enhancements

Comments

@evenyag
Copy link
Contributor

evenyag commented Dec 4, 2024

What type of enhancement is this?

Tech debt reduction

What does the enhancement do?

If users don't specify time window explicitly, the compactor will infer a time window and apply it to the version. But the region can only set that window to the memtable after reopening the region.

let part_duration = options.compaction.time_window();
// Initial memtable id is 0.
let mutable = Arc::new(TimePartitions::new(
metadata.clone(),
memtable_builder.clone(),
0,
part_duration,
));

We should set the part duration to TimePartitions according to the time window.

Implementation challenges

Setting the part duration for TimePartitions has several challenges:

  • The TimePartitions struct doesn't support updating the part_duration now.
  • If a TimePartitions struct holds partitions with the old time window, then we should not override its part duration.
  • If we alter compaction time window, should we also alter the part duration?

Instead of altering the part duration, maybe we can create a new TimePartitions with the new time window. It's possible to set this in MemtableVersion::freeze_mutable.

if self.mutable.is_empty() {
// No need to freeze the mutable memtable.
return Ok(None);
}
// Marks the mutable memtable as immutable so it can free the memory usage from our
// soft limit.
self.mutable.freeze()?;
// Fork the memtable.
let mutable = Arc::new(self.mutable.fork(metadata));

@evenyag evenyag added C-enhancement Category Enhancements A-storage Involves code in storage engines labels Dec 4, 2024
@evenyag evenyag self-assigned this Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-storage Involves code in storage engines C-enhancement Category Enhancements
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant