Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid using same opendal cache during compaction #5137

Open
v0y4g3r opened this issue Dec 11, 2024 · 1 comment
Open

Avoid using same opendal cache during compaction #5137

v0y4g3r opened this issue Dec 11, 2024 · 1 comment
Assignees
Labels
C-enhancement Category Enhancements

Comments

@v0y4g3r
Copy link
Contributor

v0y4g3r commented Dec 11, 2024

What type of enhancement is this?

Performance

What does the enhancement do?

If compactor use the same opendal backend (and cache directory) as datanode, it may pollute the cache in opendal backend especially compaction legacy sst files.

Implementation challenges

Separate opendal backend in compactor and datanode. Backend in compactor should only read file cache but not write to it.

@v0y4g3r v0y4g3r added the C-enhancement Category Enhancements label Dec 11, 2024
@v0y4g3r v0y4g3r self-assigned this Dec 11, 2024
@v0y4g3r v0y4g3r changed the title Avoid using same opendal backend during compaction Avoid using same opendal cache during compaction Dec 11, 2024
@v0y4g3r
Copy link
Contributor Author

v0y4g3r commented Dec 11, 2024

Due to the pending optimization on access layer cache, we can now remove the cache layer during compaction. Theoretically, this change won't affect compaction performance, in that most reads in compaction do not retrieve repeated ranges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Category Enhancements
Projects
None yet
Development

No branches or pull requests

1 participant