Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(mito): Implement file cache #3022

Merged
merged 8 commits into from
Jan 3, 2024

Conversation

evenyag
Copy link
Contributor

@evenyag evenyag commented Dec 27, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This PR implements a cache for files in the WriteCache. The design is similar to the ReadCache.

The differences are

  • This cache always caches the whole SST file as it serves as a write-through cache
  • This cache doesn't read files from remote object stores

This cache has the following features:

  • Caches files on local disk (the local store).
  • Uses {region_id}.{file_id} as cache key and cached file name
  • Uses an in-memory LRU cache as file index
  • If the LRU cache removes the key, it also evicts the file on the local store.
  • Recovers the file index from the local store

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR does not require documentation updates.

Refer to a related PR or issue link (optional)

@github-actions github-actions bot added docs-not-required This change does not impact docs. Size: M labels Dec 27, 2023
@evenyag evenyag changed the base branch from develop to main December 28, 2023 03:53
@evenyag evenyag marked this pull request as ready for review December 28, 2023 08:01
Copy link

codecov bot commented Dec 28, 2023

Codecov Report

Attention: 20 lines in your changes are missing coverage. Please review.

Comparison is base (b58296d) 85.54% compared to head (2ae9523) 85.05%.
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3022      +/-   ##
==========================================
- Coverage   85.54%   85.05%   -0.49%     
==========================================
  Files         793      794       +1     
  Lines      128447   128743     +296     
==========================================
- Hits       109880   109508     -372     
- Misses      18567    19235     +668     

Copy link
Contributor

@zhongzc zhongzc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused about the responsibilities. Cache is not responsible for storing files but deleting files, but it's not a big problem. LGTM

Copy link
Contributor

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@killme2008 killme2008 added this pull request to the merge queue Jan 3, 2024
Merged via the queue into GreptimeTeam:main with commit 2e686fe Jan 3, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-not-required This change does not impact docs.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants