Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pixels-cache] multi-table caching #636

Open
1 of 5 tasks
bianhq opened this issue Feb 1, 2024 · 0 comments
Open
1 of 5 tasks

[pixels-cache] multi-table caching #636

bianhq opened this issue Feb 1, 2024 · 0 comments
Assignees
@bianhq
Labels
enhancement New feature or request
Milestone
Cache Enhancement

Comments

@bianhq
Copy link
Contributor

bianhq commented Feb 1, 2024 •
edited
Loading

Currently, pixels can only cache one table specified in the pixels.properties. We should finish the following tasks for this issue:

  • remove cache.schema and cache.table from pixels.properties;
  • Remove the cacheBorder in the layout of a table, cached column chunks should be directly specified by the cache plan (job);
  • A cache plan will be related to a path and a layout of a table;
  • Redesign the data model for cache plan in etcd. A cache plan records the set of column chunks to be cached in a file and a list of paths on which the cache plan is applied;
  • Redesign the cache coordinator and cache worker, apply the new data model for cache plan.
@bianhq bianhq added the enhancement New feature or request label Feb 1, 2024
@bianhq bianhq added this to the Cache Enhancement milestone Feb 1, 2024
@bianhq bianhq self-assigned this Feb 1, 2024
bianhq added a commit that referenced this issue Sep 22, 2024
@bianhq
[Issue #636] remove cache schema and table from config file and fix m…
09a0166 
…emory leak in cache writer (#728)

1. Now, the cache schema and table names are specified in the layout_version and cache_version values.
2. Close PixelsPhysicalReader in PixelsCacheWriter to release the off-heap buffers allocated for localFs.
3. Revise the pixels cache document.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bianhq bianhq

Labels
enhancement New feature or request
Projects
None yet
Milestone
Cache Enhancement
Development

No branches or pull requests
1 participant
@bianhq