-
Notifications
You must be signed in to change notification settings - Fork 912
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor dictionary encoding in PQ writer to migrate to the new cuco::static_map
#16541
Merged
rapids-bot
merged 39 commits into
rapidsai:branch-24.10
from
mhaseeb123:fea-refactor-dict-pq
Aug 30, 2024
Merged
Refactor dictionary encoding in PQ writer to migrate to the new cuco::static_map
#16541
rapids-bot
merged 39 commits into
rapidsai:branch-24.10
from
mhaseeb123:fea-refactor-dict-pq
Aug 30, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mhaseeb123
changed the title
Refactor PQ writer's dictionary encoder to migrate from
[WIP]: Refactor dictionary encoding in PQ writer to migrate to the new Aug 13, 2024
cuco::legacy::static_map
to cuco::static_map
cuco::static_map
PointKernel
reviewed
Aug 13, 2024
PointKernel
reviewed
Aug 13, 2024
mhaseeb123
commented
Aug 13, 2024
mhaseeb123
added
2 - In Progress
Currently a work in progress
cuIO
cuIO issue
improvement
Improvement / enhancement to an existing function
breaking
Breaking change
labels
Aug 13, 2024
mhaseeb123
changed the title
[WIP]: Refactor dictionary encoding in PQ writer to migrate to the new
Refactor dictionary encoding in PQ writer to migrate to the new Aug 19, 2024
cuco::static_map
cuco::static_map
mhaseeb123
commented
Aug 20, 2024
mhaseeb123
commented
Aug 20, 2024
mhaseeb123
commented
Aug 20, 2024
/ok to test |
davidwendt
reviewed
Aug 20, 2024
3 tasks
Redundant capacity issue is tracked via NVIDIA/cuCollections#581 |
PointKernel
reviewed
Aug 21, 2024
/ok to test |
PointKernel
approved these changes
Aug 26, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One small question otherwise looks good to me.
Great work!
mhaseeb123
added
4 - Needs Review
Waiting for reviewer to review or respond
and removed
3 - Ready for Review
Ready for review by team
labels
Aug 28, 2024
Co-authored-by: Yunsong Wang <[email protected]>
/ok to test |
davidwendt
approved these changes
Aug 29, 2024
/merge |
mhaseeb123
added
cuco
cuCollections related issue
5 - Ready to Merge
Testing and reviews complete, ready to merge
and removed
4 - Needs Review
Waiting for reviewer to review or respond
labels
Oct 10, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
5 - Ready to Merge
Testing and reviews complete, ready to merge
breaking
Breaking change
cuco
cuCollections related issue
cuIO
cuIO issue
improvement
Improvement / enhancement to an existing function
libcudf
Affects libcudf (C++/CUDA) code.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Part of #12261. This PR refactors the dictionary encoding in Parquet writers to migrate from
cuco::legacy::static_map
tocuco::static_map
to build the dictionaries.Performance Results
The changes result in +0.08% average speed improvement and +16.22% average memory footprint increase (stems from the adjusted sizes by
cuco::make_window_extent
due to prime gap) across the benchmark cases extended from #16591Currently, we do see a roughly 8% speed improvement in map insert and find kernels which is counteracted by the map init and map collect kernels as they have to process 16.22% more slots. With a cuco bump, the average speed improvement will increase from 0.08% to +3% and the memory footprint change will go back from 16.22% to +0%.
Hardware used for benchmarking
Checklist