Add multi-partition Scan support to cuDF-Polars
#17494
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
github-actions
bot
added
Python
rjzamora
added
feature request
github-actions
bot
added
Python
rjzamora
added
cudf.polars
|
/ok to test |
rjzamora
added
3 - Ready for Review
| config_options["parquet_options"] = config_options.get( | ||
| "parquet_options", {} | ||
| ).copy() | ||
| config_options["parquet_options"]["chunked"] = False |
There was a problem hiding this comment.
Since we require py 3.10 now, I think this simpler as:
| config_options["parquet_options"] = config_options.get( | |
| "parquet_options", {} | |
| ).copy() | |
| config_options["parquet_options"]["chunked"] = False | |
| config_options["parquet_options"] |= {"chunked": False} |
There was a problem hiding this comment.
I don't think this works if the "parquet_options" key is missing?
| file_size: float = 0 | ||
| # TODO: Use system info to set default blocksize | ||
| parallel_options = ir.config_options.get("executor_options", {}) | ||
| blocksize: int = parallel_options.get("parquet_blocksize", 1024**3) |
There was a problem hiding this comment.
Is 1GiB a good size, or should we pick something larger?
There was a problem hiding this comment.
Is 1GiB a good size, or should we pick something larger?
My experience tells me that 1GiB is a good default, but that most users with datacenter-class GPUs will usually want to go bigger. In Dask cuDF we use pynvml to query the "real" device size. The details of this can get sticky, so I'd rather revisit this kind of improvement after we start benchmarking.
|
@wence- Any sense for how far away we are on this one? |
| class ScanPartitionFlavor(IntEnum): | ||
| """Flavor of Scan partitioning.""" |
There was a problem hiding this comment.
| class ScanPartitionFlavor(IntEnum): | |
| """Flavor of Scan partitioning.""" | |
| class ScanPartitionFlavour(IntEnum): | |
| """Flavour of Scan partitioning.""" |
😉
rjzamora
added
5 - Ready to Merge
|
/merge |
Description
Adds multi-partition
Scansupport following the same design as #17441Checklist