Add in option for Java JSON APIs to do column pruning in CUDF #16796
Conversation
Signed-off-by: Robert (Bobby) Evans <[email protected]>
revans2
added
0 - Blocked
|
This makes a big difference in performance on a number of real world use cases we have seen (160 seconds without vs 16.7 seconds with pruning) In order to not have any accuracy regressions this depends on #16545 going in. |
| @@ -1313,7 +1315,8 @@ public static Table readJSON(Schema schema, JSONOptions opts, File path) { | |||
| opts.strictValidation(), | |||
| opts.leadingZerosAllowed(), | |||
| opts.nonNumericNumbersAllowed(), | |||
| opts.unquotedControlChars()))) { | |||
| opts.unquotedControlChars(), | |||
| true))) { | |||
There was a problem hiding this comment.
Why it is always true while in the other overload it is a parameter?
There was a problem hiding this comment.
Good catch I missed that one
|
@ttnghia please take another look |
| boolean cudfPruneSchema = schema.getColumnNames() != null && | ||
| schema.getColumnNames().length != 0 && |
There was a problem hiding this comment.
Since getColumnNames calls toArray, consider storing it in a local variable. Maybe we could introduce a utils method since this PR does it a few times.
revans2
added
3 - Ready for Review
|
I need to clarify the dependency a bit. This code should not break anything if it goes in before #16797, but it will not hurt anything because pruning in CUDF is disabled by default. |
|
/merge |
Description
This adds in the options to enable column_pruning when reading JSON using the java APIs.
This is still in draft because there are test failures if this is turned on for those tests.
#16797
That said the performance impact from enabling column pruning on some queries is huge. For one query in particular the current code takes 161.5 seconds and with CUDF column pruning it is just 16.5 seconds. That is a 10x speedup for something that is fairly real world.
Checklist