Improve support for reading CSV and JSON floating-point values

[andygrove]

Signed-off-by: Andy Grove [email protected]

Closes #124 and partially addresses #125, #126, and #1986 (only for floating-point types)

This PR affects both CSV and JSON readers and reads floating-point columns as strings and then casts them to a floating-point type. This means that we can now support special values such as NaN and Inf more consistently with Spark.

If this approach is acceptable then I will follow up with PRs to do this for other data types.

Note that this isn't perfect and there are still some follow-on issues (some existing and some new):

• [BUG] CSV Parseing Inf, -Inf, and NaN not supported #125
• [FEA] Add support for CSV reader options positiveInf, negativeInf, and nanValue #4644
• [FEA] JSON reader should parse quoted floating point values as null #4647
• [FEA] Make JSON parsing of NaN and Infinity values fully compatible with Spark #4646

Out of these, issue 4647 bothers me the most.

Status:

• Implement basic functionality
• Enable related tests that were previously XFAIL
• Fix resource leaks
• Add new tests for better coverage of edge cases
• Add support (and tests) for JSON option allowNonNumericNumbers
• Add tests with ansi mode enabled for JSON and CSV with invalid inputs
• Update compatibility guide

[revans2]

This is a really great step. Ultimately I would like to use this same trick to support other types, like decimal and date/time. But I am not convinced that we want to make it common everywhere because JSON has some odd configs/requirements with how it parses some numbers, which will be different from CSV and from casting. But these are corner cases and it might not matter that much anyways.

On a side note @GaryShen2008 we need to make sure that we take this into account when we are gathering requirements for JSON parsing in CUDF. We already know that they are not going to want to provide all of the parsing options that Spark supports, so we probably want to instead work on supporting the parsing ourselves as much as we can.

[revans2]

Sorry we should also update the compatibility doc so it explains that the parsing code has the same limitations as casting.

https://github.com/NVIDIA/spark-rapids/blob/branch-22.04/docs/compatibility.md#csv-floating-point

But reading it again, I think we want to do with for almost all types. Even integers to get the proper overflow checks.

Also can you add in a test for ANSI mode with invalid float values? I want to be sure that we are doing the same thing. Either both throw or not.

[andygrove]

Nan and Inf values are already covered in nan_and_inf.csv

[andygrove]

build

[andygrove]

build

[andygrove]

There are test failures in HashAggregatesSuite when running against Spark 3.1.1 - I am investigating.

[andygrove]

build

[andygrove]

build