Skip to content

Commit

Permalink
Merge branch 'branch-23.10' into update-ci-whitelist
Browse files Browse the repository at this point in the history
  • Loading branch information
@viadea
viadea committed Sep 29, 2023
2 parents aff3f15 + 199b35c commit 7c7f42a
Show file tree
Hide file tree
Showing 21 changed files with 1,284 additions and 168 deletions.
5 changes: 3 additions & 2 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,9 @@ specifying the environment variable `BUILD_PARALLEL=<n>`.
### Building against different CUDA Toolkit versions
You can build against different versions of the CUDA Toolkit by using one of the following profiles:
* `-Pcuda11` (CUDA 11.0/11.1/11.2, default)
You can build against different versions of the CUDA Toolkit by modifying the variable `cuda.version`:
* `-Dcuda.version=cuda11` (CUDA 11.x, default)
* `-Dcuda.version=cuda12` (CUDA 12.x)
### Building a Distribution for a Single Spark Release
Expand Down
2 changes: 2 additions & 0 deletions docs/additional-functionality/advanced_configs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ Name | Description | Default Value | Applicable at
<a name="sql.format.parquet.reader.type"></a>spark.rapids.sql.format.parquet.reader.type|Sets the Parquet reader type. We support different types that are optimized for different environments. The original Spark style reader can be selected by setting this to PERFILE which individually reads and copies files to the GPU. Loading many small files individually has high overhead, and using either COALESCING or MULTITHREADED is recommended instead. The COALESCING reader is good when using a local file system where the executors are on the same nodes or close to the nodes the data is being read on. This reader coalesces all the files assigned to a task into a single host buffer before sending it down to the GPU. It copies blocks from a single file into a host buffer in separate threads in parallel, see spark.rapids.sql.multiThreadedRead.numThreads. MULTITHREADED is good for cloud environments where you are reading from a blobstore that is totally separate and likely has a higher I/O read cost. Many times the cloud environments also get better throughput when you have multiple readers in parallel. This reader uses multiple threads to read each file in parallel and each file is sent to the GPU separately. This allows the CPU to keep reading while GPU is also doing work. See spark.rapids.sql.multiThreadedRead.numThreads and spark.rapids.sql.format.parquet.multiThreadedRead.maxNumFilesParallel to control the number of threads and amount of memory used. By default this is set to AUTO so we select the reader we think is best. This will either be the COALESCING or the MULTITHREADED based on whether we think the file is in the cloud. See spark.rapids.cloudSchemes.|AUTO|Runtime
<a name="sql.format.parquet.write.enabled"></a>spark.rapids.sql.format.parquet.write.enabled|When set to false disables parquet output acceleration|true|Runtime
<a name="sql.format.parquet.writer.int96.enabled"></a>spark.rapids.sql.format.parquet.writer.int96.enabled|When set to false, disables accelerated parquet write if the spark.sql.parquet.outputTimestampType is set to INT96|true|Runtime
<a name="sql.formatNumberFloat.enabled"></a>spark.rapids.sql.formatNumberFloat.enabled|format_number with floating point types on the GPU returns results that have a different precision than the default results of Spark.|false|Runtime
<a name="sql.hasExtendedYearValues"></a>spark.rapids.sql.hasExtendedYearValues|Spark 3.2.0+ extended parsing of years in dates and timestamps to support the full range of possible values. Prior to this it was limited to a positive 4 digit year. The Accelerator does not support the extended range yet. This config indicates if your data includes this extended range or not, or if you don't care about getting the correct values on values with the extended range.|true|Runtime
<a name="sql.hashOptimizeSort.enabled"></a>spark.rapids.sql.hashOptimizeSort.enabled|Whether sorts should be inserted after some hashed operations to improve output ordering. This can improve output file sizes when saving to columnar formats.|false|Runtime
<a name="sql.improvedFloatOps.enabled"></a>spark.rapids.sql.improvedFloatOps.enabled|For some floating point operations spark uses one way to compute the value and the underlying cudf implementation can use an improved algorithm. In some cases this can result in cudf producing an answer when spark overflows.|true|Runtime
Expand Down Expand Up @@ -234,6 +235,7 @@ Name | SQL Function(s) | Description | Default Value | Notes
<a name="sql.expression.Expm1"></a>spark.rapids.sql.expression.Expm1|`expm1`|Euler's number e raised to a power minus 1|true|None|
<a name="sql.expression.Flatten"></a>spark.rapids.sql.expression.Flatten|`flatten`|Creates a single array from an array of arrays|true|None|
<a name="sql.expression.Floor"></a>spark.rapids.sql.expression.Floor|`floor`|Floor of a number|true|None|
<a name="sql.expression.FormatNumber"></a>spark.rapids.sql.expression.FormatNumber|`format_number`|Formats the number x like '#,###,###.##', rounded to d decimal places.|true|None|
<a name="sql.expression.FromUTCTimestamp"></a>spark.rapids.sql.expression.FromUTCTimestamp|`from_utc_timestamp`|Render the input UTC timestamp in the input timezone|true|None|
<a name="sql.expression.FromUnixTime"></a>spark.rapids.sql.expression.FromUnixTime|`from_unixtime`|Get the string from a unix timestamp|true|None|
<a name="sql.expression.GetArrayItem"></a>spark.rapids.sql.expression.GetArrayItem| |Gets the field at `ordinal` in the Array|true|None|
Expand Down
4 changes: 4 additions & 0 deletions docs/compatibility.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -664,6 +664,10 @@ The GPU will use different precision than Java's toString method when converting
types to strings. The GPU uses a lowercase `e` prefix for an exponent while Spark uses uppercase
`E`. As a result the computed string can differ from the default behavior in Spark.

The `format_number` function will retain 10 digits of precision for the GPU when the input is a floating
point number, but Spark will retain up to 17 digits of precision, i.e. `format_number(1234567890.1234567890, 5)`
will return `1,234,567,890.00000` on the GPU and `1,234,567,890.12346` on the CPU. To enable this on the GPU, set [`spark.rapids.sql.formatNumberFloat.enabled`](additional-functionality/advanced_configs.md#sql.formatNumberFloat.enabled) to `true`.

Starting from 22.06 this conf is enabled by default, to disable this operation on the GPU, set
[`spark.rapids.sql.castFloatToString.enabled`](additional-functionality/advanced_configs.md#sql.castFloatToString.enabled) to `false`.

Expand Down
150 changes: 109 additions & 41 deletions docs/supported_ops.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6461,23 +6461,23 @@ are limited.
<td> </td>
</tr>
<tr>
<td rowSpan="3">FromUTCTimestamp</td>
<td rowSpan="3">`from_utc_timestamp`</td>
<td rowSpan="3">Render the input UTC timestamp in the input timezone</td>
<td rowSpan="3">FormatNumber</td>
<td rowSpan="3">`format_number`</td>
<td rowSpan="3">Formats the number x like '#,###,###.##', rounded to d decimal places.</td>
<td rowSpan="3">None</td>
<td rowSpan="3">project</td>
<td>timestamp</td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td>x</td>
<td> </td>
<td>S</td>
<td>S</td>
<td>S</td>
<td>S</td>
<td>S</td>
<td>S</td>
<td> </td>
<td><em>PS<br/>UTC is only supported TZ for TIMESTAMP</em></td>
<td> </td>
<td> </td>
<td>S</td>
<td> </td>
<td> </td>
<td> </td>
Expand All @@ -6487,17 +6487,17 @@ are limited.
<td> </td>
</tr>
<tr>
<td>timezone</td>
<td> </td>
<td>d</td>
<td> </td>
<td> </td>
<td> </td>
<td><em>PS<br/>Literal value only</em></td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td><em>PS<br/>Only timezones equivalent to UTC are supported</em></td>
<td><b>NS</b></td>
<td> </td>
<td> </td>
<td> </td>
Expand All @@ -6517,8 +6517,8 @@ are limited.
<td> </td>
<td> </td>
<td> </td>
<td><em>PS<br/>UTC is only supported TZ for TIMESTAMP</em></td>
<td> </td>
<td>S</td>
<td> </td>
<td> </td>
<td> </td>
Expand Down Expand Up @@ -6555,6 +6555,74 @@ are limited.
<th>UDT</th>
</tr>
<tr>
<td rowSpan="3">FromUTCTimestamp</td>
<td rowSpan="3">`from_utc_timestamp`</td>
<td rowSpan="3">Render the input UTC timestamp in the input timezone</td>
<td rowSpan="3">None</td>
<td rowSpan="3">project</td>
<td>timestamp</td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td><em>PS<br/>UTC is only supported TZ for TIMESTAMP</em></td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
</tr>
<tr>
<td>timezone</td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td><em>PS<br/>Only timezones equivalent to UTC are supported</em></td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
</tr>
<tr>
<td>result</td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td><em>PS<br/>UTC is only supported TZ for TIMESTAMP</em></td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
</tr>
<tr>
<td rowSpan="3">FromUnixTime</td>
<td rowSpan="3">`from_unixtime`</td>
<td rowSpan="3">Get the string from a unix timestamp</td>
Expand Down Expand Up @@ -6874,6 +6942,32 @@ are limited.
<td><b>NS</b></td>
</tr>
<tr>
<th>Expression</th>
<th>SQL Functions(s)</th>
<th>Description</th>
<th>Notes</th>
<th>Context</th>
<th>Param/Output</th>
<th>BOOLEAN</th>
<th>BYTE</th>
<th>SHORT</th>
<th>INT</th>
<th>LONG</th>
<th>FLOAT</th>
<th>DOUBLE</th>
<th>DATE</th>
<th>TIMESTAMP</th>
<th>STRING</th>
<th>DECIMAL</th>
<th>NULL</th>
<th>BINARY</th>
<th>CALENDAR</th>
<th>ARRAY</th>
<th>MAP</th>
<th>STRUCT</th>
<th>UDT</th>
</tr>
<tr>
<td rowSpan="2">GetStructField</td>
<td rowSpan="2"> </td>
<td rowSpan="2">Gets the named field of the struct</td>
Expand Down Expand Up @@ -6921,32 +7015,6 @@ are limited.
<td><b>NS</b></td>
</tr>
<tr>
<th>Expression</th>
<th>SQL Functions(s)</th>
<th>Description</th>
<th>Notes</th>
<th>Context</th>
<th>Param/Output</th>
<th>BOOLEAN</th>
<th>BYTE</th>
<th>SHORT</th>
<th>INT</th>
<th>LONG</th>
<th>FLOAT</th>
<th>DOUBLE</th>
<th>DATE</th>
<th>TIMESTAMP</th>
<th>STRING</th>
<th>DECIMAL</th>
<th>NULL</th>
<th>BINARY</th>
<th>CALENDAR</th>
<th>ARRAY</th>
<th>MAP</th>
<th>STRUCT</th>
<th>UDT</th>
</tr>
<tr>
<td rowSpan="3">GetTimestamp</td>
<td rowSpan="3"> </td>
<td rowSpan="3">Gets timestamps from strings using given pattern.</td>
Expand Down
7 changes: 2 additions & 5 deletions integration_tests/src/main/python/cast_test.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@

from asserts import assert_gpu_and_cpu_are_equal_collect, assert_gpu_and_cpu_are_equal_sql, assert_gpu_and_cpu_error, assert_gpu_fallback_collect, assert_spark_exception
from data_gen import *
from spark_session import is_before_spark_320, is_before_spark_330, is_spark_340_or_later, is_spark_350_or_later, \
is_databricks113_or_later, with_gpu_session
from spark_session import is_before_spark_320, is_before_spark_330, is_spark_340_or_later, \
is_databricks113_or_later
from marks import allow_non_gpu, approximate_float
from pyspark.sql.types import *
from spark_init_internal import spark_version
Expand Down Expand Up @@ -297,7 +297,6 @@ def _assert_cast_to_string_equal (data_gen, conf):

@pytest.mark.parametrize('data_gen', all_array_gens_for_cast_to_string, ids=idfn)
@pytest.mark.parametrize('legacy', ['true', 'false'])
@pytest.mark.xfail(condition=is_spark_350_or_later(), reason='https://github.com/NVIDIA/spark-rapids/issues/9065')
def test_cast_array_to_string(data_gen, legacy):
_assert_cast_to_string_equal(
data_gen,
Expand All @@ -317,7 +316,6 @@ def test_cast_array_with_unmatched_element_to_string(data_gen, legacy):

@pytest.mark.parametrize('data_gen', basic_map_gens_for_cast_to_string, ids=idfn)
@pytest.mark.parametrize('legacy', ['true', 'false'])
@pytest.mark.xfail(condition=is_spark_350_or_later(), reason='https://github.com/NVIDIA/spark-rapids/issues/9065')
def test_cast_map_to_string(data_gen, legacy):
_assert_cast_to_string_equal(
data_gen,
Expand All @@ -337,7 +335,6 @@ def test_cast_map_with_unmatched_element_to_string(data_gen, legacy):

@pytest.mark.parametrize('data_gen', [StructGen([[str(i), gen] for i, gen in enumerate(basic_array_struct_gens_for_cast_to_string)] + [["map", MapGen(ByteGen(nullable=False), null_gen)]])], ids=idfn)
@pytest.mark.parametrize('legacy', ['true', 'false'])
@pytest.mark.xfail(condition=is_spark_350_or_later(), reason='https://github.com/NVIDIA/spark-rapids/issues/9065')
def test_cast_struct_to_string(data_gen, legacy):
_assert_cast_to_string_equal(
data_gen,
Expand Down
2 changes: 2 additions & 0 deletions integration_tests/src/main/python/csv_test.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -561,6 +561,7 @@ def test_csv_read_count(spark_tmp_path):
conf = {'spark.rapids.sql.explain': 'ALL'})

@allow_non_gpu('FileSourceScanExec', 'ProjectExec', 'CollectLimitExec', 'DeserializeToObjectExec')
@pytest.mark.skip(reason="https://github.com/NVIDIA/spark-rapids/issues/9325")
@pytest.mark.skipif(is_before_spark_340(), reason='`TIMESTAMP_NTZ` is only supported in Spark 340+')
@pytest.mark.parametrize('date_format', csv_supported_date_formats)
@pytest.mark.parametrize('ts_part', csv_supported_ts_parts)
Expand All @@ -569,6 +570,7 @@ def test_csv_infer_schema_timestamp_ntz_v1(spark_tmp_path, date_format, ts_part,
csv_infer_schema_timestamp_ntz(spark_tmp_path, date_format, ts_part, timestamp_type, 'csv', 'FileSourceScanExec')

@allow_non_gpu('BatchScanExec', 'FileSourceScanExec', 'ProjectExec', 'CollectLimitExec', 'DeserializeToObjectExec')
@pytest.mark.skip(reason="https://github.com/NVIDIA/spark-rapids/issues/9325")
@pytest.mark.skipif(is_before_spark_340(), reason='`TIMESTAMP_NTZ` is only supported in Spark 340+')
@pytest.mark.parametrize('date_format', csv_supported_date_formats)
@pytest.mark.parametrize('ts_part', csv_supported_ts_parts)
Expand Down
54 changes: 54 additions & 0 deletions integration_tests/src/main/python/string_test.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -797,3 +797,57 @@ def test_conv_dec_to_from_hex(from_base, to_base, pattern):
lambda spark: unary_op_df(spark, gen).select('a', f.conv(f.col('a'), from_base, to_base)),
conf={'spark.rapids.sql.expression.Conv': True}
)

format_number_gens = integral_gens + [DecimalGen(precision=7, scale=7), DecimalGen(precision=18, scale=0),
DecimalGen(precision=18, scale=3), DecimalGen(precision=36, scale=5),
DecimalGen(precision=36, scale=-5), DecimalGen(precision=38, scale=10),
DecimalGen(precision=38, scale=-10),
DecimalGen(precision=38, scale=30, special_cases=[Decimal('0.000125')]),
DecimalGen(precision=38, scale=32, special_cases=[Decimal('0.000125')])]

@pytest.mark.parametrize('data_gen', format_number_gens, ids=idfn)
def test_format_number_supported(data_gen):
gen = data_gen
assert_gpu_and_cpu_are_equal_collect(
lambda spark: unary_op_df(spark, gen).selectExpr(
'format_number(a, -2)',
'format_number(a, 0)',
'format_number(a, 1)',
'format_number(a, 5)',
'format_number(a, 10)',
'format_number(a, 100)')
)

float_format_number_conf = {'spark.rapids.sql.formatNumberFloat.enabled': 'true'}
format_number_float_gens = [DoubleGen(min_exp=-300, max_exp=15)]

@pytest.mark.parametrize('data_gen', format_number_float_gens, ids=idfn)
def test_format_number_float_limited(data_gen):
gen = data_gen
assert_gpu_and_cpu_are_equal_collect(
lambda spark: unary_op_df(spark, gen).selectExpr(
'format_number(a, 5)'),
conf = float_format_number_conf
)

# format_number for float/double is disabled by default due to compatibility issue
# GPU will generate result with less precision than CPU
@allow_non_gpu('ProjectExec')
@pytest.mark.parametrize('data_gen', [float_gen, double_gen], ids=idfn)
def test_format_number_float_fallback(data_gen):
assert_gpu_fallback_collect(
lambda spark: unary_op_df(spark, data_gen).selectExpr(
'format_number(a, 5)'),
'FormatNumber'
)

# fallback due to https://github.com/NVIDIA/spark-rapids/issues/9309
@allow_non_gpu('ProjectExec')
@pytest.mark.parametrize('data_gen', [float_gen, double_gen], ids=idfn)
def test_format_number_decimal_big_scale_fallback(data_gen):
data_gen = DecimalGen(precision=38, scale=37)
assert_gpu_fallback_collect(
lambda spark: unary_op_df(spark, data_gen).selectExpr(
'format_number(a, 5)'),
'FormatNumber'
)
Loading

0 comments on commit 7c7f42a

Please sign in to comment.