Skip to content

Commit

Permalink
Fix from_utc_timestamp failure on Cloudera when TZ is Iran (#10119)
Browse files Browse the repository at this point in the history
Signed-off-by: Chong Gao <[email protected]>
  • Loading branch information
res-life authored Dec 29, 2023
1 parent 7c861f1 commit 384b2a9
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 5 deletions.
4 changes: 2 additions & 2 deletions integration_tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -350,10 +350,10 @@ But only a small amout of cases are verifing with non-UTC time zone in the pre-m
When adding cases, should also check non-UTC is working besides the default UTC time zone.
Please test the following time zones:
```shell
$ TZ=Iran ./integration_tests/run_pyspark_from_build.sh
$ TZ=Asia/Shanghai ./integration_tests/run_pyspark_from_build.sh
$ TZ=America/Los_Angeles ./integration_tests/run_pyspark_from_build.sh
```
`Iran` is non-DST(Daylight Savings Time) time zone and `America/Los_Angeles` is DST time zone.
`Asia/Shanghai` is non-DST(Daylight Savings Time) time zone and `America/Los_Angeles` is DST time zone.
If the new added cases failed with non-UTC, then should allow the operator(does not support non-UTC) fallback,
For example, add the following annotation to the case:
Expand Down
2 changes: 1 addition & 1 deletion integration_tests/src/main/python/date_time_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ def test_unsupported_fallback_to_unix_timestamp(data_gen):
"to_unix_timestamp(a, b)"),
"ToUnixTimestamp")

@pytest.mark.parametrize('time_zone', ["Asia/Shanghai", "Iran", "UTC", "UTC+0", "UTC-0", "GMT", "GMT+0", "GMT-0"], ids=idfn)
@pytest.mark.parametrize('time_zone', ["Asia/Shanghai", "UTC", "UTC+0", "UTC-0", "GMT", "GMT+0", "GMT-0"], ids=idfn)
@pytest.mark.parametrize('data_gen', [timestamp_gen], ids=idfn)
@tz_sensitive_test
@allow_non_gpu(*non_utc_allow)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ import org.apache.spark.sql.types._
* A simple test performance framework for non-UTC features.
* Usage:
*
* argLine="-DTZs=Iran,Asia/Shanghai -DenableTimeZonePerf=true" \
* argLine="-DTZs=Asia/Shanghai,Japan -DenableTimeZonePerf=true" \
* mvn test -Dbuildver=311 -DwildcardSuites=com.nvidia.spark.rapids.timezone.TimeZonePerfSuite
* Note:
* Generate a Parquet file with 6 columns:
Expand All @@ -51,7 +51,7 @@ import org.apache.spark.sql.types._
class TimeZonePerfSuite extends SparkQueryCompareTestSuite with BeforeAndAfterAll {
// perf test is disabled by default since it's a long running time in UT.
private val enablePerfTest = java.lang.Boolean.getBoolean("enableTimeZonePerf")
private val timeZoneStrings = System.getProperty("TZs", "Iran")
private val timeZoneStrings = System.getProperty("TZs", "Asia/Shanghai")

// rows for perf test
private val numRows: Long = 1024L * 1024L * 10L
Expand Down

0 comments on commit 384b2a9

Please sign in to comment.