Skip to content

Commit

Permalink
Increase Java -Xmx memory to 2g for non-UTC time zone
Browse files Browse the repository at this point in the history
  • Loading branch information
Chong Gao committed Nov 26, 2023
1 parent 6a02eb7 commit 2bf2c96
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 5 deletions.
2 changes: 2 additions & 0 deletions integration_tests/run_pyspark_from_build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,8 @@ EOF
echo "SUCCESS spark-shell smoke test"
elif ((${#TEST_PARALLEL_OPTS[@]} > 0));
then
# will set -Xmx=2g
export PYSP_TEST_spark_driver_memory=2g
exec python "${RUN_TESTS_COMMAND[@]}" "${TEST_PARALLEL_OPTS[@]}" "${TEST_COMMON_OPTS[@]}"
else
# We set the GPU memory size to be a constant value even if only running with a parallelism of 1
Expand Down
7 changes: 2 additions & 5 deletions jenkins/spark-premerge-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -154,12 +154,9 @@ ci_2() {

# Download a Scala 2.12 build of spark
prepare_spark $SPARK_VER 2.12
./integration_tests/run_pyspark_from_build.sh

# Test a portion of cases for non-UTC time zone because of limited GPU resources.
# Here testing: parquet scan, orc scan, csv scan, cast, TimeZoneAwareExpression, FromUTCTimestamp
# Nightly CIs will cover all the cases.
TZ=Iran TEST='test_parquet_read_round_trip or test_read_round_trip or test_basic_csv_read or test_cast_string_ts_valid_format or test_unix_timestamp or test_from_utc_timestamp' ./integration_tests/run_pyspark_from_build.sh
# Test non-UTC time zone.
TZ=Iran ./integration_tests/run_pyspark_from_build.sh

# enable avro test separately
INCLUDE_SPARK_AVRO_JAR=true TEST='avro_test.py' ./integration_tests/run_pyspark_from_build.sh
Expand Down

0 comments on commit 2bf2c96

Please sign in to comment.