Code

Merge branch-24.12 into main #11848

Code

Merged

YanxuanLiu merged 129 commits intomainNVIDIA/spark-rapids:mainfrom merge-branch-24.12-to-mainNVIDIA/spark-rapids:merge-branch-24.12-to-mainCopy head branch name to clipboardDec 16, 2024

+20,375-5,304

Commits

Commits on Sep 24, 2024

• Init version 24.12.0-SNAPSHOT

  Show description for 614d8f5
  nvauto

  committed
  614d8f5

• Merge pull request #11494 from NVIDIA/branch-24.10

  Show description for 86d0f60
  nvauto

  authored
  86d0f60

• Merge pull request #11495 from NVIDIA/branch-24.10

  Show description for 9ed9b94
  nvauto

  authored
  9ed9b94

Commits on Sep 26, 2024

• Merge pull request #11504 from NVIDIA/branch-24.10

  Show description for bae19e3
  nvauto

  authored
  bae19e3

• Support legacy mode for yyyymmdd format [databricks] (#11493)

  Show description for 01c3003
  res-life

  authored
  01c3003

• Merge pull request #11508 from NVIDIA/branch-24.10

  Show description for 7661abb
  nvauto

  authored
  7661abb

• Merge pull request #11513 from NVIDIA/branch-24.10

  Show description for 4b61b45
  nvauto

  authored
  4b61b45

Commits on Sep 27, 2024

• Merge pull request #11517 from NVIDIA/branch-24.10

  Show description for 0e446ad
  nvauto

  authored
  0e446ad

• Merge pull request #11518 from NVIDIA/branch-24.10

  Show description for 9089b7f
  nvauto

  authored
  9089b7f

• Merge pull request #11523 from NVIDIA/branch-24.10

  Show description for e9b89ff
  nvauto

  authored
  e9b89ff

Commits on Sep 28, 2024

• Merge pull request #11540 from NVIDIA/branch-24.10

  Show description for 6692d17
  nvauto

  authored
  6692d17

Commits on Sep 30, 2024

• Update rapids JNI and private dependency to 24.12.0-SNAPSHOT [skip ci] (#11542)

  Show description for 2036f16
  

  nvauto

  and

  NvTimLiu

  authored
  2036f16

Commits on Oct 7, 2024

• Merge branch 'branch-24.10' into fix_merge_conflict

  Show description for 1432649
  revans2

  committed
  1432649

• Merge pull request #11563 from revans2/fix_merge_conflict

  Show description for 7a78951
  revans2

  authored
  7a78951

Commits on Oct 8, 2024

• Spark 4: Fix parquet_test.py [databricks] (#11519)

  Show description for 5eeddc6
  mythrocks

  authored
  5eeddc6

• Update test case related to LEACY datetime format to unblock nightly CI (#11544)

  Show description for cd46572
  

  res-life

  and

  Chong Gao

  authored
  cd46572

• Add in a basic plugin for dataframe UDF support in Apache Spark (#11561)

  Show description for 6897713
  revans2

  authored
  6897713

• Disk spill metric (#11564)

  Show description for 506d212
  zpuller

  authored
  506d212

Commits on Oct 9, 2024

• Merge branch 'branch-24.10' into fixmerge

  Show description for 58eb33f
  jlowe

  committed
  58eb33f

• Log reconfigure multi-file thread pool only once (#11571)

  Show description for 180da0f
  gerashegalov

  authored
  180da0f

• Merge pull request #11579 from jlowe/fixmerge

  Show description for 025e62e
  jlowe

  authored
  025e62e

• [Spark 4.0] Address test failures in cast_test.py [databricks] (#11559)

  Show description for e8b78c0
  mythrocks

  authored
  e8b78c0

Commits on Oct 10, 2024

• avoid long tail tasks due to PrioritySemaphore (#11574)

  Show description for e8ac073
  binmahone

  authored
  e8ac073

Commits on Oct 11, 2024

• addressing jason's comment (#11587)

  Show description for 0ba4fd2
  binmahone

  authored
  0ba4fd2

• Merge pull request #11594 from NVIDIA/branch-24.10

  Show description for 4866941
  nvauto

  authored
  4866941

Commits on Oct 12, 2024

• Fix collection_ops_tests for Spark 4.0 [databricks] (#11414)

  Show description for aca15ab
  mythrocks

  authored
  aca15ab

Commits on Oct 14, 2024

• Merge pull request #11601 from NVIDIA/branch-24.10

  Show description for 3744ad2
  nvauto

  authored
  3744ad2

• Merge branch-24.10 into branch-24.12

  NvTimLiu

  committed
  adc4e95

• Merge pull request #11605 from NvTimLiu/fix-auto-merge-conflict-11604

  Show description for 8c55ef3
  jlowe

  authored
  8c55ef3

• Disable regex tests to unblock CI (#11606)

  Show description for 2d3e0ec
  jlowe

  authored
  2d3e0ec

• Remove an unused config shuffle.spillThreads (#11595)

  Show description for 11964ae
  abellina

  authored
  11964ae

Commits on Oct 15, 2024

• Adopt JSONUtils.concatenateJsonStrings for concatenating JSON strings (#11549)

  Show description for 0510a78
  ttnghia

  authored
  0510a78

• Change DataSource calendar interval error to fix spark400 build (#11610)

  Show description for f8c386e
  jlowe

  authored
  f8c386e

Commits on Oct 16, 2024

• Use mvn -f scala2.13/ in the build scripts to build the 2.13 jars (#11608)

  Show description for ed4c878
  NvTimLiu

  authored
  ed4c878

• install_deps changes for Databricks 14.3 [databricks] (#11597)

  Show description for 7c18198
  

  razajafri

  and

  jlowe

  authored
  7c18198

• Revert "Disable regex tests to unblock CI (#11606)" (#11612)

  Show description for 0089d25
  jlowe

  authored
  0089d25

• Ensure repartition overflow test always overflows (#11614)

  Show description for e3f3f51
  jlowe

  authored
  e3f3f51

Commits on Oct 17, 2024

• Quick fix for the build script failure of Scala 2.13 jars (#11617)

  Show description for 52c91d3
  NvTimLiu

  authored
  52c91d3

Commits on Oct 18, 2024

• Merge pull request #11625 from NVIDIA/branch-24.10

  Show description for 00fe174
  nvauto

  authored
  00fe174

Commits on Oct 19, 2024

• Update JSON tests based on a closed/fixed issues (#11631)

  Show description for b5c2868
  revans2

  authored
  b5c2868

Commits on Oct 21, 2024

• Merge pull request #11636 from NVIDIA/branch-24.10

  Show description for e9f0e04
  nvauto

  authored
  e9f0e04

• Spark UT framework: Read Parquet file generated by parquet-thrift Rapids, UT case adjust. (#11591)

  Show description for f84b593
  Feng-Jiang28

  authored
  f84b593

• Add support for Spark 3.5.3 [databricks] (#11570)

  Show description for a24b575
  razajafri

  authored
  a24b575

Commits on Oct 22, 2024

• Disable date/timestamp types by default when parsing JSON (#11640)

  Show description for 36ae266
  ttnghia

  authored
  36ae266

• Fix udf-compiler scala2.13 internal return statements (#11553)

  Show description for b9a1a49
  abellina

  authored
  b9a1a49

• Fix collection_ops_test for [databricks] 14.3 (#11623)

  Show description for 732b25b
  mythrocks

  authored
  732b25b

• Spark 4 parquet_writer_test.py fixes (#11615)

  Show description for 8e2e627
  rwlee

  authored
  8e2e627

Commits on Oct 23, 2024

• Update to_json to be more generic and fix some bugs (#11642)

  Show description for a071efe
  revans2

  authored
  a071efe

Commits on Oct 24, 2024

• Datetime rebasing issue fixed (#11521)

  Show description for 5ed0a12
  Feng-Jiang28

  authored
  5ed0a12

• UT adjust test SPARK-26677: negated null-safe equality comparison (#11522)

  Show description for db15a61
  Feng-Jiang28

  authored
  db15a61

• Put DF_UDF plugin code into the main uber jar. (#11634)

  Show description for 05f40b5
  revans2

  authored
  05f40b5

Commits on Oct 25, 2024

• UT adjust override checkScanSchemata & enabling ut of exclude_by_suffix fea. (#11520)

  Show description for 910b64d
  Feng-Jiang28

  authored
  910b64d

• Support invalid partToExtract for parse_url (#11661)

  Show description for e31a710
  thirtiseven

  authored
  e31a710

Commits on Oct 28, 2024

• Support format 'yyyyMMdd HH:mm:ss' for legacy mode (#11658)

  Show description for 91db040
  res-life

  authored
  91db040

• Fix a NPE issue in GpuRand (#11647)

  Show description for b653ce2
  firestarman

  authored
  b653ce2

• Generate classes identical up to the shim package name [databricks] (#11665)

  Show description for 986eb5d
  gerashegalov

  authored
  986eb5d

Commits on Oct 29, 2024

• [DOC] update the supported OS in download page [skip ci] (#11656)

  Show description for 103e009
  

  nvliyuan

  and

  sameerz

  authored
  103e009

• Fix orc_write_test.py for [databricks] 14.3 (#11664)

  Show description for a6c4b34
  mythrocks

  authored
  a6c4b34

• Add a new NVTX range for task GPU ownership (#11596)

  Show description for 6b27556
  jihoonson

  authored
  6b27556

Commits on Oct 30, 2024

• Fix race condition with Parquet filter pushdown modifying shared hadoop Configuration (#11672)

  Show description for 81d1a3d
  tgravescs

  authored
  81d1a3d

• Simplify Transpilation of $ with Extended Line Separator Support in cuDF Regex (#11663)

  Show description for f0ae2ba
  SurajAralihalli

  authored
  f0ae2ba

Commits on Oct 31, 2024

• Merge pull request #11677 from NVIDIA/branch-24.10

  Show description for 5486a8d
  nvauto

  authored
  5486a8d

• Merge remote-tracking branch 'upstream/branch-24.10' into fix-auto-merge-conflict-11679

  pxLi

  committed
  7f8ff1b

• Merge pull request #11682 from pxLi/fix-auto-merge-conflict-11679

  Show description for 2134f2e
  pxLi

  authored
  2134f2e

Commits on Nov 1, 2024

• Use the new host memory allocation API (#11671)

  Show description for 372ca80
  revans2

  authored
  372ca80

Commits on Nov 4, 2024

• Merge pull request #11688 from NVIDIA/branch-24.10

  Show description for fcede85
  nvauto

  authored
  fcede85

• Reserve allocation should be displayed when erroring due to lack of memory on startup (#11282)

  Show description for 4a1baa5
  kuhushukla

  authored
  4a1baa5

• Exclude shimplify-generated files from scalastyle (#11685)

  Show description for 4d38dba
  gerashegalov

  authored
  4d38dba

• Preparation for the coming Kudo support (#11667)

  Show description for 35980d6
  

  firestarman

  and

  liurenjie1024

  authored
  35980d6

• Skip AQE-join-DPP tests for [databricks] 14.3 (#11644)

  Show description for 6e82c44
  mythrocks

  authored
  6e82c44

• Fix skipping fixed_length_char ORC tests on [databricks] > 13.3 (#11652)

  Show description for f533fc9
  mythrocks

  authored
  f533fc9

• Fix misc_expr_test for [databricks] 14.3 (#11670)

  Show description for 2e16ff2
  mythrocks

  authored
  2e16ff2

Commits on Nov 5, 2024

• Update the Maven repository to download Spark JAR files (#11689)

  Show description for 5afee5b
  NvTimLiu

  authored
  5afee5b

Commits on Nov 6, 2024

• Fix spark400 build due to LogicalRelation signature changes (#11695)

  Show description for ad4233d
  jlowe

  authored
  ad4233d

• Add Spark 3.4.4 Shim (#11692)

  Show description for ddbbba3
  gerashegalov

  authored
  ddbbba3

• Fix string_test for [databricks] 14.3 (#11669)

  Show description for 6100334
  mythrocks

  authored
  6100334

• Fix Parquet Writer tests on [databricks] 14.3 (#11673)

  Show description for 61acf56
  mythrocks

  authored
  61acf56

Commits on Nov 7, 2024

• Add retry in sub hash join (#11706)

  Show description for e13cd55
  firestarman

  authored
  e13cd55

Commits on Nov 8, 2024

• Add shim version 344 to LogicalPlanShims.scala (#11710)

  Show description for d208004
  SurajAralihalli

  authored
  d208004

• Make delta-lake shim dependencies parametrizable [databricks] (#11697)

  Show description for 7d2fec9
  gerashegalov

  authored
  7d2fec9

Commits on Nov 11, 2024

• impalaFile cannot be found by UT framework. (#11707)

  Show description for 3762569
  Feng-Jiang28

  authored
  3762569

Commits on Nov 12, 2024

• Simplify $ transpiling and fix newline character bug (#11703)

  Show description for 894b636
  SurajAralihalli

  authored
  894b636

• Let AWS Databricks automatically choose an Availability Zone (#11714)

  Show description for 862dab0
  NvTimLiu

  authored
  862dab0

Commits on Nov 13, 2024

• Added Shims for adding Databricks 14.3 Support [databricks] (#11635)

  Show description for 57b8caa
  

  3 peopleauthored
  57b8caa

Commits on Nov 14, 2024

• Improve JSON scan and from_json (#11702)

  Show description for a8010cc
  ttnghia

  authored
  a8010cc

Commits on Nov 18, 2024

• Change Databricks 14.3 shim name to spark350db143 (#11728)

  Show description for 9b06ae3
  NvTimLiu

  authored
  9b06ae3

Commits on Nov 19, 2024

• Support multi string contains [databricks] (#11413)

  Show description for b16d107
  res-life

  authored
  b16d107

• Add NullIntolerantShim to adapt to Spark 4.0 removing NullIntolerant (#11731)

  Show description for fd0781d
  jlowe

  authored
  fd0781d

Commits on Nov 20, 2024

• Support profiling for specific stages on a limited number of tasks (#11708)

  Show description for 3d26c4c
  

  thirtiseven

  and

  jlowe

  authored
  3d26c4c

• Skip from_json overflow tests for [databricks] 14.3 (#11719)

  Show description for 45a54ac
  mythrocks

  authored
  45a54ac

• Widen type promotion for decimals with larger scale in Parquet Read [databricks] (#11727)

  Show description for a5413e9
  nartal1

  authored
  a5413e9

• Fix leak with RapidsHostColumnBuilder in GpuUserDefinedFunction (#11700)

  Show description for e1fefa5
  abellina

  authored
  e1fefa5

• Add in support for months_between (#11737)

  Show description for 20c5281
  revans2

  authored
  20c5281

Commits on Nov 21, 2024

• Update to Spark 4.0 changing signature of SupportsV1Write.writeWithV1 (#11739)

  Show description for d3cda26
  jlowe

  authored
  d3cda26

Commits on Nov 22, 2024

• Integrate with kudo (#11724)

  liurenjie1024

  authored
  f2ea943

• Do not package the Databricks 14.3 shim into the dist jar [skip ci] (#11744)

  Show description for 7110cf4
  NvTimLiu

  authored
  7110cf4

• Add a few more JSON tests for MAP<STRING,STRING> (#11721)

  Show description for 6e6ce33
  

  revans2

  and

  ttnghia

  authored
  6e6ce33

• Add NVIDIA Copyright (#11723)

  Show description for a847575
  gerashegalov

  authored
  a847575

• Remove batch size bytes limits (#11746)

  Show description for e5547a1
  zpuller

  authored
  e5547a1

• host watermark metric (#11725)

  Show description for cacc3ae
  zpuller

  authored
  cacc3ae

Commits on Nov 23, 2024

• Execute from_json with struct schema using JSONUtils.fromJSONToStructs (#11618)

  Show description for daaaf24
  ttnghia

  authored
  daaaf24

Commits on Nov 25, 2024

• Print out the current attempt object when OOM inside a retry block (#11733)

  Show description for 6cba00d
  firestarman

  authored
  6cba00d

• Enable JSON Scan and from_json by default (#11753)

  Show description for 6539441
  

  revans2

  and

  ttnghia

  authored
  6539441

• Fix aqe_test failures on [databricks] 14.3. (#11750)

  Show description for 938db21
  mythrocks

  authored
  938db21

• Add support for asynchronous writing for parquet (#11730)

  Show description for 6b90b2f
  

  jihoonson

  and

  gerashegalov

  authored
  6b90b2f

Commits on Nov 26, 2024

• Fix Kudo batch serializer to only read header in hasNext (#11766)

  Show description for f5be35e
  jlowe

  authored
  f5be35e

• Avoid using StringBuffer in single-threaded methods. (#11759)

  Show description for 2b6ac11
  gerashegalov

  authored
  2b6ac11

• Fix query hang when using rapids multithread shuffle manager with kudo (#11771)

  Show description for e3dce9e
  liurenjie1024

  authored
  e3dce9e

• repartition-based fallback for hash aggregate v3 (#11712)

  Show description for 4fa0a1d
  

  binmahone

  and

  firestarman

  authored
  4fa0a1d

• Append knoguchi22 to blossom-ci whitelist [skip ci] (#11777)

  Show description for 82c26f1
  knoguchi22

  authored
  82c26f1

• Ability to decompress snappy and zstd Parquet files via CPU [databricks] (#11752)

  Show description for ff0ca0f
  jlowe

  authored
  ff0ca0f

• Fix dpp_test.py failures on [databricks] 14.3 (#11768)

  Show description for ed02cfe
  mythrocks

  authored
  ed02cfe

Commits on Nov 29, 2024

• fix issue 11790 (#11792)

  Show description for aa2da41
  binmahone

  authored
  aa2da41

Commits on Dec 3, 2024

• Fall back to CPU for non-UTC months_between (#11802)

  Show description for cb31afb
  revans2

  authored
  cb31afb

• exclude previous operator's time out of firstBatchHeuristic (#11794)

  Show description for 738c8e3
  binmahone

  authored
  738c8e3

Commits on Dec 7, 2024

• Orc writes don't fully support Booleans with nulls (#11763)

  kuhushukla

  authored
  fb2f72d

Commits on Dec 8, 2024

• Fixes a leak for the empty nlj iterator (#11832)

  Show description for 3449c8a
  abellina

  authored
  3449c8a

Commits on Dec 9, 2024

• Fix for lead/lag window test failures. (#11823)

  Show description for 45cdac3
  mythrocks

  authored
  45cdac3

Commits on Dec 10, 2024

• Fix leak in isTimeStamp (#11845)

  Show description for 96a58d1
  kuhushukla

  authored
  96a58d1

• Merge branch-24.12 into main

  nvauto

  committed
  2cb5a18

• Change version to 24.12.0

  Show description for 1c540c1
  nvauto

  committed
  1c540c1

Commits on Dec 11, 2024

• Increase the pre-merge CI timeout to 6 hours (#11857)

  Show description for 81b0b98
  NvTimLiu

  authored
  81b0b98

Commits on Dec 13, 2024

• [DOC] update doc for 24.12 release [skip ci] (#11841)

  Show description for 4b9bb23
  nvliyuan

  authored
  4b9bb23

Commits on Dec 16, 2024

• Update rapids JNI and private dependency to 24.12.0 (#11849)

  Show description for 4d7373b
  nvauto

  authored
  4d7373b

• Update latest changelog [skip ci] (#11851)

  Show description for 22680f5
  

  3 peopleauthored
  22680f5

• Merge remote-tracking branch 'upstream/branch-24.12' into merge-branch-24.12-to-main

  YanxuanLiu

  committed
  51a811f

• Remove 350db143 shim's build (#11874)

  Show description for 795aef8
  NvTimLiu

  authored
  795aef8

• Update latest changelog [skip ci] (#11876)

  Show description for d1ea935
  nvauto

  authored
  d1ea935

• Merge branch 'branch-24.12' into merge-branch-24.12-to-main

  YanxuanLiu

  committed
  8a8bd3d