[1.2] Gluten 1.2.0 Issue Tracker

[weiting-chen]

Backend

VL (Velox)

Bug description

This is a tracker to collect all the critical issues in Gluten v1.2.0 and plan to be fix in v1.2.1.
Discussion in here: #7125

Port List

Shuffle, Spill related bug fix

• Spill Hang fix has been done in [VL] Branch 1.2: Backport fixes for #7243 #7943
• [VL] Fix shuffle spill triggered by evicting buffers during stop #6422 has been done in [GLUTEN-7126][VL][1.2] Port Fix shuffle spill triggered by evicting buffers during stop (apache#6422) #7991
• [GLUTEN-7860][CORE] In shuffle writer, replace MemoryMappedFile to avoid OOM #7861 is working in [CORE][BRANCH-1.2] Port #7861 to fix OOM in shuffle writer #8078

Celeborn related bug fix

• [CELEBORN] CHCelebornColumnarShuffleWriter supports celeborn.client.spark.shuffle.writer to use memory sort shuffle in ClickHouse backend #6432 has been done in [VL][1.2] Port 6432 6657 for Celeborn bug fix in branch 1.2 #7922
• [GLUTEN-6656][CELEBORN] Fix CelebornColumnarShuffleWriter assertion failed #6657 has been done in [VL][1.2] Port 6432 6657 for Celeborn bug fix in branch 1.2 #7922

Uniffle related bug fix

• [GLUTEN-6656][UNIFFLE] VeloxUniffleColumnarShuffleWriter should send commit for all ColumnBatch with empty rows #6698 has been done in [GLUTEN-7126][VL][1.2] Port #6698 #7525 #7560 for Uniffle bug fix #7994
• [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle #7525 has been done in [GLUTEN-7126][VL][1.2] Port #6698 #7525 #7560 for Uniffle bug fix #7994
• [GLUTEN-7559][VL][UNIFFLE] Set rss.enabled to true in UniffleShuffleManager #7560 has been done in [GLUTEN-7126][VL][1.2] Port #6698 #7525 #7560 for Uniffle bug fix #7994

Compile & Package related bug fix

• [MINOR][VL] Remove duplicate mvn packages #6560 has been done in [VL][1.2] Port #6560 #6569 #6730 #7117 for vcpkg issue fix #7974
• [VL] Improve package scripts #6569 has been done in [VL][1.2] Port #6560 #6569 #6730 #7117 for vcpkg issue fix #7974
• [GLUTEN-6569][FOLLOWUP][VL] Delete unnecessary gcc9 enable of package script #6730 has been done in [VL][1.2] Port #6560 #6569 #6730 #7117 for vcpkg issue fix #7974
• [GLUTEN-7118][VL] Fix duckdb target issue when vcpkg is enabled #7117 has been done in [VL][1.2] Port #6560 #6569 #6730 #7117 for vcpkg issue fix #7974

CI/CD bug fix

• [GLUTEN-6915][MISC]Fix workflow permission issue. #6911 has been done in [VL][1.2] Port 6911 6919 for github workflow resource not accessible issue fix #6945
• [GLUTEN-6915][CORE]Fix listComments TypeError #6919 has been done in [VL][1.2] Port 6911 6919 for github workflow resource not accessible issue fix #6945
• Upgrade GHA artifacts version has been done in [VL][1.2] Upgrade GHA artifacts version to 3 #7293
• Port CI changes has been done in [VL] Port CI changes to branch-1.2 and pick simdjson related fix #7314
• skip data gen in OOM test has been done in [VL] Follow-up fix for #7314 on branch-1.2: skip data gen in oom test #7329
• CK Backend CI/CD refresh has been done in [CK][1.2] Support trigger CK Backend CI/CD in branch1.2 #7936
• [GLUTEN-7950][VL] Keep Core module's build flag consistent with Velox #8027 has been done in [VL][Branch-1.2] Port #8034 & #8027 for fixing march flag setting and #8042 for fixing GHA failure on centos-7 #8075
• [VL] Clean up duplicate CMake code for setting CMAKE_CXX_FLAGS #8034 has been done in [VL][Branch-1.2] Port #8034 & #8027 for fixing march flag setting and #8042 for fixing GHA failure on centos-7 #8075
• [GLUTEN-6887][VL] Daily Update Velox Version (2024_11_26) #8042 has been done in [VL][Branch-1.2] Port #8034 & #8027 for fixing march flag setting and #8042 for fixing GHA failure on centos-7 #8075
• [VL] vcpkg: Broken libelf mirror #8047 has been done in [VL] Branch 1.2: Port #8047 to fix libelf link #8059

Other bug fix

• [GLUTEN-6562][VL] Decouple BUILD_BENCHMARKS and BUILD_TESTS build options #6563 has been done in [VL][1.2] Port 6563 6679 for build options and collectQueryExecutionFallbackSummary fix #7919
• [VL] Add config for show velox task metrics when finished #6573 has been done in [VL][1.2] Port #6573 #7025 #7132 #7973
• [Core] Make collectQueryExecutionFallbackSummary as a public util method #6679 has been done in [VL][1.2] Port 6563 6679 for build options and collectQueryExecutionFallbackSummary fix #7919
• [GLUTEN-6902][VL]fix: Update to copy new LICENSE, NOTICE into jar #6901 has been done in [VL][1.2]Port PR #6901 for license and notice issue in binary jar and upgrade OAP Velox version to gluten-1.2.1 #6941
• [GLUTEN-7024][VL] Skip call collectMetrics when the task does not call next() #7025 has been done in [VL][1.2] Port #6573 #7025 #7132 #7973
• [GLUTEN-7037][VL] Add dwarf dependency to folly when building with vcpkg #7038 has been done in [GLUTEN-7037][VL][1.2] Add dwarf dependency to folly when building with vcpkg #7699
• [GLUTEN-6748][CORE] Search stack trace to infer adaptive execution context #7121 has been done in [VL][1.2] Port #7121 #7448 #7988
• [GLUTEN-7130][CORE] Skip command execution when collect qe fallback summary #7132 has been done in [VL][1.2] Port #6573 #7025 #7132 #7973
• [CORE] Fix GH security issues #7448 has been done in [VL][1.2] Port #7121 #7448 #7988
• BNLJ not fallback issue has been done in [GLUTEN-7126][CORE][1.2] Fix issue that unsupported join type in BNLJ is not fallback #7569
• devtoolset-9 for centos7 has been done in [VL][1.2] Install devtoolset-9 for centos7 build native lib GHA #7678

OAP Velox branch-1.2

• Add min, max Spark aggregate functions facebookincubator/velox#9868 has been done in [Branch-1.2] Port #9868 for min/max agg func support oap-project/velox#503
• Fix NaN handling for min/max aggreates pushed down to scan facebookincubator/velox#10583 has been done in [Branch-1.2] Port #10583 #10755 #11298 oap-project/velox#505
• Schema evolution support for reader value hooks facebookincubator/velox#10755 has been done in [Branch-1.2] Port #10583 #10755 #11298 oap-project/velox#505
• Disable aggregate pushdown for decimal type facebookincubator/velox#11298 has been done in [Branch-1.2] Port #10583 #10755 #11298 oap-project/velox#505

[zhouyuan]

Note
1.2.1 is planned for Nov/30

[weiting-chen]

1.2.1 code freeze is planned on Nov. 25