PoC: [GLUTEN-7745][VL] Incorporate SQL Union operator into Velox execution pipeline #7842

zhztheplayer · 2024-11-07T08:37:43Z

This is currently a PoC and not yet workable because of a blocker from Velox's single-threaded execution design:

I20241107 15:06:23.310680 1849235 VeloxPlanConverter.cc:128] Plan Node: 
-- Aggregation[7][PARTIAL n7_0 := count_partial("n0_0")] -> n7_0:BIGINT
  -- LocalPartition[6][GATHER] -> n0_0:BIGINT
    -- Project[4][expressions: (n0_0:BIGINT, "n1_1")] -> n0_0:BIGINT
      -- Project[1][expressions: (n1_1:BIGINT, "n0_0")] -> n1_1:BIGINT
        -- TableScan[0][table: hive_table] -> n0_0:BIGINT
    -- Project[5][expressions: (n0_0:BIGINT, "n3_1")] -> n0_0:BIGINT
      -- Project[3][expressions: (n3_1:BIGINT, "n2_0")] -> n3_1:BIGINT
        -- TableScan[2][table: hive_table] -> n2_0:BIGINT
24/11/07 15:06:23 ERROR TaskResources: Task 11 failed by error: 
org.apache.gluten.exception.GlutenException: Task doesn't support single thread execution: -- Aggregation[7]

	at org.apache.gluten.vectorized.PlanEvaluatorJniWrapper.nativeCreateKernelWithIterator(Native Method)
	at org.apache.gluten.vectorized.NativePlanEvaluator.createKernelWithBatchIterator(NativePlanEvaluator.java:66)
	at org.apache.gluten.backendsapi.velox.VeloxIteratorApi.genFirstStageIterator(VeloxIteratorApi.scala:214)
	at org.apache.gluten.execution.GlutenWholeStageColumnarRDD.$anonfun$compute$1(GlutenWholeStageColumnarRDD.scala:88)
	at org.apache.gluten.utils.Arm$.withResource(Arm.scala:25)
	at org.apache.gluten.metrics.GlutenTimeMetric$.millis(GlutenTimeMetric.scala:37)
	at org.apache.gluten.execution.GlutenWholeStageColumnarRDD.compute(GlutenWholeStageColumnarRDD.scala:77)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
	at org.apache.spark.scheduler.Task.run(Task.scala:141)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Related Velox code:

https://github.com/facebookincubator/velox/blob/12b52e70ec85ae0cdb4aa990797cddda9be5be27/velox/exec/Driver.h#L658-L660

github-actions · 2024-11-07T08:38:03Z

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Other pull requests

github-actions · 2024-11-07T08:38:14Z