Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Databricks GpuBroadcastNestedLoopJoinExec for AST splitting change [databricks] #9688

Closed
wants to merge 1 commit into from
Closed

Update Databricks GpuBroadcastNestedLoopJoinExec for AST splitting change [databricks] #9688

wants to merge 1 commit into from

Conversation

jlowe
Copy link
Member

@jlowe jlowe commented Nov 13, 2023

Fixes #9681. This ports the changes from #9635 to the Databricks version of GpuBroadcastNestedLoopJoinExec.

@jlowe jlowe added the build Related to CI / CD or cleanly building label Nov 13, 2023
@jlowe jlowe self-assigned this Nov 13, 2023
@jlowe jlowe changed the title Update Databricks GpuBroadcastNestedLoopJoinExec for AST splitting change Update Databricks GpuBroadcastNestedLoopJoinExec for AST splitting change [databricks] Nov 13, 2023
@jlowe
Copy link
Member Author

jlowe commented Nov 13, 2023

build

winningsix
winningsix approved these changes Nov 13, 2023
View reviewed changes
Copy link
Collaborator

@winningsix winningsix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@razajafri
Copy link
Collaborator

build

@razajafri
Copy link
Collaborator

I restarted the premerge by mistake!

@razajafri razajafri mentioned this pull request Nov 13, 2023
Merged
@razajafri
Copy link
Collaborator

CI failed with Databricks build result : FAILURE

@razajafri
Copy link
Collaborator

build

@pxLi
Copy link
Collaborator

pxLi commented Nov 14, 2023

failed AQE cases


[2023-11-14T06:02:38.719Z] E                   : scala.MatchError: GpuProject [date_sub(state_start#51627, 2) AS _agpu_non_ast_r_1#51716, state_start#51627]

[2023-11-14T06:02:38.719Z] E                   +- GpuShuffleCoalesce 104857600

[2023-11-14T06:02:38.719Z] E                      +- ShuffleQueryStage 2, Statistics(sizeInBytes=98.0 B, rowCount=1, isRuntime=true)

[2023-11-14T06:02:38.719Z] E                         +- GpuColumnarExchange gpusinglepartitioning$(), EXECUTOR_BROADCAST, [plan_id=85154]

[2023-11-14T06:02:38.719Z] E                            +- GpuCoalesceBatches targetsize(104857600)

[2023-11-14T06:02:38.719Z] E                               +- GpuFilter gpuisnotnull(state_start#51627)

[2023-11-14T06:02:38.719Z] E                                  +- GpuHashAggregate(keys=[], functions=[gpubasicmax(cast(ts#51641 as date))], output=[state_start#51627])

[2023-11-14T06:02:38.719Z] E                                     +- GpuShuffleCoalesce 104857600

[2023-11-14T06:02:38.719Z] E                                        +- ShuffleQueryStage 0, Statistics(sizeInBytes=456.0 B, rowCount=4, isRuntime=true)

[2023-11-14T06:02:38.719Z] E                                           +- GpuColumnarExchange gpusinglepartitioning$(), ENSURE_REQUIREMENTS, [plan_id=84919]

[2023-11-14T06:02:38.719Z] E                                              +- GpuHashAggregate(keys=[], functions=[partial_gpubasicmax(cast(ts#51641 as date))], output=[max#51651])

[2023-11-14T06:02:38.719Z] E                                                 +- GpuFileGpuScan parquet [ts#51641] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/tmp/pyspark_tests/1114-032614-fce5u4n1-10-2-128-10-gw0-3962-267437583/PAR..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ts:timestamp>

[2023-11-14T06:02:38.719Z] E                    (of class com.nvidia.spark.rapids.GpuProjectExec)

[2023-11-14T06:02:38.719Z] E                   	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.shuffleExchange(GpuBroadcastNestedLoopJoinExec.scala:159)

[2023-11-14T06:02:38.719Z] E                   	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.getBroadcastRelation(GpuBroadcastNestedLoopJoinExec.scala:171)

[2023-11-14T06:02:38.719Z] E                   	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExecBase.internalDoExecuteColumnar(GpuBroadcastNestedLoopJoinExecBase.scala:560)

[2023-11-14T06:02:38.719Z] E                   	at com.nvidia.spark.rapids.GpuExec.doExecuteColumnar(GpuExec.scala:349)

[2023-11-14T06:02:38.719Z] E                   	at com.nvidia.spark.rapids.GpuExec.doExecuteColumnar$(GpuExec.scala:348)

list of failures,

[2023-11-14T06:02:38.995Z] =========================== short test summary info ============================
[2023-11-14T06:02:38.995Z] FAILED ../../src/main/python/aqe_test.py::test_aqe_join_reused_exchange_inequality_condition[inner][INJECT_OOM, IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(BroadcastNestedLoopJoinExec,Cast,DateSub,ShuffleExchangeExec)] - py4j.protocol.Py4JJavaError: An error occurred while calling o246071.collec...
[2023-11-14T06:02:38.995Z] FAILED ../../src/main/python/aqe_test.py::test_aqe_join_reused_exchange_inequality_condition[cross][IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(BroadcastNestedLoopJoinExec,Cast,DateSub,ShuffleExchangeExec)] - py4j.protocol.Py4JJavaError: An error occurred while calling o246312.collec...
[2023-11-14T06:02:38.995Z] FAILED ../../src/main/python/aqe_test.py::test_aqe_join_reused_exchange_inequality_condition[left semi][IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(BroadcastNestedLoopJoinExec,Cast,DateSub,ShuffleExchangeExec)] - py4j.protocol.Py4JJavaError: An error occurred while calling o246553.collec...
[2023-11-14T06:02:38.995Z] FAILED ../../src/main/python/aqe_test.py::test_aqe_join_reused_exchange_inequality_condition[left anti][INJECT_OOM, IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(BroadcastNestedLoopJoinExec,Cast,DateSub,ShuffleExchangeExec)] - py4j.protocol.Py4JJavaError: An error occurred while calling o246794.collec...
[2023-11-14T06:02:38.995Z] FAILED ../../src/main/python/aqe_test.py::test_aqe_join_reused_exchange_inequality_condition[anti][IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(BroadcastNestedLoopJoinExec,Cast,DateSub,ShuffleExchangeExec)] - py4j.protocol.Py4JJavaError: An error occurred while calling o247035.collec...
[2023-11-14T06:02:38.995Z] = 5 failed, 19285 passed, 1657 skipped, 917 xfailed, 305 xpassed, 946 warnings in 8487.83s (2:21:27) =

@winningsix winningsix mentioned this pull request Nov 14, 2023
Closed
@jlowe
Copy link
Member Author

jlowe commented Nov 14, 2023

Superceded by #9695 .

@jlowe jlowe closed this Nov 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@razajafri razajafri razajafri approved these changes

@winningsix winningsix winningsix approved these changes

Assignees

@jlowe jlowe

Labels
build Related to CI / CD or cleanly building
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Databricks Build Failing For 330db+
4 participants
@jlowe @razajafri @pxLi @winningsix