Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] [Spark 4] Type mismatch Exceptions from DFUDFShims.scala with Spark-4.0.0 expressions.Expression #11822

Closed
NvTimLiu opened this issue Dec 4, 2024 · 0 comments · Fixed by #11827
Assignees
Labels
bug Something isn't working build Related to CI / CD or cleanly building

Comments

@NvTimLiu
Copy link
Collaborator

NvTimLiu commented Dec 4, 2024

Describe the bug

mvn package against spark400 shims, got below failures,

details at: https://github.com/NVIDIA/spark-rapids/actions/runs/12154774321/job/33895138679?pr=11819

[INFO] --- scala-maven-plugin:4.9.1:compile (scala-compile-first) @ rapids-4-spark-sql_2.13 ---
[INFO] Compiler bridge file: /home/runner/.m2/repository/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.10.0-bin_2.13.14__61.0-1.10.0_20240505T232140.jar
[INFO] compiling 485 Scala sources and 58 Java sources to /home/runner/work/spark-rapids/spark-rapids/scala2.13/sql-plugin/target/spark400/classes ...
Error: ] /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/spark400/scala/org/apache/spark/sql/nvidia/DFUDFShims.scala:27: type mismatch;
 found   : org.apache.spark.sql.Column
 required: org.apache.spark.sql.catalyst.expressions.Expression
[INFO] [Info] : org.apache.spark.sql.Column <: org.apache.spark.sql.catalyst.expressions.Expression?
[INFO] [Info] : false
Error: ] /home/runner/work/spark-rapids/spark-rapids/sql-plugin/src/main/spark400/scala/org/apache/spark/sql/nvidia/DFUDFShims.scala:28: type mismatch;
 found   : org.apache.spark.sql.catalyst.expressions.Expression
 required: org.apache.spark.sql.Column
[INFO] [Info] : org.apache.spark.sql.catalyst.expressions.Expression <: org.apache.spark.sql.Column?
[INFO] [Info] : false
Error: [ERROR] two errors found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for RAPIDS Accelerator for Apache Spark Root Project 25.02.0-SNAPSHOT:
[INFO] 
[INFO] RAPIDS Accelerator for Apache Spark Root Project ... SUCCESS [ 14.284 s]
[INFO] rapids-4-spark-jdk-profiles_2.13 ................... SUCCESS [  1.655 s]
[INFO] rapids-4-spark-shim-deps-parent_2.13 ............... SUCCESS [  1.946 s]
[INFO] rapids-4-spark-sql-plugin-api_2.13 ................. SUCCESS [  4.240 s]
[INFO] RAPIDS Accelerator for Apache Spark SQL Plugin ..... FAILURE [ 28.408 s]
[INFO] RAPIDS Accelerator for Apache Spark Shuffle Plugin . SKIPPED

Misc:
Similar to #11821 + #11817 + #11816 ... -- the same exceptions reported against Spark 4.0.0 shim

@NvTimLiu NvTimLiu added ? - Needs Triage Need team to review and classify bug Something isn't working labels Dec 4, 2024
@jlowe jlowe linked a pull request Dec 5, 2024 that will close this issue
@jlowe jlowe added the build Related to CI / CD or cleanly building label Dec 5, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working build Related to CI / CD or cleanly building
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants