Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

conform dependency list in 341db to previous versions style [databricks] #11321

Merged
merged 8 commits into from
Aug 19, 2024

Conversation

zpuller
Copy link
Collaborator

@zpuller zpuller commented Aug 13, 2024

Fixes #11300

Uses the pattern established in previous db shims for declaring dependencies in the pom into 341.

Relying on CI for relevant build/tests.

@jlowe jlowe changed the title conform dependency list in 341db to previous versions style conform dependency list in 341db to previous versions style [databricks] Aug 13, 2024
jlowe
jlowe previously approved these changes Aug 13, 2024
@jlowe
Copy link
Contributor

jlowe commented Aug 13, 2024

build

@zpuller
Copy link
Collaborator Author

zpuller commented Aug 14, 2024

build

@zpuller
Copy link
Collaborator Author

zpuller commented Aug 15, 2024

build

@zpuller
Copy link
Collaborator Author

zpuller commented Aug 15, 2024

build

1 similar comment
@zpuller
Copy link
Collaborator Author

zpuller commented Aug 16, 2024

build

jlowe
jlowe previously approved these changes Aug 16, 2024
@jlowe
Copy link
Contributor

jlowe commented Aug 16, 2024

CI failure seems unrelated, rekicking.

@jlowe
Copy link
Contributor

jlowe commented Aug 16, 2024

build

@zpuller
Copy link
Collaborator Author

zpuller commented Aug 16, 2024

CI failure seems unrelated, rekicking.

I think there is a real error actually

mvn -Dmaven.wagon.http.retryHandler.count=3 -B -Ddatabricks -Dbuildver=332db clean package -DskipTests

...

21:12:33  [INFO] Downloading from central: https://repo1.maven.org/maven2/org/apache/maven/maven-parent/6/maven-parent-6.pom
21:12:33  [INFO]      [java] SLF4J: Defaulting to no-operation (NOP) logger implementation
21:12:33  [INFO]      [java] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
21:12:33  [INFO]      [java] Exception in thread "main" java.lang.ExceptionInInitializerError
21:12:33  [INFO]      [java] 	at org.apache.spark.package$.<init>(package.scala:95)
21:12:33  [INFO]      [java] 	at org.apache.spark.package$.<clinit>(package.scala)
21:12:33  [INFO]      [java] 	at com.nvidia.spark.rapids.ShimLoader$.getSparkVersion(ShimLoader.scala:311)
21:12:33  [INFO]      [java] 	at com.nvidia.spark.rapids.ShimLoader$.detectShimProvider(ShimLoader.scala:205)
21:12:33  [INFO]      [java] 	at com.nvidia.spark.rapids.ShimLoader$.findShimProvider(ShimLoader.scala:299)
21:12:33  [INFO]      [java] 	at com.nvidia.spark.rapids.ShimLoader$.initShimProviderIfNeeded(ShimLoader.scala:92)
21:12:33  [INFO]      [java] 	at com.nvidia.spark.rapids.ShimLoader$.getShimClassLoader(ShimLoader.scala:184)
21:12:33  [INFO]      [java] 	at com.nvidia.spark.rapids.ShimReflectionUtils$.loadClass(ShimReflectionUtils.scala:28)
21:12:33  [INFO]      [java] 	at com.nvidia.spark.rapids.tests.DumpDefaultConfigs$.main(DumpDefaultConfigs.scala:32)
21:12:33  [INFO]      [java] 	at com.nvidia.spark.rapids.tests.DumpDefaultConfigs.main(DumpDefaultConfigs.scala)
21:12:33  [INFO]      [java] Caused by: org.apache.spark.SparkException: Could not find spark-version-info.properties
21:12:33  [INFO]      [java] 	at org.apache.spark.package$SparkBuildInfo$.<init>(package.scala:63)
21:12:33  [INFO]      [java] 	at org.apache.spark.package$SparkBuildInfo$.<clinit>(package.scala)
21:12:33  [INFO]      [java] 	... 10 more
21:12:33  [INFO] ------------------------------------------------------------------------
21:12:33  [INFO] Reactor Summary for RAPIDS Accelerator for Apache Spark Root Project 24.10.0-SNAPSHOT:
21:12:33  [INFO] 

...

21:12:33  [INFO] rapids-4-spark-integration-tests_2.12 .............. FAILURE [ 12.526 s]

@zpuller
Copy link
Collaborator Author

zpuller commented Aug 16, 2024

build

@zpuller
Copy link
Collaborator Author

zpuller commented Aug 17, 2024

build

@zpuller zpuller merged commit b23e3fa into NVIDIA:branch-24.10 Aug 19, 2024
42 of 43 checks passed
@zpuller zpuller deleted the db_shim_deps branch August 19, 2024 16:31
@sameerz sameerz added the build Related to CI / CD or cleanly building label Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Related to CI / CD or cleanly building
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use rapids-4-spark-db-bom aggregate dependency in spark341db delta-lake
4 participants