Run '--packages' only with default cuda11 jar #10279
Conversation
As '--packages' only works on the default cuda11 jar, it does not support classifier parameter, refer to issue: https://issues.apache.org/jira/browse/SPARK-20075 We can not specify classifier jar to run plugin tests with '--packages', see below error log: Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Provided Maven Coordinates must be in the form 'groupId:artifactId:version'. The coordinate provided is: com.nvidia:rapids-4-spark_2.12:23.12.0:jar:cuda12 Signed-off-by: Tim Liu <[email protected]>
NvTimLiu
requested review from
jlowe,
revans2,
tgravescs and
GaryShen2008
as code owners
|
build |
| ./run_pyspark_from_build.sh | ||
| # As '--packages' only works on the default cuda11 jar, it does not support classifiers | ||
| # refer to issue : https://issues.apache.org/jira/browse/SPARK-20075 | ||
| if [[ "$CLASSIFIER" == "" || "$CLASSIFIER" == "cuda11" ]]; then |
There was a problem hiding this comment.
Can you explain in a comment why OR-ing with "$CLASSIFIER" == "cuda11" is necessary (if it is)?
As you point out, we know that Spark does not support classifiers in Maven coordinates and it does not look used inside this if branch
There was a problem hiding this comment.
@gerashegalov , good question.
cuda11 jar the the same as main jar, though we do not put cuda11 as part of the --packages param, "CLASSIFIER==cuda11" is the most common case(e.g.https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/jenkins/version-def.sh#L30-L31) to to run integration tests. If only filter it with "CLASSIFIER==''", we may skip packages test in all CI jobs.
There was a problem hiding this comment.
I understand that the default artifact without a classifier is the same as the cuda11-classified artifact.
IIUC,
"$CLASSIFIER" == ''" is for the case when the script is run by a developer. "$CLASSIFIER" == "cuda11" is for cases when run on CI. If so, please add a comment about this. It is strange to see a check for the cuda11 classifier next to the comment saying it's not supported.
There was a problem hiding this comment.
Good suggestion, let me add comment for it.
|
build |
|
@gerashegalov @jlowe can you help review? thanks! |
| PYSP_TEST_spark_jars_packages=com.nvidia:rapids-4-spark_${SCALA_BINARY_VER}:${PROJECT_VER} \ | ||
| PYSP_TEST_spark_jars_repositories=${PROJECT_REPO} \ | ||
| ./run_pyspark_from_build.sh | ||
| if |
As '--packages' only works on the default cuda11 jar, it does not support classifier parameter, refer to issue: https://issues.apache.org/jira/browse/SPARK-20075
We can not specify classifier jar to run plugin tests with '--packages', see below error log:
Exception in thread "main" java.lang.IllegalArgumentException:
requirement failed: Provided Maven Coordinates must be in the form 'groupId:artifactId:version'.
The coordinate provided is: com.nvidia:rapids-4-spark_2.12:23.12.0:jar:cuda12
Follow up #10238 (comment)
Signed-off-by: Tim Liu [email protected]