Deploy release candidates to local maven repo for dependency check[skip ci] #10201

NvTimLiu · 2024-01-15T15:48:41Z

Deploy release candidates to local maven repo for dependency check

To fix : #10164

Deploy release candidates to local maven repo leveraging the release deploy script.

Instead of using internal snapshots maven repo(having all the intermediate artifacts),
we check dependencies for the release candidates using the local maven repo,
which only contains release candidates.

Signed-off-by: Tim Liu [email protected]

To fix : NVIDIA#10164 Deploy release candidates to local maven repo leveraging the release deploy script. Instead of using internal snapshots maven repo(having all the intermediate artifacts), we check dependencies for the release candidates using the local maven repo, which only contains release candidates. Signed-off-by: Tim Liu <[email protected]>

NvTimLiu · 2024-01-15T15:51:45Z

jenkins/deploy.sh

+CLASSLIST="$CLASSIFIERS,sources,javadoc"
+CLASSLIST=(${CLASSLIST//','/' '})
+for class in ${CLASSLIST[@]}; do
+    echo "$ART_GROUP_ID:$ART_ID:$ART_VER:jar:$class" >> $ARTIFACT_FILE


The artifact file is like below, and our plugin nightly build CI will check all these artifacts dependencies.

com.nvidia:rapids-4-spark-parent_2.12:24.02.0-SNAPSHOT:pom com.nvidia:rapids-4-spark-jdk-profiles_2.12:24.02.0-SNAPSHOT:pom com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:pom com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:cuda11 com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:sources com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:javadoc

NvTimLiu · 2024-01-15T15:53:19Z

jenkins/deploy.sh

+PARENT_ART_ID=$(mvnEval "./" project.artifactId)
+echo "$ART_GROUP_ID:$PARENT_ART_ID:$ART_VER:pom" >> $ARTIFACT_FILE
+
+###### Deploy the jdk-profile pom file ######


Deploy pom.xml file of the module jdk-profiles here, to PASS the release artifact's dependency check

NvTimLiu · 2024-01-15T15:57:17Z

jenkins/spark-nightly-build.sh

@@ -30,7 +30,7 @@ WORKSPACE=${WORKSPACE:-$(pwd)}
 export M2DIR=${M2DIR:-"$WORKSPACE/.m2"}

 ## MVN_OPT : maven options environment, e.g. MVN_OPT='-Dspark-rapids-jni.version=xxx' to specify spark-rapids-jni dependency's version.
-MVN="mvn -Dmaven.wagon.http.retryHandler.count=3 -DretryFailedDeploymentCount=3 ${MVN_OPT}"
+MVN="mvn -Dmaven.wagon.http.retryHandler.count=3 -DretryFailedDeploymentCount=3 ${MVN_OPT} -Psource-javadoc"


sources.jar and javadoc.jar files are required by OSS release

Generate these 2 files to mimic OSS release in our nightly build CI

NvTimLiu · 2024-01-15T15:59:07Z

After this change get merged, we'll enable the release candidates dependency checking in our nightly Build CI job

@gerashegalov @jlowe Can you help review? Thanks!

Signed-off-by: Tim Liu <[email protected]>

NvTimLiu · 2024-01-16T14:40:58Z

jenkins/dependency-check.sh

+
+set -ex
+
+ARTIFACT_FILE=${1:-"/tmp/artifacts-list"}


artifact list file created by deploy.sh, e.g.,

com.nvidia:rapids-4-spark-parent_2.12:24.02.0-SNAPSHOT:pom com.nvidia:rapids-4-spark-jdk-profiles_2.12:24.02.0-SNAPSHOT:pom com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:pom com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:cuda11 com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:sources com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:javadoc

should

com.nvidia:rapids-4-spark-parent_2.12:24.02.0-SNAPSHOT:pom com.nvidia:rapids-4-spark-jdk-profiles_2.12:24.02.0-SNAPSHOT:pom

be in this list?

When we do the test with mvn dependency:get for the documented artifacts such as com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:cuda11 and one of the pom-type artifacts above is missing we should see a failure anyways, correct?

@gerashegalov Yes, you're right, jar's dependency contain the pom files!

My original thought was to check anything we deployed, though pom check would be duplicated.

Let me remove pom from the check list.

Updated, artifact list will be like

com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:cuda11 com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:cuda12 com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:sources com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:javadoc

similar list for scala-2.13 jars, and for arm64 jars

NvTimLiu · 2024-01-18T14:17:49Z

jenkins/deploy.sh

@@ -96,6 +102,14 @@ echo "Deploy CMD: $DEPLOY_CMD"

 ###### Deploy the parent pom file ######
 $DEPLOY_CMD -Dfile=./pom.xml -DpomFile=./pom.xml
+PARENT_ART_ID=$(mvnEval "./" project.artifactId)
+echo "$ART_GROUP_ID:$PARENT_ART_ID:$ART_VER:pom" >> $ARTIFACT_FILE


save artifact like com.nvidia:rapids-4-spark-parent_2.12:24.02.0-SNAPSHOT:pom into $ARTIFACT_FILE

To elaborate on the previous comment, we don't have to register intermediate pom for an explicit check via an $ARTIFACT_FILE line.

gerashegalov

Looks good already, just need to double-check if the test fails if we omit deploy-file of jdk-profiles

gerashegalov · 2024-01-19T23:21:21Z

jenkins/dependency-check.sh

+
+set -ex
+
+ARTIFACT_FILE=${1:-"/tmp/artifacts-list"}


should

com.nvidia:rapids-4-spark-parent_2.12:24.02.0-SNAPSHOT:pom com.nvidia:rapids-4-spark-jdk-profiles_2.12:24.02.0-SNAPSHOT:pom

be in this list?

When we do the test with mvn dependency:get for the documented artifacts such as com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT:jar:cuda11 and one of the pom-type artifacts above is missing we should see a failure anyways, correct?

gerashegalov · 2024-01-19T23:32:33Z

jenkins/deploy.sh

@@ -96,6 +102,14 @@ echo "Deploy CMD: $DEPLOY_CMD"

 ###### Deploy the parent pom file ######
 $DEPLOY_CMD -Dfile=./pom.xml -DpomFile=./pom.xml
+PARENT_ART_ID=$(mvnEval "./" project.artifactId)
+echo "$ART_GROUP_ID:$PARENT_ART_ID:$ART_VER:pom" >> $ARTIFACT_FILE


To elaborate on the previous comment, we don't have to register intermediate pom for an explicit check via an $ARTIFACT_FILE line.

Signed-off-by: Tim Liu <[email protected]>

jenkins/dependency-check.sh

NvTimLiu · 2024-01-22T15:28:11Z

Looks good already, just need to double-check if the test fails if we omit deploy-file of jdk-profiles

I ran the jenkins/dependency-check.sh script in CI the job, it would FAIL unless all the dependency jar/pom are available in the local-remote maven repo

NvTimLiu · 2024-01-22T15:48:04Z

build

gerashegalov

LGTM

Is there a reason not to call dependency-check.sh from spark-nightly-build.sh so we do not have to guess how these two scripts are related? Are you going to add the Jenkins pipeline step for it in this repo?

NvTimLiu · 2024-01-23T01:36:38Z

LGTM

Is there a reason not to call dependency-check.sh from spark-nightly-build.sh so we do not have to guess how these two scripts are related? Are you going to add the Jenkins pipeline step for it in this repo?

Yes, I will add the dependency check into the Jenkins pipeline into another stage(dependency check).

The reason why not calling from spark-nightly-build.sh is that:

spark-nightly-build.sh has script to deploy anything into maven internal repo, if we add this check into nightly build script, there is another deploy call (deploy.sh[only for public artifacts]) plus dependency-check.sh, making nightly build script odd and complexed

NvTimLiu · 2024-01-23T01:38:27Z

build

NvTimLiu · 2024-01-24T02:16:19Z

I'll enable the dependency check in our internal nightly build CI, it will run in tonight's plugin nightly job @gerashegalov @sameerz

gerashegalov · 2024-01-26T01:38:28Z

jenkins/dependency-check.sh

+fi
+while read line; do
+    artifact=$line # artifact=groupId:artifactId:version:[[packaging]:classifier]
+    mvn dependency:get -DremoteRepositories=$remote_maven_repo -Dmaven.repo.local=$M2_CACHE -Dartifact=$artifact


Missed in the original review: we should rm -r $M2_CACHE/com/nvidia before this line or even the whole $M2_CACHE to avoid side-effect of previous dependency:get

Good catch, filed a follow-up PR for it #10278

Follow up of NVIDIA#10201 (comment) Signed-off-by: Tim Liu <[email protected]>

…10278) Follow up of #10201 (comment) Signed-off-by: Tim Liu <[email protected]>

NvTimLiu added feature request New feature or request build Related to CI / CD or cleanly building labels Jan 15, 2024

NvTimLiu requested a review from gerashegalov January 15, 2024 15:48

NvTimLiu self-assigned this Jan 15, 2024

NvTimLiu requested review from jlowe, revans2, tgravescs and GaryShen2008 as code owners January 15, 2024 15:48

NvTimLiu changed the title ~~Deploy release candidates to local maven repo for dependency check~~ Deploy release candidates to local maven repo for dependency check[skip ci] Jan 15, 2024

NvTimLiu commented Jan 15, 2024

View reviewed changes

Add the dependency checking script

4442749

Signed-off-by: Tim Liu <[email protected]>

NvTimLiu commented Jan 16, 2024

View reviewed changes

sameerz removed the feature request New feature or request label Jan 17, 2024

NvTimLiu mentioned this pull request Jan 18, 2024

Add tests that use Spark's --packages to fetch the plugin #10160

Closed

NvTimLiu commented Jan 18, 2024

View reviewed changes

gerashegalov reviewed Jan 19, 2024

View reviewed changes

NvTimLiu added 2 commits January 22, 2024 09:50

Remove pom files from the dependency file list

9ae974f

Signed-off-by: Tim Liu <[email protected]>

Remove another pom in the dependency check list

dd32789

Signed-off-by: Tim Liu <[email protected]>

tgravescs reviewed Jan 22, 2024

View reviewed changes

jenkins/dependency-check.sh Show resolved Hide resolved

description what jenkions/dependency-check.sh does

94f3078

gerashegalov approved these changes Jan 23, 2024

View reviewed changes

GaryShen2008 approved these changes Jan 23, 2024

View reviewed changes

NvTimLiu merged commit e51481e into NVIDIA:branch-24.02 Jan 23, 2024
40 checks passed

gerashegalov reviewed Jan 26, 2024

View reviewed changes

NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this pull request Jan 26, 2024

Clean up $M2_CACHE to avoid side-effect of previous dependency:get

3bc7657

Follow up of NVIDIA#10201 (comment) Signed-off-by: Tim Liu <[email protected]>

NvTimLiu mentioned this pull request Jan 26, 2024

Clean up $M2_CACHE to avoid side-effect of previous dependency:get [skip ci] #10278

Merged

NvTimLiu added a commit that referenced this pull request Jan 27, 2024

Clean up $M2_CACHE to avoid side-effect of previous dependency:get (#…

787a768

…10278) Follow up of #10201 (comment) Signed-off-by: Tim Liu <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploy release candidates to local maven repo for dependency check[skip ci] #10201

Deploy release candidates to local maven repo for dependency check[skip ci] #10201

NvTimLiu commented Jan 15, 2024

NvTimLiu Jan 15, 2024

NvTimLiu Jan 15, 2024

NvTimLiu Jan 15, 2024

NvTimLiu commented Jan 15, 2024 •

edited

Loading

NvTimLiu Jan 16, 2024 •

edited

Loading

gerashegalov Jan 19, 2024

NvTimLiu Jan 22, 2024

NvTimLiu Jan 22, 2024

NvTimLiu Jan 18, 2024

gerashegalov Jan 19, 2024

NvTimLiu Jan 22, 2024

gerashegalov left a comment

gerashegalov Jan 19, 2024

gerashegalov Jan 19, 2024

NvTimLiu commented Jan 22, 2024

NvTimLiu commented Jan 22, 2024

gerashegalov left a comment

NvTimLiu commented Jan 23, 2024

NvTimLiu commented Jan 23, 2024

NvTimLiu commented Jan 24, 2024

gerashegalov Jan 26, 2024

NvTimLiu Jan 26, 2024


		set -ex

		ARTIFACT_FILE=${1:-"/tmp/artifacts-list"}


		set -ex

		ARTIFACT_FILE=${1:-"/tmp/artifacts-list"}

Deploy release candidates to local maven repo for dependency check[skip ci] #10201

Deploy release candidates to local maven repo for dependency check[skip ci] #10201

Conversation

NvTimLiu commented Jan 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NvTimLiu commented Jan 15, 2024 • edited Loading

NvTimLiu Jan 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gerashegalov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NvTimLiu commented Jan 22, 2024

NvTimLiu commented Jan 22, 2024

gerashegalov left a comment

Choose a reason for hiding this comment

NvTimLiu commented Jan 23, 2024

NvTimLiu commented Jan 23, 2024

NvTimLiu commented Jan 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NvTimLiu commented Jan 15, 2024 •

edited

Loading

NvTimLiu Jan 16, 2024 •

edited

Loading