[BUG] scala:compile is not thread safe unless compiler bridge already exists

[gerashegalov]

Describe the bug
CI executing jenkins/spark-nightly-build.sh failed with

[ERROR] ## Exception when compiling 4 sources to /home/jenkins/agent/workspace/jenkins-rapids_scala213_nightly-dev-github-53/scala2.13/datagen/target/classes
java.lang.ClassNotFoundException: xsbt.CompilerInterface

This is due to unexpected concurrency recently introduced by #9888 in

spark-rapids/jenkins/spark-nightly-build.sh

Lines 107 to 108 in a9c1d68

	# if skip test, we could try speed up build with multiple-threads
	MVN="${MVN} -T1C"

Clobbering of the compiler bridge jar built on demand has been a known issue but slipped through the cracks. Previously it has been occurring only in the multi-shim builds, and we deal with this by de-conflicting the path used by each shim build flow.

We have no mechanism for this in the single shim flow. We can look into manual tricks or see if the recent scala-maven-plugin versions have a fix.

Until recently we could not have scala:compile invocation concurrency inside a single shim flow because of the dependency graph regardless of the -T1C parameter because we had a single module/reactor with Scala (like sql-plugin-api) that had to be built before every other submodule. Now we have independent datagen and sql-plugin-api that will be kicked off simultaneously in a multi-threaded build.

[gerashegalov]

On the CI side, compilation is always (? as far as I know) done from scratch. Thus, we could/should get rid of the incremental recompileMode requiring rebuild of that said compiler bridge artifact. However, the recompileMode=all is failing with:

15:44:24,187 [ERROR] /home/gshegalov/gits/NVIDIA/spark-rapids.worktrees/wt1/sql-plugin/src/main/scala/com/nvidia/spark/rapids/JoinGatherer.scala:20: error: object Retryable is not a member of package com.nvidia.spark
15:44:24,188 [ERROR] import com.nvidia.spark.Retryable
15:44:24,188 [ERROR]        ^
15:44:24,188 [ERROR] /home/gshegalov/gits/NVIDIA/spark-rapids.worktrees/wt1/sql-plugin/src/main/scala/com/nvidia/spark/rapids/JoinGatherer.scala:39: error: not found: type Retryable
15:44:24,188 [ERROR] trait LazySpillable extends AutoCloseable with Retryable {

seemingly not taking into account compileOrder=Mixed . Given the search result https://github.com/search?q=repo%3AdavidB/scala-maven-plugin%20CompileOrder&type=code it applies only to the incremental recompileMode.