Bloop updates - require JDK11 in buildall + docs, build bloop for all…

… targets. (NVIDIA#9615) * Bloop updates - require JDK11 in buildall + docs, build bloop for all targets. Signed-off-by: Phil Dakin <[email protected]> * Address PR comments. Signed-off-by: Phil Dakin <[email protected]> * Add a bloopInstall profile (#1) * Add bloopInstall profile Signed-off-by: Gera Shegalov <[email protected]> * Add bloopInstall profile Fixes NVIDIA#9621 Signed-off-by: Gera Shegalov <[email protected]> * add comment for install in scala2.13 Signed-off-by: Gera Shegalov <[email protected]> * simplify Signed-off-by: Gera Shegalov <[email protected]> * undo unrelated Signed-off-by: Gera Shegalov <[email protected]> * Apply suggestions from code review Co-authored-by: PhilDakin <[email protected]> * review 1 Signed-off-by: Gera Shegalov <[email protected]> * Configurable bloopInstall phase Signed-off-by: Gera Shegalov <[email protected]> * move bloop config dir to execution conf Signed-off-by: Gera Shegalov <[email protected]> --------- Signed-off-by: Gera Shegalov <[email protected]> Co-authored-by: PhilDakin <[email protected]> * Propagate bloopInstall profile changes, make config directory overridable from command line. Signed-off-by: Phil Dakin <[email protected]> * Cleanup. Signed-off-by: Phil Dakin <[email protected]> --------- Signed-off-by: Phil Dakin <[email protected]> Signed-off-by: Gera Shegalov <[email protected]> Co-authored-by: Gera Shegalov <[email protected]>
YanxuanLiu · Nov 7, 2023 · d25f1fa · d25f1fa
1 parent d6d9d5a
commit d25f1fa
Show file tree

Hide file tree

Showing 4 changed files with 156 additions and 26 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -353,13 +353,10 @@ interested in. For example, to generate the Bloop projects for the Spark 3.2.0 d
 just for the production code run:
 
 ```shell script
-mvn install ch.epfl.scala:bloop-maven-plugin:bloopInstall -pl aggregator -am \
-  -DdownloadSources=true \
-  -Dbuildver=320 \
-  -DskipTests \
-  -Dskip \
-  -Dmaven.scalastyle.skip=true \
-  -Dmaven.updateconfig.skip=true
+mvn -B clean install \
+    -DbloopInstall \
+    -DdownloadSources=true \
+    -Dbuildver=320
 ```
 
 With `--generate-bloop` we integrated Bloop project generation into `buildall`. It makes it easier
@@ -384,6 +381,11 @@ You can now open the spark-rapids as a
 
 Read on for VS Code Scala Metals instructions.
 
+##### JDK 11 Requirement
+
+It is known that Bloop's SemanticDB generation with JDK 8 is broken for spark-rapids. Please use JDK
+11 or later for Bloop builds.
+
 #### Bloop, Scala Metals, and Visual Studio Code
 
 _Last tested with 1.63.0-insider (Universal) Commit: bedf867b5b02c1c800fbaf4d6ce09cefba_
@@ -431,18 +433,11 @@ jps -l
 ###### java.lang.RuntimeException: boom
 
 Metals background compilation process status appears to be resetting to 0% after reaching 99%
-and you see a peculiar error message [`java.lang.RuntimeException: boom`][1]. You can work around
-it by making sure Metals Server (Bloop client) and Bloop Server are both running on Java 11+.
-
-1. To this end make sure that Bloop projects are generated using Java 11+
-
-    ```bash
-    JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 \
-      mvn install ch.epfl.scala:bloop-maven-plugin:bloopInstall \
-      -DdownloadSources=true \
-      -Dbuildver=331 \
-      -Dskip -DskipTests
-    ```
+and you see a peculiar error message [`java.lang.RuntimeException: boom`][1]. This is a known issue
+when running Metals/Bloop on Java 8. To work around it, ensure Metals and Bloop are both running on
+Java 11+.
+
+1. The `-DbloopInstall` profile will enforce Java 11+ compliance.
 
 1. Add [`metals.javaHome`][2] to VSCode preferences to point to Java 11+.
 

diff --git a/build/buildall b/build/buildall
@@ -65,15 +65,10 @@ function bloopInstall() {
       bloopTmpConfigDir="$bloopTmpDir/.bloop$bv"
       mkdir -p $bloopTmpConfigDir
       $MVN -B clean install \
-        ch.epfl.scala:bloop-maven-plugin:bloopInstall \
-        -pl aggregator -am \
+        -DbloopInstall \
         -Dbloop.configDirectory="$bloopTmpConfigDir" \
         -DdownloadSources=true \
-        -Dbuildver="$bv" \
-        -DskipTests \
-        -Dskip \
-        -Dmaven.scalastyle.skip=true \
-        -Dmaven.updateconfig.skip=true
+        -Dbuildver="$bv"
 
       specifier="spark$bv"
       bloopDir=$PWD/.bloop-$specifier

diff --git a/pom.xml b/pom.xml
@@ -569,6 +569,57 @@
                 <jni.classifier>${cuda.version}-arm64</jni.classifier>
             </properties>
         </profile>
+        <profile>
+            <id>bloopInstall</id>
+            <activation>
+                <property>
+                    <name>bloopInstall</name>
+                    <value>true</value>
+                </property>
+            </activation>
+            <properties>
+                <skipTests>true</skipTests>
+                <maven.scaladoc.skip>true</maven.scaladoc.skip>
+                <maven.scalastyle.skip>true</maven.scalastyle.skip>
+            </properties>
+            <build>
+                <plugins>
+                    <plugin>
+                        <groupId>org.apache.maven.plugins</groupId>
+                        <artifactId>maven-enforcer-plugin</artifactId>
+                        <executions>
+                            <execution>
+                                <id>enforce-bloop-rules</id>
+                                <goals><goal>enforce</goal></goals>
+                                <configuration>
+                                    <rules>
+                                        <requireJavaVersion>
+                                            <message>Metals semantic database requires JAVA_HOME pointing to JDK 11+, actual Java version: ${java.version}</message>
+                                            <version>[11,)</version>
+                                        </requireJavaVersion>
+                                    </rules>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
+                    <plugin>
+                        <groupId>ch.epfl.scala</groupId>
+                        <artifactId>bloop-maven-plugin</artifactId>
+                        <executions>
+                            <execution>
+                                <id>generate-bloop-projects</id>
+                                <goals><goal>bloopInstall</goal></goals>
+                                <phase>${bloop.installPhase}</phase>
+                                <configuration>
+                                    <!-- Metals looks at the repo root. so to accomodate scala 2.13  define -->
+                                    <bloopConfigDir>${bloop.configDirectory}</bloopConfigDir>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
+                </plugins>
+            </build>
+        </profile>
     </profiles>
 
     <properties>
@@ -655,6 +706,7 @@
         <scala.local-lib.path>org/scala-lang/scala-library/${scala.version}/scala-library-${scala.version}.jar</scala.local-lib.path>
         <target.classifier>${spark.version.classifier}</target.classifier>
         <maven.clean.plugin.version>3.1.0</maven.clean.plugin.version>
+        <maven.scaladoc.skip>false</maven.scaladoc.skip>
         <maven.scalastyle.skip>false</maven.scalastyle.skip>
         <dist.jar.compress>true</dist.jar.compress>
         <spark330.iceberg.version>0.14.1</spark330.iceberg.version>
@@ -765,6 +817,8 @@
             -Djdk.reflect.useDirectMethodHandle=false
         </extraJavaTestArgs>
         <cloudera.repo.enabled>false</cloudera.repo.enabled>
+        <bloop.installPhase>install</bloop.installPhase>
+        <bloop.configDirectory>${spark.rapids.source.basedir}/.bloop</bloop.configDirectory>
     </properties>
 
     <dependencyManagement>
@@ -1043,6 +1097,7 @@
                                     <arg>-doc-external-doc:${settings.localRepository}/${scala.local-lib.path}#https://scala-lang.org/api/${scala.version}/</arg>
                                     <arg>-doc-external-doc:${settings.localRepository}/org/apache/spark/spark-sql_${scala.binary.version}/${spark.version}/spark-sql_${scala.binary.version}-${spark.version}.jar#https://spark.apache.org/docs/${spark.version}/api/scala/index.html</arg>
                                 </args>
+                                <skip>${maven.scaladoc.skip}</skip>
                             </configuration>
                         </execution>
                     </executions>
@@ -1212,6 +1267,21 @@
         </pluginManagement>
 
         <plugins>
+            <plugin>
+                <groupId>ch.epfl.scala</groupId>
+                <artifactId>bloop-maven-plugin</artifactId>
+                <version>2.0.0</version>
+                <executions>
+                    <execution>
+                        <id>default-cli</id>
+                        <configuration>
+                            <skip>true</skip>
+                            <!-- workaround: skip is not skipping -->
+                            <bloopConfigDir>/dev/null/ERROR: Do not specify the bloop-maven-plugin on the command line. Instead invoke `mvn install -DbloopInstall ...`</bloopConfigDir>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-enforcer-plugin</artifactId>

diff --git a/scala2.13/pom.xml b/scala2.13/pom.xml
@@ -569,6 +569,57 @@
                 <jni.classifier>${cuda.version}-arm64</jni.classifier>
             </properties>
         </profile>
+        <profile>
+            <id>bloopInstall</id>
+            <activation>
+                <property>
+                    <name>bloopInstall</name>
+                    <value>true</value>
+                </property>
+            </activation>
+            <properties>
+                <skipTests>true</skipTests>
+                <maven.scaladoc.skip>true</maven.scaladoc.skip>
+                <maven.scalastyle.skip>true</maven.scalastyle.skip>
+            </properties>
+            <build>
+                <plugins>
+                    <plugin>
+                        <groupId>org.apache.maven.plugins</groupId>
+                        <artifactId>maven-enforcer-plugin</artifactId>
+                        <executions>
+                            <execution>
+                                <id>enforce-bloop-rules</id>
+                                <goals><goal>enforce</goal></goals>
+                                <configuration>
+                                    <rules>
+                                        <requireJavaVersion>
+                                            <message>Metals semantic database requires JAVA_HOME pointing to JDK 11+, actual Java version: ${java.version}</message>
+                                            <version>[11,)</version>
+                                        </requireJavaVersion>
+                                    </rules>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
+                    <plugin>
+                        <groupId>ch.epfl.scala</groupId>
+                        <artifactId>bloop-maven-plugin</artifactId>
+                        <executions>
+                            <execution>
+                                <id>generate-bloop-projects</id>
+                                <goals><goal>bloopInstall</goal></goals>
+                                <phase>${bloop.installPhase}</phase>
+                                <configuration>
+                                    <!-- Metals looks at the repo root. so to accomodate scala 2.13  define -->
+                                    <bloopConfigDir>${bloop.configDirectory}</bloopConfigDir>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
+                </plugins>
+            </build>
+        </profile>
     </profiles>
 
     <properties>
@@ -655,6 +706,7 @@
         <scala.local-lib.path>org/scala-lang/scala-library/${scala.version}/scala-library-${scala.version}.jar</scala.local-lib.path>
         <target.classifier>${spark.version.classifier}</target.classifier>
         <maven.clean.plugin.version>3.1.0</maven.clean.plugin.version>
+        <maven.scaladoc.skip>false</maven.scaladoc.skip>
         <maven.scalastyle.skip>false</maven.scalastyle.skip>
         <dist.jar.compress>true</dist.jar.compress>
         <spark330.iceberg.version>0.14.1</spark330.iceberg.version>
@@ -765,6 +817,8 @@
             -Djdk.reflect.useDirectMethodHandle=false
         </extraJavaTestArgs>
         <cloudera.repo.enabled>false</cloudera.repo.enabled>
+        <bloop.installPhase>install</bloop.installPhase>
+        <bloop.configDirectory>${spark.rapids.source.basedir}/.bloop</bloop.configDirectory>
     </properties>
 
     <dependencyManagement>
@@ -1043,6 +1097,7 @@
                                     <arg>-doc-external-doc:${settings.localRepository}/${scala.local-lib.path}#https://scala-lang.org/api/${scala.version}/</arg>
                                     <arg>-doc-external-doc:${settings.localRepository}/org/apache/spark/spark-sql_${scala.binary.version}/${spark.version}/spark-sql_${scala.binary.version}-${spark.version}.jar#https://spark.apache.org/docs/${spark.version}/api/scala/index.html</arg>
                                 </args>
+                                <skip>${maven.scaladoc.skip}</skip>
                             </configuration>
                         </execution>
                     </executions>
@@ -1212,6 +1267,21 @@
         </pluginManagement>
 
         <plugins>
+            <plugin>
+                <groupId>ch.epfl.scala</groupId>
+                <artifactId>bloop-maven-plugin</artifactId>
+                <version>2.0.0</version>
+                <executions>
+                    <execution>
+                        <id>default-cli</id>
+                        <configuration>
+                            <skip>true</skip>
+                            <!-- workaround: skip is not skipping -->
+                            <bloopConfigDir>/dev/null/ERROR: Do not specify the bloop-maven-plugin on the command line. Instead invoke `mvn install -DbloopInstall ...`</bloopConfigDir>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-enforcer-plugin</artifactId>