Skip to content

Latest commit

 

History

History
25 lines (13 loc) · 1.78 KB

spark-class.adoc

File metadata and controls

25 lines (13 loc) · 1.78 KB
Raw

spark-class script

bin/spark-class shell script is the script launcher for internal Spark classes.

Note
 Ultimately, any shell script in Spark calls spark-class script.

When started, it loads $SPARK_HOME/bin/load-spark-env.sh, searches for the Spark assembly jar, and starts org.apache.spark.launcher.Main.

Tip
 spark-class script loads additional environment settings (see spark-env.sh section in this document). And then spark-class searches for so-called the Spark assembly jar ( spark-assembly.hadoop..jar) in SPARK_HOME/lib or SPARK_HOME/assembly/target/scala-$SPARK_SCALA_VERSION for a binary distribution or Spark built from sources, respectively.
Note
 Set SPARK_PREPEND_CLASSES to have the Spark launcher classes (from $SPARK_HOME/launcher/target/scala-$SPARK_SCALA_VERSION/classes) to appear before the Spark assembly jar. It’s useful for development so your changes don’t require rebuilding Spark from the beginning.

As the last step in the process, org.apache.spark.launcher.Main class is executed. The Main class programmatically computes the final command to be executed.

org.apache.spark.launcher.Main

org.apache.spark.launcher.Main is the command-line launcher used in Spark scripts, like spark-class.

It uses SPARK_PRINT_LAUNCH_COMMAND to print launch command to standard output.

It builds the command line for a Spark class using the environment variables:

  • SPARK_DAEMON_JAVA_OPTS and SPARK_MASTER_OPTS to be added to the command line of the command.

  • SPARK_DAEMON_MEMORY (default: 1g) for -Xms and -Xmx.