spark-yarn-YarnSparkHadoopUtil.adoc

YarnSparkHadoopUtil

YarnSparkHadoopUtil is…​FIXME

It can only be created when SPARK_YARN_MODE flag is enabled.

Note	It belongs to org.apache.spark.deploy.yarn package.

Tip	Enable DEBUG logging level for org.apache.spark.deploy.yarn.YarnSparkHadoopUtil logger to see what happens inside. Add the following line to conf/log4j.properties: log4j.logger.org.apache.spark.deploy.yarn.YarnSparkHadoopUtil=DEBUG Refer to Logging.

MEMORY_OVERHEAD_FACTOR

MEMORY_OVERHEAD_FACTOR is a constant that equals to 10% for memory overhead.

MEMORY_OVERHEAD_MIN

MEMORY_OVERHEAD_MIN is a constant that equals to 384L for memory overhead.

getApplicationAclsForYarn

Caution

FIXME

Resolving Environment Variable (expandEnvironment method)

expandEnvironment(environment: Environment): String

expandEnvironment resolves environment variable using YARN’s Environment.$ or Environment.$$ methods (depending on the version of Hadoop used).

Getting YarnSparkHadoopUtil Instance (get method)

Caution

FIXME

Computing YARN’s ContainerId (getContainerId method)

getContainerId: ContainerId

getContainerId is a private[spark] method that gets YARN’s ContainerId from the YARN environment variable ApplicationConstants.Environment.CONTAINER_ID and converts it to the return object using YARN’s ConverterUtils.toContainerId.

startExecutorDelegationTokenRenewer

Caution

FIXME

stopExecutorDelegationTokenRenewer

Caution

FIXME

Calculating Initial Number of Executors (getInitialTargetExecutorNumber method)

getInitialTargetExecutorNumber(
  conf: SparkConf,
  numExecutors: Int = DEFAULT_NUMBER_EXECUTORS): Int

getInitialTargetExecutorNumber calculates the initial number of executors for Spark on YARN. It varies by whether dynamic allocation is enabled or not.

Note	The default number of executors (aka DEFAULT_NUMBER_EXECUTORS) is 2.

If dynamic allocation is enabled, getInitialTargetExecutorNumber returns the value of spark.dynamicAllocation.initialExecutors or spark.dynamicAllocation.minExecutors or 0.

If however dynamic allocation is disabled, getInitialTargetExecutorNumber returns the value of spark.executor.instances setting or SPARK_EXECUTOR_INSTANCES environment variable, or the default value (of the input parameter numExecutors) 2.

Note	It is used to calculate totalExpectedExecutors to start Spark on YARN in client mode or cluster mode.

addPathToEnvironment

addPathToEnvironment(env: HashMap[String, String], key: String, value: String): Unit

Caution

FIXME