spark-yarn-YarnClusterManager.adoc

YarnClusterManager — ExternalClusterManager for YARN

YarnClusterManager is the only currently known ExternalClusterManager in Spark. It creates a TaskScheduler and a SchedulerBackend for YARN.

canCreate method

YarnClusterManager can handle the yarn master URL only.

createTaskScheduler method

createTaskScheduler creates a YarnClusterScheduler for cluster deploy mode and a YarnScheduler for client deploy mode.

It throws a SparkException for unknown deploy modes.

Unknown deploy mode '[deployMode]' for Yarn

createSchedulerBackend method

createSchedulerBackend creates a YarnClusterSchedulerBackend for cluster deploy mode and a YarnClientSchedulerBackend for client deploy mode.

It throws a SparkException for unknown deploy modes.

Unknown deploy mode '[deployMode]' for Yarn

initialize method

initialize simply initializes the input TaskSchedulerImpl.