spark-yarn-yarnrmclient.adoc

YarnRMClient

YarnRMClient is responsible for registering and unregistering a Spark application (in the form of ApplicationMaster) with YARN ResourceManager (and hence RM in the name). It is a mere wrapper for AMRMClient[ContainerRequest] that is started when registering ApplicationMaster (and never stopped explicitly!).

Besides being responsible for registration and unregistration, it also knows the application attempt identifiers and tracks the maximum number of attempts to register ApplicationMaster.

Tip	Enable INFO logging level for org.apache.spark.deploy.yarn.YarnRMClient logger to see what happens inside. Add the following line to conf/log4j.properties: log4j.logger.org.apache.spark.deploy.yarn.YarnRMClient=INFO Refer to Logging.

Registering ApplicationMaster with YARN ResourceManager (register method)

To register ApplicationMaster (for a Spark application) with the YARN ResourceManager, Spark uses register.

register(
  driverUrl: String,
  driverRef: RpcEndpointRef,
  conf: YarnConfiguration,
  sparkConf: SparkConf,
  uiAddress: String,
  uiHistoryAddress: String,
  securityMgr: SecurityManager,
  localResources: Map[String, LocalResource]): YarnAllocator

register instantiates YARN’s AMRMClient, initializes it (using conf input parameter) and starts immediately. It saves uiHistoryAddress input parameter internally for later use.

Figure 1. Registering ApplicationMaster with YARN ResourceManager

You should see the following INFO message in the logs (in stderr in YARN):

INFO YarnRMClient: Registering the ApplicationMaster

It then registers the application master on the local host with port 0 and uiAddress input parameter for the URL at which the master info can be seen.

The internal registered flag is enabled.

Ultimately, it creates a new YarnAllocator with the input parameters of register passed in and the just-created YARN AMRMClient.

Unregistering ApplicationMaster from YARN ResourceManager (unregister method)

unregister(status: FinalApplicationStatus, diagnostics: String = ""): Unit

unregister unregisters the ApplicationMaster of a Spark application.

It basically checks that ApplicationMaster is registered and only when it is requests the internal AMRMClient to unregister.

unregister is called when ApplicationMaster wants to unregister.

Maximum Number of Attempts to Register ApplicationMaster (getMaxRegAttempts method)

getMaxRegAttempts(sparkConf: SparkConf, yarnConf: YarnConfiguration): Int

getMaxRegAttempts uses SparkConf and YARN’s YarnConfiguration to read configuration settings and return the maximum number of application attempts before ApplicationMaster registration with YARN is considered unsuccessful (and so the Spark application).

It reads YARN’s yarn.resourcemanager.am.max-attempts (available as YarnConfiguration.RM_AM_MAX_ATTEMPTS) or falls back to YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS (which is 2).

The return value is the minimum of the configuration settings of YARN and Spark.

Getting ApplicationAttemptId of Spark Application (getAttemptId method)

getAttemptId(): ApplicationAttemptId

getAttemptId returns YARN’s ApplicationAttemptId (of the Spark application to which the container was assigned).

Internally, it uses YARN-specific methods like ConverterUtils.toContainerId and ContainerId.getApplicationAttemptId.

getAmIpFilterParams

Caution

FIXME