spark-yarn-ClientDistributedCacheManager.adoc

ClientDistributedCacheManager

ClientDistributedCacheManager is a mere wrapper to hold the collection of cache-related resource entries CacheEntry (as distCacheEntries) to add resources to and later update Spark configuration with files to distribute.

Caution

FIXME What is a resource? Is this a file only?

Adding Cache-Related Resource (addResource method)

addResource(
  fs: FileSystem,
  conf: Configuration,
  destPath: Path,
  localResources: HashMap[String, LocalResource],
  resourceType: LocalResourceType,
  link: String,
  statCache: Map[URI, FileStatus],
  appMasterOnly: Boolean = false): Unit

Updating Spark Configuration with Resources to Distribute (updateConfiguration method)

updateConfiguration(conf: SparkConf): Unit

updateConfiguration sets the following internal Spark configuration settings in the input conf Spark configuration:

• spark.yarn.cache.filenames

• spark.yarn.cache.sizes

• spark.yarn.cache.timestamps

• spark.yarn.cache.visibilities

• spark.yarn.cache.types

It uses the internal distCacheEntries with resources to distribute.

Note	It is later used in ApplicationMaster when it prepares local resources.