spark-sql-streaming-StateStoreAwareZipPartitionsRDD.adoc

StateStoreAwareZipPartitionsRDD

StateStoreAwareZipPartitionsRDD is a ZippedPartitionsRDD2 with the left and right parent RDDs.

StateStoreAwareZipPartitionsRDD is created exclusively when StreamingSymmetricHashJoinExec physical operator is requested to execute (and requests StateStoreAwareZipPartitionsHelper for one).

Creating StateStoreAwareZipPartitionsRDD Instance

StateStoreAwareZipPartitionsRDD takes the following to be created:

• SparkContext

• Function ((Iterator[A], Iterator[B]) ⇒ Iterator[V])

• Left RDD - the RDD of the left side of a join (RDD[A])

• Right RDD - the RDD of the right side of a join (RDD[B])

• StatefulOperatorStateInfo

• Names of the state stores

• StateStoreCoordinatorRef

Placement Preferences of Partition — getPreferredLocations Method

getPreferredLocations(partition: Partition): Seq[String]

Note	getPreferredLocations is a part of the RDD Contract to specify placement preferences (aka preferred task locations), i.e. where tasks should be executed to be as close to the data as possible.

getPreferredLocations…​FIXME