spark-sql-streaming-OffsetSeqMetadata.adoc

OffsetSeqMetadata

OffsetSeqMetadata holds the metadata for the current values of the following:

• Event-time watermark (in millis) that is used to bound the lateness of data that will processed

• Current timestamp (in millis)

• Streaming configuration with spark.sql.shuffle.partitions and spark.sql.streaming.stateStore.providerClass Spark properties

Note	OffsetSeqMetadata is used mainly when IncrementalExecution is created.

OffsetSeqMetadata considers some configuration properties as relevantSQLConfs:

• SHUFFLE_PARTITIONS

• STATE_STORE_PROVIDER_CLASS

• STREAMING_MULTIPLE_WATERMARK_POLICY

• FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION

• STREAMING_AGGREGATION_STATE_FORMAT_VERSION

relevantSQLConfs are used when OffsetSeqMetadata is created and is requested to setSessionConf.

Creating OffsetSeqMetadata — apply Factory Method

apply(
  batchWatermarkMs: Long,
  batchTimestampMs: Long,
  sessionConf: RuntimeConfig): OffsetSeqMetadata

apply…​FIXME

Note	apply is used when…​FIXME

setSessionConf Method

setSessionConf(metadata: OffsetSeqMetadata, sessionConf: RuntimeConfig): Unit

setSessionConf…​FIXME

Note	setSessionConf is used when…​FIXME