spark-sql-streaming-MetadataLog.adoc

MetadataLog Contract — Metadata Storage

MetadataLog is the abstraction of store metadata that can persist, retrieve, and remove metadata (of type T).

Table 1. MetadataLog Contract

Method	Description
add	add( batchId: Long, metadata: T): Boolean Adds metadata for a batch
get	get(batchId: Long): Option[T] get( startId: Option[Long], endId: Option[Long]): Array[(Long, T)] Retrieves metadata by batch id or batch range Used when…​FIXME
getLatest	getLatest(): Option[(Long, T)] Retrieves the latest-committed metadata (if available) Used when…​FIXME
purge	purge(thresholdBatchId: Long): Unit Used when…​FIXME

Note	HDFSMetadataLog is the direct implementation of the MetadataLog Contract in Spark Structured Streaming.