spark-sql-streaming-StreamingAggregationStateManagerBaseImpl.adoc

StreamingAggregationStateManagerBaseImpl — Base State Manager for Streaming Aggregation

StreamingAggregationStateManagerBaseImpl is the base implementation of the StreamingAggregationStateManager contract for state managers for streaming aggregations that use UnsafeProjection to getKey.

StreamingAggregationStateManagerBaseImpl uses UnsafeProjection to getKey.

Table 1. StreamingAggregationStateManagerBaseImpls

StreamingAggregationStateManagerBaseImpl	Description
StreamingAggregationStateManagerImplV1	Legacy StreamingAggregationStateManager (used when spark.sql.streaming.aggregation.stateFormatVersion configuration property is 1)
StreamingAggregationStateManagerImplV2	Default StreamingAggregationStateManager (used when spark.sql.streaming.aggregation.stateFormatVersion configuration property is 2)

StreamingAggregationStateManagerBaseImpl takes the following to be created:

• Catalyst expressions for the keys (Seq[Attribute])

• Catalyst expressions for the input rows (Seq[Attribute])

Note	StreamingAggregationStateManagerBaseImpl is a Scala abstract class and cannot be created directly. It is created indirectly for the concrete StreamingAggregationStateManagerBaseImpls.

Committing (Changes to) State Store — commit Method

commit(store: StateStore): Long

Note	commit is part of the StreamingAggregationStateManager Contract to commit changes to a state store.

commit simply requests the state store to commit state changes.

Removing Key From State Store — remove Method

remove(store: StateStore, key: UnsafeRow): Unit

Note	remove is part of the StreamingAggregationStateManager Contract to remove a key from a state store.

remove…​FIXME

getKey Method

getKey(row: UnsafeRow): UnsafeRow

Note	getKey is part of the StreamingAggregationStateManager Contract to…​FIXME

getKey…​FIXME

Getting All Keys in State Store — keys Method

keys(store: StateStore): Iterator[UnsafeRow]

Note	keys is part of the StreamingAggregationStateManager Contract to get all keys in a state store (as an iterator).

keys…​FIXME