spark-sql-streaming-StreamingQueryProgress.adoc

StreamingQueryProgress

StreamingQueryProgress holds information about the progress of a streaming query.

StreamingQueryProgress is created exclusively when StreamExecution finishes a trigger.

Note	Use lastProgress property of a StreamingQuery to access the most recent StreamingQueryProgress update. val sq: StreamingQuery = ... sq.lastProgress

Note	Use recentProgress property of a StreamingQuery to access the most recent StreamingQueryProgress updates. val sq: StreamingQuery = ... sq.recentProgress

Note	Use StreamingQueryListener to get notified about StreamingQueryProgress updates while a streaming query is executed.

Table 1. StreamingQueryProgress’s Properties

Name	Description
id	Unique identifier of a streaming query
runId	Unique identifier of the current execution of a streaming query
name	Optional query name
timestamp	Time when the trigger has started (in ISO8601 format).
batchId	Unique id of the current batch
durationMs	Durations of the internal phases (in milliseconds)
eventTime	Statistics of event time seen in this batch
stateOperators	Information about stateful operators in the query that store state.
sources	Statistics about the data read from every streaming source in a streaming query
sink	Information about progress made for a sink