spark-sql-streaming-SQLConf.adoc

SQLConf — Internal Configuration Store

SQLConf is an internal key-value configuration store for parameters and hints used to configure a Spark Structured Streaming application (and Spark SQL applications in general).

The parameters and hints are accessible as property accessor methods.

SQLConf is available as the conf property of the SessionState of a SparkSession.

scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.sessionState.conf
org.apache.spark.sql.internal.SQLConf

Table 1. SQLConf’s Property Accessor Methods

Method Name / Property	Description
continuousStreamingExecutorQueueSize spark.sql.streaming.continuous.executorQueueSize	Used when: DataSourceV2ScanExec leaf physical operator is requested for the input RDDs (and creates a ContinuousDataSourceRDD) ContinuousCoalesceExec unary physical operator is requested to execute
continuousStreamingExecutorPollIntervalMs spark.sql.streaming.continuous.executorPollIntervalMs	Used exclusively when DataSourceV2ScanExec leaf physical operator is requested for the input RDDs (and creates a ContinuousDataSourceRDD)
disabledV2StreamingMicroBatchReaders spark.sql.streaming.disabledV2MicroBatchReaders	Used exclusively when MicroBatchExecution is requested for the analyzed logical plan (of a streaming query)
FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion
minBatchesToRetain spark.sql.streaming.minBatchesToRetain	Used when: CompactibleFileStreamLog is created StreamExecution is created StateStoreConf is created
SHUFFLE_PARTITIONS spark.sql.shuffle.partitions	See spark.sql.shuffle.partitions in The Internals of Spark SQL.
stateStoreProviderClass spark.sql.streaming.stateStore.providerClass	Used when: StateStoreWriter is requested to stateStoreCustomMetrics (when StateStoreWriter is requested for the metrics and getProgress) StateStoreConf is created
STREAMING_AGGREGATION_STATE_FORMAT_VERSION spark.sql.streaming.aggregation.stateFormatVersion	Used when: StatefulAggregationStrategy execution planning strategy is executed OffsetSeqMetadata is requested for the relevantSQLConfs and the relevantSQLConfDefaultValues
streamingMetricsEnabled spark.sql.streaming.metricsEnabled	Used exclusively when StreamExecution is requested for runStream (to control whether to register a metrics reporter for a streaming query)
STREAMING_MULTIPLE_WATERMARK_POLICY spark.sql.streaming.multipleWatermarkPolicy
streamingNoDataProgressEventInterval spark.sql.streaming.noDataProgressEventInterval	Used exclusively for ProgressReporter