SUMMARY.adoc

Summary

1. Introduction

2. Spark Structured Streaming — Streaming Datasets

3. Internals of Streaming Datasets

Streaming Join

1. Streaming Join

2. StateStoreAwareZipPartitionsRDD

3. SymmetricHashJoinStateManager

   1. StateStoreHandler

   2. KeyToNumValuesStore

   3. KeyWithIndexToValueStore

4. OneSideHashJoiner

5. StateStoreAwareZipPartitionsHelper

Stateful Stream Processing

1. Stateful Stream Processing

2. StateStore

   1. HDFSBackedStateStore

3. StateStoreProvider

   1. HDFSBackedStateStoreProvider

4. StateStoreId

5. StateStoreCoordinator

   1. StateStoreCoordinatorRef

6. WatermarkSupport

7. GroupStateImpl

8. StatefulOperator

   1. StateStoreReader

   2. StateStoreWriter

9. StatefulOperatorStateInfo

10. StateStoreMetrics

11. StateStoreCustomMetric

12. StateStoreUpdater

13. EventTimeStatsAccum

14. StateStoreConf

Streaming Aggregation

1. Streaming Aggregation

2. StateStoreRDD

   1. StateStoreOps

3. StreamingAggregationStateManager

   1. StreamingAggregationStateManagerBaseImpl

   2. StreamingAggregationStateManagerImplV1

   3. StreamingAggregationStateManagerImplV2

Developing Streaming Applications

1. DataStreamReader

2. DataStreamWriter

   1. OutputMode

   2. Trigger

3. StreamingQuery

4. Streaming Operators

   1. dropDuplicates Operator

   2. explain Operator

   3. groupBy Operator

   4. groupByKey Operator

   5. withWatermark Operator

5. window Function

6. KeyValueGroupedDataset

   1. mapGroupsWithState Operator

   2. flatMapGroupsWithState Operator

7. GroupState

   1. GroupStateTimeout

8. StreamingQueryManager

9. SQLConf

10. Configuration Properties

Monitoring

1. StreamingQueryListener — Intercepting Streaming Events

   1. StreamingQueryProgress

2. MetricsReporter

3. ProgressReporter Contract

   1. ExecutionStats

   2. StreamingQueryStatus

   3. SourceProgress

   4. SinkProgress

4. StreamProgress Custom Scala Map

5. Web UI

6. Logging

Demos

1. Demo: Streaming Join and StreamingSymmetricHashJoinExec Physical Operator

2. Demo: Streaming Aggregation with Dataset.groupBy High-Level Operator and Memory Data Source (Append Output Mode)

3. Demo: Streaming Aggregation with Dataset.groupBy High-Level Operator and Append Output Mode

4. Demo: groupByKey Streaming Aggregation in Update Mode

5. Demo: StateStoreSaveExec with Complete Output Mode

6. Demo: StateStoreSaveExec with Update Output Mode

7. Developing Custom Streaming Sink (and Monitoring SQL Queries in web UI)

8. current_timestamp Function For Processing Time in Streaming Queries

9. Using StreamingQueryManager for Query Termination Management

Extending Structured Streaming

1. DataSource

2. BaseStreamingSource Contract

3. BaseStreamingSink Contract

4. Streaming Source

   1. StreamSourceProvider

5. Streaming Sink

   1. StreamSinkProvider

6. StreamWriteSupport Contract

   1. StreamWriter Contract

File-Based Data Source

1. FileStreamSource

2. FileStreamSink

3. FileStreamSinkLog

   1. SinkFileStatus

   2. ManifestFileCommitProtocol

Kafka Data Source

1. Kafka Data Source — Streaming Data Source for Apache Kafka

2. KafkaSourceProvider — Data Source Provider for Apache Kafka

3. KafkaSource

4. KafkaRelation

5. KafkaSourceRDD

6. CachedKafkaConsumer

7. KafkaOffsetReader

8. ConsumerStrategy Contract for KafkaConsumer Providers

9. KafkaSourceOffset

10. KafkaSink

11. KafkaOffsetRangeLimit — Desired Offset Range Limits

12. KafkaContinuousReader — ContinuousReader for Kafka Data Source in Continuous Stream Processing

13. KafkaMicroBatchReader

    1. KafkaOffsetRangeCalculator

14. KafkaContinuousInputPartition

Text Socket Data Source

1. TextSocketSourceProvider

2. TextSocketSource

Rate Data Source

1. RateSourceProvider

2. RateStreamSource

Console Data Sink

1. ConsoleSinkProvider

2. ConsoleWriter

Foreach Data Sink

1. ForeachWriterProvider

2. ForeachWriter

3. ForeachSink

ForeachBatch Data Sink

1. ForeachBatchSink

Memory Data Sink

1. MemorySinkV2

2. MemorySink

3. MemoryStream

Micro-Batch Stream Processing (Structured Streaming V1)

1. Micro-Batch Stream Processing

2. MicroBatchExecution

   1. MicroBatchWriter

3. MicroBatchReadSupport Contract

   1. MicroBatchReader Contract

4. WatermarkTracker

Continuous Stream Processing (Structured Streaming V2)

1. Continuous Stream Processing

2. ContinuousExecution

3. ContinuousReadSupport Contract

4. ContinuousReader Contract

5. ContinuousMemoryStream

6. RateStreamContinuousReader

7. EpochCoordinator RPC Endpoint

   1. EpochCoordinatorRef

   2. EpochTracker

8. ContinuousQueuedDataReader

   1. DataReaderThread

   2. EpochMarkerGenerator

9. PartitionOffset

10. ContinuousExecutionRelation Leaf Logical Operator

11. WriteToContinuousDataSource Unary Logical Operator

12. WriteToContinuousDataSourceExec Unary Physical Operator

    1. ContinuousWriteRDD

13. ContinuousDataSourceRDD

Query Planning and Execution

1. StreamExecution — Base of Streaming Query Execution Engines

   1. StreamingQueryWrapper — Serializable StreamExecution

2. TriggerExecutor

3. IncrementalExecution

4. StreamingQueryListenerBus — Notification Bus for Streaming Events

5. StreamMetadata

Logical Operators

1. EventTimeWatermark Unary Logical Operator

2. FlatMapGroupsWithState Unary Logical Operator

3. Deduplicate Unary Logical Operator

4. MemoryPlan Logical Query Plan

5. StreamingRelation Leaf Logical Operator for Streaming Source

6. StreamingRelationV2 Leaf Logical Operator

7. StreamingExecutionRelation Leaf Logical Operator for Streaming Source At Execution

Physical Operators

1. EventTimeWatermarkExec

2. FlatMapGroupsWithStateExec

3. StateStoreRestoreExec

4. StateStoreSaveExec

5. StreamingDeduplicateExec

6. StreamingGlobalLimitExec

7. StreamingRelationExec

8. StreamingSymmetricHashJoinExec

Execution Planning Strategies

1. FlatMapGroupsWithStateStrategy

2. StatefulAggregationStrategy

3. StreamingDeduplicationStrategy

4. StreamingGlobalLimitStrategy

5. StreamingJoinStrategy

6. StreamingRelationStrategy

Offsets and Metadata Checkpointing

1. Offset

2. MetadataLog Contract

   1. HDFSMetadataLog

3. CommitLog

4. OffsetSeqLog

5. OffsetSeq

6. CompactibleFileStreamLog Contract

   1. FileStreamSourceLog

7. OffsetSeqMetadata

8. CheckpointFileManager Contract

   1. FileContextBasedCheckpointFileManager

   2. FileSystemBasedCheckpointFileManager

Varia

1. UnsupportedOperationChecker

2. InputProcessor Helper Class of FlatMapGroupsWithStateExec Physical Operator