spark-sql-streaming-ConsoleSink.adoc

ConsoleSink

ConsoleSink is a streaming sink that is registered as the console format.

val spark: SparkSession = ...
spark.readStream
  .format("text")
  .load("server-logs/*.out")
  .as[String]
  .writeStream
  .queryName("server-logs processor")
  .format("console")  // <-- uses ConsoleSink
  .start

scala> spark.streams.active.foreach(println)
Streaming Query - server-logs processor [state = ACTIVE]

// in another terminal
$ echo hello > server-logs/hello.out

// in the terminal with Spark
-------------------------------------------
Batch: 0
-------------------------------------------
+-----+
|value|
+-----+
|hello|
+-----+

ConsoleSinkProvider

ConsoleSinkProvider is a StreamSinkProvider for ConsoleSink. As a DataSourceRegister, it registers the ConsoleSink streaming sink as console format.