Skip to content

Latest commit

 

History

History
29 lines (25 loc) · 2.14 KB

spark-scheduler-listeners-statsreportlistener.adoc

File metadata and controls

29 lines (25 loc) · 2.14 KB
Raw

StatsReportListener — Logging Summary Statistics

org.apache.spark.scheduler.StatsReportListener (see the class' scaladoc) is a SparkListener that logs a few summary statistics when each stage completes.

$ ./bin/spark-shell --conf \
      spark.extraListeners=org.apache.spark.scheduler.StatsReportListener
...
INFO SparkContext: Registered listener org.apache.spark.scheduler.StatsReportListener
...

scala> sc.parallelize(0 to 10).count
...
15/11/04 15:39:45 INFO StatsReportListener: Finished stage: org.apache.spark.scheduler.StageInfo@4d3956a4
15/11/04 15:39:45 INFO StatsReportListener: task runtime:(count: 8, mean: 36.625000, stdev: 5.893588, max: 52.000000, min: 33.000000)
15/11/04 15:39:45 INFO StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
15/11/04 15:39:45 INFO StatsReportListener: 	33.0 ms	33.0 ms	33.0 ms	34.0 ms	35.0 ms	36.0 ms	52.0 ms	52.0 ms	52.0 ms
15/11/04 15:39:45 INFO StatsReportListener: task result size:(count: 8, mean: 953.000000, stdev: 0.000000, max: 953.000000, min: 953.000000)
15/11/04 15:39:45 INFO StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
15/11/04 15:39:45 INFO StatsReportListener: 	953.0 B	953.0 B	953.0 B	953.0 B	953.0 B	953.0 B	953.0 B	953.0 B	953.0 B
15/11/04 15:39:45 INFO StatsReportListener: executor (non-fetch) time pct: (count: 8, mean: 17.660220, stdev: 1.948627, max: 20.000000, min: 13.461538)
15/11/04 15:39:45 INFO StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
15/11/04 15:39:45 INFO StatsReportListener: 	13 %	13 %	13 %	17 %	18 %	20 %	20 %	20 %	20 %
15/11/04 15:39:45 INFO StatsReportListener: other time pct: (count: 8, mean: 82.339780, stdev: 1.948627, max: 86.538462, min: 80.000000)
15/11/04 15:39:45 INFO StatsReportListener: 	0%	5%	10%	25%	50%	75%	90%	95%	100%
15/11/04 15:39:45 INFO StatsReportListener: 	80 %	80 %	80 %	82 %	82 %	83 %	87 %	87 %	87 %