SUMMARY.adoc

Summary

1. Introduction

2. Overview of Spark

3. Anatomy of Spark Application

   1. SparkConf - Configuration for Spark Applications

      1. Deploy Mode

   2. SparkContext

      1. HeartbeatReceiver RPC Endpoint

      2. Inside Creating SparkContext

   3. RDD - Resilient Distributed Dataset

      1. Operators

         1. Transformations

         2. Actions

         3. RDD Lineage — Logical Execution Plan

      2. Partitions and Partitioning

      3. Shuffling

      4. Checkpointing

      5. Dependencies

      6. Types of RDDs

         1. ParallelCollectionRDD

         2. MapPartitionsRDD

         3. PairRDDFunctions

         4. CoGroupedRDD

         5. HadoopRDD

         6. ShuffledRDD

         7. BlockRDD

4. Spark Tools

   1. Spark Shell

   2. WebUI — Spark Application’s web UI

      1. Stages Tab

         1. Stages for All Jobs

         2. Stage Details

         3. Pool Details

      2. Storage Tab

      3. Executors Tab

      4. SQL Tab

         1. SQLListener

      5. JobProgressListener

   3. spark-submit

   4. spark-class

5. Spark Architecture

   1. Driver

   2. Master

   3. Workers

   4. Executors

      1. TaskRunner

6. Spark Services

   1. MemoryManager — Memory Management

      1. UnifiedMemoryManager

   2. SparkEnv — Spark Runtime Environment

   3. DAGScheduler

      1. Jobs

      2. Stages

         1. ShuffleMapStage — Intermediate Stage in Job

         2. ResultStage — Final Stage in Job

   4. Task Scheduler

      1. Tasks

      2. TaskSets

      3. Schedulable

         1. TaskSetManager

         2. Schedulable Pool

         3. Schedulable Builders

            1. FIFOSchedulableBuilder

            2. FairSchedulableBuilder

         4. Scheduling Mode

      4. TaskSchedulerImpl - Default TaskScheduler

      5. TaskContext

      6. TaskMemoryManager

         1. MemoryConsumer

      7. TaskMetrics

   5. Scheduler Backend

      1. CoarseGrainedSchedulerBackend

   6. Executor Backend

      1. CoarseGrainedExecutorBackend

   7. BlockManager

      1. MemoryStore

      2. DiskStore

      3. BlockDataManager

      4. ShuffleClient

      5. BlockTransferService

      6. BlockManagerMaster

      7. BlockInfoManager

         1. BlockInfo

   8. Dynamic Allocation (of Executors)

      1. ExecutorAllocationManager

      2. ExecutorAllocationClient

      3. ExecutorAllocationListener

      4. ExecutorAllocationManagerSource

   9. Shuffle Manager

      1. ExternalShuffleService

   10. ExternalClusterManager

   11. HTTP File Server

   12. Broadcast Manager

   13. Data Locality

   14. Cache Manager

   15. Spark, Akka and Netty

   16. OutputCommitCoordinator

   17. RPC Environment (RpcEnv)

       1. Netty-based RpcEnv

   18. ContextCleaner

   19. MapOutputTracker

7. Deployment Environments — Run Modes

   1. Spark local (pseudo-cluster)

   2. Spark on cluster

      1. Spark on YARN

         1. YarnShuffleService — ExternalShuffleService on YARN

         2. ExecutorRunnable

         3. Client

         4. YarnRMClient

         5. ApplicationMaster

            1. AMEndpoint — ApplicationMaster RPC Endpoint

         6. YarnClusterManager — ExternalClusterManager for YARN

         7. TaskSchedulers for YARN

            1. YarnScheduler

            2. YarnClusterScheduler

         8. SchedulerBackends for YARN

            1. YarnSchedulerBackend

            2. YarnClientSchedulerBackend

            3. YarnClusterSchedulerBackend

            4. YarnSchedulerEndpoint RPC Endpoint

         9. YarnAllocator

         10. Introduction to Hadoop YARN

         11. Setting up YARN Cluster

         12. Kerberos

             1. ConfigurableCredentialManager

         13. ClientDistributedCacheManager

         14. YarnSparkHadoopUtil

         15. Settings

      2. Spark Standalone

         1. Standalone Master

         2. Standalone Worker

         3. web UI

         4. Submission Gateways

         5. Management Scripts for Standalone Master

         6. Management Scripts for Standalone Workers

         7. Checking Status

         8. Example 2-workers-on-1-node Standalone Cluster (one executor per worker)

         9. StandaloneSchedulerBackend

      3. Spark on Mesos

         1. MesosCoarseGrainedSchedulerBackend

         2. About Mesos

8. Execution Model

9. Optimising Spark

   1. Caching and Persistence

   2. Broadcast variables

   3. Accumulators

10. Security

    1. Spark Security

    2. Securing Web UI

11. Data Sources in Spark

    1. Using Input and Output (I/O)

       1. Spark and Parquet

       2. Serialization

    2. Spark and Cassandra

    3. Spark and Kafka

    4. Couchbase Spark Connector

12. Spark Application Frameworks

    1. Spark SQL

       1. SparkSession — Entry Point to Datasets

       2. SQLConf

       3. Catalog

       4. Dataset

          1. LogicalPlan

          2. Encoder

          3. Columns

          4. DataFrame (Dataset[Row])

             1. Schema

             2. Explicit Schema using StructType and StructField

          5. Row

       5. DataSource API — Loading and Saving Datasets

          1. DataFrameReader

          2. DataFrameWriter

          3. DataSource

          4. DataSourceRegister

       6. Functions - Computations on Rows

          1. Standard Functions (functions object)

          2. Aggregation (GroupedData)

          3. User-Defined Functions (UDFs)

          4. Window Aggregates (Windows)

       7. Structured Streaming

          1. DataStreamReader

          2. DataStreamWriter

          3. Source

             1. FileStreamSource

          4. Streaming Sinks

             1. ConsoleSink

             2. ForeachSink

          5. StreamSinkProvider

          6. StreamingQueryManager

          7. StreamingQuery

          8. Trigger

          9. StreamExecution

          10. StreamingRelation

          11. StreamingQueryListenerBus

       8. Joins

       9. Hive Integration

          1. Spark SQL CLI - spark-sql

       10. SQL Parsers

       11. Caching

       12. Datasets vs RDDs

       13. SessionState

       14. SQLExecution Helper Object

       15. SQLContext

       16. Performance Optimizations

           1. Catalyst Query Optimizer

              1. Predicate Pushdown

              2. Constant Folding

              3. Nullability (NULL Value) Propagation

              4. Vectorized Parquet Decoder

              5. QueryPlan

              6. SparkPlan

              7. QueryPlanner

              8. QueryExecution

              9. Whole-Stage Code Generation

           2. Project Tungsten

       17. Settings

    2. Spark Streaming

       1. StreamingContext

       2. Stream Operators

          1. Windowed Operators

          2. SaveAs Operators

          3. Stateful Operators

       3. web UI and Streaming Statistics Page

       4. Streaming Listeners

       5. Checkpointing

       6. JobScheduler

       7. JobGenerator

       8. DStreamGraph

       9. Discretized Streams (DStreams)

          1. Input DStreams

          2. ReceiverInputDStreams

          3. ConstantInputDStreams

          4. ForEachDStreams

          5. WindowedDStreams

          6. MapWithStateDStreams

          7. StateDStreams

          8. TransformedDStream

       10. Receivers

           1. ReceiverTracker

           2. ReceiverSupervisors

           3. ReceivedBlockHandlers

       11. Ingesting Data from Kafka

           1. KafkaRDD

       12. RecurringTimer

       13. Backpressure

       14. Dynamic Allocation (Elastic Scaling)

           1. ExecutorAllocationManager

       15. Settings

    3. Spark MLlib - Machine Learning in Spark

       1. ML Pipelines (spark.ml)

          1. Transformers

          2. Estimators

          3. Models

          4. Evaluators

          5. CrossValidator

          6. Persistence (MLWriter and MLReader)

          7. Example — Text Classification

          8. Example — Linear Regression

       2. Latent Dirichlet Allocation (LDA)

       3. Vector

       4. LabeledPoint

       5. Streaming MLlib

    4. Spark GraphX - Distributed Graph Computations

       1. Graph Algorithms

13. Monitoring, Tuning and Debugging

    1. Unified Memory Management

    2. HistoryServer

       1. SQLHistoryListener

       2. FsHistoryProvider

    3. Logging

    4. Performance Tuning

    5. Spark Metrics System

    6. Spark Listeners

       1. LiveListenerBus

       2. ReplayListenerBus

       3. EventLoggingListener — Event Logging

       4. StatsReportListener — Logging Summary Statistics

    7. Debugging Spark using sbt

14. Varia

    1. Building Spark

    2. Spark and Hadoop

    3. Spark and software in-memory file systems

    4. Spark and The Others

    5. Distributed Deep Learning on Spark

    6. Spark Packages

    7. TransportConf — Transport Configuration

15. Interactive Notebooks

    1. Apache Zeppelin

    2. Spark Notebook

16. Spark Tips and Tricks

    1. Access private members in Scala in Spark shell

    2. SparkException: Task not serializable

    3. Running Spark on Windows

17. Exercises

    1. One-liners using PairRDDFunctions

    2. Learning Jobs and Partitions Using take Action

    3. Spark Standalone - Using ZooKeeper for High-Availability of Master

    4. Spark’s Hello World using Spark shell and Scala

    5. WordCount using Spark shell

    6. Your first complete Spark application (using Scala and sbt)

    7. Spark (notable) use cases

    8. Using Spark SQL to update data in Hive using ORC files

    9. Developing Custom SparkListener to monitor DAGScheduler in Scala

    10. Developing RPC Environment

    11. Developing Custom RDD

    12. Creating DataFrames from Tables using JDBC and PostgreSQL

    13. Causing Stage to Fail

18. Further Learning

    1. Courses

    2. Books

19. Spark Distributions

    1. DataStax Enterprise

    2. MapR Sandbox for Hadoop (Spark 1.5.2 only)

20. Commercial Products using Apache Spark

    1. IBM Analytics for Apache Spark

    2. Google Cloud Dataproc

21. Spark Advanced Workshop

    1. Requirements

    2. Day 1

    3. Day 2

22. Spark Talks Ideas (STI)

    1. 10 Lesser-Known Tidbits about Spark Standalone

    2. Learning Spark internals using groupBy (to cause shuffle)