Base Docker image with just essentials: Hadoop, Hive and Spark.
-
Hadoop 3.2.0 in Fully Distributed (Multi-node) Mode
-
Hive 3.1.2 with HiveServer2 exposed to host.
-
Spark 2.4.5 in YARN mode (Spark Scala, PySpark and SparkR)
Take a look at this repo to see how I use it as a part of a Docker Compose cluster.
Hive JDBC port is exposed to host:
- URI:
jdbc:hive2://localhost:10000 - Driver:
org.apache.hive.jdbc.HiveDriver(org.apache.hive:hive-jdbc:3.1.2) - User and password: unused.
- Hadoop 3.2.1 and Hive 3.1.2 are incompatible due to Guava version
mismatch (Hadoop: Guava 27.0, Hive: Guava 19.0). Hive fails with
java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;) - Spark 2.4.4 can not
use Hive higher than 1.2.2 as a SparkSQL engine
because of this bug: Spark need to support reading data from Hive 2.0.0 metastore
and associated issue Dealing with TimeVars removed in Hive 2.x.
Trying to make it happen results in this exception:
java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT. When this is fixed in Spark 3.0, it will be able to use Hive as a backend for SparkSQL. Alternatively you can try to downgrade Hive :)
- Docker file code linting:
docker run --rm -i hadolint/hadolint < Dockerfile - To trim the fat from Docker image
- Upgrade spark to 3.0
- When upgraded, enable Spark-Hive integration.