docker-hive

This is a docker container for Apache Hive 2.3.2. It is based on https://github.com/big-data-europe/docker-hadoop so check there for Hadoop configurations. This deploys Hive and starts a hiveserver2 on port 10000. Metastore is running with a connection to postgresql database. The hive configuration is performed with HIVE_SITE_CONF_ variables (see hadoop-hive.env for an example).

To run Hive with postgresql metastore:

    docker-compose up -d

To deploy in Docker Swarm:

    docker stack deploy -c docker-compose.yml hive

To run a PrestoDB 0.181 with Hive connector:

  docker-compose up -d presto-coordinator

This deploys a Presto server listens on port 8080

Testing

Load data into Hive:

  $ docker-compose exec hive-server bash
  # /opt/hive/bin/beeline -u jdbc:hive2://localhost:10000
  > CREATE TABLE pokes (foo INT, bar STRING);
  > LOAD DATA LOCAL INPATH '/opt/hive/examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;

Then query it from PrestoDB. You can get presto.jar from PrestoDB website:

  $ wget https://repo1.maven.org/maven2/io/prestosql/presto-cli/308/presto-cli-308-executable.jar
  $ mv presto-cli-308-executable.jar presto.jar
  $ chmod +x presto.jar
  $ ./presto.jar --server localhost:8080 --catalog hive --schema default
  presto> select * from pokes;

测试PARQUET格式：

CREATE EXTERNAL TABLE parquet_table (
    foo INT,
    bar STRING
)
STORED AS PARQUET;
insert into parquet_table (foo,bar) values(1,'xxxx' );

PARQUET格式，带分区

CREATE TABLE IF NOT EXISTS sales_data (
    sale_id STRING,
    product_name STRING,
    quantity_sold INT,
    price DOUBLE
)
PARTITIONED BY (year INT, month INT)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS PARQUET;

INSERT INTO TABLE sales_data PARTITION (year=2023, month=1)
VALUES 
('s001', 'ProductA', 15, 19.99),
('s002', 'ProductB', 20, 29.99),
('s003', 'ProductC', 25, 9.99);

Contributors

• Ivan Ermilov @earthquakesan (maintainer)
• Yiannis Mouchakis @gmouchakis
• Ke Zhu @shawnzhu