2.5.0: S3 Endpoint Region Support

Summary

In this release, we added support for accessing S3 buckets using VPC interface endpoints that are behind PrivateLink. Moreover, we updated documentation and dependencies.

Features

• #216: Added support for S3 endpoint region

Documentation

• #213: Added parallelism parameter examples to user guide
• #214: Updated build instructions in the user guide

Refactoring

• #205: Fixed missing source files for Sonar coverage report
• #219: Fixed HDFS client version mismatch

Dependency Updates

Compile Dependency Updates

• Updated com.exasol:error-reporting-java:0.4.1 to 1.0.0
• Updated com.exasol:parquet-io-java:1.3.1 to 1.3.3
• Updated com.fasterxml.jackson.core:jackson-databind:2.13.3 to 2.13.4
• Updated com.fasterxml.jackson.module:jackson-module-scala_2.13:2.13.3 to 2.13.4
• Updated com.google.protobuf:protobuf-java:3.21.2 to 3.21.6
• Updated io.delta:delta-core_2.13:1.2.1 to 2.1.0
• Updated io.grpc:grpc-netty:1.47.0 to 1.49.1
• Updated io.netty:netty-all:4.1.79.Final to 4.1.82.Final
• Updated org.alluxio:alluxio-core-client-hdfs:2.8.0-2 to 2.8.1
• Updated org.apache.avro:avro:1.11.0 to 1.11.1
• Updated org.apache.hadoop:hadoop-aws:3.3.3 to 3.3.4
• Updated org.apache.hadoop:hadoop-azure-datalake:3.3.3 to 3.3.4
• Updated org.apache.hadoop:hadoop-azure:3.3.3 to 3.3.4
• Updated org.apache.hadoop:hadoop-common:3.3.3 to 3.3.4
• Added org.apache.hadoop:hadoop-hdfs-client:3.3.4
• Updated org.apache.hadoop:hadoop-hdfs:3.3.3 to 3.3.4
• Updated org.apache.logging.log4j:log4j-1.2-api:2.17.2 to 2.19.0
• Updated org.apache.logging.log4j:log4j-api:2.17.2 to 2.19.0
• Added org.apache.logging.log4j:log4j-core:2.19.0
• Updated org.apache.orc:orc-core:1.7.5 to 1.8.0
• Updated org.scala-lang:scala-library:2.13.8 to 2.13.9
• Updated org.slf4j:jul-to-slf4j:1.7.36 to 2.0.3
• Updated org.slf4j:slf4j-log4j12:1.7.36 to 2.0.3

Test Dependency Updates

• Updated com.dimafeng:testcontainers-scala-scalatest_2.13:0.40.8 to 0.40.10
• Updated com.exasol:exasol-testcontainers:6.1.2 to 6.2.0
• Updated com.exasol:hamcrest-resultset-matcher:1.5.1 to 1.5.2
• Updated com.exasol:test-db-builder-java:3.3.3 to 3.3.4
• Added nl.jqno.equalsverifier:equalsverifier:3.10.1
• Updated org.mockito:mockito-core:4.6.1 to 4.8.0

Plugin Dependency Updates

• Updated com.diffplug.spotless:spotless-maven-plugin:2.23.0 to 2.27.0
• Updated com.exasol:error-code-crawler-maven-plugin:1.1.1 to 1.1.2
• Updated com.exasol:project-keeper-maven-plugin:2.4.6 to 2.8.0
• Updated io.github.evis:scalafix-maven-plugin_2.13:0.1.4_0.9.33 to 0.1.7_0.10.3
• Updated net.alchim31.maven:scala-maven-plugin:4.6.3 to 4.7.1
• Updated org.apache.maven.plugins:maven-enforcer-plugin:3.0.0 to 3.1.0
• Updated org.apache.maven.plugins:maven-javadoc-plugin:3.4.0 to 3.4.1
• Updated org.itsallcode:openfasttrace-maven-plugin:1.5.0 to 1.6.1

2.4.0: Support for automatic parallelism

Summary

In this release, we added support to automatically calculate parallelism parameter based on the vCPUs and RAM size on the database nodes. This way customers need to set PARALLELISM parameter only in rare cases.

Feature

• #209: Added auto calculated parallelism parameter

Dependency Updates

Compile Dependency Updates

• Updated com.google.protobuf:protobuf-java:3.21.1 to 3.21.2
• Updated io.netty:netty-all:4.1.78.Final to 4.1.79.Final

Test Dependency Updates

• Updated org.testcontainers:localstack:1.17.2 to 1.17.3

Plugin Dependency Updates

• Updated com.diffplug.spotless:spotless-maven-plugin:2.22.8 to 2.23.0

2.3.2: Upgrade dependencies

Refactorings

• #191: Enabled Java 17 javadoc CI checks

Bugfixes

• #207: Upgrade dependencies to fix vulnerabilities

Dependency Updates

Compile Dependency Updates

• Updated com.exasol:parquet-io-java:1.3.0 to 1.3.1
• Updated com.fasterxml.jackson.core:jackson-databind:2.13.2.2 to 2.13.3
• Updated com.fasterxml.jackson.module:jackson-module-scala_2.13:2.13.2 to 2.13.3
• Added com.google.oauth-client:google-oauth-client:1.34.1
• Updated com.google.protobuf:protobuf-java:3.20.0 to 3.21.1
• Updated com.typesafe.scala-logging:scala-logging_2.13:3.9.4 to 3.9.5
• Updated io.delta:delta-core_2.13:1.1.0 to 1.2.1
• Updated io.grpc:grpc-netty:1.45.1 to 1.47.0
• Updated io.netty:netty-all:4.1.75.Final to 4.1.78.Final
• Updated org.alluxio:alluxio-core-client-hdfs:2.7.4 to 2.8.0-2
• Updated org.apache.hadoop:hadoop-aws:3.3.2 to 3.3.3
• Updated org.apache.hadoop:hadoop-azure-datalake:3.3.2 to 3.3.3
• Updated org.apache.hadoop:hadoop-azure:3.3.2 to 3.3.3
• Updated org.apache.hadoop:hadoop-common:3.3.2 to 3.3.3
• Updated org.apache.hadoop:hadoop-hdfs:3.3.2 to 3.3.3
• Updated org.apache.orc:orc-core:1.7.3 to 1.7.5
• Updated org.apache.spark:spark-sql_2.13:3.2.1 to 3.3.0

Test Dependency Updates

• Updated com.dimafeng:testcontainers-scala-scalatest_2.13:0.40.5 to 0.40.8
• Updated com.exasol:exasol-testcontainers:6.1.1 to 6.1.2
• Updated com.exasol:test-db-builder-java:3.3.1 to 3.3.3
• Updated org.mockito:mockito-core:4.4.0 to 4.6.1
• Updated org.testcontainers:localstack:1.16.3 to 1.17.2

Plugin Dependency Updates

• Updated com.diffplug.spotless:spotless-maven-plugin:2.22.0 to 2.22.8
• Updated com.exasol:artifact-reference-checker-maven-plugin:0.4.1 to 0.4.0
• Updated com.exasol:project-keeper-maven-plugin:1.3.4 to 2.4.6
• Updated net.alchim31.maven:scala-maven-plugin:4.6.1 to 4.6.3
• Updated org.apache.maven.plugins:maven-clean-plugin:3.2.0 to 2.5
• Added org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M5
• Removed org.apache.maven.plugins:maven-gpg-plugin:3.0.1
• Updated org.apache.maven.plugins:maven-install-plugin:2.5.2 to 2.4
• Updated org.apache.maven.plugins:maven-javadoc-plugin:3.3.2 to 3.4.0
• Updated org.apache.maven.plugins:maven-resources-plugin:3.2.0 to 2.6
• Updated org.apache.maven.plugins:maven-site-plugin:3.11.0 to 3.3
• Removed org.apache.maven.plugins:maven-source-plugin:3.2.1
• Updated org.apache.maven.plugins:maven-surefire-plugin:2.12.4 to 3.0.0-M5
• Added org.codehaus.mojo:flatten-maven-plugin:1.2.7
• Added org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184

2.3.1: String Truncation and Bug Fixing

Summary

In this release we added a new parameter TIMEZONE_UTC, when set to true, the timestamp data from data sources will be set to 'UTC' timezone. Without this parameter, or it is set to false, the timestamp values are imported or exported using the datanode timezone (DBTIMEZONE). Additionally we added another parameter to truncate string values that exceed maximum allowed VARCHAR size.

Features

• #196: Added truncation support if string values exceed Exasol maximum length

Bug Fixes

• #190: Added S3 bucket name validation
• #193: Fixed bug reading delta formatted data
• #194: Added timezone UTC parameter
• #195: Fixed bug with missing class reading from Azure Data Lake Gen2 storage

Refactoring

• #177: Migrated to maven based build

Dependency Updates

Compile Dependency Updates

• Added com.exasol:error-reporting-java:0.4.1
• Added com.exasol:import-export-udf-common-scala_2.13:0.2.0
• Added com.exasol:parquet-io-java:1.3.0
• Added com.fasterxml.jackson.core:jackson-databind:2.13.2.2
• Added com.fasterxml.jackson.module:jackson-module-scala_2.13:2.13.2
• Added com.google.cloud.bigdataoss:gcs-connector:1.9.4-hadoop3
• Added com.google.guava:guava:31.1-jre
• Added com.google.protobuf:protobuf-java:3.20.0
• Added com.typesafe.scala-logging:scala-logging_2.13:3.9.4
• Added io.delta:delta-core_2.13:1.1.0
• Added io.grpc:grpc-netty:1.45.1
• Added io.netty:netty-all:4.1.75.Final
• Added org.alluxio:alluxio-core-client-hdfs:2.7.4
• Added org.apache.avro:avro:1.11.0
• Added org.apache.commons:commons-lang3:3.12.0
• Added org.apache.hadoop:hadoop-aws:3.3.2
• Added org.apache.hadoop:hadoop-azure-datalake:3.3.2
• Added org.apache.hadoop:hadoop-azure:3.3.2
• Added org.apache.hadoop:hadoop-common:3.3.2
• Added org.apache.hadoop:hadoop-hdfs:3.3.2
• Added org.apache.logging.log4j:log4j-1.2-api:2.17.2
• Added org.apache.logging.log4j:log4j-api:2.17.2
• Added org.apache.orc:orc-core:1.7.3
• Added org.apache.spark:spark-sql_2.13:3.2.1
• Added org.codehaus.jackson:jackson-mapper-asl:1.9.13
• Added org.scala-lang:scala-library:2.13.8
• Added org.slf4j:jul-to-slf4j:1.7.36
• Added org.slf4j:slf4j-log4j12:1.7.36

Test Dependency Updates

• Added com.dimafeng:testcontainers-scala-scalatest_2.13:0.40.5
• Added com.exasol:exasol-testcontainers:6.1.1
• Added com.exasol:hamcrest-resultset-matcher:1.5.1
• Added com.exasol:test-db-builder-java:3.3.1
• Added org.hamcrest:hamcrest:2.2
• Added org.mockito:mockito-core:4.4.0
• Added org.scalatestplus:scalatestplus-mockito_2.13:1.0.0-M2
• Added org.scalatest:scalatest_2.13:3.2.10
• Added org.testcontainers:localstack:1.16.3

Plugin Dependency Updates

• Added com.diffplug.spotless:spotless-maven-plugin:2.22.0
• Added com.exasol:artifact-reference-checker-maven-plugin:0.4.1
• Added com.exasol:error-code-crawler-maven-plugin:1.1.1
• Added com.exasol:project-keeper-maven-plugin:1.3.4
• Added io.github.evis:scalafix-maven-plugin_2.13:0.1.4_0.9.33
• Added io.github.zlika:reproducible-build-maven-plugin:0.15
• Added net.alchim31.maven:scala-maven-plugin:4.6.1
• Added org.apache.maven.plugins:maven-assembly-plugin:3.3.0
• Added org.apache.maven.plugins:maven-clean-plugin:3.2.0
• Added org.apache.maven.plugins:maven-compiler-plugin:3.10.1
• Added org.apache.maven.plugins:maven-deploy-plugin:2.7
• Added org.apache.maven.plugins:maven-enforcer-plugin:3.0.0
• Added org.apache.maven.plugins:maven-gpg-plugin:3.0.1
• Added org.apache.maven.plugins:maven-install-plugin:2.5.2
• Added org.apache.maven.plugins:maven-jar-plugin:3.2.2
• Added org.apache.maven.plugins:maven-javadoc-plugin:3.3.2
• Added org.apache.maven.plugins:maven-resources-plugin:3.2.0
• Added org.apache.maven.plugins:maven-site-plugin:3.11.0
• Added org.apache.maven.plugins:maven-source-plugin:3.2.1
• Added org.apache.maven.plugins:maven-surefire-plugin:2.12.4
• Added org.codehaus.mojo:versions-maven-plugin:2.10.0
• Added org.itsallcode:openfasttrace-maven-plugin:1.5.0
• Added org.jacoco:jacoco-maven-plugin:0.8.8
• Added org.scalastyle:scalastyle-maven-plugin:1.0.0
• Added org.scalatest:scalatest-maven-plugin:2.0.2
• Added org.sonatype.ossindex.maven:ossindex-maven-plugin:3.2.0

Added support for Parquet `TIMESTAMP_MICROS` and `UUID` logical types

Summary

In this release, we added support for reading Parquet INT64 (TIMESTAMP_MICROS) and FIXED_LEN_BYTE_ARRAY (UUID) types. We also introduced an S3 SSL enabled parameter that can be used to disable secure connection to S3 bucket.

Features

• #181: Added support for reading Parquet timestamp micros values
• #183: Added support for reading Parquet UUID values
• #184: Added S3 SSL enabled parameter

Dependency Updates

Compile Dependency Updates

• Updated io.grpc:grpc-netty:1.43.1 to 1.43.2
• Updated io.netty:netty-all:4.1.72.Final to 4.1.73.Final
• Updated org.alluxio:alluxio-core-client-hdfs:2.7.1 to 2.7.2
• Updated org.apache.logging.log4j:log4j-api:2.17.0 to 2.17.1
• Updated org.apache.logging.log4j:log4j-1.2-api:2.17.0 to 2.17.1

Test Dependency Updates

Plugin Updates

• Updated org.scalameta:sbt-scalafmt:2.4.5 to 2.4.6
• Updated org.scoverage:sbt-scoverage:1.9.2 to 1.9.3
• Updated com.timushev.sbt:sbt-updates:0.6.0 to 0.6.1

Fixed Log4J CVE-2021-45105 vulnerability

Summary

In this release we fixed Log4J CVE-2021-45105 vulnerability.

Bug Fixes

• #178: Updated log4j dependency to 2.17.0 version

Dependency Updates

Compile Dependency Updates

• Updated io.grpc:grpc-netty:1.43.0 to 1.43.1
• Updated org.apache.orc:orc-core:1.7.1 to 1.7.2
• Updated org.apache.logging.log4j:log4j-api:2.16.0 to 2.17.0
• Updated org.apache.logging.log4j:log4j-1.2-api:2.16.0 to 2.17.0
• Updated com.fasterxml.jackson.core:jackson-databind:2.13.0 to 2.13.1
• Updated com.fasterxml.jackson.module:jackson-module-scala_2.13:2.13.0 to 2.13.1

Test Dependency Updates

• Updated org.mockito:mockito-core:4.1.0 to 4.2.0

Plugin Updates

Fixed the Log4J Vulnerability

Summary

In this release we updated dependencies and fixed the Log4J CVE-2021-44228 vulnerability.

Bug Fixes

• #175: Updated dependencies and fixed the log4j vulnerability

Dependency Updates

Compile Dependency Updates

• Added io.netty:netty-all:4.1.72.Final
• Added org.apache.avro:avro:1.11.0
• Added com.fasterxml.jackson.core:jackson-databind:2.13.0
• Added com.fasterxml.jackson.module:jackson-module-scala_2.13:2.13.0
• Added org.slf4j:jul-to-slf4j:1.7.32
• Added org.apache.logging.log4j:log4j-api:2.16.0
• Added org.apache.logging.log4j:log4j-1.2-api:2.16.0
• Updated io.grpc:grpc-netty:1.41.0 to 1.43.0
• Updated com.google.guava:guava:30.0.1-jre to 31.0.1-jre
• Updated com.exasol:error-reporting-java:0.4.1
• Updated com.exasol:parquet-io-java:1.1.0 to 1.2.1
• Updated org.apache.orc:orc-core:1.7.0 to 1.7.1
• Updated org.alluxio:alluxio-core-client-hdfs:2.6.2 to 2.7.1
• Updated io.delta:delta-core_2.13:0.7.0 to 1.1.0
• Updated org.apache.spark:spark-sql_2.13:3.0.1 to 3.2.0

Test Dependency Updates

• Updated org.mockito:mockito-core:3.12.4 to 4.1.0
• Updated com.exasol:hamcrest-resultset-matcher:1.4.1 to 1.5.1
• Updated com.exasol:exasol-testcontainers:5.0.0 to 5.1.1
• Updated com.dimafeng:testcontainers-scala-scalatest:0.39.8 to 0.39.12
• Updated org.testcontainers:localstack:0.16.0 to 0.16.2

Plugin Updates

• Updated org.scalameta:sbt-scalafmt:2.4.3 to 2.4.5
• Updated org.scoverage:sbt-scoverage:1.9.0 to 1.9.2
• Updated org.wartremover:sbt-wartremover-contrib:1.3.12 to 1.3.13

Improved Parquet Reader

Summary

In this release we add optimized Parquet file importer. Previous versions read single Parquet file in a single importer process, in this version we improved it by virtually splitting files into fixed sized chunks that then can be imported in many parallel processes. For that, please do not forget to update to the latest deployment scripts in the user guide.

In addition, we added support for connections behind proxies when accessing cloud storage systems and added unified error codes.

Features

• #173: Added improved chunked Parquet reader
• Added support for connection behind proxy (PR #172)

Refactoring

• #113: Added unified error codes

Dependency Updates

Compile Dependency Updates

• Added com.exasol:error-reporting-java:0.4.0
• Updated com.exasol:parquet-io-java:1.0.3 to 1.1.0
• Updated org.apache.orc:orc-core:1.6.9 to 1.7.0
• Updated org.alluxio:alluxio-core-client-hdfs:2.6.1 to 2.6.2
• Updated io.grpc:grpc-netty:1.39.0 to 1.41.0

Test Dependency Updates

• Updated com.dimafeng:testcontainers-scala-scalatest:0.39.5 to 0.39.8
• Updated com.exasol:exasol-testcontainers:4.0.0 to 5.0.0
• Updated org.mockito:mockito-core:3.11.2 to 3.12.4
• Updated org.scalatest:scalatest:3.2.9 to 3.2.10

Plugin Updates

• Updated com.eed3si9n:sbt-assembly:1.0.0 to 1.1.0
• Updated net.bzzt:sbt-reproducible-builds:0.28 to 0.30
• Updated org.scoverage:sbt-scoverage:1.8.2 to 1.9.0

Return Exported Records Count

Summary

This release fixes an issue in the export so that the number of exported records are shown after the export process.

Bug Fixes

• #168: Return the number of rows exported after export

Dependency Updates

Runtime Dependency Updates

• Updated com.exasol:parquet-io-java:1.0.2 to 1.0.3
• Updated org.alluxio:alluxio-core-client-hdfs:2.6.0 to 2.6.1

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers:3.5.3 to 4.0.0
• Updated com.exasol:hamcrest-resultset-matcher:1.4.0 to 1.4.1
• Updated com.exasol:test-db-builder-java:3.2.0 to 3.2.1

Plugin Updates

• Updated com.timushev.sbt:sbt-updates:0.5.3 to 0.6.0

Bug Fixes and Refactorings

Summary

In this releases we added support for exporting Hive compatible Parquet schema. Also, this release fixes several bugs. We fixed a bug related to delimited identifiers and another bug with missing trailing zeros in decimals when doing export.

We also added several refactorings such as extracting parquet-io-java library, improving integration tests and migrating continuous integration (CI) setup to Github actions.

Features

• #37: Added support for Hive compatible Parquet schema when exporting Exasol tables

Bug Fixes

• #145: Fixed issue exporting tables with delimited identifiers
• #156: Fixed bug related to logging backend
• #166: Fixed bug with missing decimal trailing zeros on export

Refactoring

• #140: Refactored importer integration tests into multiple suites
• #150: Added extracted parquet-io-java library
• #157: Refactored build setup
• #162: Fixed code smells reported by Sonar cloud

Dependency Updates

Runtime Dependency Updates

• Added com.exasol:parquet-io-java:1.0.2
• Added com.google.guava:guava:30.1.1-jre
• Added com.typesafe.scala-logging:scala-logging:3.9.4
• Added org.apache.commons:commons-lang3:3.12.0
• Added org.slf4j:slf4j-log4j12:1.7.32
• Added org.apache.hadoop:hadoop-common:3.3.1
• Removed org.apache.hadoop:hadoop-client:3.3.1
• Removed org.apache.parquet:parquet-hadoop:1.12.0
• Updated org.apache.orc:orc-core:1.6.8 to 1.6.9
• Updated org.alluxio:alluxio-core-client-hdfs:2.5.0 to 2.6.0

Test Dependency Updates

• Updated org.mockito:mockito-core:3.11.1 to 3.11.2
• Updates org.testcontainers:localstack:1.15.3 to 1.16.0

Plugin Updates

• Updated org.wartremover:sbt-wartremover:2.4.13 to 2.4.16
• Updated org.wartremover:sbt-wartremover-contrib:1.3.11 to 1.3.12
• Updated org.scoverage:sbt-coveralls:1.3.0 to 1.3.1
• Updated net.bzzt:sbt-reproducible-builds:0.25 to 0.28
• Removed com.lucidchart:sbt-scalafmt-coursier:1.16
• Removed com.typesafe.sbt:sbt-git:1.0.1