Fixed bug listing files on a path using asterisk

Summary

In this release we fixed a bug related to listing files using asterisk on a path. We also updated user guide and added link checker to detect broken links in the project documentation.

Bugfixes

• #146: Fixed issue listing files when asterisk is missing on a path

Documentation

• #141: Updated readme to reflect recent changes
• #143: Added broken links checker Github actions workflow
• #151: Update user guide overview with additional sources

Dependency Updates

Runtime Dependency Updates

• Updated Scala version 2.12.12 to 2.12.14
• Updated org.apache.orc:orc-core:1.6.7 to 1.6.8
• Updated org.apache.hadoop:hadoop-aws:3.3.0 to 3.3.1
• Updated org.apache.hadoop:hadoop-azure:3.3.0 to 3.3.1
• Updated org.apache.hadoop:hadoop-azure-datalake:3.3.0 to 3.3.1
• Updated org.apache.hadoop:hadoop-client:3.3.0 to 3.3.1
• Updated org.apache.hadoop:hadoop-hdfs:3.3.0 to 3.3.1

Test Dependency Updates

• Updated com.exasol:test-db-builder-java:3.1.1 to 3.2.0
• Updated com.exasol:exasol-testcontainers:3.5.1 to 3.5.3
• Updated org.scalatest:scalatest:3.2.6 to 3.2.9
• Updated org.mockito:mockito-core:3.8.0 to 3.11.1
• Updated org.testcontainers:localstack:1.15.2 to 1.15.3
• Updated com.dimafeng:testcontainers-scala-scalatest:0.39.3 to 0.39.5

Plugin Updates

• Updated com.eed3si9n:sbt-assembly:0.15.0 to 1.0.0
• Updated com.timushev.sbt:sbt-updates:0.5.2 to 0.5.3
• Updated com.typesafe.sbt:sbt-git:1.0.0 to 1.0.1
• Updated org.scoverage:sbt-coveralls:1.2.7 to 1.3.0
• Updated org.scoverage:sbt-scoverage:1.6.1 to 1.8.2

1.1.0 - Added support for Alluxio and HDFS filesystems

Summary

The release 1.1.0 offers support for Alluxio and Hadoop Distributed Filesystem (HDFS) filesystems. We also fixed bug in Parquet dictionary encoded decimal converter and improved the documentation.

Features

• #134: Added support for Hadoop Distributed Filesystem (HDFS) (PR #136).
• #135: Added support for Alluxio filesystem (PR #139).

Bug Fixes

• #137: Fixed Parquet dictionary encoded decimal converter bug (PR #138).

Documentation

• #131: Added access privilege to connection object (PR #132).

Dependency Updates

Runtime Dependency Updates

• Updated org.apache.parquet:parquet-hadoop:1.11.1 to 1.12.0

Test Dependency Updates

• Added org.apache.hadoop:hadoop-hdfs:3.3.0
• Added org.alluxio:alluxio-core-client-hdfs:2.5.0
• Updated org.scalatest:scalatest:3.2.3 to 3.2.6
• Updated org.mockito:mockito-core:3.7.7 to 3.8.0
• Updated org.testcontainers:localstack:1.15.1 to 1.15.2
• Updated com.exasol:test-db-builder-java:3.0.0 to 3.1.1
• Updated com.exasol:exasol-testcontainers:3.5.0 to 3.5.1

Plugin Updates

• Updated com.timushev.sbt:sbt-updates:0.5.1 to 0.5.2

1.0.0 - Support complex types

Summary

In this release, we have added several features and fixed several bugs.

We have added support for importing complex data types as JSON string of different storage formats. Added support for S3 path style access and fixed bugs related to Parquet repeated types.

We have also changed the UDF script class names, so please make sure to update your deployment scripts. You can check out the user guide for the updated deployment guide.

Features / Enhancements

• #106: Updated parquet-hadoop version that includes api changes (PR #107).
• #108: Added dictionary aware Parquet decoders (PR #109).
• #11: Added support for importing Parquet complex (LIST, MAP) types (PR #111).
• #115: Added support for importing Orc complex (LIST, STRUCT) types (PR #116).
• #118: Added support for docker based Exasol installations (PR #119).
• #120: Added support for S3 path style access (PR #126).

Bug Fixes

• #112: Fixed Parquet repeated converter bugs (PR #121)
• #35: Fixed the hidden and metadata files filter bug (PR #122).

Refactoring

• #117: Added Exasol docker containers for integration tests (PR #119).
• #114: Enabled linter to check the platform charsets or locales (PR #126).
• #123: Renamed the UDF script class names (PR #126).

Documentation

• #89: Increased the default number of characters for file path (PR #105).
• #110: Added overview table with supported features (PR #124).

Dependency Updates

Runtime Dependency Updates

• Updated org.apache.orc:orc-core:1.6.4 to 1.6.7
• Updated org.apache.parquet:parquet-hadoop:1.10.1 to 1.11.1
• Updated com.exasol:import-export-udf-common-scala:0.1.0 to 0.2.0
• Updated org.apache.spark:spark-sql:3.0.0 to 3.0.1
• Removed com.exasol:exasol-script-api
• Removed com.typesafe.scala-logging:scala-logging
• Removed com.fasterxml.jackson.core:jackson-core
• Removed com.fasterxml.jackson.core:jackson-databind
• Removed com.fasterxml.jackson.core:jackson-annotations
• Removed com.fasterxml.jackson.module:"jackson-module-scala
• Removed libraries are included in import-export-udf-common-scala dependency

Test Dependency Updates

• Added org.hamcrest:hamcrest:2.2
• Added com.exasol:hamcrest-resultset-matcher:1.4.0
• Added com.exasol:test-db-builder-java:3.0.0
• Added com.exasol:exasol-testcontainers:3.5.0
• Added org.testcontainers:localstack:1.15.1
• Updates org.scalatest:scalatest:3.2.2 to 3.2.3
• Updated org.mockito:mockito-core:3.5.13 to 3.7.7

Plugin Updates

• Updated SBT sbtx wrapper to latest
• Updated org.wartremover:sbt-wartremover:2.4.10 to 2.4.13
• Updated org.wartremover:sbt-wartremover-contrib:1.3.8 to 1.3.11
• Updated com.github.cb372:sbt-explicit-dependencies:0.2.13 to 0.2.16

0.9.0

Cloud Storage Extension 0.9.0, released 2020-10-01

In this release, we moved the Kafka and Kinesis modules into their separate repositories. In addition, we disabled using access credentials as user provided parameters.

Please check out the user guide for more information.

Features

• #98: Removed Kafka and Kinesis modules (PR #99).
• #103: Removed credentials from user parameters (PR #104).

Documentation

• #97: Updated documentation (PR #101).
• #100: Renamed the repository (PR #102).

Dependency Updates

Runtime Dependency Updates

• Updated org.apache.orc:orc-core from 1.6.3 to 1.6.4.

Test Dependency Updates

• Updated org.mockito:mockito-core from 3.5.10 to 3.5.13.

v0.8.0

Summary

This is major release step in separating the different modules into separate repositories. After this release, kafka-connector-extension and kinesis-connector-extension will have their own separate repositories. And this repository will be renamed to cloud-storage-extension.

This allows quick release cycle and better visibility of integration connectors for our users and customers.

In this release we have added support for nested data structures in Kinesis connector, added support for connection object in Kafka connector.

Features

• #84: Changed to multi module setup repository (PR #85).
• #87: Added nested data support for Kinesis connector (PR #88).
• Added support for connection object in Kafka connector (PR #90).
• Added checks for keystore and truststore files in Kafka connector (PR #92).

Dependency Updates

Runtime Dependency Updates

• Updated org.apache.hadoop:hadoop-aws from 3.2.1 to 3.3.0.
• Updated org.apache.hadoop:hadoop-azure from 3.2.1 to 3.3.0.
• Updated org.apache.hadoop:hadoop-azure-datalake from 3.2.1 to 3.3.0.
• Updated org.apache.hadoop:hadoop-client from 3.2.1 to 3.3.0.
• Updated org.apache.parquet:parquet-hadoop from 1.10.1 to 1.11.1.
• Updated org.scala-lang:scala-library from 2.12.10 to 2.12.12.
• Updated org.apache.avro:avro from 1.9.2 to 1.10.0.
• Updated org.apache.orc:orc-core from 1.6.2 to 1.6.3.
• Updated io.delta:delta-core from 0.5.0 to 0.7.0.
• Updated org.apache.spark:spark-sql from 2.4.5 to 3.0.0.
• Updated org.apache.kafka:kafka-clients from 2.4.0 to 2.5.0.
• Updated io.confluent:kafka-avro-serializer from 5.4.0 to 5.5.1.

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers from 2.0.0 to 2.1.0.
• Updated org.testcontainers:localstack from 1.13.0 to 1.14.3.
• Updated org.scalatest:scalatest from 3.1.0 to 3.2.2.
• Updated org.mockito:mockito-core from 3.2.4 to 3.5.10.
• Updated com.fasterxml.jackson.core:jackson-core from 2.6.7 to 2.11.2.
• Updated com.fasterxml.jackson.core:jackson-databind from 2.6.7.3 to 2.11.2.
• Updated com.fasterxml.jackson.core:jackson-annotations from 2.6.7 to 2.11.2.
• Updated com.fasterxml.jackson.module:"jackson-module-scala from 2.6.7.1 to 2.11.2.

Plugin Updates

• Updated sbt.version from 1.3.7 to 1.3.13.
• Updated org.wartremover:sbt-wartremover from 2.4.3 to 2.4.10.
• Updated org.wartremover:sbt-wartremover-contrib from 1.3.2 to 1.3.8.
• Updated com.eed3si9n:sbt-assembly from 0.14.10 to 0.15.0.
• Updated com.timushev.sbt:sbt-updates from 0.5.0 to 0.5.1.
• Updated com.github.cb372:sbt-explicit-dependencies from 0.2.12 to 0.2.13.

v0.7.1

2020 MAY 25

New features:

• Adds OVERWRITE user parameter to be used with Export UDF. #36 #81

Improvements:

• Adds compression codec as an additional extension to the exported file. #82 #83

v0.7.0

2020 MAR 27

• Adds import JSON formatted data from AWS Kinesis Streams. #74 #77

v0.6.2

2020 MAR 16

New Features:

• Adds Delta Lake format import integration. #31 #78

Documentation Improvements:

• Adds initial system requirements document for the upcoming Amazon Kinesis
  Connector. #72 #73
• Improve the user guide document with an additional Exasol connection object
  usage example. #66 #76

Development Improvements:

• Updated dependencies to make the project run with IntelliJ IDEA. #70 #71

v0.6.2-rc1

Release candidate version v0.6.2-rc1

v0.6.1

17 FEB 2020

Changes:

• Add Azure Data Lake Gen2 Storage support #58 #69.

• Fixes a bug with regex pattern matching on the paths #65 #67.

• Adds list of dependencies with their licenses to the README.md #63 #64.

• Refactors code linting and styling checks #51 #62.

• Adds a logo to the project #60.

• Fixes bug on BigDecimal typ cast exception #56 #57.