dbt-spark v1.9.0

dbt-spark 1.9.0 - December 10, 2024

Breaking Changes

• Drop support for Python 3.8 (#1121)

Features

• Add tests for cross-database cast macro (#1028)
• Cross-database date macro (#1031)
• Allow configuring snapshot column names (#1096)
• Support custom ODBC connection parameters via connection_string_suffix config (#1092)
• Add Microbatch Strategy to dbt-spark (#1109)
• Enable setting current value of dbt_valid_to (#1112)

Fixes

• Fix incremental python models error where Databricks could not find the temp table transaction logs (#1033)

Under the Hood

• Lazy load agate to improve performance (#1049)
• Isolating distribution testing (#1069)

Dependencies

• Update freezegun requirement from ~=1.3 to ~=1.4 (#966)
• Bump actions/download-artifact from 3 to 4 (#1010)
• Bump actions/upload-artifact from 3 to 4 (#1011)
• Bump dbt-labs/actions from 1.1.0 to 1.1.1 (#1012)
• Update wheel requirement from ~=0.42 to ~=0.43 (#1035)
• Update pytest-xdist requirement from ~=3.5 to ~=3.6 (#1073)
• Loosen pin on pre-commit from 3.7.0 to 3.7 (#1074)

Contributors

• @jpoley (#1092)
• @leahwicz (#1069)
• @nilan3 (#1092)

dbt-spark v1.9.0rc1

dbt-spark 1.9.0-rc1 - December 02, 2024

Breaking Changes

• Drop support for Python 3.8 (#1121)

Features

• Enable setting current value of dbt_valid_to (#1112)

Under the Hood

• Isolating distribution testing (#1069)

Contributors

• @leahwicz (#1069)

dbt-spark v1.9.0b1

dbt-spark 1.9.0-b1 - October 01, 2024

Features

• Add tests for cross-database cast macro (#1028)
• Cross-database date macro (#1031)
• Allow configuring snapshot column names (#1096)
• Support custom ODBC connection parameters via connection_string_suffix config (#1092)
• Add Microbatch Strategy to dbt-spark (#1109)

Fixes

• Fix incremental python models error where Databricks could not find the temp table transaction logs (#1033)

Under the Hood

• Lazy load agate to improve performance (#1049)

Dependencies

• Update freezegun requirement from ~=1.3 to ~=1.4 (#966)
• Bump actions/download-artifact from 3 to 4 (#1010)
• Bump actions/upload-artifact from 3 to 4 (#1011)
• Bump dbt-labs/actions from 1.1.0 to 1.1.1 (#1012)
• Update wheel requirement from ~=0.42 to ~=0.43 (#1035)
• Update pytest-xdist requirement from ~=3.5 to ~=3.6 (#1073)
• Loosen pin on pre-commit from 3.7.0 to 3.7 (#1074)

Contributors

• @jpoley (#1092)
• @nilan3 (#1092)

dbt-spark v1.8.0

dbt-spark 1.8.0 - May 09, 2024

Features

• Implement spark__safe_cast and add functional tests for unit testing (#987)
• : Add new workflow for internal patch releases (#38)

Fixes

• Support new agate Integer type and empty seed test (#935)
• Fix hardcoded file format for python models (#803)

Under the Hood

• Add GitHub action for integration testing and use dagger-io to run tests. Remove CircleCI workflow. (#719)
• Add tests for --empty flag (#949)
• Remove unused invalid_insert_overwrite_delta_msg message (#962)
• Update import paths and list_relations to support decoupling adapters/core (#972)
• Update dependabot config to cover GHA and Docker (#1006)

Dependencies

• Update pre-commit-hooks requirement from ~=4.4 to ~=4.5 (#903)
• Bump mypy from 1.5.1 to 1.6.0 (#904)
• Update pyodbc requirement from ~=4.0.39 to ~=5.0.0 (#905)
• Update pre-commit requirement from ~=3.4 to ~=3.5 (#914)
• Update pyodbc requirement from ~=5.0.0 to ~=5.0.1 (#925)
• Bump mypy from 1.6.0 to 1.6.1 (#926)
• Update black requirement from ~=23.9 to ~=23.10 (#927)
• Update black requirement from ~=23.10 to ~=23.11 (#942)
• Bump mypy from 1.6.1 to 1.7.0 (#946)
• Update pytest-xdist requirement from ~=3.3 to ~=3.4 (#947)
• Update pytest-xdist requirement from ~=3.4 to ~=3.5 (#951)
• Update wheel requirement from ~=0.41 to ~=0.42 (#952)
• Bump mypy from 1.7.0 to 1.7.1 (#953)
• Update freezegun requirement from ~=1.2 to ~=1.3 (#956)
• Update black requirement from ~=23.11 to ~=23.12 (#959)
• Add dbt-core as a dependency to preserve backwards compatibility for installation (#1002)
• Bump actions/setup-python from 4 to 5 (#1013)
• Bump actions/checkout from 3 to 4 (#1014)

Security

• Pin black>=24.3 in dev-requirements.txt (#1000)

Contributors

• @JCZuurmond, (#719)
• @ben-schreiber (#803)

dbt-spark v1.8.0rc1

dbt-spark 1.8.0-rc1 - May 06, 2024

Under the Hood

• Update dependabot config to cover GHA and Docker (#1006)

Dependencies

• Bump actions/setup-python from 4 to 5 (#1013)
• Bump actions/checkout from 3 to 4 (#1014)

dbt-spark v1.8.0b1

dbt-spark 1.8.0-b1 - March 01, 2024

Features

• Implement spark__safe_cast and add functional tests for unit testing (#987)

Fixes

• Support new agate Integer type and empty seed test (#935)
• Fix hardcoded file format for python models (#803)

Under the Hood

• Add GitHub action for integration testing and use dagger-io to run tests. Remove CircleCI workflow. (#719)
• Add tests for --empty flag (#949)
• Remove unused invalid_insert_overwrite_delta_msg message (#962)
• Update import paths and list_relations to support decoupling adapters/core (#972)

Dependencies

• Update pre-commit-hooks requirement from ~=4.4 to ~=4.5 (#903)
• Bump mypy from 1.5.1 to 1.6.0 (#904)
• Update pyodbc requirement from ~=4.0.39 to ~=5.0.0 (#905)
• Update pre-commit requirement from ~=3.4 to ~=3.5 (#914)
• Update pyodbc requirement from ~=5.0.0 to ~=5.0.1 (#925)
• Bump mypy from 1.6.0 to 1.6.1 (#926)
• Update black requirement from ~=23.9 to ~=23.10 (#927)
• Update black requirement from ~=23.10 to ~=23.11 (#942)
• Bump mypy from 1.6.1 to 1.7.0 (#946)
• Update pytest-xdist requirement from ~=3.3 to ~=3.4 (#947)
• Update pytest-xdist requirement from ~=3.4 to ~=3.5 (#951)
• Update wheel requirement from ~=0.41 to ~=0.42 (#952)
• Bump mypy from 1.7.0 to 1.7.1 (#953)
• Update freezegun requirement from ~=1.2 to ~=1.3 (#956)
• Update black requirement from ~=23.11 to ~=23.12 (#959)

Contributors

• @JCZuurmond, (#719)
• @ben-schreiber (#803)

dbt-spark v1.6.2

dbt-spark 1.6.2 - December 13, 2023

Dependencies

• Replace sasl with pure-sasl for PyHive (#818)

dbt-spark v1.5.3

dbt-spark 1.5.3 - December 13, 2023

Features

• Persist Column level comments when creating views (#372)

Dependencies

• Replace sasl with pure-sasl for PyHive (#818)

Contributors

• @jurasan (#372)

dbt-spark v1.6.1

dbt-spark 1.6.1 - November 09, 2023

Features

• Persist Column level comments when creating views (#372)

Fixes

• Support new agate Integer type and empty seed test (#935)

Contributors

• @jurasan (#372)

dbt-spark v1.7.1

dbt-spark 1.7.1 - November 07, 2023

Fixes

• Support new agate Integer type and empty seed test (#935)