Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix hardcoded file format in python materializations #955

Merged
merged 2 commits into from
Dec 21, 2023
Merged

Fix hardcoded file format in python materializations #955

merged 2 commits into from
Dec 21, 2023

Conversation

ben-schreiber
Copy link
Contributor

resolves #803
docs dbt-labs/docs.getdbt.com/#

Problem

Solves the bug in #803

Solution

Maintains delta as the default file_format

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

@ben-schreiber ben-schreiber requested a review from a team as a code owner December 3, 2023 18:37
@cla-bot cla-bot bot added the cla:yes label Dec 3, 2023
@colin-rogers-dbt
Copy link
Contributor

@ben-schreiber thanks can you add a changelog?

@colin-rogers-dbt colin-rogers-dbt self-assigned this Dec 20, 2023
@ben-schreiber ben-schreiber force-pushed the bugfix/python-materialize-file-format branch from dce720e to faeb729 Compare December 21, 2023 06:22
@ben-schreiber
Copy link
Contributor Author

@colin-rogers-dbt ✅

colin-rogers-dbt
colin-rogers-dbt reviewed Dec 21, 2023
View reviewed changes
dbt/include/spark/macros/materializations/table.sql
@@ -98,7 +98,7 @@ else:
msg = f"{type(df)} is not a supported type for dbt Python materialization"
raise Exception(msg)

df.write.mode("overwrite").format("delta").option("overwriteSchema", "true").saveAsTable("{{ target_relation }}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benc-db any issues with making this configurable for dbt-databricks?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It already is: https://github.com/databricks/dbt-databricks/blob/e28ced5d32b8f2102284588a0dd293b142f792c6/dbt/include/databricks/macros/adapters/python.sql#L56

@colin-rogers-dbt colin-rogers-dbt merged commit 5210d0a into dbt-labs:main Dec 21, 2023
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@benc-db benc-db benc-db left review comments

@colin-rogers-dbt colin-rogers-dbt colin-rogers-dbt approved these changes

@mikealfare mikealfare Awaiting requested review from mikealfare mikealfare is a code owner automatically assigned from dbt-labs/adapters

Assignees

@colin-rogers-dbt colin-rogers-dbt

Labels
cla:yes ok to test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[ADAP 619][Bug] Unable to materialize table in anything else than delta
3 participants
@ben-schreiber @colin-rogers-dbt @benc-db