Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump dbt version to 1.4 #144

Closed
wants to merge 5 commits into from
Closed

Bump dbt version to 1.4 #144

wants to merge 5 commits into from

Conversation

geoHeil
Copy link

@geoHeil geoHeil commented Feb 27, 2023

bump DBT version

@geoHeil geoHeil mentioned this pull request Feb 27, 2023
4 tasks
@geoHeil geoHeil changed the title Bump dbt version Bump dbt version to 1.4 Feb 28, 2023
@geoHeil
Copy link
Author

geoHeil commented Feb 28, 2023

something is not yet fully working:

dbt compile --target dev
15:45:48  Encountered an error:
'_MISSING_TYPE' object is not callable
15:45:48  Traceback (most recent call last):
  File "python3.10/site-packages/dbt/main.py", line 135, in main
    results, succeeded = handle_and_check(args)
  File "python3.10/site-packages/dbt/main.py", line 198, in handle_and_check
    task, res = run_from_args(parsed)
  File "python3.10/site-packages/dbt/main.py", line 225, in run_from_args
    task = parsed.cls.from_args(args=parsed)
  File "python3.10/site-packages/dbt/task/base.py", line 163, in from_args
    return super().from_args(args)
  File "python3.10/site-packages/dbt/task/base.py", line 98, in from_args
    config = cls.ConfigType.from_args(args)
  File "python3.10/site-packages/dbt/config/runtime.py", line 265, in from_args
    return cls.from_parts(
  File "python3.10/site-packages/dbt/config/runtime.py", line 83, in from_parts
    .get_default_quote_policy()
  File "python3.10/site-packages/dbt/adapters/base/relation.py", line 60, in get_default_quote_policy
    return cls._get_field_named("quote_policy").default_factory()
TypeError: '_MISSING_TYPE' object is not callable

@geoHeil
Copy link
Author

geoHeil commented Feb 28, 2023

dbt-labs/dbt-core#6451

Also for 1.4.1 of DBT core this issue persists. How can this be fixed?

@dbeaty suggested the mainline master branch of DBT for dbt-labs/dbt-core#6451 a while ago

@geoHeil
Copy link
Author

geoHeil commented Mar 2, 2023

This issue is fixed now and it looks like dbt-impala is compatible with dbt 1.4.x now.

@geoHeil
Copy link
Author

geoHeil commented Mar 2, 2023

However, I still cannot use dbt-impala due to #146 (for both 1.3.1 and 1.4.0)

@geoHeil
Copy link
Author

geoHeil commented Mar 6, 2023

File "python3.10/site-packages/dbt/adapters/base/impl.py", line 691, in expand_target_column_types
    self.expand_column_types(from_relation, to_relation)
  File "python3.10/site-packages/dbt/adapters/sql/impl.py", line 98, in expand_column_types
    reference_columns = {c.name: c for c in self.get_columns_in_relation(goal)}
  File "python3.10/site-packages/dbt/adapters/impala/impl.py", line 163, in get_columns_in_relation
    columns = self.parse_describe_extended(relation, rows)
  File "python3.10/site-packages/dbt/adapters/impala/impl.py", line 187, in parse_describe_extended
    pos = ImpalaAdapter.find_table_information_separator(dict_rows)  # ensure that the class method is called
  File "python3.10/site-packages/dbt/adapters/impala/impl.py", line 219, in find_table_information_separator
    if row['name'].startswith('# Detailed Table Information'):
KeyError: 'name'

is still open for incremental models

@geoHeil
Copy link
Author

geoHeil commented Mar 6, 2023

After changing all references of row['name'] to row['col_name'] and row['type'] to row['data_type'] , most of these errors go away. But, the generated SQL code is weird:

strange output:

insert overwrite heilerg_db.example (foo, Detailed Table Information) partition(baz)
    select foo, Detailed Table Information
    from heilerg_db.example__dbt_tmp

most likely expected output:

insert overwrite heilerg_db.example (bar) partition(baz)
            select bar
            from heilerg_db.foo

input:

{{
    config(
        materialized='incremental',
        incremental_strategy = 'insert_overwrite',
        partition_by = 'baz',
    )
}}
{% set partition_key = '2023-01-01' %}

SELECT * FROM heilerg_db.foo

{% if is_incremental() %}

  -- this filter will only be applied on an incremental run
  WHERE baz = '{{ partition_key }}'

{% endif %}

Observations:

  • why is a non existing column foo compiled into the select foo?
  • why is detailed table observations rendered into the SQL and killing the SQL?

@geoHeil
Copy link
Author

geoHeil commented Mar 14, 2023

@hpasumarthi can you take a look at this?

@myloginid myloginid requested a review from niteshy May 14, 2023 07:10
@niteshy
Copy link
Collaborator

niteshy commented Jun 15, 2023

This is closed by #175

@niteshy niteshy closed this Jun 15, 2023
@niteshy niteshy self-assigned this Jun 15, 2023
@niteshy niteshy added the duplicate This issue or pull request already exists label Jun 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue or pull request already exists
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants