Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] py_extract_from_source syntax errors for a seemingly correct dbt model sql #103

Open
2 tasks done
kansuke-at-trimble opened this issue May 22, 2024 · 0 comments
Open
2 tasks done
Labels
bug Something isn't working

Comments

@kansuke-at-trimble
Copy link

Is this a new bug in dbt-extractor?

  • I believe this is a new bug in dbt-extractor
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When I am executing py_extract_from_source(), it yields dbt_extractor.ExtractionError: Source Error: Syntax error in source which I have no clue where it's pointing to. The following dbt sql was fed into the function (note, some parts are obfuscated for security reason):

{{
    config(
        materialized="incremental",
        unique_key=['partition_number','partition_offset'],
        incremental_strategy="merge"
    )
}}

-- comment 
WITH final AS (
    SELECT
        produced_at,
        key AS message_key,
        -- comment
        JSON_EXTRACT_SCALAR(message, '$.json.field1') AS error_type,
        JSON_EXTRACT_SCALAR(message, '$.json.field2') AS error_code,
        partition_number,
        partition_offset,
    FROM 
        {{ ref("table1") }}
    WHERE 
        -- comment
        JSON_EXTRACT(message, '$.field3') IS NOT NULL
        {% if is_incremental() %}
        -- comment
        AND produced_at > (SELECT TIMESTAMP_SUB(MAX(produced_at), INTERVAL 12 HOUR) FROM {{ this }})
        {% endif %}
)

SELECT 
    * 
FROM 
    final

Expected Behavior

It should yield a dictionary containing all the related information.

Steps To Reproduce

import dbt_extractor

c = """
{{
    config(
        materialized="incremental",
        unique_key=['partition_number','partition_offset'],
        incremental_strategy="merge"
    )
}}

-- comment 
WITH final AS (
    SELECT
        produced_at,
        key AS message_key,
        -- comment
        JSON_EXTRACT_SCALAR(message, '$.json.field1') AS error_type,
        JSON_EXTRACT_SCALAR(message, '$.json.field2') AS error_code,
        partition_number,
        partition_offset,
    FROM 
        {{ ref("table1") }}
    WHERE 
        -- comment
        JSON_EXTRACT(message, '$.field3') IS NOT NULL
        {% if is_incremental() %}
        -- comment
        AND produced_at > (SELECT TIMESTAMP_SUB(MAX(produced_at), INTERVAL 12 HOUR) FROM {{ this }})
        {% endif %}
)

SELECT 
    * 
FROM 
    final

"""
dbt_extractor.py_extract_from_source(c)

Relevant log output

No response

Additional Context

No response

@kansuke-at-trimble kansuke-at-trimble added the bug Something isn't working label May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant