-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support render for versioned dbt models. #516
Conversation
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@binhnq94, thanks a lot for your contribution!
I will still read the docs to understand more about aliases and versions, but I have two initial questions:
- Would there be any benefit in introducing a new attribute to
DbtNode
calledalias
instead of overloading the propertyname
with two possible sources? We could also introduce anidentifier
if it helped. What are the pros and cons of overridingname
? - When we render dbt nodes as Airflow nodes, what would be the benefit of rendering models using aliases instead of their names?
Codecov ReportAll modified lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #516 +/- ##
=======================================
Coverage 92.83% 92.83%
=======================================
Files 49 49
Lines 1983 1983
=======================================
Hits 1841 1841
Misses 142 142
☔ View full report in Codecov by Sentry. |
|
c107c07
to
80be88d
Compare
@binhnq94 let's aim to have this change released as part of 1.2. Please, let me know if you'd like any further support on anything - I'm looking forward to merging this change. |
67eda36
to
ce3d0eb
Compare
8e15561
to
deb9964
Compare
@tatiana I have already fixed the fail in integrate tests (sorry, last time I only run model_version dag only.) |
**Features** * Add support to model versioning available since dbt 1.6 by @binhnq94 in #516 * Add AWS Athena profile mapping by @benjamin-awd in #578 * Support customizing how dbt nodes are converted to Airflow by @tatiana in #503 * Make the arg ``dbt_project_path`` in the ``ProjectConfig`` optional by @MrBones757 in #581 **Bug fixes** * Fix Cosmos custom selector to support filtering a single model by @jlaneve and @harels in #576 * Fix using ``GoogleCloudServiceAccountDictProfileMapping`` together with ``LoadMethod.DBT_LS`` by @joppevos in #587 * Fix using the ``full_refresh`` argument in projects that contain tests by @EgorSemenov and @tatiana in #590 * Stop creating symbolic links for ``dbt_packages`` (solves ``LocalExecutor`` concurrency issue) by @tatiana in #600 **Others** * Docs: add reference to original Jaffle Shop project by @erdos2n in #583 * Docs: retries & note about DagBag error by @TJaniF in #592 * pre-commit updates in #575 and #585
This was released as part of 1.2: |
Description
Model version is supported by dbt-core >= 1.5.
As cosmos v1.1.0a1, it not support to render task for model have versions.
In this PR I propose to use
node_dict["alias"]
asDbtNode.name
instead ofnode_dict["name"]
ifnode_dict["resource_type"]
==model
.This PR also help cosmos render the names of nodes to be correctly the same as
identifier
if users config alias or custom thegenerate_alias_name
macro. Refer: alias document and thegenerate_alias_name
implementation.Related Issue(s)
#260
Breaking Change?
When model have version config, each model version will create a database relation with alias
<model_name>_v<v>
, for examples:dim_customers_v1
,dim_customers_v2
...So I expected that cosmos will created run/test tasks for each versions, for example: dim_customers_v1_run, dim_customers_v2_run, dim_customers_v1_test, dim_customers_v2_test...
The second expectation is the dag will render right depedences, for example: stg_customers_v1 >> customers_v1, stg_customers_v2 >> customer_v2...
Checklist