dbt Cloud: no asset checks or model metadata

[jairus-m]

Hello!

I'm currently scoping out Dagster for my team. I know Dagster makes the most sense when integrated with dbt-core and that the dbt-core integration has more features, is non-experimental, and has more flexibility. However, my organization just signed up for enterprise this year and has been enjoying CICD/dbt Explore and other quality of life features. In addition, many of our users use the dbt Cloud IDE which is why I'm limited to the dbt Cloud integration.

Having said that, I've been able to get started with and have used load_assets_from_dbt_cloud_job to connect our dbt Cloud assets to other parts of our stack and am able to materialize through Dagster. However, there are two big things that are not present in the integration with dbt Cloud:

1. Tests are not recognized as asset checks
   • They are asset observations and the results of the tests are only logged as event metadata
   • Therefore, there is no real observability of the test results in the UI
2. Model metadata is not surfaced at all to Dagster:
   • None of the following is available in the UI:
     • Raw model SQL
     • Model/column descriptions
     • dbt Tags
     • Compute kinds + storage details
     • Seeds

Because of this, it looks like Dagster may not work for us and our constraints. Is there a future feature roadmap that can implement the addition of these important asset_checks / metadata in the dbt Cloud integration?

Are there any workarounds here? or is the only way forward currently to migrate off dbt Cloud to dbt Core?

[jairus-m]

I have been doing more digging and for metadata, it looks like the dbt Cloud documentation outlines a potential, useful param for load_assets_from_dbt_cloud_job:

https://docs.dagster.io/_apidocs/libraries/dagster-dbt#assets-dbt-cloud:

• node_info_to_definition_metadata_fn (Dict[str, Any] -> Optional[Dict[str, RawMetadataMapping]]) – A function that takes a dictionary of dbt node info and optionally returns a dictionary of metadata to be attached to the corresponding definition. This is added to the default metadata assigned to the node, which consists of the node’s schema (if present).

However, trying to use this as well as looking at the source code, this param does not look to be implemented. I also found a GitHub issue that notes exactly this as well:

• Soure code:
  • https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cloud/asset_defs.py
• Issue:
  • Dagster (cloud) dbt cloud add custom meta data  #18664