[FEATURE] Add support for spark processing

[rishabh1815769]

1. We are trying to build lakehouse architecture on AWS S3 using dbtvault and dbt-glue. for this approach we need support with spark processing with dbtvault plugin. Currently, dbtvault only has support for snowflake, bigquery.
2. Are there any future plans or timelines on this feature?

AB#5367

[DVAlexHiggs]

Hi. Where are you reading that dbtvault only supports BigQuery and Snowflake please? We currently support Postgres and Databricks as well. You should be able to use the databricks macros in place of spark

[rishabh1815769]

Hi Alex, Thank you for the quick response. Could you please direct me toward documentation regarding dbtvautl usage with databricks or spark preferably? Thank you. Regards

…

On Thu, Feb 16, 2023 at 3:47 PM Alex Higgs ***@***.***> wrote: Hi. Where are you reading that dbtvault only supports BigQuery and Snowflake please? We currently support Postgres and Databricks as well. You should be able to use the databricks macros in place of spark — Reply to this email directly, view it on GitHub <#185 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOBFD5YF7H5J43CMWCHEGY3WXY4W7ANCNFSM6AAAAAAU6INLOU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[DVAlexHiggs]

Hi Alex, Thank you for the quick response. Could you please direct me toward documentation regarding dbtvautl usage with databricks or spark preferably? Thank you. Regards
…
On Thu, Feb 16, 2023 at 3:47 PM Alex Higgs @.> wrote: Hi. Where are you reading that dbtvault only supports BigQuery and Snowflake please? We currently support Postgres and Databricks as well. You should be able to use the databricks macros in place of spark — Reply to this email directly, view it on GitHub <#185 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOBFD5YF7H5J43CMWCHEGY3WXY4W7ANCNFSM6AAAAAAU6INLOU . You are receiving this because you authored the thread.Message ID: @.>

There is no databricks specific documentation, as dbtvault uses dbt's multi-dispatch to work cross-platform. You may use dbtvault as-is with a databricks connection for Hubs, Links and Satellites at this time. We will be adding support for more structures in the future