dbx support for pipelines (delta live tables) #250
Replies: 4 comments 6 replies
-
|
hi @benbauer89 , |
Beta Was this translation helpful? Give feedback.
-
|
I agree it would be a great feature :) @renardeinside what are the main roadblocks? Just curious. |
Beta Was this translation helpful? Give feedback.
-
|
yes, I am also really interested in this feature. Many of my clients are interested in using dbx but also DLTs... it feels like a natural thing that dbx supports pipelines! Can we help somewhere @renardeinside? |
Beta Was this translation helpful? Give feedback.
-
|
Hi everyone on this thread! We're in progress of implementation, most probably DLT support will be delivered until end of first week of October (7-8 approx.) |
Beta Was this translation helpful? Give feedback.
-
|
implemented in #531 will be soonish released in 0.8.0 |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
yes, it definitely does apply the permissions. Please note that they shall be provided on the level of the pipeline definition, e.g.: environments:
default:
workflows:
- name: "[dev][dbx-dlt-test]"
workflow_type: "pipeline"
development: true
libraries:
- notebook:
path: "/Repos/some/path"
clusters:
- label: "default"
autoscale:
min_workers: 1
max_workers: 3
mode: "ENHANCED"
driver_instance_pool_id: "instance-pool://itrusov-pool"
instance_pool_id: "instance-pool://itrusov-pool"
access_control_list:
- user_name: "[email protected]"
permission_level: "IS_OWNER"
- group_name: "some-group"
permission_level: "CAN_VIEW" |
Beta Was this translation helpful? Give feedback.
-
|
Good to know; thanks! I had trouble deploying both the pipeline and the workflow that has a pipeline task at once - when using [“some-pipeline”, “some-workflow"] the workflow cannot find the pipeline. Ended up creating two deployment files. Is that expected?
Christina Taylor
***@***.***
… On Nov 2, 2022, at 12:50 PM, Ivan Trusov ***@***.***> wrote:
yes, it definitely does apply the permissions. Please note that they shall be provided on the level of the pipeline definition, e.g.:
environments:
default:
workflows:
- name: "[dev][dbx-dlt-test]"
workflow_type: "pipeline"
development: true
libraries:
- notebook:
path: "/Repos/some/path"
clusters:
- label: "default"
autoscale:
min_workers: 1
max_workers: 3
mode: "ENHANCED"
driver_instance_pool_id: "instance-pool://itrusov-pool"
instance_pool_id: "instance-pool://itrusov-pool"
access_control_list:
- user_name: ***@***.***"
permission_level: "IS_OWNER"
- group_name: "some-group"
permission_level: "CAN_VIEW"
—
Reply to this email directly, view it on GitHub <https://www.google.com/url?q=https://github.com/databrickslabs/dbx/discussions/250%23discussioncomment-4040964&source=gmail-imap&ust=1668019848000000&usg=AOvVaw2JTBaw3hum1Pp54DjtuOJe>, or unsubscribe <https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AYPPEBEESPZRSINOK7XNVYLWGKZYNANCNFSM5WLEQHCA&source=gmail-imap&ust=1668019848000000&usg=AOvVaw3l7sPJXLsqsLdYX4ZwGCSL>.
You are receiving this because you are subscribed to this thread.
|
Beta Was this translation helpful? Give feedback.
-
|
nope, that's not - I've created #539 to fix that one |
Beta Was this translation helpful? Give feedback.
-
|
Implemented and released in 0.8.0! 🔥 |
Beta Was this translation helpful? Give feedback.
-
Hey, I was wondering whether it's already possible to use dbx to deploy pipelines (delta live tables).
If not it would be a great feature :)
Beta Was this translation helpful? Give feedback.
All reactions