Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Missing common task parameters for Notebook Task #178

Merged
merged 1 commit into from
Nov 11, 2024
Merged

[BUG] Missing common task parameters for Notebook Task #178

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions brickflow/codegen/databricks_bundle.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -540,6 +540,16 @@ def _build_native_notebook_task(
f"Make sure {task_name} returns a NotebookTask object."
) from e

# Setting common task parameters if present
workflow: Optional[Workflow] = _kwargs.get("workflow")
common_task_parameters = workflow.common_task_parameters if workflow else None

if common_task_parameters:
notebook_task.base_parameters = notebook_task.base_parameters or {}
for k, v in common_task_parameters.items():
if k not in notebook_task.base_parameters:
notebook_task.base_parameters[k] = v

jt = JobsTasks(
**task_settings.to_tf_dict(),
notebook_task=notebook_task,
Expand Down
3 changes: 3 additions & 0 deletions tests/codegen/expected_bundles/dev_bundle_monorepo.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,9 @@ targets:
min_retry_interval_millis: null
notebook_task:
notebook_path: notebooks/notebook_a
base_parameters:
all_tasks1: test
all_tasks3: "123"
retry_on_timeout: null
task_key: notebook_task_a
timeout_seconds: null
Expand Down
3 changes: 3 additions & 0 deletions tests/codegen/expected_bundles/dev_bundle_polyrepo.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,9 @@ targets:
min_retry_interval_millis: null
notebook_task:
notebook_path: notebooks/notebook_a
base_parameters:
all_tasks1: test
all_tasks3: "123"
retry_on_timeout: null
task_key: notebook_task_a
timeout_seconds: null
Expand Down
3 changes: 3 additions & 0 deletions tests/codegen/expected_bundles/dev_bundle_polyrepo_with_auto_libs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,9 @@ targets:
min_retry_interval_millis: null
notebook_task:
notebook_path: notebooks/notebook_a
base_parameters:
all_tasks1: test
all_tasks3: "123"
retry_on_timeout: null
task_key: notebook_task_a
timeout_seconds: null
Expand Down
3 changes: 3 additions & 0 deletions tests/codegen/expected_bundles/local_bundle.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,9 @@
"min_retry_interval_millis": null
"notebook_task":
"notebook_path": "notebooks/notebook_a"
base_parameters:
all_tasks1: test
all_tasks3: "123"
"retry_on_timeout": null
"task_key": "notebook_task_a"
"timeout_seconds": null
Expand Down
3 changes: 3 additions & 0 deletions tests/codegen/expected_bundles/local_bundle_prefix_suffix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,9 @@ targets:
min_retry_interval_millis: null
notebook_task:
notebook_path: notebooks/notebook_a
base_parameters:
all_tasks1: test
all_tasks3: "123"
retry_on_timeout: null
task_key: notebook_task_a
timeout_seconds: null
Expand Down
Loading