Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swap dataproc batch_id declaration to model config #804

Merged
merged 14 commits into from
Aug 11, 2023
Merged

Swap dataproc batch_id declaration to model config #804

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
1e13b6d
swap batch_id declaration to model config
nickozilla Jun 29, 2023
ef3d398
address changie req, fix python submission
nickozilla Jun 30, 2023
711e45a
Merge branch 'main' into python-dbt-model-batch-id
dbeatty10 Jul 12, 2023
222ace6
Update bug that is being resolved
dbeatty10 Jul 12, 2023
4da3436
merge master
nickozilla Jul 16, 2023
6233c52
implement dbeatty's suggestion
nickozilla Jul 16, 2023
01ff3ca
Update .changes/unreleased/Fixes-20230630-092618.yaml
nickozilla Jul 23, 2023
b7ef6ce
Merge branch 'main' into python-dbt-model-batch-id
colin-rogers-dbt Jul 28, 2023
8b57e2a
Add 2 tests
nickozilla Aug 4, 2023
facaaf4
Merge branch 'main' into python-dbt-model-batch-id
dbeatty10 Aug 8, 2023
fab30b3
Merge branch 'main' into python-dbt-model-batch-id
nickozilla Aug 11, 2023
c6dcc2b
Merge branch 'main' into python-dbt-model-batch-id
mikealfare Aug 11, 2023
feebb36
Merge branch 'main' into python-dbt-model-batch-id
colin-rogers-dbt Aug 11, 2023
f26c253
Merge branch 'main' into python-dbt-model-batch-id
colin-rogers-dbt Aug 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/unreleased/Fixes-20230630-092618.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: Change batch_id to model override
time: 2023-06-30T09:26:18.854492+01:00
custom:
Author: nickozilla
Issue: "671"
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 4 additions & 0 deletions dbt/adapters/bigquery/python_submissions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,13 +115,17 @@ def _get_job_client(self) -> dataproc_v1.BatchControllerClient:
client_options=self.client_options, credentials=self.GoogleCredentials
)

def _get_batch_id(self) -> str:
return self.parsed_model["config"].get("batch_id")

def _submit_dataproc_job(self) -> dataproc_v1.types.jobs.Job:
batch = self._configure_batch()
parent = f"projects/{self.credential.execution_project}/locations/{self.credential.dataproc_region}"

request = dataproc_v1.CreateBatchRequest(
parent=parent,
batch=batch,
batch_id=self._get_batch_id(),
)
# make the request
operation = self.job_client.create_batch(request=request) # type: ignore
Expand Down