Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Microbatch first last batch serial #11072

Merged
merged 21 commits into from
Dec 7, 2024
Merged
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
0d61609
microbatch: split out first and last batch to run in serial
MichelleArk Nov 28, 2024
bec5d57
use Task.get_runner
MichelleArk Nov 28, 2024
32002ea
only run pre_hook on first batch, post_hook on last batch
MichelleArk Nov 28, 2024
37dbd11
refactor: internalize parallel to RunTask._submit_batch
MichelleArk Nov 28, 2024
23271b9
Add optional `force_sequential` to `_submit_batch` to allow for skipp…
QMalcolm Dec 4, 2024
9c903f7
Force last batch to run sequentially
QMalcolm Dec 4, 2024
e624057
Force first batch to run sequentially
QMalcolm Dec 4, 2024
0c2e327
Fixup
QMalcolm Dec 4, 2024
a37f3a6
Remove batch_idx check in `should_run_in_parallel`
QMalcolm Dec 4, 2024
cfe1dcf
Begin skipping batches if first batch fails
QMalcolm Dec 6, 2024
8424209
Write custom `on_skip` for `MicrobatchModelRunner` to better handle w…
QMalcolm Dec 6, 2024
50abb45
Add microbatch pre-hook, post-hook, and sequential first/last batch t…
QMalcolm Dec 6, 2024
9bc3816
Fix/Add tests around first batch failure vs latter batch failure
QMalcolm Dec 6, 2024
7699f52
Fix MicrobatchModelRunner.on_skip to handle skipping the entire node
QMalcolm Dec 6, 2024
74a76c4
Fix conditional logic for setting pre and post hooks for batches
QMalcolm Dec 6, 2024
0c16d07
Revert back to using the MicrobatchModelRunner initializer directly
QMalcolm Dec 6, 2024
3fa7bbc
Add two new event types `LogStartBatch` and `LogBatchResult`
QMalcolm Dec 7, 2024
f4315ef
Update MicrobatchModelRunner to use new batch specific log events
QMalcolm Dec 7, 2024
bec65b8
Fix event testing
QMalcolm Dec 7, 2024
c0743e0
Update microbatch integrationt tests to catch batch specific event types
QMalcolm Dec 7, 2024
0198741
Add changie doc
QMalcolm Dec 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Force last batch to run sequentially
QMalcolm committed Dec 4, 2024
commit 9c903f7efa10c08996746554aa6ed4344cfa2dad
10 changes: 9 additions & 1 deletion core/dbt/task/run.py
Original file line number Diff line number Diff line change
@@ -726,7 +726,15 @@ def handle_microbatch_model(
while len(batch_results) != batch_idx:
pass
# Final batch runs once all others complete to ensure post_hook runs at the end
self._submit_batch(node, relation_exists, batches, batch_idx, batch_results, pool)
self._submit_batch(
node,
relation_exists,
batches,
batch_idx,
batch_results,
pool,
force_sequential_run=True,
)

# Finalize run: merge results, track model run, and print final result line
runner.merge_batch_results(result, batch_results)