Skip to content

Commit

Permalink
Fix walltime computation bug
Browse files Browse the repository at this point in the history
Because we logged finished step events before the current batch
of events were processed, there were situations where the snapshot from a previous
failed realization was out of date.

This solves that problem. There is still a residual problem that will surface if
walltime is to be computed for all steps: If a failure event for a step and its
success event from the subsequent run is in the same batch of events, we can get
a negative walltime computed. This problem is ignored as the walltime computation
is only interesting (currently) for walltimes > 120 seconds, where this cannot
happen (max time between batches is 2 seconds).
  • Loading branch information
berland committed Nov 15, 2024
1 parent 58c1b1f commit 90c7a4c
Showing 1 changed file with 9 additions and 6 deletions.
15 changes: 9 additions & 6 deletions src/ert/ensemble_evaluator/_ensemble.py
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,14 @@ def _log_completed_fm_step(

def update_snapshot(self, events: Sequence[Event]) -> EnsembleSnapshot:
snapshot_mutate_event = EnsembleSnapshot()
for event in events:
snapshot_mutate_event = snapshot_mutate_event.update_from_event(
event, source_snapshot=self.snapshot
)
self.snapshot.merge_snapshot(snapshot_mutate_event)
if self.snapshot.status is not None and self.status != self.snapshot.status:
self.status = self._status_tracker.update_state(self.snapshot.status)

for event in events:
if isinstance(event, (ForwardModelStepSuccess, ForwardModelStepFailure)):
step = (
Expand All @@ -188,12 +196,7 @@ def update_snapshot(self, events: Sequence[Event]) -> EnsembleSnapshot:
.get(event.fm_step)
)
self._log_completed_fm_step(event, step)
snapshot_mutate_event = snapshot_mutate_event.update_from_event(
event, source_snapshot=self.snapshot
)
self.snapshot.merge_snapshot(snapshot_mutate_event)
if self.snapshot.status is not None and self.status != self.snapshot.status:
self.status = self._status_tracker.update_state(self.snapshot.status)

return snapshot_mutate_event

async def send_event(
Expand Down

0 comments on commit 90c7a4c

Please sign in to comment.