Skip to content

Commit

Permalink
Remove instances from instance manager if terminated before starting
Browse files Browse the repository at this point in the history
Ensure that if an instance is terminated before its Propolis zone is started,
it is removed from the instance manager's instance map so that the instance can
be restarted on that sled.

Found via omicron-stress, which is much happier with this fix in place.

Also improve a little bit of tracing.
  • Loading branch information
gjcolombo committed Oct 3, 2023
1 parent f635ba1 commit 71dc6e9
Show file tree
Hide file tree
Showing 3 changed files with 7 additions and 2 deletions.
2 changes: 1 addition & 1 deletion nexus/src/app/instance.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1133,7 +1133,7 @@ impl super::Nexus {

error!(
self.log,
"failed to set instance to Failed after bad put";
"attempted to set instance to Failed after bad put";
"instance_id" => %instance_id,
"result" => ?result,
);
Expand Down
3 changes: 2 additions & 1 deletion nexus/src/app/sagas/instance_start.rs
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,8 @@ async fn sis_move_to_starting(
let instance_id = params.db_instance.id();
let propolis_id = sagactx.lookup::<Uuid>("propolis_id")?;
info!(osagactx.log(), "moving instance to Starting state via saga";
"instance_id" => %instance_id);
"instance_id" => %instance_id,
"propolis_id" => %propolis_id);

let opctx = crate::context::op_context_for_saga_action(
&sagactx,
Expand Down
4 changes: 4 additions & 0 deletions sled-agent/src/instance.rs
Original file line number Diff line number Diff line change
Expand Up @@ -506,6 +506,10 @@ impl InstanceInner {
self.log,
"Instance::terminate() called with no running state"
);

// Ensure the instance is removed from the instance manager's table
// so that a new instance can take its place.
self.instance_ticket.terminate();
return Ok(());
};

Expand Down

0 comments on commit 71dc6e9

Please sign in to comment.