Skip to content

Commit

Permalink
Fix: Waiting for instance init sometimes timed out
Browse files Browse the repository at this point in the history
The initialization of an instance can take more time than a program in normal circumstances.

This increases the timeout and uses an exponential timeout function to keep the total number of attempts low.
  • Loading branch information
hoh committed Oct 9, 2023
1 parent 1849b83 commit 837d83a
Showing 1 changed file with 10 additions and 6 deletions.
16 changes: 10 additions & 6 deletions vm_supervisor/vm/firecracker/instance.py
Original file line number Diff line number Diff line change
Expand Up @@ -142,18 +142,22 @@ async def wait_for_init(self) -> None:

ip = ip.split("/", 1)[0]

attempts = 30
timeout_seconds = 2.0
# Use an exponential value for the timeout, in seconds, based on exponential base 1.5
# attempt_timeouts = [math.floor(x**1.5) for x in range(1,16)]
attempt_timeouts = [1, 2, 5, 8, 11, 14, 18, 22, 27, 31, 36, 41, 46, 52, 58]

for attempt in range(attempts):
for timeout_seconds in attempt_timeouts:
try:
await ping(ip, packets=1, timeout=timeout_seconds)
return
except HostNotFoundError:
if attempt < (attempts - 1):
continue
else:
# Only raise if we are on the last attempt
if attempt_timeouts == attempt_timeouts[-1]:
raise
else:
continue
else:
assert False, "The timeout logic is broken, this should never be reached"

async def configure(self):
"""Configure the VM by sending configuration info to it's init"""
Expand Down

0 comments on commit 837d83a

Please sign in to comment.