Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Cloud fleet instance gets stuck in unreachable state #2041

Open
un-def opened this issue Nov 28, 2024 · 0 comments · May be fixed by #2043
Open

[Bug]: Cloud fleet instance gets stuck in unreachable state #2041

un-def opened this issue Nov 28, 2024 · 0 comments · May be fixed by #2043
Assignees
@un-def
Labels
bug Something isn't working fleets

Comments

@un-def
Copy link
Collaborator

un-def commented Nov 28, 2024 •
edited
Loading

Steps to reproduce

  1. Create a cloud fleet with termination_policy: destroy-after-idle (the default value)
  2. Start a run using this fleet
  3. Imitate server <-> instance connectivity issue, e.g., disconnect the server from network
  4. Wait for ≈1 minute, restore connectivity

Actual behaviour

  1. The run is terminated: job status is FAILED, reason: INTERRUPTED_BY_NO_CAPACITY, run status has changed TERMINATING -> FAILED, reason: JOB_FAILED
  2. The instance is marked as unreachable: idle (unreachable) and is stuck in this state
  3. The instance cannot be reused, the only options are either delete the fleet manually or wait until it's terminated by the server due to idle timeout

Expected behaviour

No response

dstack version

0.18.28

Server logs

No response

Additional information

No response
@un-def un-def added bug Something isn't working fleets labels Nov 28, 2024
@un-def un-def self-assigned this Nov 28, 2024
un-def added a commit that referenced this issue Nov 28, 2024
@un-def
Recover unreachable instances
fd3d13f 
* Check instances regardless of termination_policy -- gives unreachable
  instances a chance to become reachable again
* Stop a previous job container possibly still running on the insance when
  submitting a new one

Fixes: #2041
@un-def un-def linked a pull request Nov 28, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@un-def un-def

Labels
bug Something isn't working fleets
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Recover unreachable instances dstackai/dstack
1 participant
@un-def