Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report SLURM runner status as launch-slurm-runner job outcome #540

Open
yhtang opened this issue Feb 14, 2024 · 0 comments
Open

Report SLURM runner status as launch-slurm-runner job outcome #540

yhtang opened this issue Feb 14, 2024 · 0 comments
Assignees

Comments

@yhtang
Copy link
Collaborator

yhtang commented Feb 14, 2024

We provision GitHub Actions runners for running JAX/Pallas/Levanter A100 unit tests using SLURM jobs that launch ephemeral runner instances inside the job. Currently, the launch-slurm-runner job will succeed as long as it is able to submit the job and wait for the job to finish/terminate. If the runner fails to register, e.g. due to network issue, then launch-slurm-runner would succeed, while the actual unit test job that needs to use the runner will wait forever. Even cancelling and then restarting the workflow cannot work around this, as the launcher job is marked as successfully and won't be rerun.

To fix that, we need the launch-slurm-runner job to query the status of the runner running inside the job. If the runner errors out without being able to handle a job, then the CI job should also fail.

Note the difference between

  1. launch-slurm-runner successfully starts an A100 runner, which picks up a job that run and failed, and
  2. launch-slurm-runner fails to start an A100 runner.

The latter is what we need to address in this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants