Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add test cases for scaling with arrangement and no shuffle backfilling #17015

Open
lmatz opened this issue May 30, 2024 · 0 comments
Open

add test cases for scaling with arrangement and no shuffle backfilling #17015

lmatz opened this issue May 30, 2024 · 0 comments

Comments

@lmatz
Copy link
Contributor

lmatz commented May 30, 2024

I am going through all the recent incidents on Notion to see if we can enhance the tests.
Is it possible to add such a case to the existing deterministic scale test?

          > Any test to repro this case?

This is a sporadic issue that can be reproduced as follows:

  1. Create a basic job.
  2. Use arrangement backfill and no shuffle backfill to create two secondary jobs.
  3. Then, shut down the meta and try to scale up the compute nodes.
  4. Start the meta. At this point, if the job on the no shuffle backfill path undergoes scale out during the recovery phase first, it can cause misjudgment for the job on the arrangement backfill path.

Originally posted by @shanicky in #16923 (comment)

@github-actions github-actions bot added this to the release-1.10 milestone May 30, 2024
@lmatz lmatz removed this from the release-1.10 milestone Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant