Switch pytest-xdist algo to worksteal

[galipremsagar]

Description

This PR switches pytest-xdist distribution algorithm to a much more efficient algorithm worksteal, that will assign any idle pytest worker to pickup remaining pytests.

I see a 25% time savings when this switch is made locally:

`loadscope`:
== 101421 passed, 2115 skipped, 867 xfailed in 1179.48s (0:19:39) ==
`worksteal`:
== 101423 passed, 2115 skipped, 867 xfailed in 891.79s (0:14:51) ==

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[bdice]

Nice! This is a new feature in pytest-xdist 3.2.0, from about a year ago (after I had last investigated the options for --dist). https://pytest-xdist.readthedocs.io/en/latest/changelog.html#pytest-xdist-3-2-0-2023-02-07

I vaguely recall there might have been some issues caused by other --dist options in the past. 🤔 I tried to see if I could find a past issue/PR about this but didn't see one.

[galipremsagar]

Nice! This is a new feature in pytest-xdist 3.2.0, from about a year ago (after I had last investigated the options for --dist). https://pytest-xdist.readthedocs.io/en/latest/changelog.html#pytest-xdist-3-2-0-2023-02-07

I vaguely recall there might have been some issues caused by other --dist options in the past. 🤔 I tried to see if I could find a past issue/PR about this but didn't see one.

Yes, loadscope was the only best distribution I found more than a year ago. While going through pytest-xdist docs today, I found this algorithm which was perfect for our setup.

[bdice]

@galipremsagar For awareness, I have prepared a series of PRs (locally) that will apply the worksteal strategy across RAPIDS repos that support pytest-xdist. I will open those PRs after this one is merged.

[galipremsagar]

@galipremsagar For awareness, I have prepared a series of PRs (locally) that will apply the worksteal strategy across RAPIDS repos that support pytest-xdist. I will open those PRs after this one is merged.

Thanks @bdice ! For reference here are the time savings:
branch-24.04:

======== 101174 passed, 2091 skipped, 1136 xfailed in 853.34s (0:14:13) ========

This PR:

======== 101424 passed, 2115 skipped, 869 xfailed in 689.92s (0:11:29) =========

An entire job that took 27 mins now takes just 17 mins! - 10mins time reduction.

[galipremsagar]

/merge

[vyasr]

That's an amazing speedup @galipremsagar!

@bdice when you make your PRs, can you also double-check and align the parallelism across wheel/conda builds? For example, cuml wheel tests are currently using -n 4 while conda tests use -n 8. Adjusting those can also save us time.

There are larger improvements we could make to unify the testing logic more generally, but for now I'm just focusing on the quick wins!

[bdice]

Follow-up PRs across RAPIDS:

• Switch pytest-xdist algorithm to worksteal cuspatial#1355
• https://github.com/rapidsai/cugraph-ops/pull/609
• Switch pytest-xdist algorithm to worksteal cucim#708
• Switch pytest-xdist algorithm to worksteal cuml#5792
• Switch pytest-xdist algorithm to worksteal cuxfilter#576
• https://github.com/rapidsai/cudf-pandas-integration/pull/100

I'll work on the manual changes needed for each of these and prepare them for review.