Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] raft v24.10 #2460

Merged
merged 40 commits into from
Oct 9, 2024
Merged

[RELEASE] raft v24.10 #2460

merged 40 commits into from
Oct 9, 2024

Conversation

raydouglass
Copy link
Member

❄️ Code freeze for branch-24.10 and v24.10 release

What does this mean?

Only critical/hotfix level issues should be merged into branch-24.10 until release (merging of this PR).

What is the purpose of this PR?

  • Update documentation
  • Allow testing for the new release
  • Enable a means to merge branch-24.10 into main for the release

raydouglass and others added 30 commits July 19, 2024 17:26
@raydouglass
DOC v24.10 Updates [skip ci]
a7730b1
@GPUtester
Merge pull request #2387 from rapidsai/branch-24.08
9ea5e22 
Forward-merge branch-24.08 into branch-24.10
@GPUtester
Merge pull request #2390 from rapidsai/branch-24.08
9a3b7d1 
Forward-merge branch-24.08 into branch-24.10
@GPUtester
Merge pull request #2391 from rapidsai/branch-24.08
80e68cb 
Forward-merge branch-24.08 into branch-24.10
@GPUtester
Merge pull request #2392 from rapidsai/branch-24.08
b4caea0 
Forward-merge branch-24.08 into branch-24.10
@GPUtester
Merge pull request #2393 from rapidsai/branch-24.08
11a49c8 
Forward-merge branch-24.08 into branch-24.10
@jameslamb
Merge branch-24.08 into branch-24.10
7179cc2
@raydouglass
Merge pull request #2397 from jameslamb/branch-24.10-merge-24.08
1682700 
Merge branch-24.08 into branch-24.10
@raydouglass
Merge pull request #2395 from rapidsai/branch-24.08
401935b 
Forward-merge branch-24.08 into branch-24.10
@jameslamb
Update pip devcontainers to UCX v1.17.0 (#2401)
f520235 
Contributes to rapidsai/build-planning#77.

Follow-up to rapidsai/devcontainers#338

Proposes updating the pip devcontainers to use v1.17.0, as part of an effort to use that version across RAPIDS.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)
  - Peter Andreas Entschev (https://github.com/pentschev)

URL: #2401
@GPUtester
Merge pull request #2407 from rapidsai/branch-24.08
e993320 
Forward-merge branch-24.08 into branch-24.10
@bdice
Improve update-version.sh (#2408)
dbdde82 
This PR improves `update-version.sh` by clarifying suffix handling and ucxx/ucx-py version handling.

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #2408
@jameslamb
Use tool.scikit-build.cmake.version, set scikit-build-core minimum-ve…
a1f4844 
…rsion (#2406)

Contributes to rapidsai/build-planning#58.

`scikit-build-core==0.10.0` was released today (https://github.com/scikit-build/scikit-build-core/releases/tag/v0.10.0), and wheel-building configurations across RAPIDS are incompatible with it.

This proposes upgrading to that version and fixing configuration here in a way that:

* is compatible with that new `scikit-build-core` version
* takes advantage of the forward-compatibility mechanism (`minimum-version`) that `scikit-build-core` provides, to reduce the risk of needing to do this again in the future

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - https://github.com/jakirkham

URL: #2406
@KyleFromNVIDIA
Update pre-commit hooks (#2409)
32f3703 
This PR updates pre-commit hooks to the latest versions that are supported without causing style check errors.

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #2409
@rhdong
[FEA] Support for half-float mixed precise in brute-force (#2382)
db07998 
- distance supports half-float
- SDDMM support half-float
- gemm supports multi-type compose
- transpose & copy support half
- random supports half

Authors:
  - rhdong (https://github.com/rhdong)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #2382
@KyleFromNVIDIA
Use CUDA math wheels (#2415)
f1cc0fb 
Use CUDA math wheels to reduce wheel size by not statically linking CUDA math libraries.

Contributes to rapidsai/build-planning#35

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - Robert Maynard (https://github.com/robertmaynard)
  - Bradley Dice (https://github.com/bdice)
  - James Lamb (https://github.com/jameslamb)

URL: #2415
@jinsolp
[FEA] Batching NN Descent (#2403)
2f587b1 
This PR implements batching NN Descent. It will be helpful for reducing device memory usages for large datasets (specifically if the dataset is kept on host).
`index_params` now has...
- `n_clusters`: number of clusters to make. Larger clusters reduce device memory usage. Default is 1, in which case it doesn't do the batched NND.

### Notes
- The batching approach may have duplicate indices in the knn graph (in rare cases) because sometimes distances calculated for the same pair may be slightly different. This results in putting the same index far apart after sorting by distances, making it difficult to get unique indices (which is done by looking at 2 indices before the current one). 
  - handled by adding a `max_duplicates` for `check_unique_indices` in tests

### Benchmarks
- Dataset for NND (no batch and batch) is on host
- Dataset for brute force knn is on device (but still won't be able to run with large datasets even if the data is put on the host because it brings the entire dataset to device anyway)
- The dataset is just a slice of the wiki-all dataset (88M, 768) to test for different sizes
<img width="773" alt="Screenshot 2024-08-02 at 8 35 58 AM" src="https://github.com/user-attachments/assets/2d2d4bb2-a7a9-4731-9b3d-7d0eccf875ea">

Authors:
  - Jinsol Park (https://github.com/jinsolp)

Approvers:
  - Divye Gala (https://github.com/divyegala)
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #2403
@seberg
Remove NumPy <2 pin (#2414)
2dafe79 
This PR removes the NumPy<2 pin which is expected to work for
RAPIDS projects once CuPy 13.3.0 is released (CuPy 13.2.0 had
some issues preventing the use with NumPy 2).

Authors:
  - Sebastian Berg (https://github.com/seberg)
  - https://github.com/jakirkham

Approvers:
  - James Lamb (https://github.com/jameslamb)
  - https://github.com/jakirkham

URL: #2414
@jameslamb
Drop Python 3.9 support (#2417)
73dc36c 
Contributes to rapidsai/build-planning#88

Finishes the work of dropping Python 3.9 support.

This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235.
This PR updates configuration and docs to reflect that.

## Notes for Reviewers

### How I tested this

Checked that there were no remaining uses like this:

```shell
git grep -E '3\.9'
git grep '39'
git grep 'py39'
```

And similar for variations on Python 3.8 (to catch things that were missed the last time this was done).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - https://github.com/jakirkham

URL: #2417
@KyleFromNVIDIA
Update rapidsai/pre-commit-hooks (#2420)
e17dcdf 
This PR updates rapidsai/pre-commit-hooks to the version 0.4.0.

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #2420
@robertmaynard
Exclude any kernel symbol that uses cutlass (#2425)
e04a640 
Authors:
  - Robert Maynard (https://github.com/robertmaynard)

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

URL: #2425
@jameslamb
Add support for Python 3.12 (#2428)
8a71b98 
Contributes to rapidsai/build-planning#40

This PR adds support for Python 3.12.

## Notes for Reviewers

This is part of ongoing work to add Python 3.12 support across RAPIDS.
It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213.

A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all
RAPIDS repos have added Python 3.12 support.

### This will fail until all dependencies have been updates to Python 3.12

CI here is expected to fail until all of this project's upstream dependencies support Python 3.12.

This can be merged whenever all CI jobs are passing.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #2428
@lowener
[BUG] Fix bitset function visibility (#2429)
ac53a0f 
`raft::ceildiv` is also being replaced with `raft::div_rounding_up_safe` to avoid including CUDA headers when not needed.

Authors:
  - Micka (https://github.com/lowener)

Approvers:
  - rhdong (https://github.com/rhdong)
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #2429
@benfred
Allow coo_sort to work on int64_t indices (#2432)
2071841 
Authors:
  - Ben Frederickson (https://github.com/benfred)

Approvers:
  - Divye Gala (https://github.com/divyegala)

URL: #2432
@bdice
Update to flake8 7.1.1. (#2435)
e60cd7d 
We need to update flake8 to fix a false-positive that appears with older flake8 versions on Python 3.12.

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #2435
@jameslamb
Use CI workflow branch 'branch-24.10' again [skip ci] (#2437)
c5dcba0
@raydouglass
Fix sed syntax [skip-ci] (#2441)
cd02a8f
@lowener
Use runtime check of cudart version for eig (#2430)
f49567e 
Authors:
  - Micka (https://github.com/lowener)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #2430
@jameslamb
Update fmt (to 11.0.2) and spdlog (to 1.14.1). (#2433)
b79f15d 
* Update fmt (to 11.0.2) and spdlog (to 1.14.1).

* use rmm and ucxx CI artifacts

* try using librmm wheels

* try again to use rmm wheels

* rapids-get-pr-wheel-artifact was missing RAPIDS_PY_WHEEL_NAME

* ok you do not need to provide the Python slug yourself for rapids-get-pr-wheel-artifact

* constraints need 'file://' protocol

* try suppressing unreachable-code diagnostics from nvcc (this should be narrowed down / upstreamed before merging)

* fix checks

* Revert "try suppressing unreachable-code diagnostics from nvcc (this should be narrowed down / upstreamed before merging)"

This reverts commit 3ba2201.

* copyright

* move rapids-cmake overrides [skip ci]

* kick off a build

* fix dependency graph

* devcontainer

* run all CI

* remove testing-only changes [skip ci]
@AyodeAwe
update update-version.sh to use packaging lib (#2447)
878cefc
galipremsagar and others added 9 commits September 25, 2024 18:49
@galipremsagar
Switch traceback to native (#2446)
17757b8 
In cudf we have observed a ~10% speed up of pytest suite execution by switching pytest traceback to `--native`:

```
currently:

102474 passed, 2117 skipped, 902 xfailed in 892.16s (0:14:52)

--tb=short:

102474 passed, 2117 skipped, 902 xfailed in 898.99s (0:14:58)

--tb=no:

102474 passed, 2117 skipped, 902 xfailed in 815.98s (0:13:35)

--tb=native:

102474 passed, 2117 skipped, 902 xfailed in 820.92s (0:13:40)
```

This PR makes a similar change to `raft` repo.

xref: rapidsai/cudf#16851

Authors:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #2446
@divyegala
Disable NN Descent Batch tests temporarily (#2453)
0f8b097 
Linked issue #2450

Authors:
  - Divye Gala (https://github.com/divyegala)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #2453
@trxcllnt
Add missing cuda_suffixed: true (#2440)
2e705fb 
This change ensures `rmm-cuXX` is not in the package dependencies when building the pylibraft wheel with `-C "rapidsai.matrix-entry=cuda_suffixed=false`.

Authors:
  - Paul Taylor (https://github.com/trxcllnt)
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #2440
@rhdong
[Feat] add repeat, sparsity, eval_n_elements APIs to bitset (#…
0284b42 
…2439)

- This PR is a part of the feature that applies the prefilter brute-force in Cagra.

Authors:
  - rhdong (https://github.com/rhdong)
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Micka (https://github.com/lowener)

URL: #2439
@jameslamb
bump NCCL floor to 2.18.1.1 (#2443)
f37c41c 
Contributes to rapidsai/build-planning#102

Some RAPIDS libraries are using `ncclCommSplit()`, which was introduced in `nccl==2.18.1.1`. This is part of a series of PRs across RAPIDS updating libraries' pins to `nccl>=2.18.1.1` to ensure they get a new-enough version that supports that.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #2443
@rhdong
[Opt] Enforce the UT Coverity and add benchmark for transpose (#2438)
704feb1 
Authors:
  - rhdong (https://github.com/rhdong)

Approvers:
  - Artem M. Chirkin (https://github.com/achirkin)
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #2438
@viclafargue
Adding NCCL clique to the RAFT handle (#2431)
b78a07d 
Authors:
  - Victor Lafargue (https://github.com/viclafargue)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #2431
@jameslamb
bump NCCL floor to 2.19 (#2458)
5ee0e79 
Follow-up to #2443

As part of the work to support NumPy 2 across RAPIDS, we found reason to upgrade some libraries like `cugraph` to slightly newer NCCL (`>=2.19`). Context: rapidsai/build-planning#102 (comment)

This applies that same bump here, to keep the range of NCCL versions consistent across RAPIDS.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - https://github.com/jakirkham
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #2458
@cjnolet
Deprecating vector search APIs and updating README accordingly (#2448)
6c4fdfb 
I opted to deprecate just the necessary pieces, such as the `index` classes instead of deprecating every single function.

Authors:
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Ben Frederickson (https://github.com/benfred)

URL: #2448
@raydouglass raydouglass requested review from a team as code owners September 27, 2024 14:36
@raydouglass raydouglass requested review from bdice and removed request for a team September 27, 2024 14:36
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Sep 27, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@raydouglass raydouglass merged commit 226c82e into main Oct 9, 2024
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bdice bdice Awaiting requested review from bdice bdice is a code owner automatically assigned from rapidsai/packaging-codeowners

Assignees
No one assigned
Labels
ci CMake cpp python
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.