[RELEASE] raft v24.08 #2399
Merged
[RELEASE] raft v24.08 #2399
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Forward-merge branch-24.06 into branch-24.08
Forward-merge branch-24.06 into branch-24.08
Forward-merge branch-24.06 into branch-24.08
Forward-merge branch-24.06 into branch-24.08
Forward-merge branch-24.06 into branch-24.08
Forward-merge branch-24.06 into branch-24.08
Forward-merge branch-24.06 into branch-24.08
Forward-merge branch-24.06 into branch-24.08
Forward-merge branch-24.06 into branch-24.08
Contributes to rapidsai/build-planning#31 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Corey J. Nolet (https://github.com/cjnolet) URL: #2331
Contributes to rapidsai/build-planning#62. It looks like this some of this project's `conda` recipes have unnecessary dependencies on `setuptools`. I suspect those are left over from before the project was cut over to `scikit-build-core`. ## Notes for Reviewers How I confirmed there were no direct uses of `setuptools` in `pylibraft` and `raft-dask`: ```shell git grep -i setuptools ``` Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Jake Awe (https://github.com/AyodeAwe) URL: #2343
… run, fix wheel dependencies (#2349) Fixes #2348 #2331 introduced `rapids-build-backend` (https://github.com/rapidsai/rapids-build-backend) as the build backend for `pylibraft`, `raft-dask`, and `raft-ann-bench`. That library handles automatically modifying a wheel's dependencies based on the target CUDA version. Unfortunately, we missed a few cases in #2331, and as a result the last few days of nightly `raft-dask` wheels had the following issues: * depending on `pylibraft` - *(does not exist, it's called `pylibraft-cu12`)* * depending on `ucx-py==0.39.*` - *(does not exist, it's called `ucx-py-cu12`)* * depending on `distributed-ucxx-cu11==0.39.*` instead of `distributed-ucxx-cu11==0.39.*,>=0.0.0a0` - *(without that alpha specifier, `pip install --pre` is required to install pre-release versions)* This wasn't caught in `raft`'s CI, but in downstream CI like `cuml` and `cugraph`, with errors like this: ```text ERROR: ResolutionImpossible: The conflict is caused by: raft-dask-cu12 24.8.0a20 depends on pylibraft==24.8.* and >=0.0.0a0 raft-dask-cu12 24.8.0a19 depends on pylibraft==24.8.* and >=0.0.0a0 ``` ([example cugraph build](https://github.com/rapidsai/cugraph/actions/runs/9315062495/job/25656684762?pr=4454#step:7:1811)) This PR: * fixes those dependency issues * modifies `raft`'s CI so that similar issues would be caught here in the future, before publishing wheels ## Notes for Reviewers ### What was the root cause of CI missing this, and how does this PR fix it? The `raft-dask` test CI jobs use this pattern to install the `raft-dask` wheel built earlier in the CI pipeline. ```shell pip install "raft_dask-cu12[test]>=0.0.0a0" --find-links dist/ ``` As described in the `pip` docs ([link](https://pip.pypa.io/en/stable/cli/pip_install/#finding-packages)), `--find-links` just adds a directory to the list of other places `pip` searches for packages. Because the wheel there had unsatisfiable constraints (e.g. `pylibraft==24.8.*` does not exist anywhere), `pip install` silently disregarded that locally-downloaded `raft_dask` wheel and backtracked (i.e. downloaded older and older wheels from https://pypi.anaconda.org/rapidsai-wheels-nightly/simple/) until it found one that wasn't problematic. This PR ensures that won't happen by telling `pip` to install **exactly that locally-downloaded file**, like this ```shell pip install "$(echo ./dist/raft_dask_cu12*.whl)[test]" ``` If that file is uninstallable, `pip install` fails and you find out via a CI failure. ### How I tested this Initially pushed a commit with just the changes to the test script. Saw the `wheel-tests-raft-dask` CI jobs fail in the expected way, instead of silently falling back to an older wheel and passing 🎉 . ```text ERROR: Could not find a version that satisfies the requirement ucx-py-cu12==0.39.* (from raft-dask-cu12[test]) (from versions: 0.32.0, 0.33.0, 0.34.0, 0.35.0, 0.36.0, 0.37.0, 0.38.0a4, 0.38.0a5, 0.38.0a6, 0.39.0a0) ERROR: No matching distribution found for ucx-py-cu12==0.39.* ``` ([build link](https://github.com/rapidsai/raft/actions/runs/9323598882/job/25668146747?pr=2349)) Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Dante Gama Dessavre (https://github.com/dantegd) URL: #2349
…onstants (#2344) Contributes to rapidsai/build-planning#31 Follow-up to #2331 * ensures that `update-version.sh` does not remove alpha specs like `,>=0.0.0a0` in `pyproject.toml` and conda environment files * consolidates `rapids-build-backend` versions in `dependencies.yaml` - *since I was pushing a new commit here anyway, figured I'd take the opportunity to include that simplification recommended in rapidsai/cudf#15245 (comment) * adds tests that `__git_commit__` and `__version__` constants are present and that `__version__` is populated Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Corey J. Nolet (https://github.com/cjnolet) URL: #2344
pip install uses --config-settings as its argument. Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - James Lamb (https://github.com/jameslamb) Approvers: - James Lamb (https://github.com/jameslamb) URL: #2342
During reduction in device code (reduction.cuh), the value assigned in the residual threads during last stage are zero initilized. However, if we want to reduce some custom type, it might not have the appropriate constructor. Thus, this PR makes the change so that we call the default constructor for the residual values. Authors: - Akif ÇÖRDÜK (https://github.com/akifcorduk) Approvers: - Dante Gama Dessavre (https://github.com/dantegd) - Tamas Bela Feher (https://github.com/tfeher) URL: #2351
Forward-merge branch-24.06 into branch-24.08
This PR removes text builds of the documentation, which we do not currently use for anything. Contributes to rapidsai/build-planning#71. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Ben Frederickson (https://github.com/benfred) - James Lamb (https://github.com/jameslamb) URL: #2354
Recently devcontainer names were updated to include the current user's name. However, in GitHub Codespaces, the username is not defined. As a result, the container name starts with a dash. This is not allowed by GitHub Codespaces, so it fails to launch. This PR adds a default value of `anon` to the devcontainer username. See rapidsai/cudf#15784 for more information. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Paul Taylor (https://github.com/trxcllnt) - James Lamb (https://github.com/jameslamb) URL: #2355
This PR fixes a bug uncovered by CCCL version bump #2358 Authors: - Yunsong Wang (https://github.com/PointKernel) Approvers: - Ben Frederickson (https://github.com/benfred) URL: #2361
… followup (#2360) Contributes to rapidsai/build-planning#31 Contributes to rapidsai/dependency-file-generator#89 Since #2331 was merged, we've made some small adjustments to the approach for `rapids-build-backend`. This catches `raft` up with those changes: * consolidates version-handling in `ci/` scripts * uses `--file-key` instead of `--file_key` in `rapids-dependency-file-generator` calls * reduces duplication of `rapids-build-backend` version constraint in `dependencies.yaml`, and renames some lists there to match the patterns used in other projects (e.g. `rapids_build_skbuild` instead of `build`) Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #2360
- Enable NN Descent to return the distances array as well (previously only returning indices array) - Added a `return_distances` flag in `index_params`. When set to 1 (true), allocates a distance array to return. - Test for checking distances recall compared to naive knn Authors: - Jinsol Park (https://github.com/jinsolp) Approvers: - Divye Gala (https://github.com/divyegala) URL: #2345
- [x] RMM pooled resource for RAFT enabled FAISS - [x] Update yaml config for `faiss_gpu_ivf_flat`, `faiss_gpu_ivf_pq` - [x] Small fix FAISS GPU IVFPQ params - [x] Update `get_faiss.cmake` - [x] Fix SIGSEGV Notes: The `StandardGpuResources` object is part of FAISS' index classes. As a result, there is no way of creating a separate Raft handle for each thread without creating a new instance of the whole index object for a new thread. As such, multi-threaded benchmarking for GPU indices will not work. For CPU indices, @divyegala had seen some other issues. Though this PR is mainly about raft-enabled FAISS GPU indices. Authors: - Tarang Jain (https://github.com/tarang-jain) - Corey J. Nolet (https://github.com/cjnolet) - Tamas Bela Feher (https://github.com/tfeher) Approvers: - Tamas Bela Feher (https://github.com/tfeher) - James Lamb (https://github.com/jameslamb) - Ben Frederickson (https://github.com/benfred) URL: #2026
backport of rapidsai/cuvs#192 There is a bug in the current CAGRA graph rank-based neighbor reordering process. A low recall or illegal memory access can occur if there are many detourable nodes from a node to its neighbors, e.g. there is a small subgraph in the initial kNN graph. This PR fixes this. Authors: - tsuki (https://github.com/enp1s0) Approvers: - Tamas Bela Feher (https://github.com/tfeher) URL: #2365
Adds support for the `DistanceType::DiceExpanded` for dense inputs. 1. Naive Kernel Implementation (unexpanded form) 2. Expanded form for dice distance that follows ground truth from `scipy.spatial.distance.dice` 3. Gtests in `cpp/test/distance/dist-dice.cu` Related to rapidsai/cuml#5129 Authors: - Anupam (https://github.com/aamijar) Approvers: - Divye Gala (https://github.com/divyegala) URL: #2359
Closes #2363 Bugs fixed: 1. Setting `ef` in search, it was not being set at all before 2. `from_cagra` used a hard coded filename to serialize CAGRA graph and deserialize to HNSW graph. The PR changes the hardcoded filename to a random string so that multiple graphs may be converted concurrently cc @Presburger thank you for reporting these bugs Authors: - Divye Gala (https://github.com/divyegala) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #2367
PR #1740 forgot to rename `BLOCK_SIZE` in `#ifdef _CLK_BREAKDOWN` blocks. The use of `RAFT_LOG_DEBUG` in kernel function results in compilation errors, replace it with `printf`. Also remove an unused function in search_single_cta_kernel-inl.cuh After merging: - [x] port to cuVS rapidsai/cuvs#202 Authors: - Yinzuo Jiang (https://github.com/jiangyinzuo) - Tamas Bela Feher (https://github.com/tfeher) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - tsuki (https://github.com/enp1s0) - Tamas Bela Feher (https://github.com/tfeher) URL: #2350
`raft_cagra` wrapper stopped including the dataset in the index to save memory, but this adversely affected `raft_cagra_hnswlib` wrapper because the dataset needed to be included in the index. The need for inclusion of the dataset is because we need the dataset to be serialized when writing to the `hnswlib` format. Authors: - Divye Gala (https://github.com/divyegala) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #2369
With the deployment of rapids-build-backend, we need to make sure our dependencies have alpha specs. Contributes to rapidsai/build-planning#31 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - James Lamb (https://github.com/jameslamb) URL: #2373
This PR fixes a bug in the `bitmap_view`. Closes #2368 Authors: - rhdong (https://github.com/rhdong) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #2371
Contributes to rapidsai/build-planning#80 Adds constraints to avoid pulling in CMake 3.30.0, for the reasons described in that issue. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #2375
Usage of the CUDA math libraries is independent of the CUDA runtime. Make their static/shared status separately controllable. Contributes to rapidsai/build-planning#35 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Bradley Dice (https://github.com/bdice) - Robert Maynard (https://github.com/robertmaynard) URL: #2376
Instead of binarizing in `pairwise_distances()` in cuml, do it in the distance function in raft. Authors: - Anupam (https://github.com/aamijar) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Micka (https://github.com/lowener) URL: #2370
Adding distance epilogue to NN Descent. Planning to use them for the following use cases as of now; - Calculating mutual reachability distance for HDBSCAN (possibly for usage with HDBSCAN - [related PR here](rapidsai/cuml#5939)) - Enabling `L2SqrtExpanded` distance metric, by `sqrt`-ing the current supported metric (`L2Expanded`) of NN Descent in the distance epilogue. (for usage with UMAP - [related PR here](rapidsai/cuml#5910)) Authors: - Jinsol Park (https://github.com/jinsolp) Approvers: - Tarang Jain (https://github.com/tarang-jain) - Tamas Bela Feher (https://github.com/tfeher) - Divye Gala (https://github.com/divyegala) URL: #2364
This PR updates the latest CUDA build/test version 12.2.2 to 12.5.1. Contributes to rapidsai/build-planning#73 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - https://github.com/jakirkham Approvers: - James Lamb (https://github.com/jameslamb) URL: #2378
After updating everything to CUDA 12.5.1, use `[email protected]` again. Contributes to rapidsai/build-planning#73 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - James Lamb (https://github.com/jameslamb) - https://github.com/jakirkham
This make use of raft's slicing kernel within NN Descent build. I found that my previous implementation was inefficient (merged in [this PR](#2345)). ### Improvements Time to call NN Descent `build()` with `return_distances=True` before and after using this kernel | Dataset | Before |After| | ------------- | ------------- |---| | mnist (60000, 784) | 1550ms | 1020ms| | sift (1M, 128) | 11342ms |5546ms| | gist (1M, 960) | 13508ms |9278ms| Authors: - Jinsol Park (https://github.com/jinsolp) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Micka (https://github.com/lowener) URL: #2380
- For resolving the issue of cuVS: rapidsai/cuvs#158 Authors: - rhdong (https://github.com/rhdong) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #2346
#2336 Authors: - rhdong (https://github.com/rhdong) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #2362
Add cusparseSpMV_preprocess to cusparse wrapper Authors: - Nicolas Blin (https://github.com/Kh4ster) Approvers: - Micka (https://github.com/lowener) URL: #2384
This PR consists of multiple parts: 1. redirect custom reduction kernels within `stats `namespace to `linalg::reduce` 2. Specialize reduction kernels for addition utilizing the _Kahan-Babushka-Neumaier-Sum_ [link](https://en.wikipedia.org/wiki/Kahan_summation_algorithm) 3. Slightly adjust kernel heuristics for coalesced reductions This should address #2366 and #2205. With the kernel heuristics adjusted the maximum performance drop is 4%. FYI, @tfeher Authors: - Malte Förster (https://github.com/mfoerste4) Approvers: - Tamas Bela Feher (https://github.com/tfeher) - Corey J. Nolet (https://github.com/cjnolet) URL: #2381
Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Bradley Dice (https://github.com/bdice) URL: #2388
Bug 4580093 of cusolver is causing an issue with `cusolverDnXsyevd`. This bug has been seen in rapidsai/cuml#5555 and impact PCA and Linear Regression with CUDA 12.0+. Setting the stream to a different value than `cudaStreamPerThread` seems to solve it as a workaround. Authors: - Micka (https://github.com/lowener) Approvers: - Tamas Bela Feher (https://github.com/tfeher) - Artem M. Chirkin (https://github.com/achirkin) - Corey J. Nolet (https://github.com/cjnolet) URL: #2332
Authors: - rhdong (https://github.com/rhdong) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Divye Gala (https://github.com/divyegala) URL: #2394
rapidsai/ucxx#238 introduced a new timeout argument for `registerGeneric{Pre,Post}` that can be used to prevent blocking indefinitely should there be no UCX worker progress wakeup events. This should also result in new RAFT packages with updated symbols. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #2398
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
❄️ Code freeze for
branch-24.08and v24.08 releaseWhat does this mean?
Only critical/hotfix level issues should be merged into
branch-24.08until release (merging of this PR).What is the purpose of this PR?
branch-24.08intomainfor the release