Merge branch 'branch-24.04' into rhdong/bitmap

rapidsai · Feb 23, 2024 · f8960d9 · f8960d9
2 parents 70cce47 + 9fb05a2
commit f8960d9
Show file tree

Hide file tree

Showing 109 changed files with 3,172 additions and 1,152 deletions.
diff --git a/.github/ops-bot.yaml b/.github/ops-bot.yaml
@@ -6,3 +6,4 @@ branch_checker: true
 label_checker: true
 release_drafter: true
 recently_updated: true
+forward_merger: true
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -27,7 +27,7 @@ repos:
                 types_or: [python, cython]
                 additional_dependencies: ["flake8-force"]
       - repo: https://github.com/pre-commit/mirrors-mypy
-        rev: 'v0.971'
+        rev: 'v1.3.0'
         hooks:
               - id: mypy
                 additional_dependencies: [types-cachetools]

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,92 @@
+# raft 24.02.00 (12 Feb 2024)
+
+## 🚨 Breaking Changes
+
+- Switch to scikit-build-core ([#2051](https://github.com/rapidsai/raft/pull/2051)) [@vyasr](https://github.com/vyasr)
+- Update to CCCL 2.2.0. ([#2049](https://github.com/rapidsai/raft/pull/2049)) [@bdice](https://github.com/bdice)
+- Update `raft-ann-bench` output filenames and add features to plotting ([#2043](https://github.com/rapidsai/raft/pull/2043)) [@divyegala](https://github.com/divyegala)
+- Remove selection_faiss ([#2027](https://github.com/rapidsai/raft/pull/2027)) [@benfred](https://github.com/benfred)
+
+## 🐛 Bug Fixes
+
+- fix is_row/col_order for strided layouts ([#2173](https://github.com/rapidsai/raft/pull/2173)) [@mfoerste4](https://github.com/mfoerste4)
+- Fix failing C++ tests and revert #2097, #2085. ([#2168](https://github.com/rapidsai/raft/pull/2168)) [@cjnolet](https://github.com/cjnolet)
+- Exclude tests from builds ([#2162](https://github.com/rapidsai/raft/pull/2162)) [@vyasr](https://github.com/vyasr)
+- [HOTFIX] 24.02 Revert Random Sampling ([#2144](https://github.com/rapidsai/raft/pull/2144)) [@cjnolet](https://github.com/cjnolet)
+- Pin to pytest 7. ([#2137](https://github.com/rapidsai/raft/pull/2137)) [@bdice](https://github.com/bdice)
+- Conditionally include `hnsw` wrapper source in CMake ([#2135](https://github.com/rapidsai/raft/pull/2135)) [@divyegala](https://github.com/divyegala)
+- [BUG] Fix `SPMM` strided view ([#2124](https://github.com/rapidsai/raft/pull/2124)) [@lowener](https://github.com/lowener)
+- Fixing small bug in CUSPARSE spmm w/ CUDA 12.2 ([#2117](https://github.com/rapidsai/raft/pull/2117)) [@cjnolet](https://github.com/cjnolet)
+- [BUG] Fix `num_cta_per_query` div ([#2107](https://github.com/rapidsai/raft/pull/2107)) [@lowener](https://github.com/lowener)
+- Remove extraneous host pinnings from libraft-headers-only. ([#2102](https://github.com/rapidsai/raft/pull/2102)) [@bdice](https://github.com/bdice)
+- Remove unneeded CI symbol excludes ([#2098](https://github.com/rapidsai/raft/pull/2098)) [@robertmaynard](https://github.com/robertmaynard)
+- Properly taking ownership of nccl subcomm (and destroying it) ([#2094](https://github.com/rapidsai/raft/pull/2094)) [@cjnolet](https://github.com/cjnolet)
+- Fix `max_queries` for CAGRA ([#2081](https://github.com/rapidsai/raft/pull/2081)) [@lowener](https://github.com/lowener)
+- Fix compile failure on RTX 4090 ([#2076](https://github.com/rapidsai/raft/pull/2076)) [@JieFengWang](https://github.com/JieFengWang)
+- Fix a crash in FAISS benchmark wrapper introduced in #2021 ([#2062](https://github.com/rapidsai/raft/pull/2062)) [@achirkin](https://github.com/achirkin)
+- Correct function that wasn&#39;t returning a value ([#2045](https://github.com/rapidsai/raft/pull/2045)) [@robertmaynard](https://github.com/robertmaynard)
+- Fixing small bug in raft-ann-bench ([#2041](https://github.com/rapidsai/raft/pull/2041)) [@cjnolet](https://github.com/cjnolet)
+- Make device_resources accessed from device_resources_manager thread-safe ([#2030](https://github.com/rapidsai/raft/pull/2030)) [@wphicks](https://github.com/wphicks)
+- Fix ann-bench multithreading ([#2021](https://github.com/rapidsai/raft/pull/2021)) [@achirkin](https://github.com/achirkin)
+- Fix `ci/checks/copyright.py` to mirror RAPIDS reference ([#2008](https://github.com/rapidsai/raft/pull/2008)) [@divyegala](https://github.com/divyegala)
+- Fix pyproject versions ([#2002](https://github.com/rapidsai/raft/pull/2002)) [@vyasr](https://github.com/vyasr)
+
+## 📖 Documentation
+
+- Adding license info for wiki-all dataset ([#2129](https://github.com/rapidsai/raft/pull/2129)) [@cjnolet](https://github.com/cjnolet)
+- [DOC] Documentation updates for release 24.02 ([#2093](https://github.com/rapidsai/raft/pull/2093)) [@cjnolet](https://github.com/cjnolet)
+- Fix errors with ingroup exposed by doxygen 1.10 ([#2079](https://github.com/rapidsai/raft/pull/2079)) [@wphicks](https://github.com/wphicks)
+- Fix a typo ([#2070](https://github.com/rapidsai/raft/pull/2070)) [@narangvivek10](https://github.com/narangvivek10)
+- Add usage example for brute_force::build ([#2029](https://github.com/rapidsai/raft/pull/2029)) [@benfred](https://github.com/benfred)
+- Add filtering to vector search tutorial ([#1996](https://github.com/rapidsai/raft/pull/1996)) [@lowener](https://github.com/lowener)
+
+## 🚀 New Features
+
+- Update to use rapids-cmake for all deps ([#2096](https://github.com/rapidsai/raft/pull/2096)) [@robertmaynard](https://github.com/robertmaynard)
+- Add IVF-PQ example into the template project ([#2091](https://github.com/rapidsai/raft/pull/2091)) [@achirkin](https://github.com/achirkin)
+- Support for fp16 in CAGRA and IVF-PQ ([#2085](https://github.com/rapidsai/raft/pull/2085)) [@achirkin](https://github.com/achirkin)
+- Add random subsampling for IVF methods ([#2077](https://github.com/rapidsai/raft/pull/2077)) [@tfeher](https://github.com/tfeher)
+- Update `raft-ann-bench` output filenames and add features to plotting ([#2043](https://github.com/rapidsai/raft/pull/2043)) [@divyegala](https://github.com/divyegala)
+- Add brute_force index serialization ([#2036](https://github.com/rapidsai/raft/pull/2036)) [@wphicks](https://github.com/wphicks)
+- Add eps-neighbor search via RBC ([#2028](https://github.com/rapidsai/raft/pull/2028)) [@mfoerste4](https://github.com/mfoerste4)
+- `libraft` and `pylibraft` API for CAGRA build and HNSW search ([#2022](https://github.com/rapidsai/raft/pull/2022)) [@divyegala](https://github.com/divyegala)
+- Export Pareto frontier in `raft-ann-bench.data_export` ([#2009](https://github.com/rapidsai/raft/pull/2009)) [@divyegala](https://github.com/divyegala)
+- Implement maybe-owning multi-dimensional container (mdbuffer) ([#1999](https://github.com/rapidsai/raft/pull/1999)) [@wphicks](https://github.com/wphicks)
+- Add support for 1024+ dim vectors in CAGRA search ([#1994](https://github.com/rapidsai/raft/pull/1994)) [@enp1s0](https://github.com/enp1s0)
+- Replace GEMM backend: cublas.gemm -&gt; cublaslt.matmul ([#1736](https://github.com/rapidsai/raft/pull/1736)) [@achirkin](https://github.com/achirkin)
+
+## 🛠️ Improvements
+
+- Remove get_mem_info functions from RAFT custom memory resources ([#2108](https://github.com/rapidsai/raft/pull/2108)) [@harrism](https://github.com/harrism)
+- Replace call to mr::get_mem_info() ([#2099](https://github.com/rapidsai/raft/pull/2099)) [@harrism](https://github.com/harrism)
+- Allow topk larger than 1024 in CAGRA ([#2097](https://github.com/rapidsai/raft/pull/2097)) [@benfred](https://github.com/benfred)
+- Remove usages of rapids-env-update ([#2095](https://github.com/rapidsai/raft/pull/2095)) [@KyleFromNVIDIA](https://github.com/KyleFromNVIDIA)
+- Provide explicit pool size for pool_memory_resources and clean up includes ([#2088](https://github.com/rapidsai/raft/pull/2088)) [@harrism](https://github.com/harrism)
+- refactor CUDA versions in dependencies.yaml ([#2086](https://github.com/rapidsai/raft/pull/2086)) [@jameslamb](https://github.com/jameslamb)
+- ANN bench fix latency measurement overhead ([#2084](https://github.com/rapidsai/raft/pull/2084)) [@tfeher](https://github.com/tfeher)
+- Remove hardcoded limit in `print_results` function ([#2080](https://github.com/rapidsai/raft/pull/2080)) [@narangvivek10](https://github.com/narangvivek10)
+- [FEA] Add support for SDDMM by wrapping the cusparseSDDMM ([#2067) (#2067](https://github.com/rapidsai/raft/pull/2067) (#2067)) [@rhdong](https://github.com/rhdong)
+- Benchmark brute force knn ([#2063](https://github.com/rapidsai/raft/pull/2063)) [@benfred](https://github.com/benfred)
+- [BUG] fix empty initialization of device_ndarray in pylibraft ([#2061](https://github.com/rapidsai/raft/pull/2061)) [@mfoerste4](https://github.com/mfoerste4)
+- Improve parallelism of refine host ([#2059](https://github.com/rapidsai/raft/pull/2059)) [@anaruse](https://github.com/anaruse)
+- Subsampling for IVF-PQ codebook generation ([#2052](https://github.com/rapidsai/raft/pull/2052)) [@abc99lr](https://github.com/abc99lr)
+- Switch to scikit-build-core ([#2051](https://github.com/rapidsai/raft/pull/2051)) [@vyasr](https://github.com/vyasr)
+- Update to CCCL 2.2.0. ([#2049](https://github.com/rapidsai/raft/pull/2049)) [@bdice](https://github.com/bdice)
+- Use cuda::proclaim_return_type on device lambda. ([#2048](https://github.com/rapidsai/raft/pull/2048)) [@bdice](https://github.com/bdice)
+- Removing code that explicitly compares equality of rmm memory resources ([#2047](https://github.com/rapidsai/raft/pull/2047)) [@cjnolet](https://github.com/cjnolet)
+- Add public enum for select-k algorithm selection ([#2046](https://github.com/rapidsai/raft/pull/2046)) [@benfred](https://github.com/benfred)
+- Update dependencies.yaml to new pip index ([#2042](https://github.com/rapidsai/raft/pull/2042)) [@vyasr](https://github.com/vyasr)
+- Remove RAFT_BUILD_WHEELS and standardize Python builds ([#2040](https://github.com/rapidsai/raft/pull/2040)) [@vyasr](https://github.com/vyasr)
+- Fix ucx-py version pinning in dependencies.yaml. ([#2035](https://github.com/rapidsai/raft/pull/2035)) [@bdice](https://github.com/bdice)
+- [REVIEW] Fix typos in parameter tuning guide ([#2034](https://github.com/rapidsai/raft/pull/2034)) [@abc99lr](https://github.com/abc99lr)
+- Add AIR-Top-k reference ([#2031](https://github.com/rapidsai/raft/pull/2031)) [@tfeher](https://github.com/tfeher)
+- Remove selection_faiss ([#2027](https://github.com/rapidsai/raft/pull/2027)) [@benfred](https://github.com/benfred)
+- Fixing json parse error in `raft-ann-bench.data_export` ([#2025](https://github.com/rapidsai/raft/pull/2025)) [@cjnolet](https://github.com/cjnolet)
+- Updating cagra build constraint ([#2016](https://github.com/rapidsai/raft/pull/2016)) [@cjnolet](https://github.com/cjnolet)
+- Update to fmt 10.1.1 and spdlog 1.12.0. ([#1957](https://github.com/rapidsai/raft/pull/1957)) [@bdice](https://github.com/bdice)
+- Enable host dataset for IVF-Flat ([#1635](https://github.com/rapidsai/raft/pull/1635)) [@tfeher](https://github.com/tfeher)
+- add half/bfloat support to myInf and abs ([#1592](https://github.com/rapidsai/raft/pull/1592)) [@Kh4ster](https://github.com/Kh4ster)
+
 # raft 23.12.00 (6 Dec 2023)
 
 ## 🐛 Bug Fixes

diff --git a/conda/environments/all_cuda-118_arch-aarch64.yaml b/conda/environments/all_cuda-118_arch-aarch64.yaml
@@ -44,8 +44,8 @@ dependencies:
 - nvcc_linux-aarch64=11.8
 - pre-commit
 - pydata-sphinx-theme
-- pytest
 - pytest-cov
+- pytest==7.*
 - rapids-dask-dependency==24.4.*
 - recommonmark
 - rmm==24.4.*

diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml
@@ -44,8 +44,8 @@ dependencies:
 - nvcc_linux-64=11.8
 - pre-commit
 - pydata-sphinx-theme
-- pytest
 - pytest-cov
+- pytest==7.*
 - rapids-dask-dependency==24.4.*
 - recommonmark
 - rmm==24.4.*

diff --git a/...vironments/all_cuda-120_arch-aarch64.yaml → ...vironments/all_cuda-122_arch-aarch64.yaml b/...vironments/all_cuda-120_arch-aarch64.yaml → ...vironments/all_cuda-122_arch-aarch64.yaml
@@ -17,7 +17,7 @@ dependencies:
 - cuda-nvtx-dev
 - cuda-profiler-api
 - cuda-python>=12.0,<13.0a0
-- cuda-version=12.0
+- cuda-version=12.2
 - cupy>=12.0.0
 - cxx-compiler
 - cython>=3.0.0
@@ -40,8 +40,8 @@ dependencies:
 - numpydoc
 - pre-commit
 - pydata-sphinx-theme
-- pytest
 - pytest-cov
+- pytest==7.*
 - rapids-dask-dependency==24.4.*
 - recommonmark
 - rmm==24.4.*
@@ -54,4 +54,4 @@ dependencies:
 - ucx-proc=*=gpu
 - ucx-py==0.37.*
 - ucx>=1.13.0
-name: all_cuda-120_arch-aarch64
+name: all_cuda-122_arch-aarch64
diff --git a/...nvironments/all_cuda-120_arch-x86_64.yaml → ...nvironments/all_cuda-122_arch-x86_64.yaml b/...nvironments/all_cuda-120_arch-x86_64.yaml → ...nvironments/all_cuda-122_arch-x86_64.yaml
@@ -17,7 +17,7 @@ dependencies:
 - cuda-nvtx-dev
 - cuda-profiler-api
 - cuda-python>=12.0,<13.0a0
-- cuda-version=12.0
+- cuda-version=12.2
 - cupy>=12.0.0
 - cxx-compiler
 - cython>=3.0.0
@@ -40,8 +40,8 @@ dependencies:
 - numpydoc
 - pre-commit
 - pydata-sphinx-theme
-- pytest
 - pytest-cov
+- pytest==7.*
 - rapids-dask-dependency==24.4.*
 - recommonmark
 - rmm==24.4.*
@@ -54,4 +54,4 @@ dependencies:
 - ucx-proc=*=gpu
 - ucx-py==0.37.*
 - ucx>=1.13.0
-name: all_cuda-120_arch-x86_64
+name: all_cuda-122_arch-x86_64
diff --git a/conda/recipes/libraft/meta.yaml b/conda/recipes/libraft/meta.yaml
@@ -6,7 +6,6 @@
 {% set minor_version = version.split('.')[0] + '.' + version.split('.')[1] %}
 {% set cuda_version = '.'.join(environ['RAPIDS_CUDA_VERSION'].split('.')[:2]) %}
 {% set cuda_major = cuda_version.split('.')[0] %}
-{% set cuda_spec = ">=" + cuda_major ~ ",<" + (cuda_major | int + 1) ~ ".0a0" %} # i.e. >=11,<12.0a0
 {% set date_string = environ['RAPIDS_DATE_STRING'] %}
 
 package:
@@ -42,6 +41,9 @@ outputs:
       ignore_run_exports_from:
         {% if cuda_major == "11" %}
         - {{ compiler('cuda11') }}
+        {% else %}
+        - {{ compiler('cuda') }}
+        - cuda-cudart-dev
         {% endif %}
         - librmm
     requirements:
@@ -58,14 +60,16 @@ outputs:
         - ninja
         - sysroot_{{ target_platform }} {{ sysroot_version }}
       host:
+        - cuda-version ={{ cuda_version }}
         {% if cuda_major != "11" %}
         - cuda-cudart-dev
         {% endif %}
-        - cuda-version ={{ cuda_version }}
       run:
         - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }}
         {% if cuda_major == "11" %}
         - cudatoolkit
+        {% else %}
+        - cuda-cudart
         {% endif %}
         - librmm ={{ minor_version }}
         - spdlog {{ spdlog_version }}
@@ -82,6 +86,8 @@ outputs:
       ignore_run_exports_from:
         {% if cuda_major == "11" %}
         - {{ compiler('cuda11') }}
+        {% else %}
+        - {{ compiler('cuda') }}
         {% endif %}
         - librmm
     requirements:
@@ -124,6 +130,13 @@ outputs:
       ignore_run_exports_from:
         {% if cuda_major == "11" %}
         - {{ compiler('cuda11') }}
+        {% else %}
+        - {{ compiler('cuda') }}
+        - cuda-cudart-dev
+        - libcublas-dev
+        - libcurand-dev
+        - libcusolver-dev
+        - libcusparse-dev
         {% endif %}
     requirements:
       build:
@@ -152,6 +165,7 @@ outputs:
         - libcusparse {{ cuda11_libcusparse_host_version }}
         - libcusparse-dev {{ cuda11_libcusparse_host_version }}
         {% else %}
+        - cuda-cudart-dev
         - cuda-profiler-api
         - libcublas-dev
         - libcurand-dev
@@ -161,6 +175,13 @@ outputs:
       run:
         - {{ pin_subpackage('libraft-headers', exact=True) }}
         - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }}
+        {% if cuda_major != "11" %}
+        - cuda-cudart
+        - libcublas
+        - libcurand
+        - libcusolver
+        - libcusparse
+        {% endif %}
     about:
       home: https://rapids.ai/
       license: Apache-2.0
@@ -175,6 +196,9 @@ outputs:
       ignore_run_exports_from:
         {% if cuda_major == "11" %}
         - {{ compiler('cuda11') }}
+        {% else %}
+        - {{ compiler('cuda') }}
+        - cuda-cudart-dev
         {% endif %}
     requirements:
       build:
@@ -203,6 +227,7 @@ outputs:
         - libcusparse {{ cuda11_libcusparse_host_version }}
         - libcusparse-dev {{ cuda11_libcusparse_host_version }}
         {% else %}
+        - cuda-cudart-dev
         - cuda-profiler-api
         - libcublas-dev
         - libcurand-dev
@@ -212,6 +237,13 @@ outputs:
       run:
         - {{ pin_subpackage('libraft-headers', exact=True) }}
         - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }}
+        {% if cuda_major != "11" %}
+        - cuda-cudart
+        - libcublas
+        - libcurand
+        - libcusolver
+        - libcusparse
+        {% endif %}
     about:
       home: https://rapids.ai/
       license: Apache-2.0
@@ -226,6 +258,13 @@ outputs:
       ignore_run_exports_from:
         {% if cuda_major == "11" %}
         - {{ compiler('cuda11') }}
+        {% else %}
+        - {{ compiler('cuda') }}
+        - cuda-cudart-dev
+        - libcublas-dev
+        - libcurand-dev
+        - libcusolver-dev
+        - libcusparse-dev
         {% endif %}
     requirements:
       build:
@@ -267,6 +306,12 @@ outputs:
         - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }}
         {% if cuda_major == "11" %}
         - cudatoolkit
+        {% else %}
+        - cuda-cudart
+        - libcublas
+        - libcurand
+        - libcusolver
+        - libcusparse
         {% endif %}
         - {{ pin_subpackage('libraft', exact=True) }}
         - gmock {{ gtest_version }}
@@ -285,6 +330,10 @@ outputs:
       ignore_run_exports_from:
         {% if cuda_major == "11" %}
         - {{ compiler('cuda11') }}
+        {% else %}
+        - {{ compiler('cuda') }}
+        - cuda-cudart-dev
+        - libcublas-dev
         {% endif %}
     requirements:
       build:
@@ -308,13 +357,17 @@ outputs:
         - libcublas {{ cuda11_libcublas_host_version }}
         - libcublas-dev {{ cuda11_libcublas_host_version }}
         {% else %}
+        - cuda-cudart-dev
         - cuda-profiler-api
         - libcublas-dev
         {% endif %}
       run:
         - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }}
         {% if cuda_major == "11" %}
         - cudatoolkit
+        {% else %}
+        - cuda-cudart
+        - libcublas
         {% endif %}
         - {{ pin_subpackage('libraft', exact=True) }}
     about:

diff --git a/conda/recipes/pylibraft/meta.yaml b/conda/recipes/pylibraft/meta.yaml
@@ -1,4 +1,4 @@
-# Copyright (c) 2022-2023, NVIDIA CORPORATION.
+# Copyright (c) 2022-2024, NVIDIA CORPORATION.
 
 # Usage:
 #   conda build . -c conda-forge -c numba -c rapidsai -c pytorch
@@ -22,6 +22,9 @@ build:
   ignore_run_exports_from:
     {% if cuda_major == "11" %}
     - {{ compiler('cuda11') }}
+    {% else %}
+    - {{ compiler('cuda') }}
+    - cuda-cudart-dev
     {% endif %}
 
 requirements:
@@ -43,6 +46,7 @@ requirements:
     - cudatoolkit
     {% else %}
     - cuda-python >=12.0,<13.0a0
+    - cuda-cudart-dev
     {% endif %}
     - cuda-version ={{ cuda_version }}
     - cython >=3.0.0
@@ -56,6 +60,8 @@ requirements:
     - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }}
     {% if cuda_major == "11" %}
     - cudatoolkit
+    {% else %}
+    - cuda-cudart
     {% endif %}
     - libraft {{ version }}
     - libraft-headers {{ version }}