Adds option to rapids_cpm_find for raft to disable hnswlib feature, a…

…dds updates for pytest 8 compat, temporarily skips IO intensive test in CI (#4121)

* Adds option to rapids_cpm_find for raft to disable hnswlib feature, which is not needed by cugraph.  This resolves cmake failures such as the following:
   ```
        CMake Error at /home/coder/cugraph/cpp/build/release/_deps/raft-build/raft-targets.cmake:56 (set_target_properties):
         The link interface of target "raft::raft" contains:
  
           hnswlib::hnswlib
  
  
         but the target was not found. 
   ```
* TEMPORARILY skips bulk sampling IO tests which were hanging on CI runners. These tests pass locally, so we believe this is related to IO issues on CI runners, but will investigate further in 24.04
* Adds updates for pytest >=8
  * Adds option to pytest to append source paths to PYTHONPATH instead of prepend (the default). The prepended PYTHONPATH caused the dask `LocalCUDACluster` to import cugraph modules from the source tree instead of the install location, which resulted in `ImportError`s. This appears to be a behavior change in pytest 8.
  * Adds an `xfail` for an [NX test known to be incompatible with pytest>=8](networkx/networkx#7203)

Authors:
  - Rick Ratzel (https://github.com/rlratzel)

Approvers:
  - Divye Gala (https://github.com/divyegala)
  - Chuck Hastings (https://github.com/ChuckHastings)
  - Ray Douglass (https://github.com/raydouglass)
  - Erik Welch (https://github.com/eriknw)

URL: #4121

ci/test_python.sh

@@ -63,7 +63,16 @@ pytest \
   tests
 popd
 
-# FIXME: TEMPORARILY disable single-GPU "MG" testing
+# Test runs that include tests that use dask require
+# --import-mode=append. Those tests start a LocalCUDACluster that inherits
+# changes from pytest's modifications to PYTHONPATH (which defaults to
+# prepending source tree paths to PYTHONPATH).  This causes the
+# LocalCUDACluster subprocess to import cugraph from the source tree instead of
+# the install location, and in most cases, the source tree does not have
+# extensions built in-place and will result in ImportErrors.
+#
+# FIXME: TEMPORARILY disable MG PropertyGraph tests (experimental) tests and
+# bulk sampler IO tests (hangs in CI)
 rapids-logger "pytest cugraph"
 pushd python/cugraph/cugraph
 DASK_WORKER_DEVICES="0" \
@@ -72,14 +81,15 @@ DASK_DISTRIBUTED__COMM__TIMEOUTS__CONNECT="1000s" \
 DASK_CUDA_WAIT_WORKERS_MIN_TIMEOUT="1000s" \
 pytest \
   -v \
+  --import-mode=append \
   --benchmark-disable \
   --cache-clear \
   --junitxml="${RAPIDS_TESTS_DIR}/junit-cugraph.xml" \
   --cov-config=../../.coveragerc \
   --cov=cugraph \
   --cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cugraph-coverage.xml" \
   --cov-report=term \
-  -k "not test_property_graph_mg" \
+  -k "not test_property_graph_mg and not test_bulk_sampler_io" \
   tests
 popd
 

ci/test_wheel.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-# Copyright (c) 2023, NVIDIA CORPORATION.
+# Copyright (c) 2023-2024, NVIDIA CORPORATION.
 
 set -eoxu pipefail
 
@@ -21,10 +21,21 @@ arch=$(uname -m)
 if [[ "${arch}" == "aarch64" && ${RAPIDS_BUILD_TYPE} == "pull-request" ]]; then
     python ./ci/wheel_smoke_test_${package_name}.py
 else
-    # FIXME: TEMPORARILY disable single-GPU "MG" testing
+    # Test runs that include tests that use dask require
+    # --import-mode=append. See test_python.sh for details.
+    # FIXME: Adding PY_IGNORE_IMPORTMISMATCH=1 to workaround conftest.py import
+    # mismatch error seen by nx-cugraph after using pytest 8 and
+    # --import-mode=append.
     RAPIDS_DATASET_ROOT_DIR=`pwd`/datasets \
+    PY_IGNORE_IMPORTMISMATCH=1 \
+    DASK_WORKER_DEVICES="0" \
     DASK_DISTRIBUTED__SCHEDULER__WORKER_TTL="1000s" \
     DASK_DISTRIBUTED__COMM__TIMEOUTS__CONNECT="1000s" \
     DASK_CUDA_WAIT_WORKERS_MIN_TIMEOUT="1000s" \
-    python -m pytest ./python/${package_name}/${python_package_name}/tests
+    python -m pytest \
+       -v \
+       --import-mode=append \
+       --benchmark-disable \
+       -k "not test_property_graph_mg and not test_bulk_sampler_io" \
+       ./python/${package_name}/${python_package_name}/tests
 fi

cpp/cmake/thirdparty/get_raft.cmake

@@ -1,5 +1,5 @@
 #=============================================================================
-# Copyright (c) 2022-2023, NVIDIA CORPORATION.
+# Copyright (c) 2022-2024, NVIDIA CORPORATION.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -52,6 +52,7 @@ function(find_and_configure_raft)
                 "RAFT_COMPILE_LIBRARY ${PKG_COMPILE_RAFT_LIB}"
                 "BUILD_TESTS OFF"
                 "BUILD_BENCH OFF"
+                "BUILD_CAGRA_HNSWLIB OFF"
     )
 
     if(raft_ADDED)

python/nx-cugraph/nx_cugraph/interface.py

@@ -80,18 +80,26 @@ def key(testpath):
         nxver = parse(nx.__version__)
 
         if nxver.major == 3 and nxver.minor <= 2:
-            # Networkx versions prior to 3.2.1 have tests written to expect
-            # sp.sparse.linalg.ArpackNoConvergence exceptions raised on no
-            # convergence in HITS. Newer versions since the merge of
-            # https://github.com/networkx/networkx/pull/7084 expect
-            # nx.PowerIterationFailedConvergence, which is what nx_cugraph.hits
-            # raises, so we mark them as xfail for previous versions of NX.
             xfail.update(
                 {
+                    # NetworkX versions prior to 3.2.1 have tests written to
+                    # expect sp.sparse.linalg.ArpackNoConvergence exceptions
+                    # raised on no convergence in HITS. Newer versions since
+                    # the merge of
+                    # https://github.com/networkx/networkx/pull/7084 expect
+                    # nx.PowerIterationFailedConvergence, which is what
+                    # nx_cugraph.hits raises, so we mark them as xfail for
+                    # previous versions of NX.
                     key(
                         "test_hits.py:TestHITS.test_hits_not_convergent"
                     ): "nx_cugraph.hits raises updated exceptions not caught in "
                     "these tests",
+                    # NetworkX versions 3.2 and older contain tests that fail
+                    # with pytest>=8. Assume pytest>=8 and mark xfail.
+                    key(
+                        "test_strongly_connected.py:"
+                        "TestStronglyConnected.test_connected_raise"
+                    ): "test is incompatible with pytest>=8",
                 }
             )