Merge branch 'feat/raft_logger' of github.com:vyasr/cugraph into feat…

…/raft_logger
rapidsai · Dec 18, 2024 · 43b7074 · 43b7074
2 parents d603a2a + 7cad393
commit 43b7074
Show file tree

Hide file tree

Showing 17 changed files with 176 additions and 27 deletions.
diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml
@@ -40,6 +40,7 @@ jobs:
         OTEL_SERVICE_NAME: "pr-cugraph"
     steps:
       - name: Telemetry setup
+        if: ${{ vars.TELEMETRY_ENABLED == 'true' }}
         uses: rapidsai/shared-actions/telemetry-dispatch-stash-base-env-vars@main
   changed-files:
     secrets: inherit
@@ -179,16 +180,11 @@ jobs:
         build-all --verbose -j$(nproc --ignore=1) -DBUILD_CUGRAPH_MG_TESTS=ON;
         sccache -s;
   telemetry-summarize:
-    runs-on: ubuntu-latest
+    # This job must use a self-hosted runner to record telemetry traces.
+    runs-on: linux-amd64-cpu4
     needs: pr-builder
-    if: always()
+    if: ${{ vars.TELEMETRY_ENABLED == 'true' && !cancelled() }}
     continue-on-error: true
     steps:
-      - name: Load stashed telemetry env vars
-        uses: rapidsai/shared-actions/telemetry-dispatch-load-base-env-vars@main
-        with:
-            load_service_name: true
       - name: Telemetry summarize
-        uses: rapidsai/shared-actions/telemetry-dispatch-write-summary@main
-        with:
-          cert_concat: "${{ secrets.OTEL_EXPORTER_OTLP_CA_CERTIFICATE }};${{ secrets.OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE }};${{ secrets.OTEL_EXPORTER_OTLP_CLIENT_KEY }}"
+        uses: rapidsai/shared-actions/telemetry-dispatch-summarize@main
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,79 @@
+# cugraph 24.12.00 (11 Dec 2024)
+
+## 🚨 Breaking Changes
+
+- Fix SSL Error ([#4825](https://github.com/rapidsai/cugraph/pull/4825)) [@nv-rliu](https://github.com/nv-rliu)
+- [BUG] Set Max PyTorch Version, Skip 11.4 Tests Using WholeGraph ([#4808](https://github.com/rapidsai/cugraph/pull/4808)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+- Remove GNN Packages ([#4765](https://github.com/rapidsai/cugraph/pull/4765)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+- remove cugraph-dgl and cugraph-pyg building/testing/publishing ([#4752](https://github.com/rapidsai/cugraph/pull/4752)) [@jameslamb](https://github.com/jameslamb)
+- Remove `nx-cugraph` build/test/publish from cugraph ([#4748](https://github.com/rapidsai/cugraph/pull/4748)) [@nv-rliu](https://github.com/nv-rliu)
+- Remove CMake/C++ references to cugraph-ops ([#4744](https://github.com/rapidsai/cugraph/pull/4744)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Delete the deprecated data_type_id_t enum ([#4737](https://github.com/rapidsai/cugraph/pull/4737)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Don&#39;t compile int32_t/int64_t vertex_t/edge_t combinations ([#4720](https://github.com/rapidsai/cugraph/pull/4720)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Remove deprecated C API functions for graph creation/graph free ([#4718](https://github.com/rapidsai/cugraph/pull/4718)) [@ChuckHastings](https://github.com/ChuckHastings)
+
+## 🐛 Bug Fixes
+
+- Fix SSL Error ([#4825](https://github.com/rapidsai/cugraph/pull/4825)) [@nv-rliu](https://github.com/nv-rliu)
+- [BUG] Set Max PyTorch Version, Skip 11.4 Tests Using WholeGraph ([#4808](https://github.com/rapidsai/cugraph/pull/4808)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+- Fix CCCL 2.7.0-rc2 compile issue by removing reference from values. ([#4799](https://github.com/rapidsai/cugraph/pull/4799)) [@bdice](https://github.com/bdice)
+- Re-enable balanced edge cut test ([#4781](https://github.com/rapidsai/cugraph/pull/4781)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Fix debug build failure ([#4774](https://github.com/rapidsai/cugraph/pull/4774)) [@seunghwak](https://github.com/seunghwak)
+- Add sphinx-lint pre-commit and some docs fixes ([#4771](https://github.com/rapidsai/cugraph/pull/4771)) [@eriknw](https://github.com/eriknw)
+- Fix improper initialization of degree_type ([#4755](https://github.com/rapidsai/cugraph/pull/4755)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Put a ceiling on cuda-python ([#4747](https://github.com/rapidsai/cugraph/pull/4747)) [@bdice](https://github.com/bdice)
+- Fix MG similarity issues ([#4741](https://github.com/rapidsai/cugraph/pull/4741)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Address Leiden clustering generating too many clusters ([#4730](https://github.com/rapidsai/cugraph/pull/4730)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Add support for storing results for all k-values ([#4728](https://github.com/rapidsai/cugraph/pull/4728)) [@nv-rliu](https://github.com/nv-rliu)
+- Remove `--collect-only` Option Left by Accident ([#4727](https://github.com/rapidsai/cugraph/pull/4727)) [@nv-rliu](https://github.com/nv-rliu)
+- Update nx-cugraph to NetworkX 3.4 ([#4717](https://github.com/rapidsai/cugraph/pull/4717)) [@eriknw](https://github.com/eriknw)
+- Symmetrize edgelist when creating a CSR graph ([#4716](https://github.com/rapidsai/cugraph/pull/4716)) [@jnke2016](https://github.com/jnke2016)
+
+## 📖 Documentation
+
+- nx-cugraph: add `bipartite_betweenness_centrality` to supported algorithms ([#4778](https://github.com/rapidsai/cugraph/pull/4778)) [@eriknw](https://github.com/eriknw)
+- Notebook to run Centrality against PatentsView data for Medium Blog ([#4769](https://github.com/rapidsai/cugraph/pull/4769)) [@acostadon](https://github.com/acostadon)
+- Drop support for NetworkX 3.0 and 3.1 for nx-cugraph ([#4766](https://github.com/rapidsai/cugraph/pull/4766)) [@eriknw](https://github.com/eriknw)
+- Update nx-cugraph Docs URLs ([#4761](https://github.com/rapidsai/cugraph/pull/4761)) [@nv-rliu](https://github.com/nv-rliu)
+- Update `nx-cugraph` README with New Env Variable ([#4705](https://github.com/rapidsai/cugraph/pull/4705)) [@nv-rliu](https://github.com/nv-rliu)
+
+## 🚀 New Features
+
+- [FEA] Support Edge ID Lookup in PyLibcuGraph ([#4687](https://github.com/rapidsai/cugraph/pull/4687)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+
+## 🛠️ Improvements
+
+- Increase max_iterations in MG HITS TEST ([#4783](https://github.com/rapidsai/cugraph/pull/4783)) [@seunghwak](https://github.com/seunghwak)
+- Updates READMEs, updates `core_number` to properly ignore `degree_type`, minor cleanup ([#4776](https://github.com/rapidsai/cugraph/pull/4776)) [@rlratzel](https://github.com/rlratzel)
+- Remove edge renumber map from the homogeneous sampling API ([#4775](https://github.com/rapidsai/cugraph/pull/4775)) [@jnke2016](https://github.com/jnke2016)
+- Remove GNN Packages ([#4765](https://github.com/rapidsai/cugraph/pull/4765)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+- Remove cugraph-equivariance ([#4762](https://github.com/rapidsai/cugraph/pull/4762)) [@BradReesWork](https://github.com/BradReesWork)
+- deprecate NetworkX Graph as input ([#4759](https://github.com/rapidsai/cugraph/pull/4759)) [@BradReesWork](https://github.com/BradReesWork)
+- Remove `nx-cugraph` from this repo ([#4756](https://github.com/rapidsai/cugraph/pull/4756)) [@eriknw](https://github.com/eriknw)
+- enforce wheel size limits, README formatting in CI ([#4754](https://github.com/rapidsai/cugraph/pull/4754)) [@jameslamb](https://github.com/jameslamb)
+- remove cugraph-dgl and cugraph-pyg building/testing/publishing ([#4752](https://github.com/rapidsai/cugraph/pull/4752)) [@jameslamb](https://github.com/jameslamb)
+- Primitives &amp; BFS performance improvements ([#4751](https://github.com/rapidsai/cugraph/pull/4751)) [@seunghwak](https://github.com/seunghwak)
+- extract the edgelist from the graph ([#4750](https://github.com/rapidsai/cugraph/pull/4750)) [@jnke2016](https://github.com/jnke2016)
+- Remove `nx-cugraph` build/test/publish from cugraph ([#4748](https://github.com/rapidsai/cugraph/pull/4748)) [@nv-rliu](https://github.com/nv-rliu)
+- Remove CMake/C++ references to cugraph-ops ([#4744](https://github.com/rapidsai/cugraph/pull/4744)) [@ChuckHastings](https://github.com/ChuckHastings)
+- add telemetry ([#4740](https://github.com/rapidsai/cugraph/pull/4740)) [@msarahan](https://github.com/msarahan)
+- Delete the deprecated data_type_id_t enum ([#4737](https://github.com/rapidsai/cugraph/pull/4737)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Updates README with new dataset, removes mention of script no longer used ([#4736](https://github.com/rapidsai/cugraph/pull/4736)) [@rlratzel](https://github.com/rlratzel)
+- devcontainer: replace `VAULT_HOST` with `AWS_ROLE_ARN` ([#4732](https://github.com/rapidsai/cugraph/pull/4732)) [@jjacobelli](https://github.com/jjacobelli)
+- Adds new dataset for benchmarking in the 100k node 1M edge range, adds additional k-values for BC benchmarks ([#4726](https://github.com/rapidsai/cugraph/pull/4726)) [@rlratzel](https://github.com/rlratzel)
+- re-run all CI when files in .github/workflows change ([#4723](https://github.com/rapidsai/cugraph/pull/4723)) [@jameslamb](https://github.com/jameslamb)
+- Don&#39;t compile int32_t/int64_t vertex_t/edge_t combinations ([#4720](https://github.com/rapidsai/cugraph/pull/4720)) [@ChuckHastings](https://github.com/ChuckHastings)
+- print sccache stats in builds ([#4719](https://github.com/rapidsai/cugraph/pull/4719)) [@jameslamb](https://github.com/jameslamb)
+- Remove deprecated C API functions for graph creation/graph free ([#4718](https://github.com/rapidsai/cugraph/pull/4718)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Clean up the use of cuco hash functions ([#4707](https://github.com/rapidsai/cugraph/pull/4707)) [@PointKernel](https://github.com/PointKernel)
+- Relax PyTorch upper bound (allowing 2.4) ([#4703](https://github.com/rapidsai/cugraph/pull/4703)) [@jakirkham](https://github.com/jakirkham)
+- combine pip install calls in wheel-testing scripts ([#4701](https://github.com/rapidsai/cugraph/pull/4701)) [@jameslamb](https://github.com/jameslamb)
+- Merge branch-24.10 into branch-24.12 ([#4697](https://github.com/rapidsai/cugraph/pull/4697)) [@jameslamb](https://github.com/jameslamb)
+- Merge branch-24.10 into branch-24.12 ([#4682](https://github.com/rapidsai/cugraph/pull/4682)) [@jameslamb](https://github.com/jameslamb)
+- Update all rmm imports to use pylibrmm/librmm ([#4671](https://github.com/rapidsai/cugraph/pull/4671)) [@Matt711](https://github.com/Matt711)
+- Prune workflows based on changed files ([#4634](https://github.com/rapidsai/cugraph/pull/4634)) [@KyleFromNVIDIA](https://github.com/KyleFromNVIDIA)
+- support heterogenous fanout type ([#4608](https://github.com/rapidsai/cugraph/pull/4608)) [@jnke2016](https://github.com/jnke2016)
+
 # cugraph 24.10.00 (9 Oct 2024)
 
 ## 🚨 Breaking Changes

diff --git a/ci/build_docs.sh b/ci/build_docs.sh
@@ -63,6 +63,8 @@ rapids-logger "Build CPP docs"
 pushd cpp/doxygen
 doxygen Doxyfile
 export XML_DIR_LIBCUGRAPH="$(pwd)/xml"
+mkdir -p "${RAPIDS_DOCS_DIR}/libcugraph/xml_tar"
+tar -czf "${RAPIDS_DOCS_DIR}/libcugraph/xml_tar"/xml.tar.gz -C xml .
 popd
 
 rapids-logger "Build Python docs"

diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml
@@ -10,8 +10,10 @@ dependencies:
 - aiohttp
 - breathe
 - c-compiler
+- certifi
 - cmake>=3.26.4,!=3.30.0
 - cuda-nvtx
+- cuda-python>=11.8.5,<12.0a0
 - cuda-version=11.8
 - cudatoolkit
 - cudf==25.2.*,>=0.0.0a0
@@ -51,7 +53,7 @@ dependencies:
 - pytest-cov
 - pytest-xdist
 - python-louvain
-- pytorch>=2.3
+- pytorch>=2.3,<2.5a0
 - raft-dask==25.2.*,>=0.0.0a0
 - rapids-build-backend>=0.3.1,<0.4.0.dev0
 - rapids-dask-dependency==25.2.*,>=0.0.0a0

diff --git a/conda/environments/all_cuda-125_arch-x86_64.yaml b/conda/environments/all_cuda-125_arch-x86_64.yaml
@@ -10,11 +10,13 @@ dependencies:
 - aiohttp
 - breathe
 - c-compiler
+- certifi
 - cmake>=3.26.4,!=3.30.0
 - cuda-cudart-dev
 - cuda-nvcc
 - cuda-nvtx-dev
 - cuda-profiler-api
+- cuda-python>=12.6.2,<13.0a0
 - cuda-version=12.5
 - cudf==25.2.*,>=0.0.0a0
 - cupy>=12.0.0
@@ -56,7 +58,7 @@ dependencies:
 - pytest-cov
 - pytest-xdist
 - python-louvain
-- pytorch>=2.3
+- pytorch>=2.3,<2.5a0
 - raft-dask==25.2.*,>=0.0.0a0
 - rapids-build-backend>=0.3.1,<0.4.0.dev0
 - rapids-dask-dependency==25.2.*,>=0.0.0a0

diff --git a/dependencies.yaml b/dependencies.yaml
@@ -401,6 +401,15 @@ dependencies:
             # dataset APIs require [http] extras for use with cudf.
           - fsspec[http]>=0.6.0
     specific:
+      - output_types: [conda, requirements, pyproject]
+        matrices:
+          - matrix:
+              cuda: "12.*"
+            packages:
+              - cuda-python>=12.6.2,<13.0a0
+          - matrix: # All CUDA 11 versions
+            packages:
+              - cuda-python>=11.8.5,<12.0a0
       - output_types: pyproject
         matrices:
           - matrix:
@@ -464,6 +473,7 @@ dependencies:
     common:
       - output_types: [conda, requirements]
         packages:
+          - certifi
           - ipython
           - notebook>=0.5.0
       - output_types: [conda]
@@ -483,6 +493,7 @@ dependencies:
     common:
       - output_types: [conda, pyproject]
         packages:
+          - certifi
           - networkx>=2.5.1
           - *numpy
           - python-louvain
@@ -501,7 +512,7 @@ dependencies:
     common:
       - output_types: [conda]
         packages:
-          - &pytorch_conda pytorch>=2.3
+          - &pytorch_conda pytorch>=2.3,<2.5a0
           - torchdata
           - pydantic
           - ogb

diff --git a/python/cugraph-service/server/pyproject.toml b/python/cugraph-service/server/pyproject.toml
@@ -46,6 +46,7 @@ cugraph-service-server = "cugraph_service_server.__main__:main"
 
 [project.optional-dependencies]
 test = [
+    "certifi",
     "networkx>=2.5.1",
     "numpy>=1.23,<3.0a0",
     "pandas",

diff --git a/python/cugraph/cugraph/datasets/__init__.py b/python/cugraph/cugraph/datasets/__init__.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2022-2023, NVIDIA CORPORATION.
+# Copyright (c) 2022-2024, NVIDIA CORPORATION.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
@@ -49,3 +49,4 @@
 europe_osm = Dataset(meta_path / "europe_osm.yaml")
 # 1.5 GB
 hollywood = Dataset(meta_path / "hollywood.yaml")
+amazon0302 = Dataset(meta_path / "amazon0302.yaml")
diff --git a/python/cugraph/cugraph/datasets/dataset.py b/python/cugraph/cugraph/datasets/dataset.py
@@ -11,14 +11,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import cudf
-import dask_cudf
-import yaml
 import os
 import pandas as pd
-import cugraph.dask as dcg
+import yaml
 from pathlib import Path
-import urllib.request
+from urllib.request import urlretrieve
+
+import cudf
+import cugraph.dask as dcg
+import dask_cudf
 from cugraph.structure.graph_classes import Graph
 
 
@@ -142,7 +143,7 @@ def __download_csv(self, url):
         filename = self.metadata["name"] + self.metadata["file_type"]
         if self._dl_path.path.is_dir():
             self._path = self._dl_path.path / filename
-            urllib.request.urlretrieve(url, str(self._path))
+            urlretrieve(url, str(self._path))
 
         else:
             raise RuntimeError(
@@ -458,7 +459,7 @@ def download_all(force=False):
                 filename = meta["name"] + meta["file_type"]
                 save_to = default_download_dir.path / filename
                 if not save_to.is_file() or force:
-                    urllib.request.urlretrieve(meta["url"], str(save_to))
+                    urlretrieve(meta["url"], str(save_to))
 
 
 def set_download_dir(path):

diff --git a/python/cugraph/cugraph/datasets/metadata/amazon0302.yaml b/python/cugraph/cugraph/datasets/metadata/amazon0302.yaml
@@ -0,0 +1,26 @@
+name: amazon0302
+file_type: .gz
+description:
+  This network was collected by crawling the Amazon website. It is based on the
+  "Customers Who Bought This Item Also Bought" feature of the Amazon website.
+  If product i is frequently co-purchased with product j, the graph contains a
+  directed edge from i to j. The data was collected in March 02 2003.
+author: J. Leskovec, L. Adamic and B. Adamic
+refs:
+  J. Leskovec, L. Adamic and B. Adamic. The Dynamics of Viral Marketing.
+  ACM Transactions on the Web (ACM TWEB), 1(1), 2007.
+delim: "\t"
+header: 3
+col_names:
+  - FromNodeId
+  - ToNodeId
+col_types:
+  - int32
+  - int32
+has_loop: false
+is_directed: true
+is_multigraph: false
+is_symmetric: false
+number_of_edges: 1234877
+number_of_nodes: 262111
+url: https://snap.stanford.edu/data/amazon0302.txt.gz
diff --git a/python/cugraph/cugraph/testing/__init__.py b/python/cugraph/cugraph/testing/__init__.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2023, NVIDIA CORPORATION.
+# Copyright (c) 2023-2024, NVIDIA CORPORATION.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
@@ -38,6 +38,7 @@
     cit_patents,
     europe_osm,
     hollywood,
+    amazon0302,
     # twitter,
 )
 
@@ -71,4 +72,10 @@
     toy_graph_undirected,
 ]
 DEFAULT_DATASETS = [dolphins, netscience, karate_disjoint]
-BENCHMARKING_DATASETS = [soc_livejournal, cit_patents, europe_osm, hollywood]
+BENCHMARKING_DATASETS = [
+    soc_livejournal,
+    cit_patents,
+    europe_osm,
+    hollywood,
+    amazon0302,
+]
diff --git a/python/cugraph/cugraph/testing/resultset.py b/python/cugraph/cugraph/testing/resultset.py
@@ -13,7 +13,6 @@
 
 import warnings
 import tarfile
-
 import urllib.request
 
 import cudf
@@ -22,8 +21,6 @@
     default_download_dir,
 )
 
-# results_dir_path = utils.RAPIDS_DATASET_ROOT_DIR_PATH / "tests" / "resultsets"
-
 
 class Resultset:
     """

diff --git a/python/cugraph/cugraph/tests/conftest.py b/python/cugraph/cugraph/tests/conftest.py
@@ -23,9 +23,20 @@
 # Avoid timeout during shutdown
 from dask_cuda.utils_test import IncreasedCloseTimeoutNanny
 
-# module-wide fixtures
+import certifi
+from ssl import create_default_context
+from urllib.request import build_opener, HTTPSHandler, install_opener
+
+
+# Install SSL certificates
+def pytest_sessionstart(session):
+    ssl_context = create_default_context(cafile=certifi.where())
+    https_handler = HTTPSHandler(context=ssl_context)
+    install_opener(build_opener(https_handler))
 
 
+# module-wide fixtures
+
 # Spoof the gpubenchmark fixture if it's not available so that asvdb and
 # rapids-pytest-benchmark do not need to be installed to run tests.
 if "gpubenchmark" not in globals():

diff --git a/python/cugraph/cugraph/tests/data_store/test_gnn_feat_storage_wholegraph.py b/python/cugraph/cugraph/tests/data_store/test_gnn_feat_storage_wholegraph.py
@@ -15,6 +15,8 @@
 import numpy as np
 import os
 
+import numba.cuda
+
 from cugraph.gnn import FeatureStore
 
 from cugraph.utilities.utils import import_optional, MissingModule
@@ -25,6 +27,11 @@
 wgth = import_optional("pylibwholegraph.torch")
 
 
+def get_cudart_version():
+    major, minor = numba.cuda.runtime.get_version()
+    return major * 1000 + minor * 10
+
+
 def runtest(rank: int, world_size: int):
     torch.cuda.set_device(rank)
 
@@ -66,6 +73,9 @@ def runtest(rank: int, world_size: int):
 @pytest.mark.skipif(
     isinstance(pylibwholegraph, MissingModule), reason="wholegraph not available"
 )
+@pytest.mark.skipif(
+    get_cudart_version() < 11080, reason="not compatible with CUDA < 11.8"
+)
 def test_feature_storage_wholegraph_backend():
     world_size = torch.cuda.device_count()
     print("gpu count:", world_size)
@@ -81,6 +91,9 @@ def test_feature_storage_wholegraph_backend():
 @pytest.mark.skipif(
     isinstance(pylibwholegraph, MissingModule), reason="wholegraph not available"
 )
+@pytest.mark.skipif(
+    get_cudart_version() < 11080, reason="not compatible with CUDA < 11.8"
+)
 def test_feature_storage_wholegraph_backend_mg():
     world_size = torch.cuda.device_count()
     print("gpu count:", world_size)