From b43a32efeb6861a5f1095aca747f8ee5994472b4 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Thu, 29 Aug 2024 10:55:44 -0500 Subject: [PATCH 01/28] wheels: dynamically load `libcudf.so` from `libcudf` wheel (#1447) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Contributes to https://github.com/rapidsai/build-planning/issues/33. Proposes the following for `cuspatial` wheels: * add build and runtime dependencies on `libcudf` wheels * stop vendoring copies of `libcudf.so`, `libnvcomp.so`, `libnvcomp_bitcomp.so`, and `libnvcomp_gdeflate.so` - *(load `libcudf.so` dynamically at runtime instead)* And other related changes for development/CI: * combine all `pip install` calls into 1 in wheel-testing scripts - *like https://github.com/rapidsai/cudf/pull/16575* - *to improve the chance that packaging issues are discovered in CI* * `dependencies.yaml` changes: - more use of YAML anchors = less duplication - use dedicated `depends_on_librmm` and `depends_on_libcudf` groups * explicitly pass a package type to `gha-tools` wheel uploading/downloading scripts ## Notes for Reviewers ### Benefits of these changes Unblocks CI in this repo (ref: https://github.com/rapidsai/cuspatial/pull/1444#issuecomment-2308151120, https://github.com/rapidsai/cuspatial/pull/1441#issuecomment-2308149200). Reduces wheel sizes for `cuspatial` wheels by about 125MB 😁 | wheel | size (before) | size (this PR) | |:-----------:|-------------:|---------------:| | `cuspatial` | 146.0M | 21M | | `cuproj ` | 0.9M | 0.9M | |**TOTAL** | **146.9M** | **21.9M** | *NOTES: size = compressed, "before" = 2024-08-21 nightlies (https://github.com/rapidsai/cuspatial/commit/c60bd4dfc25e62d68cdf520789c6c8bfc577724d), CUDA = 12, Python = 3.11*
how I calculated those (click me) ```shell # note: 2024-08-21 because that was the most recent date with # successfully-built cuspatial nightlies # docker run \ --rm \ -v $(pwd):/opt/work:ro \ -w /opt/work \ --network host \ --env RAPIDS_NIGHTLY_DATE=2024-08-21 \ --env RAPIDS_NIGHTLY_SHA=c60bd4d \ --env RAPIDS_PR_NUMBER=1447 \ --env RAPIDS_PY_CUDA_SUFFIX=cu12 \ --env RAPIDS_REPOSITORY=rapidsai/cuspatial \ --env WHEEL_DIR_BEFORE=/tmp/wheels-before \ --env WHEEL_DIR_AFTER=/tmp/wheels-after \ -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.11 \ bash mkdir -p "${WHEEL_DIR_BEFORE}" mkdir -p "${WHEEL_DIR_AFTER}" py_projects=( cuspatial cuproj ) for project in "${py_projects[@]}"; do # before RAPIDS_BUILD_TYPE=nightly \ RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REF_NAME="branch-24.10" \ RAPIDS_SHA=${RAPIDS_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python "${WHEEL_DIR_BEFORE}" # after RAPIDS_BUILD_TYPE=pull-request \ RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REF_NAME="pull-request/${RAPIDS_PR_NUMBER}" \ rapids-download-wheels-from-s3 python "${WHEEL_DIR_AFTER}" done du -sh ${WHEEL_DIR_BEFORE}/* du -sh ${WHEEL_DIR_BEFORE} du -sh ${WHEEL_DIR_AFTER}/* du -sh ${WHEEL_DIR_AFTER} ```
Reduces the amount of additional work required to start shipping `libcuspatial` wheels. ### Background This is part of ongoing work towards packaging `libcuspatial` as a wheel. relevant prior work: * packaging `libcudf` wheels: https://github.com/rapidsai/cudf/pull/15483 * consolidating `pip install` calls in CI scripts for `cudf`: https://github.com/rapidsai/cudf/pull/16575 * `cudf` dropping its Arrow library dependency: https://github.com/rapidsai/cudf/pull/16640 ### How I tested this Confirmed in local builds and CI logs that `cudf` is being *found*, not *built*, in `cuspatial` builds. ```text -- CPM: Using local package cudf@24.10.0 ``` ([build link](https://github.com/rapidsai/cuspatial/actions/runs/10602971716/job/29386288614?pr=1447#step:9:23472)) Built `cuspatial` wheels locally and ran all the unit tests, without issue. # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) - Matthew Roeschke (https://github.com/mroeschke) URL: https://github.com/rapidsai/cuspatial/pull/1447 --- .github/workflows/build.yaml | 2 + ci/build_wheel.sh | 8 ++- ci/build_wheel_cuproj.sh | 4 +- ci/build_wheel_cuspatial.sh | 6 +- ci/test_wheel_cuproj.sh | 16 +++-- ci/test_wheel_cuspatial.sh | 12 ++-- dependencies.yaml | 69 ++++++++++++++++--- python/cuspatial/CMakeLists.txt | 10 +-- .../cmake/Modules/WheelHelpers.cmake | 59 ---------------- python/cuspatial/cuspatial/__init__.py | 6 +- python/cuspatial/pyproject.toml | 4 ++ 11 files changed, 100 insertions(+), 96 deletions(-) delete mode 100644 python/cuspatial/cmake/Modules/WheelHelpers.cmake diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 4ecbbab45..1f3fd7922 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -85,6 +85,7 @@ jobs: sha: ${{ inputs.sha }} date: ${{ inputs.date }} package-name: cuspatial + package-type: python wheel-build-cuproj: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 @@ -104,3 +105,4 @@ jobs: sha: ${{ inputs.sha }} date: ${{ inputs.date }} package-name: cuproj + package-type: python diff --git a/ci/build_wheel.sh b/ci/build_wheel.sh index 5e6eb944e..14802741d 100755 --- a/ci/build_wheel.sh +++ b/ci/build_wheel.sh @@ -5,6 +5,7 @@ set -euo pipefail package_name=$1 package_dir=$2 +package_type=$3 source rapids-configure-sccache source rapids-date-string @@ -18,6 +19,9 @@ cd "${package_dir}" python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check mkdir -p final_dist -python -m auditwheel repair -w final_dist dist/* +python -m auditwheel repair \ + --exclude libcudf.so \ + -w final_dist \ + dist/* -RAPIDS_PY_WHEEL_NAME="${package_name}_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 final_dist +RAPIDS_PY_WHEEL_NAME="${package_name}_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 "${package_type}" final_dist diff --git a/ci/build_wheel_cuproj.sh b/ci/build_wheel_cuproj.sh index 961f4e363..6001e44e9 100755 --- a/ci/build_wheel_cuproj.sh +++ b/ci/build_wheel_cuproj.sh @@ -1,6 +1,6 @@ #!/bin/bash -# Copyright (c) 2023, NVIDIA CORPORATION. +# Copyright (c) 2023-2024, NVIDIA CORPORATION. set -euo pipefail -ci/build_wheel.sh cuproj python/cuproj +ci/build_wheel.sh cuproj python/cuproj python diff --git a/ci/build_wheel_cuspatial.sh b/ci/build_wheel_cuspatial.sh index 932665e27..171bfa6a4 100755 --- a/ci/build_wheel_cuspatial.sh +++ b/ci/build_wheel_cuspatial.sh @@ -1,8 +1,6 @@ #!/bin/bash -# Copyright (c) 2023, NVIDIA CORPORATION. +# Copyright (c) 2023-2024, NVIDIA CORPORATION. set -euo pipefail -export SKBUILD_CMAKE_ARGS="-DUSE_LIBARROW_FROM_PYARROW=ON" - -ci/build_wheel.sh cuspatial python/cuspatial +ci/build_wheel.sh cuspatial python/cuspatial python diff --git a/ci/test_wheel_cuproj.sh b/ci/test_wheel_cuproj.sh index 9451190e0..4144e8dab 100755 --- a/ci/test_wheel_cuproj.sh +++ b/ci/test_wheel_cuproj.sh @@ -5,19 +5,21 @@ set -eou pipefail mkdir -p ./dist RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})" -RAPIDS_PY_WHEEL_NAME="cuproj_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 ./dist -# Install additional dependencies +# install build dependencies for fiona apt update DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev -python -m pip install --no-binary fiona 'fiona>=1.8.19,<1.9' -# Download the cuspatial built in the previous step -RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 ./local-cuspatial-dep -python -m pip install --no-deps ./local-cuspatial-dep/cuspatial*.whl +# Download the cuproj and cuspatial built in the previous step +RAPIDS_PY_WHEEL_NAME="cuproj_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist +RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist # echo to expand wildcard before adding `[extra]` requires for pip -python -m pip install $(echo ./dist/cuproj*.whl)[test] +python -m pip install \ + --no-binary 'fiona' \ + "$(echo ./dist/cuspatial*.whl)" \ + "$(echo ./dist/cuproj*.whl)[test]" \ + 'fiona>=1.8.19,<1.9' rapids-logger "pytest cuproj" pushd python/cuproj/cuproj diff --git a/ci/test_wheel_cuspatial.sh b/ci/test_wheel_cuspatial.sh index aa36681bc..8faa4e8d3 100755 --- a/ci/test_wheel_cuspatial.sh +++ b/ci/test_wheel_cuspatial.sh @@ -5,15 +5,19 @@ set -eou pipefail mkdir -p ./dist RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})" -RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 ./dist -# Install additional dependencies +# install build dependencies for fiona apt update DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev -python -m pip install --no-binary fiona 'fiona>=1.8.19,<1.9' + +# Download the cuspatial built in the previous step +RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist # echo to expand wildcard before adding `[extra]` requires for pip -python -m pip install $(echo ./dist/cuspatial*.whl)[test] +python -m pip install \ + --no-binary 'fiona' \ + "$(echo ./dist/cuspatial*.whl)[test]" \ + 'fiona>=1.8.19,<1.9' rapids-logger "pytest cuspatial" pushd python/cuspatial/cuspatial diff --git a/dependencies.yaml b/dependencies.yaml index 3b4bee329..80822b43d 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -17,6 +17,8 @@ files: - depends_on_cudf - depends_on_cuml - depends_on_cupy + - depends_on_libcudf + - depends_on_librmm - rapids_build_skbuild - run_python_cuspatial - test_libcuspatial @@ -77,14 +79,17 @@ files: - build_cpp - build_python - build_wheels + - depends_on_libcudf + - depends_on_librmm py_run_cuspatial: output: [pyproject] pyproject_dir: python/cuspatial extras: table: project includes: - - depends_on_rmm - depends_on_cudf + - depends_on_libcudf + - depends_on_rmm - run_python_cuspatial py_test_cuspatial: output: [pyproject] @@ -141,14 +146,12 @@ dependencies: common: - output_types: [conda, requirements, pyproject] packages: - - ninja - - cmake>=3.26.4,!=3.30.0 + - &ninja ninja + - &cmake cmake>=3.26.4,!=3.30.0 - output_types: conda packages: - c-compiler - cxx-compiler - - libcudf==24.10.*,>=0.0.0a0 - - librmm==24.10.*,>=0.0.0a0 - proj - sqlite specific: @@ -184,13 +187,13 @@ dependencies: common: - output_types: [conda, requirements, pyproject] packages: - - ninja - - cmake>=3.26.4,!=3.30.0 + - *ninja + - *cmake - output_types: conda packages: - c-compiler - cxx-compiler - - librmm==24.10.*,>=0.0.0a0 + - &librmm_unsuffixed librmm==24.10.*,>=0.0.0a0 - proj - sqlite specific: @@ -443,6 +446,31 @@ dependencies: - pylibcudf-cu11==24.10.*,>=0.0.0a0 - {matrix: null, packages: [*cudf_unsuffixed]} + depends_on_libcudf: + common: + - output_types: conda + packages: + - &libcudf_unsuffixed libcudf==24.10.*,>=0.0.0a0 + - output_types: requirements + packages: + # pip recognizes the index as a global option for the requirements.txt file + - --extra-index-url=https://pypi.nvidia.com + - --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple + specific: + - output_types: [requirements, pyproject] + matrices: + - matrix: + cuda: "12.*" + cuda_suffixed: "true" + packages: + - libcudf-cu12==24.10.*,>=0.0.0a0 + - matrix: + cuda: "11.*" + cuda_suffixed: "true" + packages: + - libcudf-cu11==24.10.*,>=0.0.0a0 + - {matrix: null, packages: [*libcudf_unsuffixed]} + depends_on_cuml: common: - output_types: conda @@ -523,3 +551,28 @@ dependencies: - libcuspatial==24.10.*,>=0.0.0a0 - cuspatial==24.10.*,>=0.0.0a0 - cuproj==24.10.*,>=0.0.0a0 + + depends_on_librmm: + common: + - output_types: conda + packages: + - *librmm_unsuffixed + - output_types: requirements + packages: + # pip recognizes the index as a global option for the requirements.txt file + - --extra-index-url=https://pypi.nvidia.com + - --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple + specific: + - output_types: [requirements, pyproject] + matrices: + - matrix: + cuda: "12.*" + cuda_suffixed: "true" + packages: + - librmm-cu12==24.10.*,>=0.0.0a0 + - matrix: + cuda: "11.*" + cuda_suffixed: "true" + packages: + - librmm-cu11==24.10.*,>=0.0.0a0 + - {matrix: null, packages: [*librmm_unsuffixed]} diff --git a/python/cuspatial/CMakeLists.txt b/python/cuspatial/CMakeLists.txt index e3678c311..28fcf467d 100644 --- a/python/cuspatial/CMakeLists.txt +++ b/python/cuspatial/CMakeLists.txt @@ -43,15 +43,7 @@ if(NOT cuspatial_FOUND) add_subdirectory(../../cpp cuspatial-cpp EXCLUDE_FROM_ALL) set(cython_lib_dir cuspatial) - include(cmake/Modules/WheelHelpers.cmake) - # TODO: This install is currently overzealous. We should only install the libraries that are - # downloaded by CPM during the build, not libraries that were found on the system. However, in - # practice this would only be a problem if libcudf was not found but some of the - # dependencies were, and we have no real use cases where that happens. - install_aliased_imported_targets( - TARGETS cuspatial arrow_shared nvcomp::nvcomp nvcomp::nvcomp_gdeflate nvcomp::nvcomp_bitcomp - DESTINATION ${cython_lib_dir} - ) + install(TARGETS cuspatial DESTINATION ${cython_lib_dir}) endif() include(rapids-cython-core) diff --git a/python/cuspatial/cmake/Modules/WheelHelpers.cmake b/python/cuspatial/cmake/Modules/WheelHelpers.cmake deleted file mode 100644 index 278d6751c..000000000 --- a/python/cuspatial/cmake/Modules/WheelHelpers.cmake +++ /dev/null @@ -1,59 +0,0 @@ -# ============================================================================= -# Copyright (c) 2022-2023, NVIDIA CORPORATION. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except -# in compliance with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software distributed under the License -# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express -# or implied. See the License for the specific language governing permissions and limitations under -# the License. -# ============================================================================= -include_guard(GLOBAL) - -# Making libraries available inside wheels by installing the associated targets. -function(install_aliased_imported_targets) - list(APPEND CMAKE_MESSAGE_CONTEXT "install_aliased_imported_targets") - - set(options "") - set(one_value "DESTINATION") - set(multi_value "TARGETS") - cmake_parse_arguments(_ "${options}" "${one_value}" "${multi_value}" ${ARGN}) - - message(VERBOSE "Installing targets '${__TARGETS}' into lib_dir '${__DESTINATION}'") - - foreach(target IN LISTS __TARGETS) - - if(NOT TARGET ${target}) - message(VERBOSE "No target named ${target}") - continue() - endif() - - get_target_property(alias_target ${target} ALIASED_TARGET) - if(alias_target) - set(target ${alias_target}) - endif() - - get_target_property(is_imported ${target} IMPORTED) - if(NOT is_imported) - # If the target isn't imported, install it into the wheel - install(TARGETS ${target} DESTINATION ${__DESTINATION}) - message(VERBOSE "install(TARGETS ${target} DESTINATION ${__DESTINATION})") - else() - # If the target is imported, make sure it's global - get_target_property(type ${target} TYPE) - if(${type} STREQUAL "UNKNOWN_LIBRARY") - install(FILES $ DESTINATION ${__DESTINATION}) - message(VERBOSE "install(FILES $ DESTINATION ${__DESTINATION})") - else() - install(IMPORTED_RUNTIME_ARTIFACTS ${target} DESTINATION ${__DESTINATION}) - message( - VERBOSE - "install(IMPORTED_RUNTIME_ARTIFACTS $ DESTINATION ${__DESTINATION})" - ) - endif() - endif() - endforeach() -endfunction() diff --git a/python/cuspatial/cuspatial/__init__.py b/python/cuspatial/cuspatial/__init__.py index 069b5454d..14a5fcc4d 100644 --- a/python/cuspatial/cuspatial/__init__.py +++ b/python/cuspatial/cuspatial/__init__.py @@ -1,4 +1,8 @@ -# Copyright (c) 2023, NVIDIA CORPORATION. +# Copyright (c) 2023-2024, NVIDIA CORPORATION. + +# load cudf before any other imports, to be sure libcudf's symbols are found +# in the libcudf.so from the libcudf wheel (if one is installed) +import cudf from ._version import __git_commit__, __version__ from .core.geodataframe import GeoDataFrame diff --git a/python/cuspatial/pyproject.toml b/python/cuspatial/pyproject.toml index 80f436ea6..25ea28bfd 100644 --- a/python/cuspatial/pyproject.toml +++ b/python/cuspatial/pyproject.toml @@ -32,6 +32,7 @@ requires-python = ">=3.10" dependencies = [ "cudf==24.10.*,>=0.0.0a0", "geopandas>=0.11.0", + "libcudf==24.10.*,>=0.0.0a0", "numpy>=1.23,<2.0a0", "rmm==24.10.*,>=0.0.0a0", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. @@ -71,6 +72,7 @@ known_dask = [ known_rapids = [ "rmm", "cudf", + "libcudf", "pylibcudf", ] known_first_party = [ @@ -134,6 +136,8 @@ requires = [ "cmake>=3.26.4,!=3.30.0", "cudf==24.10.*,>=0.0.0a0", "cython>=3.0.0", + "libcudf==24.10.*,>=0.0.0a0", + "librmm==24.10.*,>=0.0.0a0", "ninja", "rmm==24.10.*,>=0.0.0a0", "wheel", From b61c052b7c20f52a47fa5c2259212491c7028c99 Mon Sep 17 00:00:00 2001 From: Kyle Edwards Date: Thu, 29 Aug 2024 12:52:00 -0400 Subject: [PATCH 02/28] Update rapidsai/pre-commit-hooks (#1445) This PR updates rapidsai/pre-commit-hooks to the version 0.4.0. Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - James Lamb (https://github.com/jameslamb) Approvers: - James Lamb (https://github.com/jameslamb) URL: https://github.com/rapidsai/cuspatial/pull/1445 --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 74f71fd7b..b96543933 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -43,7 +43,7 @@ repos: args: ["--config pyproject.toml"] additional_dependencies: ["tomli"] - repo: https://github.com/rapidsai/pre-commit-hooks - rev: v0.3.1 + rev: v0.4.0 hooks: - id: verify-copyright - id: verify-alpha-spec From 9c616dec64d13f74451e60d2f1d95dca06c1bf28 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Thu, 29 Aug 2024 14:18:21 -0500 Subject: [PATCH 03/28] cuproj: depend on librmm, not rmm (#1448) Contributes to https://github.com/rapidsai/build-planning/issues/33 `cuproj` does not need the `rmm` Python package... it only needs the RMM headers at build time. This proposes the following changes for `cuproj`: * dropping the runtime requirement on `rmm` in wheels and conda packages * switching the build requirement from `rmm` to `librmm` for wheels and conda packages * removing unnecessary imports in the `test:` environment for conda packages For more context on these changes, see https://github.com/rapidsai/build-planning/issues/92. ## Notes for Reviewers ### Benefits of these changes Faster conda builds (via dropping unnecessary dependencies). Cheaper (in terms of bandwidth and disk space) installation of wheels and conda packages (via removing an unnecessary runtime dependency). Reduces a source of network calls (and therefore CI instability) by removing some CPM downloads of RMM. Before: ```text -- CPM: Adding package rmm@24.10 (branch-24.10) ``` ([build link](https://github.com/rapidsai/cuspatial/actions/runs/10618529204/job/29434041322#step:9:16754)) After (this PR): ```text -- CPM: Using local package rmm@24.10.0 ``` ([build link](https://github.com/rapidsai/cuspatial/actions/runs/10619138604/job/29436119470?pr=1448#step:9:11256)) ### Is this required for `libcuspatial` wheel packaging? No, it's just a side thing I noticed while working on that. The two are totally independent. # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: https://github.com/rapidsai/cuspatial/pull/1448 --- conda/recipes/cuproj/meta.yaml | 8 +------- dependencies.yaml | 6 ++---- python/cuproj/pyproject.toml | 3 +-- 3 files changed, 4 insertions(+), 13 deletions(-) diff --git a/conda/recipes/cuproj/meta.yaml b/conda/recipes/cuproj/meta.yaml index 6851ab214..29d590bc0 100644 --- a/conda/recipes/cuproj/meta.yaml +++ b/conda/recipes/cuproj/meta.yaml @@ -62,7 +62,7 @@ requirements: - cython >=3.0.0 - python - rapids-build-backend >=0.3.0,<0.4.0.dev0 - - rmm ={{ minor_version }} + - librmm ={{ minor_version }} - scikit-build-core >=0.10.0 - proj - sqlite @@ -74,17 +74,11 @@ requirements: {% endif %} - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} - python - - rmm ={{ minor_version }} - cupy>=12.0.0 test: # [linux64] imports: # [linux64] - cuproj # [linux64] - requires: - - cupy>=12.0.0 - - cuspatial ={{ minor_version }} - - rmm ={{ minor_version }} - about: home: https://rapids.ai/ diff --git a/dependencies.yaml b/dependencies.yaml index 80822b43d..859009683 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -113,17 +113,16 @@ files: table: tool.rapids-build-backend key: requires includes: - - depends_on_rmm - build_cpp_cuproj - build_python - build_wheels + - depends_on_librmm py_run_cuproj: output: [pyproject] pyproject_dir: python/cuproj extras: table: project includes: - - depends_on_rmm - depends_on_cupy py_test_cuproj: output: [pyproject] @@ -193,7 +192,6 @@ dependencies: packages: - c-compiler - cxx-compiler - - &librmm_unsuffixed librmm==24.10.*,>=0.0.0a0 - proj - sqlite specific: @@ -556,7 +554,7 @@ dependencies: common: - output_types: conda packages: - - *librmm_unsuffixed + - &librmm_unsuffixed librmm==24.10.*,>=0.0.0a0 - output_types: requirements packages: # pip recognizes the index as a global option for the requirements.txt file diff --git a/python/cuproj/pyproject.toml b/python/cuproj/pyproject.toml index 7625a01f9..6c4d6cc2d 100644 --- a/python/cuproj/pyproject.toml +++ b/python/cuproj/pyproject.toml @@ -29,7 +29,6 @@ license = { text = "Apache 2.0" } requires-python = ">=3.10" dependencies = [ "cupy-cuda11x>=12.0.0", - "rmm==24.10.*,>=0.0.0a0", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. classifiers = [ "Intended Audience :: Developers", @@ -122,7 +121,7 @@ matrix-entry = "cuda_suffixed=true" requires = [ "cmake>=3.26.4,!=3.30.0", "cython>=3.0.0", + "librmm==24.10.*,>=0.0.0a0", "ninja", - "rmm==24.10.*,>=0.0.0a0", "wheel", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. From 1544e7bd16f11457436d1fb6a43da0db3aa9af57 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Thu, 29 Aug 2024 21:50:30 -0500 Subject: [PATCH 04/28] cuproj: remove pin on 'proj' in conda packages (#1449) Proposes removing the `cuproj` conda package's pin on `proj`, so that it can build using the same version that `conda-forge` pins to in https://github.com/conda-forge/conda-forge-pinning-feedstock. For context, see: * why the `proj==9.3.0` pin was originally added: https://github.com/rapidsai/cuspatial/pull/1307 * `conda-forge` completied migrations: - `9.3.1`: https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/5401 - `9.4.`: https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/6240 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - https://github.com/jakirkham URL: https://github.com/rapidsai/cuspatial/pull/1449 --- conda/recipes/cuproj/conda_build_config.yaml | 4 ---- 1 file changed, 4 deletions(-) diff --git a/conda/recipes/cuproj/conda_build_config.yaml b/conda/recipes/cuproj/conda_build_config.yaml index 2186d11ca..001878ff2 100644 --- a/conda/recipes/cuproj/conda_build_config.yaml +++ b/conda/recipes/cuproj/conda_build_config.yaml @@ -18,7 +18,3 @@ c_stdlib_version: cmake_version: - ">=3.26.4,!=3.30.0" - -# Workaround until proj 9.3.1 migration completes -proj: - - "9.3.0" From e07d66c19e085ae73ec32c0a0fd924799a18d6f2 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Wed, 4 Sep 2024 13:04:30 -0500 Subject: [PATCH 05/28] fix wheel tests on Rocky Linux (#1452) RAPIDS libraries recently started running nightly wheel tests on Rocky Linux 8: https://github.com/rapidsai/shared-workflows/pull/236/files#r1725947245 That distribution's system package manager is `yum`, not `apt`, and as a result `cuspatial`'s nightly runs are failing like this: ```text ci/test_wheel_cuspatial.sh: line 10: apt: command not found Error: Process completed with exit code 127. ``` ([build link](https://github.com/rapidsai/cuspatial/actions/runs/10678284262/job/29595047179)) This fixes that. ## Notes for Reviewers ### How I tested this ```shell docker run \ --rm \ --gpus 1 \ -v $(pwd):/opt/work \ -w /opt/work \ -it rapidsai/citestwheel:cuda12.5.1-rockylinux8-py3.11 \ bash yum update -y yum config-manager --set-enabled powertools yum update -y yum install -y gdal-devel python -m pip install \ --no-binary fiona \ 'cuproj-cu12[test]==24.10.*,>=0.0.0a0' \ 'cuspatial-cu12[test]==24.10.*,>=0.0.0a0' \ 'fiona>=1.8.19,<1.9' pushd python/cuproj/cuproj python -m pytest \ --cache-clear \ --numprocesses=8 \ --dist=worksteal \ tests popd pushd python/cuspatial/cuspatial python -m pytest \ --cache-clear \ --numprocesses=8 \ --dist=worksteal \ tests popd ``` # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cuspatial/pull/1452 --- README.md | 7 +++++++ ci/test_wheel_cuproj.sh | 12 ++++++++++-- ci/test_wheel_cuspatial.sh | 12 ++++++++++-- 3 files changed, 27 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 96ffa4502..a2a2be972 100644 --- a/README.md +++ b/README.md @@ -164,6 +164,13 @@ There are two ways to fix this: sudo -y add-apt-repository ppa:ubuntugis/ppa sudo apt install libgdal-dev ``` + * Rocky Linux users can install a newer GDAL as follows + ```shell + yum update -y + yum config-manager --set-enabled powertools + yum update -y + yum install -y gdal-devel + ``` 2. Pin fiona's version to a range that's compatible with your version of `libgdal-dev` * For Ubuntu20.04 ([GDAL v3.0.4](https://packages.ubuntu.com/focal/libgdal-dev)): ```shell diff --git a/ci/test_wheel_cuproj.sh b/ci/test_wheel_cuproj.sh index 4144e8dab..594702479 100755 --- a/ci/test_wheel_cuproj.sh +++ b/ci/test_wheel_cuproj.sh @@ -7,8 +7,16 @@ mkdir -p ./dist RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})" # install build dependencies for fiona -apt update -DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev +if type -f yum > /dev/null 2>&1; then + yum update -y + # some of gdal-devel's dependencies, like 'libdap', come from the powertools repo + yum config-manager --set-enabled powertools + yum update -y + yum install -y gdal-devel +else + apt update + DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev +fi # Download the cuproj and cuspatial built in the previous step RAPIDS_PY_WHEEL_NAME="cuproj_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist diff --git a/ci/test_wheel_cuspatial.sh b/ci/test_wheel_cuspatial.sh index 8faa4e8d3..c4708e80e 100755 --- a/ci/test_wheel_cuspatial.sh +++ b/ci/test_wheel_cuspatial.sh @@ -7,8 +7,16 @@ mkdir -p ./dist RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})" # install build dependencies for fiona -apt update -DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev +if type -f yum > /dev/null 2>&1; then + yum update -y + # some of gdal-devel's dependencies, like 'libdap', come from the powertools repo + yum config-manager --set-enabled powertools + yum update -y + yum install -y gdal-devel +else + apt update + DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev +fi # Download the cuspatial built in the previous step RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist From 0e9d36cff2c8e4fcfdd9e88e7ec3fa07a6db6184 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Wed, 4 Sep 2024 18:40:09 -0500 Subject: [PATCH 06/28] Distribute libcuspatial wheels (#1450) Contributes to https://github.com/rapidsai/build-planning/issues/33 Adds `libcuspatial` wheels, and switches `cuspatial` wheels to using them. ## Notes for Reviewers ### Benefits of these changes Faster CI runs and smaller total footprint on package repositories (because now `libcuspatial` no longer needs to be compiled once per Python version). Smaller `cuspatial` wheels. | whee. l | size (before) | size (this PR) | |:-------------:|-------------:|----------------:| | `libcuspatial` | --- | 17.0M | | `cuspatial`. | 21.0M | 4.1M | | `cuproj ` | 0.9M | 0.9M | |**TOTAL** | **21.9M** | **22.0M** | *NOTES: size = compressed, "before" = 2024-09-02 nightlies (https://github.com/rapidsai/cuspatial/commit/1544e7bd16f11457436d1fb6a43da0db3aa9af57), CUDA = 12, Python = 3.11*
how I calculated those (click me) ```shell docker run \ --rm \ -v $(pwd):/opt/work:ro \ -w /opt/work \ --network host \ --env RAPIDS_NIGHTLY_DATE=2024-09-02 \ --env RAPIDS_NIGHTLY_SHA=1544e7b \ --env RAPIDS_PR_NUMBER=1450 \ --env RAPIDS_PY_CUDA_SUFFIX=cu12 \ --env RAPIDS_REPOSITORY=rapidsai/cuspatial \ --env WHEEL_DIR_BEFORE=/tmp/wheels-before \ --env WHEEL_DIR_AFTER=/tmp/wheels-after \ -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.11 \ bash mkdir -p "${WHEEL_DIR_BEFORE}" mkdir -p "${WHEEL_DIR_AFTER}" py_projects=( cuspatial cuproj ) for project in "${py_projects[@]}"; do # before RAPIDS_BUILD_TYPE=nightly \ RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REF_NAME="branch-24.10" \ RAPIDS_SHA=${RAPIDS_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python "${WHEEL_DIR_BEFORE}" # after RAPIDS_BUILD_TYPE=pull-request \ RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REF_NAME="pull-request/${RAPIDS_PR_NUMBER}" \ rapids-download-wheels-from-s3 python "${WHEEL_DIR_AFTER}" done # after RAPIDS_BUILD_TYPE=pull-request \ RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REF_NAME="pull-request/${RAPIDS_PR_NUMBER}" \ rapids-download-wheels-from-s3 cpp "${WHEEL_DIR_AFTER}" du -sh ${WHEEL_DIR_BEFORE}/* du -sh ${WHEEL_DIR_BEFORE} du -sh ${WHEEL_DIR_AFTER}/* du -sh ${WHEEL_DIR_AFTER} ```
### devcontainers job? Once this PR is close to ready, let's merge the devcontainers PR and then re-run the devcontainers CI here. devcontainers PR: https://github.com/rapidsai/devcontainers/pull/387 ### `rapids-metadata` changes? Not necessary, `libcuspatial` is already there: https://github.com/rapidsai/rapids-metadata/blob/9b6307e708511cd9a1990d8bb36606df53bc9e1b/src/rapids_metadata/__init__.py#L89 # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Mark Harris (https://github.com/harrism) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cuspatial/pull/1450 --- .github/workflows/build.yaml | 23 ++++++ .github/workflows/pr.yaml | 12 ++- build.sh | 5 -- ci/build_wheel.sh | 21 +++++- ci/build_wheel_cuspatial.sh | 11 +++ ci/build_wheel_libcuspatial.sh | 6 ++ ci/test_wheel_cuproj.sh | 2 + ci/test_wheel_cuspatial.sh | 4 +- cpp/cmake/thirdparty/get_ranger.cmake | 4 +- dependencies.yaml | 59 ++++++++++++++- python/cuspatial/CMakeLists.txt | 27 +------ python/cuspatial/cuspatial/__init__.py | 14 +++- python/cuspatial/pyproject.toml | 2 + python/libcuspatial/CMakeLists.txt | 42 +++++++++++ python/libcuspatial/LICENSE | 1 + python/libcuspatial/README.md | 1 + python/libcuspatial/libcuspatial/VERSION | 1 + python/libcuspatial/libcuspatial/__init__.py | 16 ++++ python/libcuspatial/libcuspatial/_version.py | 32 ++++++++ python/libcuspatial/libcuspatial/load.py | 55 ++++++++++++++ python/libcuspatial/pyproject.toml | 77 ++++++++++++++++++++ 21 files changed, 373 insertions(+), 42 deletions(-) create mode 100755 ci/build_wheel_libcuspatial.sh create mode 100644 python/libcuspatial/CMakeLists.txt create mode 120000 python/libcuspatial/LICENSE create mode 120000 python/libcuspatial/README.md create mode 120000 python/libcuspatial/libcuspatial/VERSION create mode 100644 python/libcuspatial/libcuspatial/__init__.py create mode 100644 python/libcuspatial/libcuspatial/_version.py create mode 100644 python/libcuspatial/libcuspatial/load.py create mode 100644 python/libcuspatial/pyproject.toml diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 1f3fd7922..d86122f5a 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -66,7 +66,30 @@ jobs: node_type: "gpu-v100-latest-1" run_script: "ci/build_docs.sh" sha: ${{ inputs.sha }} + wheel-build-libcuspatial: + secrets: inherit + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + with: + # build for every combination of arch and CUDA version, but only for the latest Python + matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) + build_type: ${{ inputs.build_type || 'branch' }} + branch: ${{ inputs.branch }} + sha: ${{ inputs.sha }} + date: ${{ inputs.date }} + script: ci/build_wheel_libcuspatial.sh + wheel-publish-libcuspatial: + needs: wheel-build-libcuspatial + secrets: inherit + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10 + with: + build_type: ${{ inputs.build_type || 'branch' }} + branch: ${{ inputs.branch }} + sha: ${{ inputs.sha }} + date: ${{ inputs.date }} + package-name: libcuspatial + package-type: cpp wheel-build-cuspatial: + needs: wheel-publish-libcuspatial secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 with: diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index ecf4206c7..94617d953 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -21,6 +21,7 @@ jobs: - conda-notebook-tests - docs-build - wheel-build-cuspatial + - wheel-build-libcuspatial - wheel-tests-cuspatial - wheel-build-cuproj - wheel-tests-cuproj @@ -83,10 +84,19 @@ jobs: arch: "amd64" container_image: "rapidsai/ci-conda:latest" run_script: "ci/build_docs.sh" - wheel-build-cuspatial: + wheel-build-libcuspatial: needs: checks secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + with: + # build for every combination of arch and CUDA version, but only for the latest Python + matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) + build_type: pull-request + script: ci/build_wheel_libcuspatial.sh + wheel-build-cuspatial: + needs: [checks, wheel-build-libcuspatial] + secrets: inherit + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 with: build_type: pull-request script: ci/build_wheel_cuspatial.sh diff --git a/build.sh b/build.sh index ffe97da9a..77b8fa3eb 100755 --- a/build.sh +++ b/build.sh @@ -124,11 +124,6 @@ if hasArg benchmarks; then BUILD_BENCHMARKS=ON fi -# Append `-DFIND_CUSPATIAL_CPP=ON` to EXTRA_CMAKE_ARGS unless a user specified the option. -if [[ "${EXTRA_CMAKE_ARGS}" != *"DFIND_CUSPATIAL_CPP"* ]]; then - EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DFIND_CUSPATIAL_CPP=ON" -fi - # Append `-DFIND_CUPROJ_CPP=ON` to EXTRA_CMAKE_ARGS unless a user specified the option. if [[ "${EXTRA_CMAKE_ARGS}" != *"DFIND_CUPROJ_CPP"* ]]; then EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DFIND_CUPROJ_CPP=ON" diff --git a/ci/build_wheel.sh b/ci/build_wheel.sh index 14802741d..75328d272 100755 --- a/ci/build_wheel.sh +++ b/ci/build_wheel.sh @@ -7,6 +7,25 @@ package_name=$1 package_dir=$2 package_type=$3 +# The 'libcuspatial' wheel should package 'libcuspatial.so', and all others +# should exclude it (they dynamically load it if they need it). +# +# Capturing that here in argument-parsing to allow this build_wheel.sh +# script to be re-used by all wheel builds in the project. +case "${package_dir}" in + python/libcuspatial) + EXCLUDE_ARGS=( + --exclude "libcudf.so" + ) + ;; + *) + EXCLUDE_ARGS=( + --exclude "libcudf.so" + --exclude "libcuspatial.so" + ) + ;; +esac + source rapids-configure-sccache source rapids-date-string @@ -20,7 +39,7 @@ python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check mkdir -p final_dist python -m auditwheel repair \ - --exclude libcudf.so \ + "${EXCLUDE_ARGS[@]}" \ -w final_dist \ dist/* diff --git a/ci/build_wheel_cuspatial.sh b/ci/build_wheel_cuspatial.sh index 171bfa6a4..e581a3e76 100755 --- a/ci/build_wheel_cuspatial.sh +++ b/ci/build_wheel_cuspatial.sh @@ -3,4 +3,15 @@ set -euo pipefail +RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})" + +# Downloads libcuspatial wheel from this current build, +# then ensures 'cuspatial' wheel builds always use the 'libcuspatial' just built in the same CI run. +# +# Using env variable PIP_CONSTRAINT is necessary to ensure the constraints +# are used when creating the isolated build environment. +RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp /tmp/libcuspatial_dist +echo "libcuspatial-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo /tmp/libcuspatial_dist/libcuspatial_*.whl)" > /tmp/constraints.txt +export PIP_CONSTRAINT="/tmp/constraints.txt" + ci/build_wheel.sh cuspatial python/cuspatial python diff --git a/ci/build_wheel_libcuspatial.sh b/ci/build_wheel_libcuspatial.sh new file mode 100755 index 000000000..742a58caa --- /dev/null +++ b/ci/build_wheel_libcuspatial.sh @@ -0,0 +1,6 @@ +#!/bin/bash +# Copyright (c) 2024, NVIDIA CORPORATION. + +set -euo pipefail + +ci/build_wheel.sh libcuspatial python/libcuspatial cpp diff --git a/ci/test_wheel_cuproj.sh b/ci/test_wheel_cuproj.sh index 594702479..27cc87a2a 100755 --- a/ci/test_wheel_cuproj.sh +++ b/ci/test_wheel_cuproj.sh @@ -21,12 +21,14 @@ fi # Download the cuproj and cuspatial built in the previous step RAPIDS_PY_WHEEL_NAME="cuproj_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist +RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp ./dist # echo to expand wildcard before adding `[extra]` requires for pip python -m pip install \ --no-binary 'fiona' \ "$(echo ./dist/cuspatial*.whl)" \ "$(echo ./dist/cuproj*.whl)[test]" \ + "$(echo ./dist/libcuspatial*.whl)" \ 'fiona>=1.8.19,<1.9' rapids-logger "pytest cuproj" diff --git a/ci/test_wheel_cuspatial.sh b/ci/test_wheel_cuspatial.sh index c4708e80e..a8f79832c 100755 --- a/ci/test_wheel_cuspatial.sh +++ b/ci/test_wheel_cuspatial.sh @@ -18,13 +18,15 @@ else DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev fi -# Download the cuspatial built in the previous step +# Download the cuspatial and libcuspatial built in the previous step RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist +RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp ./dist # echo to expand wildcard before adding `[extra]` requires for pip python -m pip install \ --no-binary 'fiona' \ "$(echo ./dist/cuspatial*.whl)[test]" \ + "$(echo ./dist/libcuspatial*.whl)" \ 'fiona>=1.8.19,<1.9' rapids-logger "pytest cuspatial" diff --git a/cpp/cmake/thirdparty/get_ranger.cmake b/cpp/cmake/thirdparty/get_ranger.cmake index 563e4120a..c57131f3a 100644 --- a/cpp/cmake/thirdparty/get_ranger.cmake +++ b/cpp/cmake/thirdparty/get_ranger.cmake @@ -1,5 +1,5 @@ #============================================================================= -# Copyright (c) 2023, NVIDIA CORPORATION. +# Copyright (c) 2023-2024, NVIDIA CORPORATION. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -26,8 +26,6 @@ function(find_and_configure_ranger) rapids_cpm_find( ranger 00.01.00 GLOBAL_TARGETS "${global_targets}" - BUILD_EXPORT_SET cuspatial-exports - INSTALL_EXPORT_SET cuspatial-exports CPM_ARGS GIT_REPOSITORY https://github.com/harrism/ranger.git GIT_TAG main diff --git a/dependencies.yaml b/dependencies.yaml index 859009683..463683232 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -18,6 +18,7 @@ files: - depends_on_cuml - depends_on_cupy - depends_on_libcudf + - depends_on_libcuspatial - depends_on_librmm - rapids_build_skbuild - run_python_cuspatial @@ -30,11 +31,13 @@ files: output: none includes: - cuda_version + - depends_on_libcuspatial - test_libcuspatial test_python: output: none includes: - cuda_version + - depends_on_libcuspatial - py_version - test_python_cuspatial - test_python_cuproj @@ -44,6 +47,7 @@ files: includes: - cuda_version - depends_on_cuml + - depends_on_libcuspatial - test_notebooks - notebooks - py_version @@ -57,9 +61,35 @@ files: output: none includes: - cuda_version + - depends_on_libcuspatial - docs - py_version - test_cuspatial + py_build_libcuspatial: + output: [pyproject] + pyproject_dir: python/libcuspatial + extras: + table: build-system + includes: + - rapids_build_skbuild + py_rapids_build_libcuspatial: + output: [pyproject] + pyproject_dir: python/libcuspatial + extras: + table: tool.rapids-build-backend + key: requires + includes: + - build_cpp + - build_wheels + - depends_on_libcudf + - depends_on_librmm + py_run_libcuspatial: + output: [pyproject] + pyproject_dir: python/libcuspatial + extras: + table: project + includes: + - depends_on_libcudf py_build_cuspatial: output: [pyproject] pyproject_dir: python/cuspatial @@ -80,6 +110,7 @@ files: - build_python - build_wheels - depends_on_libcudf + - depends_on_libcuspatial - depends_on_librmm py_run_cuspatial: output: [pyproject] @@ -89,6 +120,7 @@ files: includes: - depends_on_cudf - depends_on_libcudf + - depends_on_libcuspatial - depends_on_rmm - run_python_cuspatial py_test_cuspatial: @@ -540,16 +572,39 @@ dependencies: common: - output_types: conda packages: - - libcuspatial==24.10.*,>=0.0.0a0 - libcuspatial-tests==24.10.*,>=0.0.0a0 test_cuspatial: common: - output_types: conda packages: - - libcuspatial==24.10.*,>=0.0.0a0 - cuspatial==24.10.*,>=0.0.0a0 - cuproj==24.10.*,>=0.0.0a0 + depends_on_libcuspatial: + common: + - output_types: conda + packages: + - &libcuspatial_unsuffixed libcuspatial==24.10.*,>=0.0.0a0 + - output_types: requirements + packages: + # pip recognizes the index as a global option for the requirements.txt file + - --extra-index-url=https://pypi.nvidia.com + - --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple + specific: + - output_types: [requirements, pyproject] + matrices: + - matrix: + cuda: "12.*" + cuda_suffixed: "true" + packages: + - libcuspatial-cu12==24.10.*,>=0.0.0a0 + - matrix: + cuda: "11.*" + cuda_suffixed: "true" + packages: + - libcuspatial-cu11==24.10.*,>=0.0.0a0 + - {matrix: null, packages: [*libcuspatial_unsuffixed]} + depends_on_librmm: common: - output_types: conda diff --git a/python/cuspatial/CMakeLists.txt b/python/cuspatial/CMakeLists.txt index 28fcf467d..c3dc00c8a 100644 --- a/python/cuspatial/CMakeLists.txt +++ b/python/cuspatial/CMakeLists.txt @@ -23,34 +23,9 @@ project( VERSION "${RAPIDS_VERSION}" LANGUAGES CXX CUDA) -option(FIND_CUSPATIAL_CPP "Search for existing cuspatial C++ installations before defaulting to local files" - OFF) - -# If the user requested it we attempt to find cuspatial. -if(FIND_CUSPATIAL_CPP) - find_package(cuspatial "${RAPIDS_VERSION}") -else() - set(cuspatial_FOUND OFF) -endif() - -if(NOT cuspatial_FOUND) - set(BUILD_TESTS OFF) - set(BUILD_BENCHMARKS OFF) - set(CUDA_STATIC_RUNTIME ON) - set(CUSPATIAL_USE_CUDF_STATIC ON) - set(CUSPATIAL_EXCLUDE_CUDF_FROM_ALL ON) - - add_subdirectory(../../cpp cuspatial-cpp EXCLUDE_FROM_ALL) - - set(cython_lib_dir cuspatial) - install(TARGETS cuspatial DESTINATION ${cython_lib_dir}) -endif() +find_package(cuspatial "${RAPIDS_VERSION}" REQUIRED) include(rapids-cython-core) rapids_cython_init() add_subdirectory(cuspatial/_lib) - -if(DEFINED cython_lib_dir) - rapids_cython_add_rpath_entries(TARGET cuspatial PATHS "${cython_lib_dir}") -endif() diff --git a/python/cuspatial/cuspatial/__init__.py b/python/cuspatial/cuspatial/__init__.py index 14a5fcc4d..89c193819 100644 --- a/python/cuspatial/cuspatial/__init__.py +++ b/python/cuspatial/cuspatial/__init__.py @@ -1,8 +1,16 @@ # Copyright (c) 2023-2024, NVIDIA CORPORATION. -# load cudf before any other imports, to be sure libcudf's symbols are found -# in the libcudf.so from the libcudf wheel (if one is installed) -import cudf +# If libcuspatial was installed as a wheel, we must request it to load the library +# symbols. Otherwise, we assume that the library was installed in a system path that ld +# can find. +try: + import libcuspatial +except ModuleNotFoundError: + pass +else: + libcuspatial.load_library() + del libcuspatial + from ._version import __git_commit__, __version__ from .core.geodataframe import GeoDataFrame diff --git a/python/cuspatial/pyproject.toml b/python/cuspatial/pyproject.toml index 25ea28bfd..f8aba3c4b 100644 --- a/python/cuspatial/pyproject.toml +++ b/python/cuspatial/pyproject.toml @@ -33,6 +33,7 @@ dependencies = [ "cudf==24.10.*,>=0.0.0a0", "geopandas>=0.11.0", "libcudf==24.10.*,>=0.0.0a0", + "libcuspatial==24.10.*,>=0.0.0a0", "numpy>=1.23,<2.0a0", "rmm==24.10.*,>=0.0.0a0", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. @@ -137,6 +138,7 @@ requires = [ "cudf==24.10.*,>=0.0.0a0", "cython>=3.0.0", "libcudf==24.10.*,>=0.0.0a0", + "libcuspatial==24.10.*,>=0.0.0a0", "librmm==24.10.*,>=0.0.0a0", "ninja", "rmm==24.10.*,>=0.0.0a0", diff --git a/python/libcuspatial/CMakeLists.txt b/python/libcuspatial/CMakeLists.txt new file mode 100644 index 000000000..86c3cd1ef --- /dev/null +++ b/python/libcuspatial/CMakeLists.txt @@ -0,0 +1,42 @@ +# ============================================================================= +# Copyright (c) 2024, NVIDIA CORPORATION. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except +# in compliance with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed under the License +# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express +# or implied. See the License for the specific language governing permissions and limitations under +# the License. +# ============================================================================= + +cmake_minimum_required(VERSION 3.26.4 FATAL_ERROR) + +include(../../rapids_config.cmake) + +project( + libcuspatial-python + VERSION "${RAPIDS_VERSION}" + LANGUAGES CXX +) + +# Check if cuspatial is already available. If so, it is the user's responsibility to ensure that the +# CMake package is also available at build time of the Python cuspatial package. +find_package(cuspatial "${RAPIDS_VERSION}") +if(cuspatial_FOUND) + return() +endif() + +unset(cuspatial_FOUND) + +# set options for building libcuspatial +set(BUILD_SHARED_LIBS ON) +set(BUILD_TESTS OFF) +set(BUILD_BENCHMARKS OFF) +set(CUDA_STATIC_RUNTIME ON) +set(CUSPATIAL_USE_CUDF_STATIC OFF) +set(CUSPATIAL_EXCLUDE_CUDF_FROM_ALL ON) + +add_subdirectory(../../cpp cuspatial-cpp) diff --git a/python/libcuspatial/LICENSE b/python/libcuspatial/LICENSE new file mode 120000 index 000000000..30cff7403 --- /dev/null +++ b/python/libcuspatial/LICENSE @@ -0,0 +1 @@ +../../LICENSE \ No newline at end of file diff --git a/python/libcuspatial/README.md b/python/libcuspatial/README.md new file mode 120000 index 000000000..fe8400541 --- /dev/null +++ b/python/libcuspatial/README.md @@ -0,0 +1 @@ +../../README.md \ No newline at end of file diff --git a/python/libcuspatial/libcuspatial/VERSION b/python/libcuspatial/libcuspatial/VERSION new file mode 120000 index 000000000..d62dc733e --- /dev/null +++ b/python/libcuspatial/libcuspatial/VERSION @@ -0,0 +1 @@ +../../../VERSION \ No newline at end of file diff --git a/python/libcuspatial/libcuspatial/__init__.py b/python/libcuspatial/libcuspatial/__init__.py new file mode 100644 index 000000000..49b02bb14 --- /dev/null +++ b/python/libcuspatial/libcuspatial/__init__.py @@ -0,0 +1,16 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from libcuspatial._version import __git_commit__, __version__ +from libcuspatial.load import load_library diff --git a/python/libcuspatial/libcuspatial/_version.py b/python/libcuspatial/libcuspatial/_version.py new file mode 100644 index 000000000..826e84a67 --- /dev/null +++ b/python/libcuspatial/libcuspatial/_version.py @@ -0,0 +1,32 @@ +# Copyright (c) 2023-2024, NVIDIA CORPORATION. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import importlib.resources + +__version__ = ( + importlib.resources.files(__package__) + .joinpath("VERSION") + .read_text() + .strip() +) +try: + __git_commit__ = ( + importlib.resources.files(__package__) + .joinpath("GIT_COMMIT") + .read_text() + .strip() + ) +except FileNotFoundError: + __git_commit__ = "" + +__all__ = ["__git_commit__", "__version__"] diff --git a/python/libcuspatial/libcuspatial/load.py b/python/libcuspatial/libcuspatial/load.py new file mode 100644 index 000000000..285628187 --- /dev/null +++ b/python/libcuspatial/libcuspatial/load.py @@ -0,0 +1,55 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +import ctypes +import os + +import libcudf + + +def load_library(): + # libcudf must be loaded before libcuspatial because libcuspatial + # references its symbols + libcudf.load_library() + + # Dynamically load libcuspatial.so. Prefer a system library if one is + # present to avoid clobbering symbols that other packages might expect, + # but if no other library is present use the one in the wheel. + libcuspatial_lib = None + try: + libcuspatial_lib = ctypes.CDLL("libcuspatial.so", ctypes.RTLD_GLOBAL) + except OSError: + # If neither of these directories contain the library, we assume we are + # in an environment where the C++ library is already installed + # somewhere else and the CMake build of the libcuspatial Python package + # was a no-op. + # + # Note that this approach won't work for real editable installs of the + # libcuspatial package. scikit-build-core has limited support for + # importlib.resources so there isn't a clean way to support that case + # yet. + for lib_dir in ("lib", "lib64"): + if os.path.isfile( + lib := os.path.join( + os.path.dirname(__file__), lib_dir, "libcuspatial.so" + ) + ): + libcuspatial_lib = ctypes.CDLL(lib, ctypes.RTLD_GLOBAL) + break + + # The caller almost never needs to do anything with this library, but no + # harm in offering the option since this object at least provides a handle + # to inspect where the library was loaded from. + return libcuspatial_lib diff --git a/python/libcuspatial/pyproject.toml b/python/libcuspatial/pyproject.toml new file mode 100644 index 000000000..ab64d4cb6 --- /dev/null +++ b/python/libcuspatial/pyproject.toml @@ -0,0 +1,77 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[build-system] +build-backend = "rapids_build_backend.build" +requires = [ + "rapids-build-backend>=0.3.0,<0.4.0.dev0", + "scikit-build-core[pyproject]>=0.10.0", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. + +[project] +name = "libcuspatial" +dynamic = ["version"] +description = "cuSpatial: GPU-Accelerated Spatial and Trajectory Data Management and Analytics Library (C++)" +readme = { file = "README.md", content-type = "text/markdown" } +authors = [ + { name = "NVIDIA Corporation" }, +] +license = { text = "Apache 2.0" } +requires-python = ">=3.10" +classifiers = [ + "Intended Audience :: Developers", + "Topic :: Database", + "Topic :: Scientific/Engineering", + "License :: OSI Approved :: Apache Software License", + "Programming Language :: Python", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", +] +dependencies = [ + "libcudf==24.10.*,>=0.0.0a0", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. + +[project.urls] +Homepage = "https://github.com/rapidsai/cuspatial" + +[project.entry-points."cmake.prefix"] +libcuspatial = "libcuspatial" + +[tool.scikit-build] +build-dir = "build/{wheel_tag}" +cmake.build-type = "Release" +cmake.version = "CMakeLists.txt" +minimum-version = "build-system.requires" +ninja.make-fallback = true +sdist.reproducible = true +wheel.packages = ["libcuspatial"] +wheel.install-dir = "libcuspatial" +wheel.py-api = "py3" + +[tool.scikit-build.metadata.version] +provider = "scikit_build_core.metadata.regex" +input = "libcuspatial/VERSION" +regex = "(?P.*)" + +[tool.rapids-build-backend] +build-backend = "scikit_build_core.build" +dependencies-file = "../../dependencies.yaml" +matrix-entry = "cuda_suffixed=true" +requires = [ + "cmake>=3.26.4,!=3.30.0", + "libcudf==24.10.*,>=0.0.0a0", + "librmm==24.10.*,>=0.0.0a0", + "ninja", + "wheel", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. From 9385e4937c017a3e7d5a7e9006e46cf45fb04204 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Fri, 6 Sep 2024 07:51:50 -0500 Subject: [PATCH 07/28] re-enable mixing libcuspatial wheels with libcudf conda packages (#1456) Fixes #1455 devcontainer conda CI jobs are failing in this project because of the following mix of characteristics for thoes jobs: * all build and runtime dependencies for `libcuspatial`, `cuspatial`, `cuproj` are installed via conda * `libcuspatial`, `cuspatial`, and `cuproj` wheels are then built with `pip install -e --no-deps --no-build-isolation` * `import libcuspatial` results in unconditionally running `import libcudf` * `libcudf` is provided by the `libcudf` **conda** package, which does not have any Python modules, so that import fails This fixes that, and restores the ability to mix a `pip install`'d `cuspatial` / `cuproj` with a `conda`-installed `libcudf`. ## Notes for Reviewers ### How did CI not catch this before? When https://github.com/rapidsai/devcontainers/pull/387 was merged, I only re-ran the **pip** devcontainers CI job on #1450. # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cuspatial/pull/1456 --- python/libcuspatial/libcuspatial/load.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/python/libcuspatial/libcuspatial/load.py b/python/libcuspatial/libcuspatial/load.py index 285628187..7842de9a0 100644 --- a/python/libcuspatial/libcuspatial/load.py +++ b/python/libcuspatial/libcuspatial/load.py @@ -16,13 +16,22 @@ import ctypes import os -import libcudf - def load_library(): - # libcudf must be loaded before libcuspatial because libcuspatial - # references its symbols - libcudf.load_library() + try: + # libcudf must be loaded before libcuspatial because libcuspatial + # references its symbols + import libcudf + libcudf.load_library() + except ModuleNotFoundError: + # 'libcuspatial' has a runtime dependency on 'libcudf'. However, + # that dependency might be satisfied by the 'libcudf' conda package + # (which does not have any Python modules), instead of the + # 'libcudf' wheel. + # + # In that situation, assume that 'libcudf.so' is in a place where + # the loader can find it. + pass # Dynamically load libcuspatial.so. Prefer a system library if one is # present to avoid clobbering symbols that other packages might expect, From 0bbd71da65675c3ee29e6c6c09c6eac6b8e72d7b Mon Sep 17 00:00:00 2001 From: James Lamb Date: Mon, 9 Sep 2024 08:00:26 -0700 Subject: [PATCH 08/28] rearrange dependencies.yaml, fix development instructions (#1451) Follow-up to #1448. This proposes some `dependencies.yaml` changes that I noticed while working on adding `libcuspatial` wheels in #1450. I've left inline comments with more details, but in short: * not including any cuspatial projects in the conda environment files checked into this repo (which are intended to be used to set up a development environment) * removing an unnecessary conda-only list of dependencies from pyproject.toml-specific config for `cuproj` * introduction of `depends_on_libcuspatial` and `depends_on_cuproj` lists in `dependencies.yaml`, to reduce duplication Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cuspatial/pull/1451 --- .../all_cuda-118_arch-x86_64.yaml | 2 - .../all_cuda-125_arch-x86_64.yaml | 2 - dependencies.yaml | 47 ++++++++++++------- 3 files changed, 31 insertions(+), 20 deletions(-) diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml index 0cbe9a1f7..211d27b68 100644 --- a/conda/environments/all_cuda-118_arch-x86_64.yaml +++ b/conda/environments/all_cuda-118_arch-x86_64.yaml @@ -24,8 +24,6 @@ dependencies: - ipython - ipywidgets - libcudf==24.10.*,>=0.0.0a0 -- libcuspatial-tests==24.10.*,>=0.0.0a0 -- libcuspatial==24.10.*,>=0.0.0a0 - librmm==24.10.*,>=0.0.0a0 - myst-parser - nbsphinx diff --git a/conda/environments/all_cuda-125_arch-x86_64.yaml b/conda/environments/all_cuda-125_arch-x86_64.yaml index 1001169e5..fc782ccfc 100644 --- a/conda/environments/all_cuda-125_arch-x86_64.yaml +++ b/conda/environments/all_cuda-125_arch-x86_64.yaml @@ -27,8 +27,6 @@ dependencies: - ipython - ipywidgets - libcudf==24.10.*,>=0.0.0a0 -- libcuspatial-tests==24.10.*,>=0.0.0a0 -- libcuspatial==24.10.*,>=0.0.0a0 - librmm==24.10.*,>=0.0.0a0 - myst-parser - nbsphinx diff --git a/dependencies.yaml b/dependencies.yaml index 463683232..859605b4f 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -18,11 +18,9 @@ files: - depends_on_cuml - depends_on_cupy - depends_on_libcudf - - depends_on_libcuspatial - depends_on_librmm - rapids_build_skbuild - run_python_cuspatial - - test_libcuspatial - test_notebooks - test_python_cuspatial - test_python_cuproj @@ -37,21 +35,21 @@ files: output: none includes: - cuda_version - - depends_on_libcuspatial + - depends_on_cuproj + - depends_on_cuspatial - py_version - test_python_cuspatial - test_python_cuproj - - test_cuspatial test_notebooks: output: none includes: - cuda_version - depends_on_cuml - - depends_on_libcuspatial + - depends_on_cuproj + - depends_on_cuspatial - test_notebooks - notebooks - py_version - - test_cuspatial checks: output: none includes: @@ -61,10 +59,10 @@ files: output: none includes: - cuda_version - - depends_on_libcuspatial + - depends_on_cuspatial + - depends_on_cuproj - docs - py_version - - test_cuspatial py_build_libcuspatial: output: [pyproject] pyproject_dir: python/libcuspatial @@ -165,7 +163,6 @@ files: includes: - test_python_cuproj - depends_on_cuspatial - - test_cuspatial channels: - rapidsai @@ -551,6 +548,31 @@ dependencies: - cuspatial-cu11==24.10.*,>=0.0.0a0 - {matrix: null, packages: [*cuspatial_unsuffixed]} + depends_on_cuproj: + common: + - output_types: conda + packages: + - &cuproj_unsuffixed cuproj==24.10.*,>=0.0.0a0 + - output_types: requirements + packages: + # pip recognizes the index as a global option for the requirements.txt file + - --extra-index-url=https://pypi.nvidia.com + - --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple + specific: + - output_types: [requirements, pyproject] + matrices: + - matrix: + cuda: "12.*" + cuda_suffixed: "true" + packages: + - cuproj-cu12==24.10.*,>=0.0.0a0 + - matrix: + cuda: "11.*" + cuda_suffixed: "true" + packages: + - cuproj-cu11==24.10.*,>=0.0.0a0 + - {matrix: null, packages: [*cuproj_unsuffixed]} + depends_on_cupy: common: - output_types: conda @@ -573,13 +595,6 @@ dependencies: - output_types: conda packages: - libcuspatial-tests==24.10.*,>=0.0.0a0 - test_cuspatial: - common: - - output_types: conda - packages: - - cuspatial==24.10.*,>=0.0.0a0 - - cuproj==24.10.*,>=0.0.0a0 - depends_on_libcuspatial: common: - output_types: conda From 2a30f5f3c4153a65af2a57756661cc7e96634533 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Mon, 9 Sep 2024 09:57:35 -0700 Subject: [PATCH 09/28] Add support for Python 3.12, ping proj back to 9.3.x, require geopandas>=1.0 (#1453) Contributes to https://github.com/rapidsai/build-planning/issues/40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from https://github.com/rapidsai/shared-workflows/pull/213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. Other changes required to add that support: * pinning `proj` back to 9.3.x, reverting #1449 (https://github.com/rapidsai/cuspatial/pull/1453#issuecomment-2335527542) * requiring `geopandas >= 1.0` (https://github.com/rapidsai/cuspatial/pull/1453#discussion_r1750444386) ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cuspatial/pull/1453 --- .github/workflows/build.yaml | 20 ++++----- .github/workflows/pr.yaml | 30 ++++++------- .github/workflows/test.yaml | 10 ++--- README.md | 42 ++----------------- ci/test_wheel_cuproj.sh | 16 +------ ci/test_wheel_cuspatial.sh | 16 +------ .../all_cuda-118_arch-x86_64.yaml | 7 ++-- .../all_cuda-125_arch-x86_64.yaml | 7 ++-- conda/recipes/cuproj/meta.yaml | 2 +- conda/recipes/cuspatial/meta.yaml | 2 +- conda/recipes/libcuspatial/meta.yaml | 4 +- dependencies.yaml | 15 +++---- python/cuproj/pyproject.toml | 3 +- python/cuspatial/pyproject.toml | 3 +- 14 files changed, 57 insertions(+), 120 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index d86122f5a..14f3817a0 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -28,7 +28,7 @@ concurrency: jobs: cpp-build: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@python-3.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -37,7 +37,7 @@ jobs: python-build: needs: [cpp-build] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@python-3.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -46,7 +46,7 @@ jobs: upload-conda: needs: [cpp-build, python-build] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@python-3.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -56,7 +56,7 @@ jobs: if: github.ref_type == 'branch' needs: python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12 with: arch: "amd64" branch: ${{ inputs.branch }} @@ -68,7 +68,7 @@ jobs: sha: ${{ inputs.sha }} wheel-build-libcuspatial: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 with: # build for every combination of arch and CUDA version, but only for the latest Python matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) @@ -80,7 +80,7 @@ jobs: wheel-publish-libcuspatial: needs: wheel-build-libcuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -91,7 +91,7 @@ jobs: wheel-build-cuspatial: needs: wheel-publish-libcuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -101,7 +101,7 @@ jobs: wheel-publish-cuspatial: needs: wheel-build-cuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -111,7 +111,7 @@ jobs: package-type: python wheel-build-cuproj: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -121,7 +121,7 @@ jobs: wheel-publish-cuproj: needs: wheel-build-cuproj secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index 94617d953..65a49b87d 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -27,47 +27,47 @@ jobs: - wheel-tests-cuproj - devcontainer secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@python-3.12 checks: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@python-3.12 with: enable_check_generated_files: false conda-cpp-build: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@python-3.12 with: build_type: pull-request conda-cpp-checks: needs: conda-cpp-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@python-3.12 with: build_type: pull-request enable_check_symbols: true conda-cpp-tests: needs: conda-cpp-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@python-3.12 with: build_type: pull-request conda-python-build: needs: conda-cpp-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@python-3.12 with: build_type: pull-request conda-python-tests: needs: conda-python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@python-3.12 with: build_type: pull-request conda-notebook-tests: needs: conda-python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12 with: build_type: pull-request node_type: "gpu-v100-latest-1" @@ -77,7 +77,7 @@ jobs: docs-build: needs: conda-python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12 with: build_type: pull-request node_type: "gpu-v100-latest-1" @@ -87,7 +87,7 @@ jobs: wheel-build-libcuspatial: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 with: # build for every combination of arch and CUDA version, but only for the latest Python matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) @@ -96,34 +96,34 @@ jobs: wheel-build-cuspatial: needs: [checks, wheel-build-libcuspatial] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 with: build_type: pull-request script: ci/build_wheel_cuspatial.sh wheel-tests-cuspatial: needs: wheel-build-cuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12 with: build_type: pull-request script: ci/test_wheel_cuspatial.sh wheel-build-cuproj: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 with: build_type: pull-request script: ci/build_wheel_cuproj.sh wheel-tests-cuproj: needs: [wheel-build-cuspatial, wheel-build-cuproj] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12 with: build_type: pull-request script: ci/test_wheel_cuproj.sh devcontainer: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@python-3.12 with: arch: '["amd64"]' cuda: '["12.5"]' diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 493eca446..d1e378b68 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -16,7 +16,7 @@ on: jobs: conda-cpp-checks: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@python-3.12 with: build_type: nightly branch: ${{ inputs.branch }} @@ -25,7 +25,7 @@ jobs: enable_check_symbols: true conda-cpp-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@python-3.12 with: build_type: nightly branch: ${{ inputs.branch }} @@ -33,7 +33,7 @@ jobs: sha: ${{ inputs.sha }} conda-python-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@python-3.12 with: build_type: nightly branch: ${{ inputs.branch }} @@ -41,7 +41,7 @@ jobs: sha: ${{ inputs.sha }} wheel-tests-cuspatial: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12 with: build_type: nightly branch: ${{ inputs.branch }} @@ -50,7 +50,7 @@ jobs: script: ci/test_wheel_cuspatial.sh wheel-tests-cuproj: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12 with: build_type: nightly branch: ${{ inputs.branch }} diff --git a/README.md b/README.md index a2a2be972..6086a7c9e 100644 --- a/README.md +++ b/README.md @@ -119,13 +119,13 @@ docker run --gpus all --pull always --rm -it \ ### Install with Conda To install via conda: -> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10 and 3.11. +> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10, 3.11, and 3.12. cuSpatial can be installed with conda (miniconda, or the full Anaconda distribution) from the rapidsai channel: ```shell conda install -c rapidsai -c conda-forge -c nvidia \ - cuspatial=24.10 python=3.11 cudatoolkit=11.8 + cuspatial=24.10 python=3.12 cudatoolkit=11.8 ``` We also provide nightly Conda packages built from the HEAD of our latest development branch. @@ -134,7 +134,7 @@ See the [RAPIDS installation documentation](https://docs.rapids.ai/install) for ### Install with pip To install via pip: -> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10 and 3.11. +> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10, 3.11, and 3.12. The cuSpatial pip packages can be installed from NVIDIA's PyPI index. pip installations require using the matching wheel to the system's installed CUDA toolkit. - For CUDA 11 toolkits, install the `-cu11` wheels @@ -145,42 +145,6 @@ pip install cuspatial-cu12 --extra-index-url=https://pypi.nvidia.com pip install cuspatial-cu11 --extra-index-url=https://pypi.nvidia.com ``` -#### Troubleshooting Fiona/GDAL versions - -cuSpatial depends on [`geopandas`](https://github.com/geopandas/geopandas), which uses [`fiona >= 1.8.19`](https://pypi.org/project/Fiona/), to read common GIS formats with GDAL. - -Fiona requires GDAL is already present on your system, but its minimum required version may be newer than the version of GDAL in your OS's package manager. - -Fiona checks the GDAL version at install time and fails with an error like this if a compatible version of GDAL isn't installed: -``` -ERROR: GDAL >= 3.2 is required for fiona. Please upgrade GDAL. -``` - -There are two ways to fix this: - -1. Install a version of GDAL that meets fiona's minimum required version - * Ubuntu users can install a newer GDAL with the [UbuntuGIS PPA](https://wiki.ubuntu.com/UbuntuGIS): - ```shell - sudo -y add-apt-repository ppa:ubuntugis/ppa - sudo apt install libgdal-dev - ``` - * Rocky Linux users can install a newer GDAL as follows - ```shell - yum update -y - yum config-manager --set-enabled powertools - yum update -y - yum install -y gdal-devel - ``` -2. Pin fiona's version to a range that's compatible with your version of `libgdal-dev` - * For Ubuntu20.04 ([GDAL v3.0.4](https://packages.ubuntu.com/focal/libgdal-dev)): - ```shell - pip install --no-binary fiona --extra-index-url=https://pypi.nvidia.com cuspatial-cu12 'fiona>=1.8.19,<1.9' - ``` - * For Ubuntu22.04 ([GDAL v3.4.1](https://packages.ubuntu.com/jammy/libgdal-dev)): - ```shell - pip install --no-binary fiona --extra-index-url=https://pypi.nvidia.com cuspatial-cu12 'fiona>=1.9' - ``` - ### Build/Install from source To build and install cuSpatial from source please see the [build documentation](https://docs.rapids.ai/api/cuspatial/stable/developer_guide/build.html). diff --git a/ci/test_wheel_cuproj.sh b/ci/test_wheel_cuproj.sh index 27cc87a2a..abd20ebf1 100755 --- a/ci/test_wheel_cuproj.sh +++ b/ci/test_wheel_cuproj.sh @@ -6,18 +6,6 @@ set -eou pipefail mkdir -p ./dist RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})" -# install build dependencies for fiona -if type -f yum > /dev/null 2>&1; then - yum update -y - # some of gdal-devel's dependencies, like 'libdap', come from the powertools repo - yum config-manager --set-enabled powertools - yum update -y - yum install -y gdal-devel -else - apt update - DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev -fi - # Download the cuproj and cuspatial built in the previous step RAPIDS_PY_WHEEL_NAME="cuproj_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist @@ -25,11 +13,9 @@ RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-whe # echo to expand wildcard before adding `[extra]` requires for pip python -m pip install \ - --no-binary 'fiona' \ "$(echo ./dist/cuspatial*.whl)" \ "$(echo ./dist/cuproj*.whl)[test]" \ - "$(echo ./dist/libcuspatial*.whl)" \ - 'fiona>=1.8.19,<1.9' + "$(echo ./dist/libcuspatial*.whl)" rapids-logger "pytest cuproj" pushd python/cuproj/cuproj diff --git a/ci/test_wheel_cuspatial.sh b/ci/test_wheel_cuspatial.sh index a8f79832c..a352e1bfd 100755 --- a/ci/test_wheel_cuspatial.sh +++ b/ci/test_wheel_cuspatial.sh @@ -6,28 +6,14 @@ set -eou pipefail mkdir -p ./dist RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})" -# install build dependencies for fiona -if type -f yum > /dev/null 2>&1; then - yum update -y - # some of gdal-devel's dependencies, like 'libdap', come from the powertools repo - yum config-manager --set-enabled powertools - yum update -y - yum install -y gdal-devel -else - apt update - DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev -fi - # Download the cuspatial and libcuspatial built in the previous step RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp ./dist # echo to expand wildcard before adding `[extra]` requires for pip python -m pip install \ - --no-binary 'fiona' \ "$(echo ./dist/cuspatial*.whl)[test]" \ - "$(echo ./dist/libcuspatial*.whl)" \ - 'fiona>=1.8.19,<1.9' + "$(echo ./dist/libcuspatial*.whl)" rapids-logger "pytest cuspatial" pushd python/cuspatial/cuspatial diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml index 211d27b68..5ac2a6b9c 100644 --- a/conda/environments/all_cuda-118_arch-x86_64.yaml +++ b/conda/environments/all_cuda-118_arch-x86_64.yaml @@ -19,8 +19,7 @@ dependencies: - cython>=3.0.0 - doxygen - gcc_linux-64=11.* -- geopandas<1 -- geopandas>=0.11.0 +- geopandas>=1.0.0 - ipython - ipywidgets - libcudf==24.10.*,>=0.0.0a0 @@ -34,14 +33,14 @@ dependencies: - nvcc_linux-64=11.8 - osmnx>=1.9.3 - pre-commit -- proj +- proj >=9.3.0,<9.3.1.0a0 - pydata-sphinx-theme!=0.14.2 - pydeck - pylibcudf==24.10.*,>=0.0.0a0 - pytest - pytest-cov - pytest-xdist -- python>=3.10,<3.12 +- python>=3.10,<3.13 - rapids-build-backend>=0.3.0,<0.4.0.dev0 - rmm==24.10.*,>=0.0.0a0 - scikit-build-core>=0.10.0 diff --git a/conda/environments/all_cuda-125_arch-x86_64.yaml b/conda/environments/all_cuda-125_arch-x86_64.yaml index fc782ccfc..7c2ca25d4 100644 --- a/conda/environments/all_cuda-125_arch-x86_64.yaml +++ b/conda/environments/all_cuda-125_arch-x86_64.yaml @@ -22,8 +22,7 @@ dependencies: - cython>=3.0.0 - doxygen - gcc_linux-64=11.* -- geopandas<1 -- geopandas>=0.11.0 +- geopandas>=1.0.0 - ipython - ipywidgets - libcudf==24.10.*,>=0.0.0a0 @@ -36,14 +35,14 @@ dependencies: - numpydoc - osmnx>=1.9.3 - pre-commit -- proj +- proj >=9.3.0,<9.3.1.0a0 - pydata-sphinx-theme!=0.14.2 - pydeck - pylibcudf==24.10.*,>=0.0.0a0 - pytest - pytest-cov - pytest-xdist -- python>=3.10,<3.12 +- python>=3.10,<3.13 - rapids-build-backend>=0.3.0,<0.4.0.dev0 - rmm==24.10.*,>=0.0.0a0 - scikit-build-core>=0.10.0 diff --git a/conda/recipes/cuproj/meta.yaml b/conda/recipes/cuproj/meta.yaml index 29d590bc0..0096561fe 100644 --- a/conda/recipes/cuproj/meta.yaml +++ b/conda/recipes/cuproj/meta.yaml @@ -64,7 +64,7 @@ requirements: - rapids-build-backend >=0.3.0,<0.4.0.dev0 - librmm ={{ minor_version }} - scikit-build-core >=0.10.0 - - proj + - proj >=9.3.0,<9.3.1.0a0 # TODO: proj is being held back due to needing spdlog/fmt upgrades to support recent conda-forge builds of tiledb - sqlite run: {% if cuda_major == "11" %} diff --git a/conda/recipes/cuspatial/meta.yaml b/conda/recipes/cuspatial/meta.yaml index a418d8ce0..c7c0d780c 100644 --- a/conda/recipes/cuspatial/meta.yaml +++ b/conda/recipes/cuspatial/meta.yaml @@ -76,7 +76,7 @@ requirements: - {{ pin_compatible('cuda-version', max_pin='x', min_pin='x') }} - cudf ={{ minor_version }} - pylibcudf ={{ minor_version }} - - geopandas >=0.11.0 + - geopandas >=1.0.0 - numpy >=1.23,<2.0a0 - python - rmm ={{ minor_version }} diff --git a/conda/recipes/libcuspatial/meta.yaml b/conda/recipes/libcuspatial/meta.yaml index 3e212690c..177037568 100644 --- a/conda/recipes/libcuspatial/meta.yaml +++ b/conda/recipes/libcuspatial/meta.yaml @@ -49,7 +49,7 @@ requirements: - libcudf ={{ minor_version }} - librmm ={{ minor_version }} - sqlite - - proj + - proj >=9.3.0,<9.3.1.0a0 # TODO: proj is being held back due to needing spdlog/fmt upgrades to support recent conda-forge builds of tiledb outputs: - name: libcuspatial @@ -85,7 +85,7 @@ outputs: - libcudf ={{ minor_version }} - librmm ={{ minor_version }} - sqlite - - proj + - proj >=9.3.0,<9.3.1.0a0 # TODO: proj is being held back due to needing spdlog/fmt upgrades to support recent conda-forge builds of tiledb test: commands: - test -f $PREFIX/lib/libcuspatial.so diff --git a/dependencies.yaml b/dependencies.yaml index 859605b4f..3ac92dfbf 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -180,7 +180,7 @@ dependencies: packages: - c-compiler - cxx-compiler - - proj + - proj >=9.3.0,<9.3.1.0a0 # TODO: proj is being held back due to needing spdlog/fmt upgrades to support recent conda-forge builds of tiledb - sqlite specific: - output_types: conda @@ -373,9 +373,6 @@ dependencies: # TODO: Remove geopandas.dataset usage in cuspatial_api_examples.ipynb test_notebooks: common: - - output_types: [conda, requirements, pyproject] - packages: - - geopandas<1 - output_types: conda packages: - osmnx>=1.9.3 @@ -391,14 +388,18 @@ dependencies: py: "3.11" packages: - python=3.11 + - matrix: + py: "3.12" + packages: + - python=3.12 - matrix: packages: - - python>=3.10,<3.12 + - python>=3.10,<3.13 run_python_cuspatial: common: - output_types: [conda, requirements, pyproject] packages: - - geopandas>=0.11.0 + - &geopandas geopandas>=1.0.0 - &numpy numpy>=1.23,<2.0a0 test_python_cuspatial: common: @@ -415,7 +416,7 @@ dependencies: - pytest - pytest-cov - pytest-xdist - - geopandas>=0.11.0 + - *geopandas - output_types: [requirements, pyproject] packages: - pyproj>=3.6.0,<3.7a0 diff --git a/python/cuproj/pyproject.toml b/python/cuproj/pyproject.toml index 6c4d6cc2d..fc4b99d54 100644 --- a/python/cuproj/pyproject.toml +++ b/python/cuproj/pyproject.toml @@ -38,12 +38,13 @@ classifiers = [ "Programming Language :: Python", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", ] [project.optional-dependencies] test = [ "cuspatial==24.10.*,>=0.0.0a0", - "geopandas>=0.11.0", + "geopandas>=1.0.0", "numpy>=1.23,<2.0a0", "pyproj>=3.6.0,<3.7a0", "pytest", diff --git a/python/cuspatial/pyproject.toml b/python/cuspatial/pyproject.toml index f8aba3c4b..3a850030e 100644 --- a/python/cuspatial/pyproject.toml +++ b/python/cuspatial/pyproject.toml @@ -31,7 +31,7 @@ license = { text = "Apache 2.0" } requires-python = ">=3.10" dependencies = [ "cudf==24.10.*,>=0.0.0a0", - "geopandas>=0.11.0", + "geopandas>=1.0.0", "libcudf==24.10.*,>=0.0.0a0", "libcuspatial==24.10.*,>=0.0.0a0", "numpy>=1.23,<2.0a0", @@ -45,6 +45,7 @@ classifiers = [ "Programming Language :: Python", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", ] [project.optional-dependencies] From 3df71284f90976016c5875464e81f30cc9216205 Mon Sep 17 00:00:00 2001 From: Mike McCarty Date: Tue, 17 Sep 2024 10:36:37 -0400 Subject: [PATCH 10/28] Recommending `miniforge` for conda install (#1457) Recommending `miniforge` for conda install in build instructions. Authors: - Mike McCarty (https://github.com/mmccarty) Approvers: - Mark Harris (https://github.com/harrism) URL: https://github.com/rapidsai/cuspatial/pull/1457 --- docs/source/developer_guide/build.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/developer_guide/build.md b/docs/source/developer_guide/build.md index d20e27b3d..8683d1461 100644 --- a/docs/source/developer_guide/build.md +++ b/docs/source/developer_guide/build.md @@ -4,7 +4,7 @@ - gcc >= 7.5 - cmake >= 3.26.4 -- miniconda +- miniforge ## Fetch cuSpatial repository From a2d21c94bc5e53bca030f6bb3603bd7ac2c770dd Mon Sep 17 00:00:00 2001 From: James Lamb Date: Wed, 18 Sep 2024 14:42:53 -0500 Subject: [PATCH 11/28] Use CI workflow branch 'branch-24.10' again [skip ci] (#1459) --- .github/workflows/build.yaml | 20 ++++++++++---------- .github/workflows/pr.yaml | 30 +++++++++++++++--------------- .github/workflows/test.yaml | 10 +++++----- 3 files changed, 30 insertions(+), 30 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 14f3817a0..d86122f5a 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -28,7 +28,7 @@ concurrency: jobs: cpp-build: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.10 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -37,7 +37,7 @@ jobs: python-build: needs: [cpp-build] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.10 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -46,7 +46,7 @@ jobs: upload-conda: needs: [cpp-build, python-build] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@branch-24.10 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -56,7 +56,7 @@ jobs: if: github.ref_type == 'branch' needs: python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10 with: arch: "amd64" branch: ${{ inputs.branch }} @@ -68,7 +68,7 @@ jobs: sha: ${{ inputs.sha }} wheel-build-libcuspatial: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 with: # build for every combination of arch and CUDA version, but only for the latest Python matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) @@ -80,7 +80,7 @@ jobs: wheel-publish-libcuspatial: needs: wheel-build-libcuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -91,7 +91,7 @@ jobs: wheel-build-cuspatial: needs: wheel-publish-libcuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -101,7 +101,7 @@ jobs: wheel-publish-cuspatial: needs: wheel-build-cuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -111,7 +111,7 @@ jobs: package-type: python wheel-build-cuproj: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -121,7 +121,7 @@ jobs: wheel-publish-cuproj: needs: wheel-build-cuproj secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index 65a49b87d..94617d953 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -27,47 +27,47 @@ jobs: - wheel-tests-cuproj - devcontainer secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@branch-24.10 checks: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@branch-24.10 with: enable_check_generated_files: false conda-cpp-build: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.10 with: build_type: pull-request conda-cpp-checks: needs: conda-cpp-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.10 with: build_type: pull-request enable_check_symbols: true conda-cpp-tests: needs: conda-cpp-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.10 with: build_type: pull-request conda-python-build: needs: conda-cpp-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.10 with: build_type: pull-request conda-python-tests: needs: conda-python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.10 with: build_type: pull-request conda-notebook-tests: needs: conda-python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10 with: build_type: pull-request node_type: "gpu-v100-latest-1" @@ -77,7 +77,7 @@ jobs: docs-build: needs: conda-python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10 with: build_type: pull-request node_type: "gpu-v100-latest-1" @@ -87,7 +87,7 @@ jobs: wheel-build-libcuspatial: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 with: # build for every combination of arch and CUDA version, but only for the latest Python matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) @@ -96,34 +96,34 @@ jobs: wheel-build-cuspatial: needs: [checks, wheel-build-libcuspatial] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 with: build_type: pull-request script: ci/build_wheel_cuspatial.sh wheel-tests-cuspatial: needs: wheel-build-cuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 with: build_type: pull-request script: ci/test_wheel_cuspatial.sh wheel-build-cuproj: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 with: build_type: pull-request script: ci/build_wheel_cuproj.sh wheel-tests-cuproj: needs: [wheel-build-cuspatial, wheel-build-cuproj] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 with: build_type: pull-request script: ci/test_wheel_cuproj.sh devcontainer: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@branch-24.10 with: arch: '["amd64"]' cuda: '["12.5"]' diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index d1e378b68..493eca446 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -16,7 +16,7 @@ on: jobs: conda-cpp-checks: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.10 with: build_type: nightly branch: ${{ inputs.branch }} @@ -25,7 +25,7 @@ jobs: enable_check_symbols: true conda-cpp-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.10 with: build_type: nightly branch: ${{ inputs.branch }} @@ -33,7 +33,7 @@ jobs: sha: ${{ inputs.sha }} conda-python-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.10 with: build_type: nightly branch: ${{ inputs.branch }} @@ -41,7 +41,7 @@ jobs: sha: ${{ inputs.sha }} wheel-tests-cuspatial: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 with: build_type: nightly branch: ${{ inputs.branch }} @@ -50,7 +50,7 @@ jobs: script: ci/test_wheel_cuspatial.sh wheel-tests-cuproj: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 with: build_type: nightly branch: ${{ inputs.branch }} From c22fb0425e969f8ea50fe597f7a9bf32d7667f5c Mon Sep 17 00:00:00 2001 From: Ray Douglass Date: Thu, 19 Sep 2024 12:07:42 -0400 Subject: [PATCH 12/28] DOC v24.12 Updates [skip ci] --- .../cuda11.8-conda/devcontainer.json | 6 +- .devcontainer/cuda11.8-pip/devcontainer.json | 6 +- .../cuda12.5-conda/devcontainer.json | 6 +- .devcontainer/cuda12.5-pip/devcontainer.json | 6 +- .github/workflows/build.yaml | 20 +++---- .github/workflows/pr.yaml | 30 +++++----- .github/workflows/test.yaml | 10 ++-- README.md | 6 +- VERSION | 2 +- ci/build_docs.sh | 2 +- .../all_cuda-118_arch-x86_64.yaml | 12 ++-- .../all_cuda-125_arch-x86_64.yaml | 12 ++-- dependencies.yaml | 56 +++++++++---------- .../user_guide/cuproj_api_examples.ipynb | 2 +- .../user_guide/cuspatial_api_examples.ipynb | 4 +- python/cuproj/pyproject.toml | 4 +- python/cuspatial/pyproject.toml | 18 +++--- python/libcuspatial/pyproject.toml | 6 +- 18 files changed, 104 insertions(+), 104 deletions(-) diff --git a/.devcontainer/cuda11.8-conda/devcontainer.json b/.devcontainer/cuda11.8-conda/devcontainer.json index 646640085..221f606cb 100644 --- a/.devcontainer/cuda11.8-conda/devcontainer.json +++ b/.devcontainer/cuda11.8-conda/devcontainer.json @@ -5,17 +5,17 @@ "args": { "CUDA": "11.8", "PYTHON_PACKAGE_MANAGER": "conda", - "BASE": "rapidsai/devcontainers:24.10-cpp-cuda11.8-mambaforge-ubuntu22.04" + "BASE": "rapidsai/devcontainers:24.12-cpp-cuda11.8-mambaforge-ubuntu22.04" } }, "runArgs": [ "--rm", "--name", - "${localEnv:USER:anon}-rapids-${localWorkspaceFolderBasename}-24.10-cuda11.8-conda" + "${localEnv:USER:anon}-rapids-${localWorkspaceFolderBasename}-24.12-cuda11.8-conda" ], "hostRequirements": {"gpu": "optional"}, "features": { - "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:24.10": {} + "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:24.12": {} }, "overrideFeatureInstallOrder": [ "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils" diff --git a/.devcontainer/cuda11.8-pip/devcontainer.json b/.devcontainer/cuda11.8-pip/devcontainer.json index b29f6a10e..38d7d2b10 100644 --- a/.devcontainer/cuda11.8-pip/devcontainer.json +++ b/.devcontainer/cuda11.8-pip/devcontainer.json @@ -5,17 +5,17 @@ "args": { "CUDA": "11.8", "PYTHON_PACKAGE_MANAGER": "pip", - "BASE": "rapidsai/devcontainers:24.10-cpp-cuda11.8-ubuntu22.04" + "BASE": "rapidsai/devcontainers:24.12-cpp-cuda11.8-ubuntu22.04" } }, "runArgs": [ "--rm", "--name", - "${localEnv:USER:anon}-rapids-${localWorkspaceFolderBasename}-24.10-cuda11.8-pip" + "${localEnv:USER:anon}-rapids-${localWorkspaceFolderBasename}-24.12-cuda11.8-pip" ], "hostRequirements": {"gpu": "optional"}, "features": { - "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:24.10": {} + "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:24.12": {} }, "overrideFeatureInstallOrder": [ "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils" diff --git a/.devcontainer/cuda12.5-conda/devcontainer.json b/.devcontainer/cuda12.5-conda/devcontainer.json index 7b5a6efaa..55aa94fb0 100644 --- a/.devcontainer/cuda12.5-conda/devcontainer.json +++ b/.devcontainer/cuda12.5-conda/devcontainer.json @@ -5,17 +5,17 @@ "args": { "CUDA": "12.5", "PYTHON_PACKAGE_MANAGER": "conda", - "BASE": "rapidsai/devcontainers:24.10-cpp-mambaforge-ubuntu22.04" + "BASE": "rapidsai/devcontainers:24.12-cpp-mambaforge-ubuntu22.04" } }, "runArgs": [ "--rm", "--name", - "${localEnv:USER:anon}-rapids-${localWorkspaceFolderBasename}-24.10-cuda12.5-conda" + "${localEnv:USER:anon}-rapids-${localWorkspaceFolderBasename}-24.12-cuda12.5-conda" ], "hostRequirements": {"gpu": "optional"}, "features": { - "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:24.10": {} + "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:24.12": {} }, "overrideFeatureInstallOrder": [ "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils" diff --git a/.devcontainer/cuda12.5-pip/devcontainer.json b/.devcontainer/cuda12.5-pip/devcontainer.json index 1d3a82749..be5f98bd2 100644 --- a/.devcontainer/cuda12.5-pip/devcontainer.json +++ b/.devcontainer/cuda12.5-pip/devcontainer.json @@ -5,17 +5,17 @@ "args": { "CUDA": "12.5", "PYTHON_PACKAGE_MANAGER": "pip", - "BASE": "rapidsai/devcontainers:24.10-cpp-cuda12.5-ubuntu22.04" + "BASE": "rapidsai/devcontainers:24.12-cpp-cuda12.5-ubuntu22.04" } }, "runArgs": [ "--rm", "--name", - "${localEnv:USER:anon}-rapids-${localWorkspaceFolderBasename}-24.10-cuda12.5-pip" + "${localEnv:USER:anon}-rapids-${localWorkspaceFolderBasename}-24.12-cuda12.5-pip" ], "hostRequirements": {"gpu": "optional"}, "features": { - "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:24.10": {} + "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:24.12": {} }, "overrideFeatureInstallOrder": [ "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils" diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index d86122f5a..53277786e 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -28,7 +28,7 @@ concurrency: jobs: cpp-build: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -37,7 +37,7 @@ jobs: python-build: needs: [cpp-build] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -46,7 +46,7 @@ jobs: upload-conda: needs: [cpp-build, python-build] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@branch-24.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -56,7 +56,7 @@ jobs: if: github.ref_type == 'branch' needs: python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.12 with: arch: "amd64" branch: ${{ inputs.branch }} @@ -68,7 +68,7 @@ jobs: sha: ${{ inputs.sha }} wheel-build-libcuspatial: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.12 with: # build for every combination of arch and CUDA version, but only for the latest Python matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) @@ -80,7 +80,7 @@ jobs: wheel-publish-libcuspatial: needs: wheel-build-libcuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -91,7 +91,7 @@ jobs: wheel-build-cuspatial: needs: wheel-publish-libcuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -101,7 +101,7 @@ jobs: wheel-publish-cuspatial: needs: wheel-build-cuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -111,7 +111,7 @@ jobs: package-type: python wheel-build-cuproj: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} @@ -121,7 +121,7 @@ jobs: wheel-publish-cuproj: needs: wheel-build-cuproj secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.12 with: build_type: ${{ inputs.build_type || 'branch' }} branch: ${{ inputs.branch }} diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index 94617d953..5f1278333 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -27,47 +27,47 @@ jobs: - wheel-tests-cuproj - devcontainer secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@branch-24.12 checks: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@branch-24.12 with: enable_check_generated_files: false conda-cpp-build: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.12 with: build_type: pull-request conda-cpp-checks: needs: conda-cpp-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.12 with: build_type: pull-request enable_check_symbols: true conda-cpp-tests: needs: conda-cpp-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.12 with: build_type: pull-request conda-python-build: needs: conda-cpp-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.12 with: build_type: pull-request conda-python-tests: needs: conda-python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.12 with: build_type: pull-request conda-notebook-tests: needs: conda-python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.12 with: build_type: pull-request node_type: "gpu-v100-latest-1" @@ -77,7 +77,7 @@ jobs: docs-build: needs: conda-python-build secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.12 with: build_type: pull-request node_type: "gpu-v100-latest-1" @@ -87,7 +87,7 @@ jobs: wheel-build-libcuspatial: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.12 with: # build for every combination of arch and CUDA version, but only for the latest Python matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) @@ -96,34 +96,34 @@ jobs: wheel-build-cuspatial: needs: [checks, wheel-build-libcuspatial] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.12 with: build_type: pull-request script: ci/build_wheel_cuspatial.sh wheel-tests-cuspatial: needs: wheel-build-cuspatial secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.12 with: build_type: pull-request script: ci/test_wheel_cuspatial.sh wheel-build-cuproj: needs: checks secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.12 with: build_type: pull-request script: ci/build_wheel_cuproj.sh wheel-tests-cuproj: needs: [wheel-build-cuspatial, wheel-build-cuproj] secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.12 with: build_type: pull-request script: ci/test_wheel_cuproj.sh devcontainer: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@branch-24.12 with: arch: '["amd64"]' cuda: '["12.5"]' diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 493eca446..45218c240 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -16,7 +16,7 @@ on: jobs: conda-cpp-checks: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.12 with: build_type: nightly branch: ${{ inputs.branch }} @@ -25,7 +25,7 @@ jobs: enable_check_symbols: true conda-cpp-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.12 with: build_type: nightly branch: ${{ inputs.branch }} @@ -33,7 +33,7 @@ jobs: sha: ${{ inputs.sha }} conda-python-tests: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.12 with: build_type: nightly branch: ${{ inputs.branch }} @@ -41,7 +41,7 @@ jobs: sha: ${{ inputs.sha }} wheel-tests-cuspatial: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.12 with: build_type: nightly branch: ${{ inputs.branch }} @@ -50,7 +50,7 @@ jobs: script: ci/test_wheel_cuspatial.sh wheel-tests-cuproj: secrets: inherit - uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10 + uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.12 with: build_type: nightly branch: ${{ inputs.branch }} diff --git a/README.md b/README.md index 6086a7c9e..4193373dd 100644 --- a/README.md +++ b/README.md @@ -113,7 +113,7 @@ An example command from the Release Selector: docker run --gpus all --pull always --rm -it \ --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \ -p 8888:8888 -p 8787:8787 -p 8786:8786 \ - nvcr.io/nvidia/rapidsai/notebooks:24.10-cuda11.8-py3.10 + nvcr.io/nvidia/rapidsai/notebooks:24.12-cuda11.8-py3.10 ``` ### Install with Conda @@ -125,7 +125,7 @@ cuSpatial can be installed with conda (miniconda, or the full Anaconda distribut ```shell conda install -c rapidsai -c conda-forge -c nvidia \ - cuspatial=24.10 python=3.12 cudatoolkit=11.8 + cuspatial=24.12 python=3.12 cudatoolkit=11.8 ``` We also provide nightly Conda packages built from the HEAD of our latest development branch. @@ -155,7 +155,7 @@ To build and install cuSpatial from source please see the [build documentation]( If you find cuSpatial useful in your published work, please consider citing the repository. ```bibtex -@misc{cuspatial:24.10, +@misc{cuspatial:24.12, author = {{NVIDIA Corporation}}, title = {cuSpatial: GPU-Accelerated Geospatial and Spatiotemporal Algorithms}, year = {2023}, diff --git a/VERSION b/VERSION index 7c7ba0443..af28c42b5 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -24.10.00 +24.12.00 diff --git a/ci/build_docs.sh b/ci/build_docs.sh index 80b7c6ca7..3ce57b2d8 100755 --- a/ci/build_docs.sh +++ b/ci/build_docs.sh @@ -23,7 +23,7 @@ rapids-print-env export RAPIDS_VERSION="$(rapids-version)" export RAPIDS_VERSION_MAJOR_MINOR="$(rapids-version-major-minor)" -export RAPIDS_VERSION_NUMBER="24.10" +export RAPIDS_VERSION_NUMBER="24.12" export RAPIDS_DOCS_DIR="$(mktemp -d)" rapids-logger "Build cuSpatial CPP docs" diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml index 5ac2a6b9c..cac73912d 100644 --- a/conda/environments/all_cuda-118_arch-x86_64.yaml +++ b/conda/environments/all_cuda-118_arch-x86_64.yaml @@ -11,8 +11,8 @@ dependencies: - cmake>=3.26.4,!=3.30.0 - cuda-version=11.8 - cudatoolkit -- cudf==24.10.*,>=0.0.0a0 -- cuml==24.10.*,>=0.0.0a0 +- cudf==24.12.*,>=0.0.0a0 +- cuml==24.12.*,>=0.0.0a0 - cupy>=12.0.0 - curl - cxx-compiler @@ -22,8 +22,8 @@ dependencies: - geopandas>=1.0.0 - ipython - ipywidgets -- libcudf==24.10.*,>=0.0.0a0 -- librmm==24.10.*,>=0.0.0a0 +- libcudf==24.12.*,>=0.0.0a0 +- librmm==24.12.*,>=0.0.0a0 - myst-parser - nbsphinx - ninja @@ -36,13 +36,13 @@ dependencies: - proj >=9.3.0,<9.3.1.0a0 - pydata-sphinx-theme!=0.14.2 - pydeck -- pylibcudf==24.10.*,>=0.0.0a0 +- pylibcudf==24.12.*,>=0.0.0a0 - pytest - pytest-cov - pytest-xdist - python>=3.10,<3.13 - rapids-build-backend>=0.3.0,<0.4.0.dev0 -- rmm==24.10.*,>=0.0.0a0 +- rmm==24.12.*,>=0.0.0a0 - scikit-build-core>=0.10.0 - scikit-image - shapely diff --git a/conda/environments/all_cuda-125_arch-x86_64.yaml b/conda/environments/all_cuda-125_arch-x86_64.yaml index 7c2ca25d4..9a8ec1071 100644 --- a/conda/environments/all_cuda-125_arch-x86_64.yaml +++ b/conda/environments/all_cuda-125_arch-x86_64.yaml @@ -14,8 +14,8 @@ dependencies: - cuda-nvcc - cuda-nvrtc-dev - cuda-version=12.5 -- cudf==24.10.*,>=0.0.0a0 -- cuml==24.10.*,>=0.0.0a0 +- cudf==24.12.*,>=0.0.0a0 +- cuml==24.12.*,>=0.0.0a0 - cupy>=12.0.0 - curl - cxx-compiler @@ -25,8 +25,8 @@ dependencies: - geopandas>=1.0.0 - ipython - ipywidgets -- libcudf==24.10.*,>=0.0.0a0 -- librmm==24.10.*,>=0.0.0a0 +- libcudf==24.12.*,>=0.0.0a0 +- librmm==24.12.*,>=0.0.0a0 - myst-parser - nbsphinx - ninja @@ -38,13 +38,13 @@ dependencies: - proj >=9.3.0,<9.3.1.0a0 - pydata-sphinx-theme!=0.14.2 - pydeck -- pylibcudf==24.10.*,>=0.0.0a0 +- pylibcudf==24.12.*,>=0.0.0a0 - pytest - pytest-cov - pytest-xdist - python>=3.10,<3.13 - rapids-build-backend>=0.3.0,<0.4.0.dev0 -- rmm==24.10.*,>=0.0.0a0 +- rmm==24.12.*,>=0.0.0a0 - scikit-build-core>=0.10.0 - scikit-image - shapely diff --git a/dependencies.yaml b/dependencies.yaml index 3ac92dfbf..f9796be2e 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -425,7 +425,7 @@ dependencies: common: - output_types: conda packages: - - &rmm_unsuffixed rmm==24.10.*,>=0.0.0a0 + - &rmm_unsuffixed rmm==24.12.*,>=0.0.0a0 - output_types: requirements packages: # pip recognizes the index as a global option for the requirements.txt file @@ -438,20 +438,20 @@ dependencies: cuda: "12.*" cuda_suffixed: "true" packages: - - rmm-cu12==24.10.*,>=0.0.0a0 + - rmm-cu12==24.12.*,>=0.0.0a0 - matrix: cuda: "11.*" cuda_suffixed: "true" packages: - - rmm-cu11==24.10.*,>=0.0.0a0 + - rmm-cu11==24.12.*,>=0.0.0a0 - {matrix: null, packages: [*rmm_unsuffixed]} depends_on_cudf: common: - output_types: conda packages: - - &cudf_unsuffixed cudf==24.10.*,>=0.0.0a0 - - &pylibcudf_unsuffixed pylibcudf==24.10.*,>=0.0.0a0 + - &cudf_unsuffixed cudf==24.12.*,>=0.0.0a0 + - &pylibcudf_unsuffixed pylibcudf==24.12.*,>=0.0.0a0 - output_types: requirements packages: # pip recognizes the index as a global option for the requirements.txt file @@ -464,21 +464,21 @@ dependencies: cuda: "12.*" cuda_suffixed: "true" packages: - - cudf-cu12==24.10.*,>=0.0.0a0 - - pylibcudf-cu12==24.10.*,>=0.0.0a0 + - cudf-cu12==24.12.*,>=0.0.0a0 + - pylibcudf-cu12==24.12.*,>=0.0.0a0 - matrix: cuda: "11.*" cuda_suffixed: "true" packages: - - cudf-cu11==24.10.*,>=0.0.0a0 - - pylibcudf-cu11==24.10.*,>=0.0.0a0 + - cudf-cu11==24.12.*,>=0.0.0a0 + - pylibcudf-cu11==24.12.*,>=0.0.0a0 - {matrix: null, packages: [*cudf_unsuffixed]} depends_on_libcudf: common: - output_types: conda packages: - - &libcudf_unsuffixed libcudf==24.10.*,>=0.0.0a0 + - &libcudf_unsuffixed libcudf==24.12.*,>=0.0.0a0 - output_types: requirements packages: # pip recognizes the index as a global option for the requirements.txt file @@ -491,19 +491,19 @@ dependencies: cuda: "12.*" cuda_suffixed: "true" packages: - - libcudf-cu12==24.10.*,>=0.0.0a0 + - libcudf-cu12==24.12.*,>=0.0.0a0 - matrix: cuda: "11.*" cuda_suffixed: "true" packages: - - libcudf-cu11==24.10.*,>=0.0.0a0 + - libcudf-cu11==24.12.*,>=0.0.0a0 - {matrix: null, packages: [*libcudf_unsuffixed]} depends_on_cuml: common: - output_types: conda packages: - - &cuml_unsuffixed cuml==24.10.*,>=0.0.0a0 + - &cuml_unsuffixed cuml==24.12.*,>=0.0.0a0 - output_types: requirements packages: # pip recognizes the index as a global option for the requirements.txt file @@ -516,19 +516,19 @@ dependencies: cuda: "12.*" cuda_suffixed: "true" packages: - - cuml-cu12==24.10.*,>=0.0.0a0 + - cuml-cu12==24.12.*,>=0.0.0a0 - matrix: cuda: "11.*" cuda_suffixed: "true" packages: - - cuml-cu11==24.10.*,>=0.0.0a0 + - cuml-cu11==24.12.*,>=0.0.0a0 - {matrix: null, packages: [*cuml_unsuffixed]} depends_on_cuspatial: common: - output_types: conda packages: - - &cuspatial_unsuffixed cuspatial==24.10.*,>=0.0.0a0 + - &cuspatial_unsuffixed cuspatial==24.12.*,>=0.0.0a0 - output_types: requirements packages: # pip recognizes the index as a global option for the requirements.txt file @@ -541,19 +541,19 @@ dependencies: cuda: "12.*" cuda_suffixed: "true" packages: - - cuspatial-cu12==24.10.*,>=0.0.0a0 + - cuspatial-cu12==24.12.*,>=0.0.0a0 - matrix: cuda: "11.*" cuda_suffixed: "true" packages: - - cuspatial-cu11==24.10.*,>=0.0.0a0 + - cuspatial-cu11==24.12.*,>=0.0.0a0 - {matrix: null, packages: [*cuspatial_unsuffixed]} depends_on_cuproj: common: - output_types: conda packages: - - &cuproj_unsuffixed cuproj==24.10.*,>=0.0.0a0 + - &cuproj_unsuffixed cuproj==24.12.*,>=0.0.0a0 - output_types: requirements packages: # pip recognizes the index as a global option for the requirements.txt file @@ -566,12 +566,12 @@ dependencies: cuda: "12.*" cuda_suffixed: "true" packages: - - cuproj-cu12==24.10.*,>=0.0.0a0 + - cuproj-cu12==24.12.*,>=0.0.0a0 - matrix: cuda: "11.*" cuda_suffixed: "true" packages: - - cuproj-cu11==24.10.*,>=0.0.0a0 + - cuproj-cu11==24.12.*,>=0.0.0a0 - {matrix: null, packages: [*cuproj_unsuffixed]} depends_on_cupy: @@ -595,12 +595,12 @@ dependencies: common: - output_types: conda packages: - - libcuspatial-tests==24.10.*,>=0.0.0a0 + - libcuspatial-tests==24.12.*,>=0.0.0a0 depends_on_libcuspatial: common: - output_types: conda packages: - - &libcuspatial_unsuffixed libcuspatial==24.10.*,>=0.0.0a0 + - &libcuspatial_unsuffixed libcuspatial==24.12.*,>=0.0.0a0 - output_types: requirements packages: # pip recognizes the index as a global option for the requirements.txt file @@ -613,19 +613,19 @@ dependencies: cuda: "12.*" cuda_suffixed: "true" packages: - - libcuspatial-cu12==24.10.*,>=0.0.0a0 + - libcuspatial-cu12==24.12.*,>=0.0.0a0 - matrix: cuda: "11.*" cuda_suffixed: "true" packages: - - libcuspatial-cu11==24.10.*,>=0.0.0a0 + - libcuspatial-cu11==24.12.*,>=0.0.0a0 - {matrix: null, packages: [*libcuspatial_unsuffixed]} depends_on_librmm: common: - output_types: conda packages: - - &librmm_unsuffixed librmm==24.10.*,>=0.0.0a0 + - &librmm_unsuffixed librmm==24.12.*,>=0.0.0a0 - output_types: requirements packages: # pip recognizes the index as a global option for the requirements.txt file @@ -638,10 +638,10 @@ dependencies: cuda: "12.*" cuda_suffixed: "true" packages: - - librmm-cu12==24.10.*,>=0.0.0a0 + - librmm-cu12==24.12.*,>=0.0.0a0 - matrix: cuda: "11.*" cuda_suffixed: "true" packages: - - librmm-cu11==24.10.*,>=0.0.0a0 + - librmm-cu11==24.12.*,>=0.0.0a0 - {matrix: null, packages: [*librmm_unsuffixed]} diff --git a/docs/cuproj/source/user_guide/cuproj_api_examples.ipynb b/docs/cuproj/source/user_guide/cuproj_api_examples.ipynb index 2c78ac8bb..a78452640 100644 --- a/docs/cuproj/source/user_guide/cuproj_api_examples.ipynb +++ b/docs/cuproj/source/user_guide/cuproj_api_examples.ipynb @@ -45,7 +45,7 @@ "metadata": {}, "outputs": [], "source": [ - "# !conda create -n rapids-24.10 --solver=libmamba -c rapidsai -c conda-forge -c nvidia \\ \n", + "# !conda create -n rapids-24.12 --solver=libmamba -c rapidsai -c conda-forge -c nvidia \\ \n", "# cuproj-23.12 python=3.10 cuda-version=12.0" ] }, diff --git a/docs/source/user_guide/cuspatial_api_examples.ipynb b/docs/source/user_guide/cuspatial_api_examples.ipynb index 0a34ea08a..72f7febab 100644 --- a/docs/source/user_guide/cuspatial_api_examples.ipynb +++ b/docs/source/user_guide/cuspatial_api_examples.ipynb @@ -57,8 +57,8 @@ "metadata": {}, "outputs": [], "source": [ - "# !conda create -n rapids-24.10 -c rapidsai -c conda-forge -c nvidia \\ \n", - "# cuspatial=24.10 python=3.9 cudatoolkit=11.5 " + "# !conda create -n rapids-24.12 -c rapidsai -c conda-forge -c nvidia \\ \n", + "# cuspatial=24.12 python=3.9 cudatoolkit=11.5 " ] }, { diff --git a/python/cuproj/pyproject.toml b/python/cuproj/pyproject.toml index fc4b99d54..d6deefdda 100644 --- a/python/cuproj/pyproject.toml +++ b/python/cuproj/pyproject.toml @@ -43,7 +43,7 @@ classifiers = [ [project.optional-dependencies] test = [ - "cuspatial==24.10.*,>=0.0.0a0", + "cuspatial==24.12.*,>=0.0.0a0", "geopandas>=1.0.0", "numpy>=1.23,<2.0a0", "pyproj>=3.6.0,<3.7a0", @@ -122,7 +122,7 @@ matrix-entry = "cuda_suffixed=true" requires = [ "cmake>=3.26.4,!=3.30.0", "cython>=3.0.0", - "librmm==24.10.*,>=0.0.0a0", + "librmm==24.12.*,>=0.0.0a0", "ninja", "wheel", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. diff --git a/python/cuspatial/pyproject.toml b/python/cuspatial/pyproject.toml index 3a850030e..9b8b87c8f 100644 --- a/python/cuspatial/pyproject.toml +++ b/python/cuspatial/pyproject.toml @@ -30,12 +30,12 @@ authors = [ license = { text = "Apache 2.0" } requires-python = ">=3.10" dependencies = [ - "cudf==24.10.*,>=0.0.0a0", + "cudf==24.12.*,>=0.0.0a0", "geopandas>=1.0.0", - "libcudf==24.10.*,>=0.0.0a0", - "libcuspatial==24.10.*,>=0.0.0a0", + "libcudf==24.12.*,>=0.0.0a0", + "libcuspatial==24.12.*,>=0.0.0a0", "numpy>=1.23,<2.0a0", - "rmm==24.10.*,>=0.0.0a0", + "rmm==24.12.*,>=0.0.0a0", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. classifiers = [ "Intended Audience :: Developers", @@ -136,12 +136,12 @@ dependencies-file = "../../dependencies.yaml" matrix-entry = "cuda_suffixed=true" requires = [ "cmake>=3.26.4,!=3.30.0", - "cudf==24.10.*,>=0.0.0a0", + "cudf==24.12.*,>=0.0.0a0", "cython>=3.0.0", - "libcudf==24.10.*,>=0.0.0a0", - "libcuspatial==24.10.*,>=0.0.0a0", - "librmm==24.10.*,>=0.0.0a0", + "libcudf==24.12.*,>=0.0.0a0", + "libcuspatial==24.12.*,>=0.0.0a0", + "librmm==24.12.*,>=0.0.0a0", "ninja", - "rmm==24.10.*,>=0.0.0a0", + "rmm==24.12.*,>=0.0.0a0", "wheel", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. diff --git a/python/libcuspatial/pyproject.toml b/python/libcuspatial/pyproject.toml index ab64d4cb6..1de8df8c1 100644 --- a/python/libcuspatial/pyproject.toml +++ b/python/libcuspatial/pyproject.toml @@ -39,7 +39,7 @@ classifiers = [ "Programming Language :: Python :: 3.11", ] dependencies = [ - "libcudf==24.10.*,>=0.0.0a0", + "libcudf==24.12.*,>=0.0.0a0", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. [project.urls] @@ -70,8 +70,8 @@ dependencies-file = "../../dependencies.yaml" matrix-entry = "cuda_suffixed=true" requires = [ "cmake>=3.26.4,!=3.30.0", - "libcudf==24.10.*,>=0.0.0a0", - "librmm==24.10.*,>=0.0.0a0", + "libcudf==24.12.*,>=0.0.0a0", + "librmm==24.12.*,>=0.0.0a0", "ninja", "wheel", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. From 43776f294b46d0eaedf3f9460830ecea7b795c8a Mon Sep 17 00:00:00 2001 From: Bradley Dice Date: Fri, 20 Sep 2024 13:21:04 -0500 Subject: [PATCH 13/28] Update flake8 to 7.1.1. (#1458) We need to update flake8 to fix a false-positive that appears with older flake8 versions on Python 3.12. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Matthew Roeschke (https://github.com/mroeschke) - James Lamb (https://github.com/jameslamb) URL: https://github.com/rapidsai/cuspatial/pull/1458 --- .pre-commit-config.yaml | 2 +- python/cuspatial/cuspatial/tests/test_geodataframe.py | 11 +++++------ python/cuspatial/cuspatial/tests/test_geoseries.py | 10 +++++----- 3 files changed, 11 insertions(+), 12 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index b96543933..3949b4c3d 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -18,7 +18,7 @@ repos: # Explicitly specify the pyproject.toml at the repo root, not per-project. args: ["--config", "pyproject.toml"] - repo: https://github.com/PyCQA/flake8 - rev: 5.0.4 + rev: 7.1.1 hooks: - id: flake8 args: ["--config=.flake8"] diff --git a/python/cuspatial/cuspatial/tests/test_geodataframe.py b/python/cuspatial/cuspatial/tests/test_geodataframe.py index dee2bfeb4..bd6560742 100644 --- a/python/cuspatial/cuspatial/tests/test_geodataframe.py +++ b/python/cuspatial/cuspatial/tests/test_geodataframe.py @@ -66,7 +66,7 @@ def generator(size, has_z=False): def assert_eq_point(p1, p2): - assert type(p1) == type(p2) + assert type(p1) is type(p2) assert p1.x == p2.x assert p1.y == p2.y assert p1.has_z == p2.has_z @@ -76,7 +76,7 @@ def assert_eq_point(p1, p2): def assert_eq_multipoint(p1, p2): - assert type(p1) == type(p2) + assert type(p1) is type(p2) assert len(p1) == len(p2) for i in range(len(p1)): assert_eq_point(p1[i], p2[i]) @@ -93,8 +93,7 @@ def assert_eq_multipolygon(p1, p2): def assert_eq_geo_df(geo1, geo2): - if type(geo1) != type(geo2): - assert TypeError + assert type(geo1) is type(geo2) assert geo1.columns.equals(geo2.columns) for col in geo1.columns: if geo1[col].dtype == "geometry": @@ -112,7 +111,7 @@ def test_select_multiple_columns(gpdf): def test_type_persistence(gpdf): cugpdf = cuspatial.from_geopandas(gpdf) - assert type(cugpdf["geometry"]) == cuspatial.GeoSeries + assert type(cugpdf["geometry"]) is cuspatial.GeoSeries def test_interleaved_point(gpdf, polys): @@ -462,7 +461,7 @@ def test_reset_index(level, drop, inplace, col_level, col_fill): def test_cudf_dataframe_init(): df = cudf.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]}) gdf = cuspatial.GeoDataFrame(df) - assert_eq_geo_df(gdf.to_pandas(), df.to_pandas()) + assert_eq_geo_df(gdf.to_pandas(), gpd.GeoDataFrame(df.to_pandas())) def test_apply_boolean_mask(gpdf, mask_factory): diff --git a/python/cuspatial/cuspatial/tests/test_geoseries.py b/python/cuspatial/cuspatial/tests/test_geoseries.py index 35007c57d..56f61158e 100644 --- a/python/cuspatial/cuspatial/tests/test_geoseries.py +++ b/python/cuspatial/cuspatial/tests/test_geoseries.py @@ -87,7 +87,7 @@ def generator(size: Integral, obj_type: Example_Feature_Enum = None): def assert_eq_point(p1, p2): - assert type(p1) == type(p2) + assert type(p1) is type(p2) assert p1.x == p2.x assert p1.y == p2.y assert p1.has_z == p2.has_z @@ -97,14 +97,14 @@ def assert_eq_point(p1, p2): def assert_eq_multipoint(p1, p2): - assert type(p1) == type(p2) + assert type(p1) is type(p2) assert len(p1) == len(p2) for i in range(len(p1)): assert_eq_point(p1[i], p2[i]) def assert_eq_linestring(p1, p2): - assert type(p1) == type(p2) + assert type(p1) is type(p2) assert p1 == p2 @@ -124,8 +124,8 @@ def assert_eq_multipolygon(p1, p2): def assert_eq_geo(geo1, geo2): - if type(geo1) != type(geo2): - assert TypeError + if type(geo1) is not type(geo2): + raise TypeError result = geo1.equals(geo2) if isinstance(result, bool): assert result From be73c8e4bbb9017e8b3ecc4bf501e0bdd4f41a24 Mon Sep 17 00:00:00 2001 From: Sebastian Berg Date: Tue, 24 Sep 2024 03:03:17 +0200 Subject: [PATCH 14/28] Remove NumPy <2 pin, remove proj pin (#1441) This PR removes the NumPy<2 pin which is expected to work for RAPIDS projects once CuPy 13.3.0 is released (CuPy 13.2.0 had some issues preventing the use with NumPy 2). Also contributes to https://github.com/rapidsai/build-planning/issues/56 by removing the pin on `proj` (which allows this project to work with newer `fmt` / `spdlog`, in sync with conda-forge). Authors: - Sebastian Berg (https://github.com/seberg) - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) - https://github.com/jakirkham Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cuspatial/pull/1441 --- conda/environments/all_cuda-118_arch-x86_64.yaml | 4 ++-- conda/environments/all_cuda-125_arch-x86_64.yaml | 4 ++-- conda/recipes/cuproj/meta.yaml | 2 +- conda/recipes/cuspatial/meta.yaml | 2 +- conda/recipes/libcuspatial/meta.yaml | 4 ++-- dependencies.yaml | 4 ++-- python/cuproj/pyproject.toml | 2 +- python/cuspatial/pyproject.toml | 2 +- 8 files changed, 12 insertions(+), 12 deletions(-) diff --git a/conda/environments/all_cuda-118_arch-x86_64.yaml b/conda/environments/all_cuda-118_arch-x86_64.yaml index 5ac2a6b9c..6526e9d5f 100644 --- a/conda/environments/all_cuda-118_arch-x86_64.yaml +++ b/conda/environments/all_cuda-118_arch-x86_64.yaml @@ -28,12 +28,12 @@ dependencies: - nbsphinx - ninja - notebook -- numpy>=1.23,<2.0a0 +- numpy>=1.23,<3.0a0 - numpydoc - nvcc_linux-64=11.8 - osmnx>=1.9.3 - pre-commit -- proj >=9.3.0,<9.3.1.0a0 +- proj - pydata-sphinx-theme!=0.14.2 - pydeck - pylibcudf==24.10.*,>=0.0.0a0 diff --git a/conda/environments/all_cuda-125_arch-x86_64.yaml b/conda/environments/all_cuda-125_arch-x86_64.yaml index 7c2ca25d4..9d0e65000 100644 --- a/conda/environments/all_cuda-125_arch-x86_64.yaml +++ b/conda/environments/all_cuda-125_arch-x86_64.yaml @@ -31,11 +31,11 @@ dependencies: - nbsphinx - ninja - notebook -- numpy>=1.23,<2.0a0 +- numpy>=1.23,<3.0a0 - numpydoc - osmnx>=1.9.3 - pre-commit -- proj >=9.3.0,<9.3.1.0a0 +- proj - pydata-sphinx-theme!=0.14.2 - pydeck - pylibcudf==24.10.*,>=0.0.0a0 diff --git a/conda/recipes/cuproj/meta.yaml b/conda/recipes/cuproj/meta.yaml index 0096561fe..29d590bc0 100644 --- a/conda/recipes/cuproj/meta.yaml +++ b/conda/recipes/cuproj/meta.yaml @@ -64,7 +64,7 @@ requirements: - rapids-build-backend >=0.3.0,<0.4.0.dev0 - librmm ={{ minor_version }} - scikit-build-core >=0.10.0 - - proj >=9.3.0,<9.3.1.0a0 # TODO: proj is being held back due to needing spdlog/fmt upgrades to support recent conda-forge builds of tiledb + - proj - sqlite run: {% if cuda_major == "11" %} diff --git a/conda/recipes/cuspatial/meta.yaml b/conda/recipes/cuspatial/meta.yaml index c7c0d780c..9f37d47e9 100644 --- a/conda/recipes/cuspatial/meta.yaml +++ b/conda/recipes/cuspatial/meta.yaml @@ -77,7 +77,7 @@ requirements: - cudf ={{ minor_version }} - pylibcudf ={{ minor_version }} - geopandas >=1.0.0 - - numpy >=1.23,<2.0a0 + - numpy >=1.23,<3.0a0 - python - rmm ={{ minor_version }} diff --git a/conda/recipes/libcuspatial/meta.yaml b/conda/recipes/libcuspatial/meta.yaml index 177037568..3e212690c 100644 --- a/conda/recipes/libcuspatial/meta.yaml +++ b/conda/recipes/libcuspatial/meta.yaml @@ -49,7 +49,7 @@ requirements: - libcudf ={{ minor_version }} - librmm ={{ minor_version }} - sqlite - - proj >=9.3.0,<9.3.1.0a0 # TODO: proj is being held back due to needing spdlog/fmt upgrades to support recent conda-forge builds of tiledb + - proj outputs: - name: libcuspatial @@ -85,7 +85,7 @@ outputs: - libcudf ={{ minor_version }} - librmm ={{ minor_version }} - sqlite - - proj >=9.3.0,<9.3.1.0a0 # TODO: proj is being held back due to needing spdlog/fmt upgrades to support recent conda-forge builds of tiledb + - proj test: commands: - test -f $PREFIX/lib/libcuspatial.so diff --git a/dependencies.yaml b/dependencies.yaml index 3ac92dfbf..8e4c0964e 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -180,7 +180,7 @@ dependencies: packages: - c-compiler - cxx-compiler - - proj >=9.3.0,<9.3.1.0a0 # TODO: proj is being held back due to needing spdlog/fmt upgrades to support recent conda-forge builds of tiledb + - proj - sqlite specific: - output_types: conda @@ -400,7 +400,7 @@ dependencies: - output_types: [conda, requirements, pyproject] packages: - &geopandas geopandas>=1.0.0 - - &numpy numpy>=1.23,<2.0a0 + - &numpy numpy>=1.23,<3.0a0 test_python_cuspatial: common: - output_types: [conda, requirements, pyproject] diff --git a/python/cuproj/pyproject.toml b/python/cuproj/pyproject.toml index fc4b99d54..f7e920543 100644 --- a/python/cuproj/pyproject.toml +++ b/python/cuproj/pyproject.toml @@ -45,7 +45,7 @@ classifiers = [ test = [ "cuspatial==24.10.*,>=0.0.0a0", "geopandas>=1.0.0", - "numpy>=1.23,<2.0a0", + "numpy>=1.23,<3.0a0", "pyproj>=3.6.0,<3.7a0", "pytest", "pytest-cov", diff --git a/python/cuspatial/pyproject.toml b/python/cuspatial/pyproject.toml index 3a850030e..94b401472 100644 --- a/python/cuspatial/pyproject.toml +++ b/python/cuspatial/pyproject.toml @@ -34,7 +34,7 @@ dependencies = [ "geopandas>=1.0.0", "libcudf==24.10.*,>=0.0.0a0", "libcuspatial==24.10.*,>=0.0.0a0", - "numpy>=1.23,<2.0a0", + "numpy>=1.23,<3.0a0", "rmm==24.10.*,>=0.0.0a0", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. classifiers = [ From 3d94becb14f017062c8e0fde71507f66c1dd33d4 Mon Sep 17 00:00:00 2001 From: Jake Awe <50372925+AyodeAwe@users.noreply.github.com> Date: Tue, 24 Sep 2024 14:11:20 -0500 Subject: [PATCH 15/28] update update-version.sh to use packaging lib (#1462) --- ci/release/update-version.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/release/update-version.sh b/ci/release/update-version.sh index 3f00dd7fb..03c05e9c4 100755 --- a/ci/release/update-version.sh +++ b/ci/release/update-version.sh @@ -24,7 +24,7 @@ NEXT_MINOR=$(echo $NEXT_FULL_TAG | awk '{split($0, a, "."); print a[2]}') NEXT_SHORT_TAG=${NEXT_MAJOR}.${NEXT_MINOR} # Need to distutils-normalize the original version -NEXT_SHORT_TAG_PEP440=$(python -c "from setuptools.extern import packaging; print(packaging.version.Version('${NEXT_SHORT_TAG}'))") +NEXT_SHORT_TAG_PEP440=$(python -c "from packaging.version import Version; print(Version('${NEXT_SHORT_TAG}'))") echo "Preparing release $CURRENT_TAG => $NEXT_FULL_TAG" From d4835eef065d0ee680045becebfefcfc343bf34a Mon Sep 17 00:00:00 2001 From: GALI PREM SAGAR Date: Wed, 25 Sep 2024 13:22:50 -0500 Subject: [PATCH 16/28] Switch `traceback` to `native` in pytests (#1464) In cudf & cuml we have observed a ~10% to ~20% respectively speed up of pytest suite execution by switching pytest traceback to `--native`: ``` currently: 102474 passed, 2117 skipped, 902 xfailed in 892.16s (0:14:52) --tb=short: 102474 passed, 2117 skipped, 902 xfailed in 898.99s (0:14:58) --tb=no: 102474 passed, 2117 skipped, 902 xfailed in 815.98s (0:13:35) --tb=native: 102474 passed, 2117 skipped, 902 xfailed in 820.92s (0:13:40) ``` This PR makes similar change to `cuspatial` repo. xref: https://github.com/rapidsai/cudf/pull/16851 Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Bradley Dice (https://github.com/bdice) - Paul Taylor (https://github.com/trxcllnt) URL: https://github.com/rapidsai/cuspatial/pull/1464 --- python/cuproj/cuproj/tests/pytest.ini | 5 +++++ python/cuspatial/cuspatial/tests/pytest.ini | 5 +++++ 2 files changed, 10 insertions(+) create mode 100644 python/cuproj/cuproj/tests/pytest.ini create mode 100644 python/cuspatial/cuspatial/tests/pytest.ini diff --git a/python/cuproj/cuproj/tests/pytest.ini b/python/cuproj/cuproj/tests/pytest.ini new file mode 100644 index 000000000..bf70c06f8 --- /dev/null +++ b/python/cuproj/cuproj/tests/pytest.ini @@ -0,0 +1,5 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. + +[pytest] +addopts = --tb=native + diff --git a/python/cuspatial/cuspatial/tests/pytest.ini b/python/cuspatial/cuspatial/tests/pytest.ini new file mode 100644 index 000000000..bf70c06f8 --- /dev/null +++ b/python/cuspatial/cuspatial/tests/pytest.ini @@ -0,0 +1,5 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. + +[pytest] +addopts = --tb=native + From cc8fd60b7bbc09773892bfde4b4dcd6db1e9ce3e Mon Sep 17 00:00:00 2001 From: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Date: Mon, 7 Oct 2024 12:53:30 -1000 Subject: [PATCH 17/28] xfail Python tests dependent on invalid Polygons (#1467) --- python/cuspatial/cuspatial/tests/conftest.py | 2 ++ .../cuspatial/tests/test_cudf_integration.py | 11 ++++++++++- python/cuspatial/cuspatial/tests/test_geodataframe.py | 8 +++++++- 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/python/cuspatial/cuspatial/tests/conftest.py b/python/cuspatial/cuspatial/tests/conftest.py index caa53dcea..b71b08128 100644 --- a/python/cuspatial/cuspatial/tests/conftest.py +++ b/python/cuspatial/cuspatial/tests/conftest.py @@ -29,6 +29,8 @@ def gs(): g8 = Polygon( ((35, 36), (37, 38), (39, 40), (41, 42)), ) + # TODO: g9, g10, g11 are invalid + # https://github.com/libgeos/geos/issues/1177 g9 = MultiPolygon( [ ( diff --git a/python/cuspatial/cuspatial/tests/test_cudf_integration.py b/python/cuspatial/cuspatial/tests/test_cudf_integration.py index 0457f0992..41214e816 100644 --- a/python/cuspatial/cuspatial/tests/test_cudf_integration.py +++ b/python/cuspatial/cuspatial/tests/test_cudf_integration.py @@ -1,11 +1,18 @@ -# Copyright (c) 2022 NVIDIA CORPORATION. +# Copyright (c) 2022-2024, NVIDIA CORPORATION. import geopandas as gpd import numpy as np import pandas as pd +import pytest import cuspatial +reason = ( + "gs fixture contains invalid Polygons/MultiPolygons: " + "https://github.com/libgeos/geos/issues/1177" +) + +@pytest.mark.xfail(reason=reason) def test_sort_index_series(gs): gs.index = np.random.permutation(len(gs)) cugs = cuspatial.from_geopandas(gs) @@ -14,6 +21,7 @@ def test_sort_index_series(gs): gpd.testing.assert_geoseries_equal(got, expected) +@pytest.mark.xfail(reason=reason) def test_sort_index_dataframe(gpdf): gpdf.index = np.random.permutation(len(gpdf)) cugpdf = cuspatial.from_geopandas(gpdf) @@ -22,6 +30,7 @@ def test_sort_index_dataframe(gpdf): gpd.testing.assert_geodataframe_equal(got, expected) +@pytest.mark.xfail(reason=reason) def test_sort_values(gpdf): cugpdf = cuspatial.from_geopandas(gpdf) expected = gpdf.sort_values("random") diff --git a/python/cuspatial/cuspatial/tests/test_geodataframe.py b/python/cuspatial/cuspatial/tests/test_geodataframe.py index bd6560742..f834b35e5 100644 --- a/python/cuspatial/cuspatial/tests/test_geodataframe.py +++ b/python/cuspatial/cuspatial/tests/test_geodataframe.py @@ -464,7 +464,13 @@ def test_cudf_dataframe_init(): assert_eq_geo_df(gdf.to_pandas(), gpd.GeoDataFrame(df.to_pandas())) -def test_apply_boolean_mask(gpdf, mask_factory): +def test_apply_boolean_mask(gpdf, mask_factory, request): + if "MaskNone" not in request.node.callspec.id: + reason = ( + "gs fixture contains invalid Polygons/MultiPolygons: " + "https://github.com/libgeos/geos/issues/1177" + ) + request.applymarker(pytest.mark.xfail(reason=reason)) mask = mask_factory(len(gpdf)) expected = gpdf[mask] From 77c893abac23168a8d820f87c16af7f5fa63b197 Mon Sep 17 00:00:00 2001 From: Ray Douglass Date: Wed, 9 Oct 2024 09:38:55 -0400 Subject: [PATCH 18/28] Update Changelog [skip ci] --- CHANGELOG.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index faa934484..abe227bae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,48 @@ +# cuspatial 24.10.00 (9 Oct 2024) + +## 🚨 Breaking Changes + +- Disallow GeoSeries from accepting a column in favor of `_from_column` ([#1434](https://github.com/rapidsai/cuspatial/pull/1434)) [@mroeschke](https://github.com/mroeschke) + +## 🐛 Bug Fixes + +- re-enable mixing libcuspatial wheels with libcudf conda packages ([#1456](https://github.com/rapidsai/cuspatial/pull/1456)) [@jameslamb](https://github.com/jameslamb) +- fix wheel tests on Rocky Linux ([#1452](https://github.com/rapidsai/cuspatial/pull/1452)) [@jameslamb](https://github.com/jameslamb) +- Add data=None to ListColumn constructor ([#1442](https://github.com/rapidsai/cuspatial/pull/1442)) [@mroeschke](https://github.com/mroeschke) +- Remove unused shuffle_order in GeoColumn ([#1439](https://github.com/rapidsai/cuspatial/pull/1439)) [@mroeschke](https://github.com/mroeschke) +- Fix _recombine_columns when either geocolumn or non-geocolumn was empty ([#1438](https://github.com/rapidsai/cuspatial/pull/1438)) [@mroeschke](https://github.com/mroeschke) +- Internal linkage kernels ([#1436](https://github.com/rapidsai/cuspatial/pull/1436)) [@robertmaynard](https://github.com/robertmaynard) +- Disallow GeoSeries from accepting a column in favor of `_from_column` ([#1434](https://github.com/rapidsai/cuspatial/pull/1434)) [@mroeschke](https://github.com/mroeschke) + +## 📖 Documentation + +- Recommending `miniforge` for conda install ([#1457](https://github.com/rapidsai/cuspatial/pull/1457)) [@mmccarty](https://github.com/mmccarty) + +## 🛠️ Improvements + +- xfail Python tests dependent on invalid Polygons ([#1467](https://github.com/rapidsai/cuspatial/pull/1467)) [@mroeschke](https://github.com/mroeschke) +- Switch `traceback` to `native` in pytests ([#1464](https://github.com/rapidsai/cuspatial/pull/1464)) [@galipremsagar](https://github.com/galipremsagar) +- Update update-version.sh to use packaging lib ([#1462](https://github.com/rapidsai/cuspatial/pull/1462)) [@AyodeAwe](https://github.com/AyodeAwe) +- Use CI workflow branch 'branch-24.10' again ([#1459](https://github.com/rapidsai/cuspatial/pull/1459)) [@jameslamb](https://github.com/jameslamb) +- Update flake8 to 7.1.1. ([#1458](https://github.com/rapidsai/cuspatial/pull/1458)) [@bdice](https://github.com/bdice) +- Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 ([#1453](https://github.com/rapidsai/cuspatial/pull/1453)) [@jameslamb](https://github.com/jameslamb) +- rearrange dependencies.yaml, fix development instructions ([#1451](https://github.com/rapidsai/cuspatial/pull/1451)) [@jameslamb](https://github.com/jameslamb) +- Distribute libcuspatial wheels ([#1450](https://github.com/rapidsai/cuspatial/pull/1450)) [@jameslamb](https://github.com/jameslamb) +- cuproj: remove pin on 'proj' in conda packages ([#1449](https://github.com/rapidsai/cuspatial/pull/1449)) [@jameslamb](https://github.com/jameslamb) +- cuproj: depend on librmm, not rmm ([#1448](https://github.com/rapidsai/cuspatial/pull/1448)) [@jameslamb](https://github.com/jameslamb) +- wheels: dynamically load `libcudf.so` from `libcudf` wheel ([#1447](https://github.com/rapidsai/cuspatial/pull/1447)) [@jameslamb](https://github.com/jameslamb) +- Update rapidsai/pre-commit-hooks ([#1445](https://github.com/rapidsai/cuspatial/pull/1445)) [@KyleFromNVIDIA](https://github.com/KyleFromNVIDIA) +- Drop Python 3.9 support ([#1443](https://github.com/rapidsai/cuspatial/pull/1443)) [@jameslamb](https://github.com/jameslamb) +- Remove NumPy <2 pin, remove proj pin ([#1441](https://github.com/rapidsai/cuspatial/pull/1441)) [@seberg](https://github.com/seberg) +- Update cuspatial to use pylibcudf ([#1440](https://github.com/rapidsai/cuspatial/pull/1440)) [@vyasr](https://github.com/vyasr) +- Update pre-commit hooks ([#1435](https://github.com/rapidsai/cuspatial/pull/1435)) [@KyleFromNVIDIA](https://github.com/KyleFromNVIDIA) +- Improve update-version.sh ([#1432](https://github.com/rapidsai/cuspatial/pull/1432)) [@bdice](https://github.com/bdice) +- Use tool.scikit-build.cmake.version, set scikit-build-core minimum-version ([#1430](https://github.com/rapidsai/cuspatial/pull/1430)) [@jameslamb](https://github.com/jameslamb) +- Pass columns instead of Series to `cudf.DataFrame` in split-combine workflow ([#1429](https://github.com/rapidsai/cuspatial/pull/1429)) [@mroeschke](https://github.com/mroeschke) +- Address cudf.DataFrame.insert API change ([#1428](https://github.com/rapidsai/cuspatial/pull/1428)) [@mroeschke](https://github.com/mroeschke) +- ensure notebook-testing script fails when notebooks fail ([#1424](https://github.com/rapidsai/cuspatial/pull/1424)) [@jameslamb](https://github.com/jameslamb) +- Forward-merge branch-24.08 into branch-24.10 ([#1410](https://github.com/rapidsai/cuspatial/pull/1410)) [@jakirkham](https://github.com/jakirkham) + # cuspatial 24.08.00 (7 Aug 2024) ## 🐛 Bug Fixes From c45ff5f9fa2855c7123d4cd094c48c2aee527ed0 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Fri, 11 Oct 2024 15:06:47 -0500 Subject: [PATCH 19/28] remove unused variables in build_docs script (#1469) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Contributes to https://github.com/rapidsai/build-planning/issues/106 This project doesn't need any of the changes that are being made as part of that issue across most other repos... it's already doing all of its `pip` and `conda` installs in CI in single installs (which is strict and safe) 🎉 This small PR just makes its handling of environment variables in `ci/build_docs.sh` consistent with other RAPIDS repos, in the interest of keeping those scripts looking similar so it's easy to apply changes across all RAPIDS projects. ref: https://github.com/rapidsai/cudf/pull/17013#discussion_r1792163289 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Mark Harris (https://github.com/harrism) - Mike Sarahan (https://github.com/msarahan) URL: https://github.com/rapidsai/cuspatial/pull/1469 --- ci/build_docs.sh | 3 +-- ci/release/update-version.sh | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/ci/build_docs.sh b/ci/build_docs.sh index 3ce57b2d8..ab2b56c35 100755 --- a/ci/build_docs.sh +++ b/ci/build_docs.sh @@ -23,7 +23,6 @@ rapids-print-env export RAPIDS_VERSION="$(rapids-version)" export RAPIDS_VERSION_MAJOR_MINOR="$(rapids-version-major-minor)" -export RAPIDS_VERSION_NUMBER="24.12" export RAPIDS_DOCS_DIR="$(mktemp -d)" rapids-logger "Build cuSpatial CPP docs" @@ -54,4 +53,4 @@ mkdir -p "${RAPIDS_DOCS_DIR}/cuproj/html" mv _html/* "${RAPIDS_DOCS_DIR}/cuproj/html" popd -rapids-upload-docs +RAPIDS_VERSION_NUMBER="${RAPIDS_VERSION_MAJOR_MINOR}" rapids-upload-docs diff --git a/ci/release/update-version.sh b/ci/release/update-version.sh index 03c05e9c4..dfd0fe7e3 100755 --- a/ci/release/update-version.sh +++ b/ci/release/update-version.sh @@ -40,7 +40,6 @@ echo "${NEXT_FULL_TAG}" > VERSION for FILE in .github/workflows/*.yaml; do sed_runner "/shared-workflows/ s/@.*/@branch-${NEXT_SHORT_TAG}/g" "${FILE}" done -sed_runner "s/RAPIDS_VERSION_NUMBER=\".*/RAPIDS_VERSION_NUMBER=\"${NEXT_SHORT_TAG}\"/g" ci/build_docs.sh DEPENDENCIES=( pylibcudf From 8314c1365ddde9d630a8d4e22d82fcba4ea926d8 Mon Sep 17 00:00:00 2001 From: Basit Ayantunde Date: Wed, 16 Oct 2024 15:14:39 +0100 Subject: [PATCH 20/28] Updated libcudftestutil CMake linking logic for 24.12 (#1475) This merge request fixes the CMake linking logic to cudftestutil for branch-24.12, allowing you to specify which version of GTest/GBench to use as long as the API is source compatible. Follows up on: https://github.com/rapidsai/cudf/pull/16839 Authors: - Basit Ayantunde (https://github.com/lamarrr) Approvers: - Mark Harris (https://github.com/harrism) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cuspatial/pull/1475 --- cpp/CMakeLists.txt | 2 +- cpp/benchmarks/CMakeLists.txt | 6 +++++- cpp/cmake/thirdparty/get_cudf.cmake | 4 ++-- cpp/tests/CMakeLists.txt | 28 +++++++++++++++++++++++++++- cpp/tests/test_common.cpp | 15 +++++++++++++++ 5 files changed, 50 insertions(+), 5 deletions(-) create mode 100644 cpp/tests/test_common.cpp diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt index 8908617b1..aac5aea43 100644 --- a/cpp/CMakeLists.txt +++ b/cpp/CMakeLists.txt @@ -68,7 +68,7 @@ set(CUSPATIAL_BUILD_TESTS ${BUILD_TESTS}) set(CUSPATIAL_BUILD_BENCHMARKS ${BUILD_BENCHMARKS}) set(CUSPATIAL_CXX_FLAGS "") -set(CUSPATIAL_CUDA_FLAGS "") +set(CUSPATIAL_CUDA_FLAGS --expt-extended-lambda --expt-relaxed-constexpr) set(CUSPATIAL_CXX_DEFINITIONS "") set(CUSPATIAL_CUDA_DEFINITIONS "") diff --git a/cpp/benchmarks/CMakeLists.txt b/cpp/benchmarks/CMakeLists.txt index 10e626f3f..98506d9a4 100644 --- a/cpp/benchmarks/CMakeLists.txt +++ b/cpp/benchmarks/CMakeLists.txt @@ -23,6 +23,9 @@ add_library(cuspatial_benchmark_common OBJECT target_compile_features(cuspatial_benchmark_common PUBLIC cxx_std_17 cuda_std_17) +target_compile_options(cuspatial_benchmark_common PUBLIC "$<$:${CUSPATIAL_CXX_FLAGS}>" + "$<$:${CUSPATIAL_CUDA_FLAGS}>") + set_target_properties(cuspatial_benchmark_common PROPERTIES RUNTIME_OUTPUT_DIRECTORY "$" INSTALL_RPATH "\$ORIGIN/../../../lib" @@ -38,7 +41,8 @@ target_link_libraries(cuspatial_benchmark_common PUBLIC benchmark::benchmark cudf::cudftestutil ranger::ranger - cuspatial) + cuspatial GTest::gtest GTest::gmock + PRIVATE cudf::cudftestutil_impl) target_compile_options(cuspatial_benchmark_common PUBLIC "$<$:${CUSPATIAL_CXX_FLAGS}>" diff --git a/cpp/cmake/thirdparty/get_cudf.cmake b/cpp/cmake/thirdparty/get_cudf.cmake index 3d7232fba..c5d2712c9 100644 --- a/cpp/cmake/thirdparty/get_cudf.cmake +++ b/cpp/cmake/thirdparty/get_cudf.cmake @@ -1,5 +1,5 @@ #============================================================================= -# Copyright (c) 2021-2023, NVIDIA CORPORATION. +# Copyright (c) 2021-2024, NVIDIA CORPORATION. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -27,7 +27,7 @@ function(find_and_configure_cudf) set(cudf_components "") if(BUILD_TESTS OR BUILD_BENCHMARKS) - list(APPEND global_targets cudf::cudftestutil) + list(APPEND global_targets cudf::cudftestutil cudf::cudftestutil_impl) set(cudf_components COMPONENTS testing) endif() diff --git a/cpp/tests/CMakeLists.txt b/cpp/tests/CMakeLists.txt index f6752c860..7bca36a41 100644 --- a/cpp/tests/CMakeLists.txt +++ b/cpp/tests/CMakeLists.txt @@ -17,6 +17,32 @@ ################################################################################################### # - compiler function ----------------------------------------------------------------------------- +# cudftestutil_impl is an interface source library, this empty object +# library is used to speed-up compilation and linking against it, +# otherwise we pay the non-trivial compilation cost repeatedly for each +# test executable +add_library(cuspatial_test_common OBJECT test_common.cpp) + +target_compile_features(cuspatial_test_common PUBLIC cxx_std_17 cuda_std_17) + +set_target_properties(cuspatial_test_common + PROPERTIES RUNTIME_OUTPUT_DIRECTORY "$" + INSTALL_RPATH "\$ORIGIN/../../../lib" + CXX_STANDARD 17 + CXX_STANDARD_REQUIRED ON + CUDA_STANDARD 17 + CUDA_STANDARD_REQUIRED ON + POSITION_INDEPENDENT_CODE ON + INTERFACE_POSITION_INDEPENDENT_CODE ON +) + +target_link_libraries(cuspatial_test_common + PUBLIC cudf::cudftestutil GTest::gtest GTest::gmock + PRIVATE cudf::cudftestutil_impl) + +target_compile_options(cuspatial_test_common PUBLIC "$<$:${CUSPATIAL_CXX_FLAGS}>" +"$<$:${CUSPATIAL_CUDA_FLAGS}>") + function(ConfigureTest CMAKE_TEST_NAME) add_executable(${CMAKE_TEST_NAME} ${ARGN}) target_compile_options(${CMAKE_TEST_NAME} @@ -34,7 +60,7 @@ function(ConfigureTest CMAKE_TEST_NAME) CUDA_STANDARD 17 CUDA_STANDARD_REQUIRED ON ) - target_link_libraries(${CMAKE_TEST_NAME} GTest::gtest_main GTest::gmock_main ranger::ranger cudf::cudftestutil cuspatial) + target_link_libraries(${CMAKE_TEST_NAME} GTest::gtest_main GTest::gmock_main ranger::ranger cudf::cudftestutil cuspatial cuspatial_test_common) add_test(NAME ${CMAKE_TEST_NAME} COMMAND ${CMAKE_TEST_NAME}) install( TARGETS ${CMAKE_TEST_NAME} diff --git a/cpp/tests/test_common.cpp b/cpp/tests/test_common.cpp new file mode 100644 index 000000000..244bc0c4b --- /dev/null +++ b/cpp/tests/test_common.cpp @@ -0,0 +1,15 @@ +/* + * Copyright (c) 2024, NVIDIA CORPORATION. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ From 1bf2527cc3968c178c420ad4d97d674e40c296a1 Mon Sep 17 00:00:00 2001 From: Mark Harris <783069+harrism@users.noreply.github.com> Date: Thu, 17 Oct 2024 13:22:11 +1100 Subject: [PATCH 21/28] Make polygon fixture data valid (#1472) Fixes #1471. Fixes the invalid polygon geometry and improves geometry slicing tests to not depend on hard-coded vertex positions that are also hard-coded in the fixture. Authors: - Mark Harris (https://github.com/harrism) - Bradley Dice (https://github.com/bdice) Approvers: - Michael Wang (https://github.com/isVoid) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cuspatial/pull/1472 --- python/cuspatial/cuspatial/testing/helpers.py | 15 ++ python/cuspatial/cuspatial/tests/conftest.py | 93 ++------- .../cuspatial/tests/test_cudf_integration.py | 9 - .../cuspatial/tests/test_from_geopandas.py | 11 +- .../cuspatial/tests/test_geodataframe.py | 16 +- .../cuspatial/tests/test_geoseries.py | 182 +++--------------- 6 files changed, 74 insertions(+), 252 deletions(-) create mode 100644 python/cuspatial/cuspatial/testing/helpers.py diff --git a/python/cuspatial/cuspatial/testing/helpers.py b/python/cuspatial/cuspatial/testing/helpers.py new file mode 100644 index 000000000..a03ab2d57 --- /dev/null +++ b/python/cuspatial/cuspatial/testing/helpers.py @@ -0,0 +1,15 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. +from itertools import chain + +from shapely import get_coordinates + + +def geometry_to_coords(geom, geom_types): + points_list = geom[geom.apply(lambda x: isinstance(x, geom_types))] + # flatten multigeometries, then geometries, then coordinates + points = list(chain(points_list.apply(get_coordinates))) + coords_list = list(chain(*points)) + xy = list(chain(*coords_list)) + x = xy[::2] + y = xy[1::2] + return xy, x, y diff --git a/python/cuspatial/cuspatial/tests/conftest.py b/python/cuspatial/cuspatial/tests/conftest.py index b71b08128..a8ecb11ec 100644 --- a/python/cuspatial/cuspatial/tests/conftest.py +++ b/python/cuspatial/cuspatial/tests/conftest.py @@ -19,48 +19,46 @@ @pytest.fixture def gs(): g0 = Point(-1, 0) - g1 = MultiPoint(((1, 2), (3, 4))) - g2 = MultiPoint(((5, 6), (7, 8))) - g3 = Point(9, 10) + g1 = Point(9, 10) + g2 = MultiPoint(((1, 2), (3, 4))) + g3 = MultiPoint(((5, 6), (7, 8))) g4 = LineString(((11, 12), (13, 14))) g5 = MultiLineString((((15, 16), (17, 18)), ((19, 20), (21, 22)))) g6 = MultiLineString((((23, 24), (25, 26)), ((27, 28), (29, 30)))) g7 = LineString(((31, 32), (33, 34))) g8 = Polygon( - ((35, 36), (37, 38), (39, 40), (41, 42)), + ((35, 36), (38, 36), (41, 39), (41, 42)), ) - # TODO: g9, g10, g11 are invalid - # https://github.com/libgeos/geos/issues/1177 g9 = MultiPolygon( [ ( - ((43, 44), (45, 46), (47, 48)), - [((49, 50), (51, 52), (53, 54))], + ((43, 44), (48, 44), (47, 48)), + [((45, 45), (46, 46), (47, 45))], ), ( - ((55, 56), (57, 58), (59, 60)), - [((61, 62), (63, 64), (65, 66))], + ((55, 56), (60, 56), (59, 60)), + [((57, 57), (58, 58), (59, 57))], ), ] ) g10 = MultiPolygon( [ ( - ((67, 68), (69, 70), (71, 72)), - [((73, 74), (75, 76), (77, 78))], + ((67, 68), (72, 68), (71, 72)), + [((69, 69), (70, 70), (71, 69))], ), ( - ((79, 80), (81, 82), (83, 84)), + ((79, 80), (90, 82), (83, 90)), [ - ((85, 86), (87, 88), (89, 90)), - ((91, 92), (93, 94), (95, 96)), + ((80, 81), (82, 84), (84, 82)), + ((85, 85), (88, 82), (86, 82)), ], ), ] ) g11 = Polygon( ((97, 98), (99, 101), (102, 103), (101, 108)), - [((106, 107), (108, 109), (110, 111), (113, 108))], + [((99, 102), (100, 103), (101, 103), (100, 102))], ) gs = gpd.GeoSeries([g0, g1, g2, g3, g4, g5, g6, g7, g8, g9, g10, g11]) return gs @@ -72,8 +70,10 @@ def gpdf(gs): random_col = int_col np.random.shuffle(random_col) str_col = [str(x) for x in int_col] - key_col = np.repeat(np.arange(4), len(int_col) // 4) + key_col = np.repeat(np.arange(4), (len(int_col) // 4) + 1) + key_col = key_col[: len(int_col)] np.random.shuffle(key_col) + result = gpd.GeoDataFrame( { "geometry": gs, @@ -87,65 +87,6 @@ def gpdf(gs): return result -@pytest.fixture -def polys(): - return np.array( - ( - (35, 36), - (37, 38), - (39, 40), - (41, 42), - (35, 36), - (43, 44), - (45, 46), - (47, 48), - (43, 44), - (49, 50), - (51, 52), - (53, 54), - (49, 50), - (55, 56), - (57, 58), - (59, 60), - (55, 56), - (61, 62), - (63, 64), - (65, 66), - (61, 62), - (67, 68), - (69, 70), - (71, 72), - (67, 68), - (73, 74), - (75, 76), - (77, 78), - (73, 74), - (79, 80), - (81, 82), - (83, 84), - (79, 80), - (85, 86), - (87, 88), - (89, 90), - (85, 86), - (91, 92), - (93, 94), - (95, 96), - (91, 92), - (97, 98), - (99, 101), - (102, 103), - (101, 108), - (97, 98), - (106, 107), - (108, 109), - (110, 111), - (113, 108), - (106, 107), - ) - ) - - @pytest.fixture def gs_sorted(gs): result = pd.concat( diff --git a/python/cuspatial/cuspatial/tests/test_cudf_integration.py b/python/cuspatial/cuspatial/tests/test_cudf_integration.py index 41214e816..0e67733c0 100644 --- a/python/cuspatial/cuspatial/tests/test_cudf_integration.py +++ b/python/cuspatial/cuspatial/tests/test_cudf_integration.py @@ -2,17 +2,10 @@ import geopandas as gpd import numpy as np import pandas as pd -import pytest import cuspatial -reason = ( - "gs fixture contains invalid Polygons/MultiPolygons: " - "https://github.com/libgeos/geos/issues/1177" -) - -@pytest.mark.xfail(reason=reason) def test_sort_index_series(gs): gs.index = np.random.permutation(len(gs)) cugs = cuspatial.from_geopandas(gs) @@ -21,7 +14,6 @@ def test_sort_index_series(gs): gpd.testing.assert_geoseries_equal(got, expected) -@pytest.mark.xfail(reason=reason) def test_sort_index_dataframe(gpdf): gpdf.index = np.random.permutation(len(gpdf)) cugpdf = cuspatial.from_geopandas(gpdf) @@ -30,7 +22,6 @@ def test_sort_index_dataframe(gpdf): gpd.testing.assert_geodataframe_equal(got, expected) -@pytest.mark.xfail(reason=reason) def test_sort_values(gpdf): cugpdf = cuspatial.from_geopandas(gpdf) expected = gpdf.sort_values("random") diff --git a/python/cuspatial/cuspatial/tests/test_from_geopandas.py b/python/cuspatial/cuspatial/tests/test_from_geopandas.py index 1c33e216a..a02ab4a35 100644 --- a/python/cuspatial/cuspatial/tests/test_from_geopandas.py +++ b/python/cuspatial/cuspatial/tests/test_from_geopandas.py @@ -36,15 +36,8 @@ def test_dataframe_column_access(gs): def test_from_geoseries_complex(gs): cugs = cuspatial.from_geopandas(gs) - assert cugs.points.xy.sum() == 18 - assert cugs.lines.xy.sum() == 540 - assert cugs.multipoints.xy.sum() == 36 - assert cugs.polygons.xy.sum() == 7436 - assert cugs._column.polygons._column.base_children[0].sum() == 15 - assert ( - cugs._column.polygons._column.base_children[1].base_children[0].sum() - == 38 - ) + gs_roundtrip = cugs.to_geopandas() + gpd.testing.assert_geoseries_equal(gs_roundtrip, gs) def test_from_geopandas_point(): diff --git a/python/cuspatial/cuspatial/tests/test_geodataframe.py b/python/cuspatial/cuspatial/tests/test_geodataframe.py index f834b35e5..af8b7334e 100644 --- a/python/cuspatial/cuspatial/tests/test_geodataframe.py +++ b/python/cuspatial/cuspatial/tests/test_geodataframe.py @@ -19,6 +19,7 @@ import cudf import cuspatial +from cuspatial.testing.helpers import geometry_to_coords np.random.seed(0) @@ -114,7 +115,7 @@ def test_type_persistence(gpdf): assert type(cugpdf["geometry"]) is cuspatial.GeoSeries -def test_interleaved_point(gpdf, polys): +def test_interleaved_point(gpdf): cugpdf = cuspatial.from_geopandas(gpdf) cugs = cugpdf["geometry"] gs = gpdf["geometry"] @@ -128,7 +129,7 @@ def test_interleaved_point(gpdf, polys): ) -def test_interleaved_multipoint(gpdf, polys): +def test_interleaved_multipoint(gpdf): cugpdf = cuspatial.from_geopandas(gpdf) cugs = cugpdf["geometry"] gs = gpdf["geometry"] @@ -156,7 +157,7 @@ def test_interleaved_multipoint(gpdf, polys): ) -def test_interleaved_lines(gpdf, polys): +def test_interleaved_lines(gpdf): cugpdf = cuspatial.from_geopandas(gpdf) cugs = cugpdf["geometry"] cudf.testing.assert_series_equal( @@ -175,16 +176,19 @@ def test_interleaved_lines(gpdf, polys): ) -def test_interleaved_polygons(gpdf, polys): +def test_interleaved_polygons(gpdf): cugpdf = cuspatial.from_geopandas(gpdf) cugs = cugpdf["geometry"] + gs = gpdf["geometry"] + xy, x, y = geometry_to_coords(gs, (Polygon, MultiPolygon)) + cudf.testing.assert_series_equal( cudf.Series.from_arrow(cugs.polygons.x.to_arrow()), - cudf.Series(polys[:, 0], dtype="float64"), + cudf.Series(x, dtype="float64"), ) cudf.testing.assert_series_equal( cudf.Series.from_arrow(cugs.polygons.y.to_arrow()), - cudf.Series(polys[:, 1], dtype="float64"), + cudf.Series(y, dtype="float64"), ) diff --git a/python/cuspatial/cuspatial/tests/test_geoseries.py b/python/cuspatial/cuspatial/tests/test_geoseries.py index 56f61158e..5304d7ea6 100644 --- a/python/cuspatial/cuspatial/tests/test_geoseries.py +++ b/python/cuspatial/cuspatial/tests/test_geoseries.py @@ -23,6 +23,7 @@ from cudf.testing import assert_series_equal import cuspatial +from cuspatial.testing.helpers import geometry_to_coords np.random.seed(0) @@ -133,7 +134,7 @@ def assert_eq_geo(geo1, geo2): assert result.all() -def test_interleaved_point(gs, polys): +def test_interleaved_point(gs): cugs = cuspatial.from_geopandas(gs) pd.testing.assert_series_equal( cugs.points.x.to_pandas(), @@ -181,13 +182,16 @@ def test_interleaved_point(gs, polys): dtype="float64", ).reset_index(drop=True), ) + + xy, x, y = geometry_to_coords(gs, (MultiPolygon, Polygon)) + cudf.testing.assert_series_equal( cugs.polygons.x.reset_index(drop=True), - cudf.Series(polys[:, 0], dtype="float64").reset_index(drop=True), + cudf.Series(x, dtype="float64").reset_index(drop=True), ) cudf.testing.assert_series_equal( cugs.polygons.y.reset_index(drop=True), - cudf.Series(polys[:, 1], dtype="float64").reset_index(drop=True), + cudf.Series(y, dtype="float64").reset_index(drop=True), ) @@ -350,155 +354,29 @@ def test_size(gs, series_slice): assert len(gi) == len(cugs) -def test_geometry_point_slicing(gs): - cugs = cuspatial.from_geopandas(gs) - assert (cugs[:1].points.x == cudf.Series([-1])).all() - assert (cugs[:1].points.y == cudf.Series([0])).all() - assert (cugs[:1].points.xy == cudf.Series([-1, 0])).all() - assert (cugs[3:].points.x == cudf.Series([9])).all() - assert (cugs[3:].points.y == cudf.Series([10])).all() - assert (cugs[3:].points.xy == cudf.Series([9, 10])).all() - assert (cugs[0:4].points.x == cudf.Series([-1, 9])).all() - assert (cugs[0:4].points.y == cudf.Series([0, 10])).all() - assert (cugs[0:4].points.xy == cudf.Series([-1, 0, 9, 10])).all() - - -def test_geometry_multipoint_slicing(gs): - cugs = cuspatial.from_geopandas(gs) - assert (cugs[:2].multipoints.x == cudf.Series([1, 3])).all() - assert (cugs[:2].multipoints.y == cudf.Series([2, 4])).all() - assert (cugs[:2].multipoints.xy == cudf.Series([1, 2, 3, 4])).all() - assert (cugs[2:].multipoints.x == cudf.Series([5, 7])).all() - assert (cugs[2:].multipoints.y == cudf.Series([6, 8])).all() - assert (cugs[2:].multipoints.xy == cudf.Series([5, 6, 7, 8])).all() - assert (cugs[0:4].multipoints.x == cudf.Series([1, 3, 5, 7])).all() - assert (cugs[0:4].multipoints.y == cudf.Series([2, 4, 6, 8])).all() - assert ( - cugs[0:4].multipoints.xy == cudf.Series([1, 2, 3, 4, 5, 6, 7, 8]) - ).all() - - -def test_geometry_linestring_slicing(gs): - cugs = cuspatial.from_geopandas(gs) - assert (cugs[:5].lines.x == cudf.Series([11, 13])).all() - assert (cugs[:5].lines.y == cudf.Series([12, 14])).all() - assert (cugs[:5].lines.xy == cudf.Series([11, 12, 13, 14])).all() - assert (cugs[:6].lines.x == cudf.Series([11, 13, 15, 17, 19, 21])).all() - assert (cugs[:6].lines.y == cudf.Series([12, 14, 16, 18, 20, 22])).all() - assert ( - cugs[:6].lines.xy - == cudf.Series([11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]) - ).all() - assert (cugs[7:].lines.x == cudf.Series([31, 33])).all() - assert (cugs[7:].lines.y == cudf.Series([32, 34])).all() - assert (cugs[7:].lines.xy == cudf.Series([31, 32, 33, 34])).all() - assert (cugs[6:].lines.x == cudf.Series([23, 25, 27, 29, 31, 33])).all() - assert (cugs[6:].lines.y == cudf.Series([24, 26, 28, 30, 32, 34])).all() - assert ( - cugs[6:].lines.xy - == cudf.Series([23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34]) - ).all() - - -def test_geometry_polygon_slicing(gs): - cugs = cuspatial.from_geopandas(gs) - assert (cugs[:9].polygons.x == cudf.Series([35, 37, 39, 41, 35])).all() - assert (cugs[:9].polygons.y == cudf.Series([36, 38, 40, 42, 36])).all() - assert ( - cugs[:9].polygons.xy - == cudf.Series([35, 36, 37, 38, 39, 40, 41, 42, 35, 36]) - ).all() - assert ( - cugs[:10].polygons.x - == cudf.Series( - [ - 35, - 37, - 39, - 41, - 35, - 43, - 45, - 47, - 43, - 49, - 51, - 53, - 49, - 55, - 57, - 59, - 55, - 61, - 63, - 65, - 61, - ] - ) - ).all() - assert ( - cugs[:10].polygons.y - == cudf.Series( - [ - 36, - 38, - 40, - 42, - 36, - 44, - 46, - 48, - 44, - 50, - 52, - 54, - 50, - 56, - 58, - 60, - 56, - 62, - 64, - 66, - 62, - ] - ) - ).all() - assert ( - cugs[11:].polygons.x - == cudf.Series([97, 99, 102, 101, 97, 106, 108, 110, 113, 106]) - ).all() - assert ( - cugs[11:].polygons.y - == cudf.Series([98, 101, 103, 108, 98, 107, 109, 111, 108, 107]) - ).all() - assert ( - cugs[11:].polygons.xy - == cudf.Series( - [ - 97, - 98, - 99, - 101, - 102, - 103, - 101, - 108, - 97, - 98, - 106, - 107, - 108, - 109, - 110, - 111, - 113, - 108, - 106, - 107, - ] - ) - ).all() +@pytest.mark.parametrize( + "geom_access", + [ + # Tuples: accessor, types, slice + # slices here are meant to be supersets of the range in the gs fixture + # that contains the types of geometries being accessed + # Note that cuspatial.GeoSeries provides accessors for "multipoints", + # but not for "multilinestrings" or "multipolygons" + # (inconsistent interface) + ("points", Point, slice(0, 6)), + ("multipoints", MultiPoint, slice(2, 8)), + ("lines", (LineString, MultiLineString), slice(2, 10)), + ("polygons", (Polygon, MultiPolygon), slice(6, 12)), + ], +) +def test_geometry_access_slicing(gs, geom_access): + accessor, types, slice = geom_access + xy, x, y = geometry_to_coords(gs, types) + + cugs = cuspatial.from_geopandas(gs)[slice] + assert (getattr(cugs, accessor).x == cudf.Series(x)).all() + assert (getattr(cugs, accessor).y == cudf.Series(y)).all() + assert (getattr(cugs, accessor).xy == cudf.Series(xy)).all() def test_loc(gs): From d8dd061cf1134f1da250a353ba89a8b995a14c1d Mon Sep 17 00:00:00 2001 From: Bradley Dice Date: Wed, 16 Oct 2024 22:49:27 -0500 Subject: [PATCH 22/28] Update README architecture and platform support (#1476) A few minor documentation updates to align with the rest of RAPIDS. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Mark Harris (https://github.com/harrism) URL: https://github.com/rapidsai/cuspatial/pull/1476 --- README.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 4193373dd..3fbb9b86c 100644 --- a/README.md +++ b/README.md @@ -102,8 +102,7 @@ gitGraph ## Using cuSpatial **CUDA/GPU requirements** - CUDA 11.2+ with a [compatible, supported driver](https://docs.nvidia.com/datacenter/tesla/drivers/#cuda-drivers) -- Linux native: Pascal architecture or newer ([Compute Capability >=6.0](https://developer.nvidia.com/cuda-gpus)) -- WSL2: Volta architecture or newer ([Compute Capability >=7.0](https://developer.nvidia.com/cuda-gpus)) +- Volta architecture or newer ([Compute Capability >=7.0](https://developer.nvidia.com/cuda-gpus)) ### Quick start: Docker Use the [RAPIDS Release Selector](https://docs.rapids.ai/install#selector), selecting `Docker` as the installation method. All RAPIDS Docker images contain cuSpatial. @@ -113,7 +112,7 @@ An example command from the Release Selector: docker run --gpus all --pull always --rm -it \ --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \ -p 8888:8888 -p 8787:8787 -p 8786:8786 \ - nvcr.io/nvidia/rapidsai/notebooks:24.12-cuda11.8-py3.10 + nvcr.io/nvidia/rapidsai/notebooks:24.12-cuda11.8-py3.12 ``` ### Install with Conda @@ -121,7 +120,7 @@ docker run --gpus all --pull always --rm -it \ To install via conda: > **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10, 3.11, and 3.12. -cuSpatial can be installed with conda (miniconda, or the full Anaconda distribution) from the rapidsai channel: +cuSpatial can be installed with conda from the rapidsai channel: ```shell conda install -c rapidsai -c conda-forge -c nvidia \ From 86ef6753d719a6717513ea4d2e8f88766ce7c59a Mon Sep 17 00:00:00 2001 From: James Lamb Date: Tue, 22 Oct 2024 16:57:20 -0500 Subject: [PATCH 23/28] build wheels without build isolation (#1473) Contributes to https://github.com/rapidsai/build-planning/issues/108 Contributes to https://github.com/rapidsai/build-planning/issues/111 Proposes building `libcuspatial` wheels with `--no-build-isolation`, to improve the rate of `sccache` cache hits and therefore reduce CI times. Also proposes printing `sccache` stats to CI logs after wheel and conda builds. ## Notes for Reviewers # Authors: - James Lamb (https://github.com/jameslamb) - Bradley Dice (https://github.com/bdice) Approvers: - Mike Sarahan (https://github.com/msarahan) - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Vyas Ramasubramani (https://github.com/vyasr) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cuspatial/pull/1473 --- .pre-commit-config.yaml | 2 +- ci/build_cpp.sh | 4 ++++ ci/build_python.sh | 8 ++++++++ ci/build_wheel.sh | 10 +++++++++- ci/build_wheel_libcuspatial.sh | 24 +++++++++++++++++++++++- 5 files changed, 45 insertions(+), 3 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 3949b4c3d..bd10e977a 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -48,7 +48,7 @@ repos: - id: verify-copyright - id: verify-alpha-spec - repo: https://github.com/rapidsai/dependency-file-generator - rev: v1.13.11 + rev: v1.16.0 hooks: - id: rapids-dependency-file-generator args: ["--clean"] diff --git a/ci/build_cpp.sh b/ci/build_cpp.sh index c23dac174..effab796d 100755 --- a/ci/build_cpp.sh +++ b/ci/build_cpp.sh @@ -15,7 +15,11 @@ rapids-print-env rapids-logger "Begin cpp build" +sccache --zero-stats + RAPIDS_PACKAGE_VERSION=$(rapids-generate-version) rapids-conda-retry mambabuild \ conda/recipes/libcuspatial +sccache --show-adv-stats + rapids-upload-conda-to-s3 cpp diff --git a/ci/build_python.sh b/ci/build_python.sh index fc1066fc7..e0ff96639 100755 --- a/ci/build_python.sh +++ b/ci/build_python.sh @@ -21,6 +21,8 @@ CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) rapids-logger "Begin py build cuSpatial" +sccache --zero-stats + # TODO: Remove `--no-test` flag once importing on a CPU # node works correctly RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ @@ -28,8 +30,12 @@ RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ --channel "${CPP_CHANNEL}" \ conda/recipes/cuspatial +sccache --show-adv-stats + rapids-logger "Begin py build cuProj" +sccache --zero-stats + # TODO: Remove `--no-test` flag once importing on a CPU # node works correctly RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ @@ -37,4 +43,6 @@ RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \ --channel "${CPP_CHANNEL}" \ conda/recipes/cuproj +sccache --show-adv-stats + rapids-upload-conda-to-s3 python diff --git a/ci/build_wheel.sh b/ci/build_wheel.sh index 75328d272..363d10d42 100755 --- a/ci/build_wheel.sh +++ b/ci/build_wheel.sh @@ -35,7 +35,15 @@ RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})" cd "${package_dir}" -python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check +rapids-logger "Building '${package_name}' wheel" +python -m pip wheel \ + -w dist \ + -v \ + --no-deps \ + --disable-pip-version-check \ + . + +sccache --show-adv-stats mkdir -p final_dist python -m auditwheel repair \ diff --git a/ci/build_wheel_libcuspatial.sh b/ci/build_wheel_libcuspatial.sh index 742a58caa..ed3822207 100755 --- a/ci/build_wheel_libcuspatial.sh +++ b/ci/build_wheel_libcuspatial.sh @@ -3,4 +3,26 @@ set -euo pipefail -ci/build_wheel.sh libcuspatial python/libcuspatial cpp +package_name="libcuspatial" + +rapids-logger "Generating build requirements" +matrix_selectors="cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true" + +rapids-dependency-file-generator \ + --output requirements \ + --file-key "py_build_${package_name}" \ + --file-key "py_rapids_build_${package_name}" \ + --matrix "${matrix_selectors}" \ +| tee /tmp/requirements-build.txt + +rapids-logger "Installing build requirements" +python -m pip install \ + -v \ + --prefer-binary \ + -r /tmp/requirements-build.txt + +# build with '--no-build-isolation', for better sccache hit rate +# 0 really means "add --no-build-isolation" (ref: https://github.com/pypa/pip/issues/5735) +export PIP_NO_BUILD_ISOLATION=0 + +ci/build_wheel.sh "${package_name}" python/libcuspatial cpp From 0a2685d8d9e64341a2f72ad1ee3a9e6d428a4ba7 Mon Sep 17 00:00:00 2001 From: Kyle Edwards Date: Wed, 23 Oct 2024 10:16:48 -0400 Subject: [PATCH 24/28] Prune workflows based on changed files (#1479) Contributes to https://github.com/rapidsai/build-planning/issues/94 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - James Lamb (https://github.com/jameslamb) URL: https://github.com/rapidsai/cuspatial/pull/1479 --- .github/workflows/pr.yaml | 49 +++++++++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index 5f1278333..3f6a68b8a 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -12,6 +12,7 @@ concurrency: jobs: pr-builder: needs: + - changed-files - checks - conda-cpp-build - conda-cpp-checks @@ -28,6 +29,39 @@ jobs: - devcontainer secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@branch-24.12 + if: always() + with: + needs: ${{ toJSON(needs) }} + changed-files: + secrets: inherit + uses: rapidsai/shared-workflows/.github/workflows/changed-files.yaml@branch-24.12 + with: + files_yaml: | + test_cpp: + - '**' + - '!.devcontainer/**' + - '!.pre-commit-config.yaml' + - '!CONTRIBUTING.md' + - '!README.md' + - '!docs/**' + - '!java/**' + - '!notebooks/**' + - '!python/**' + test_notebooks: + - '**' + - '!.devcontainer/**' + - '!.pre-commit-config.yaml' + - '!CONTRIBUTING.md' + - '!README.md' + test_python: + - '**' + - '!.devcontainer/**' + - '!.pre-commit-config.yaml' + - '!CONTRIBUTING.md' + - '!README.md' + - '!docs/**' + - '!java/**' + - '!notebooks/**' checks: secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@branch-24.12 @@ -47,9 +81,10 @@ jobs: build_type: pull-request enable_check_symbols: true conda-cpp-tests: - needs: conda-cpp-build + needs: [conda-cpp-build, changed-files] secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.12 + if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_cpp with: build_type: pull-request conda-python-build: @@ -59,15 +94,17 @@ jobs: with: build_type: pull-request conda-python-tests: - needs: conda-python-build + needs: [conda-python-build, changed-files] secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.12 + if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_cpp with: build_type: pull-request conda-notebook-tests: - needs: conda-python-build + needs: [conda-python-build, changed-files] secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.12 + if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_notebooks with: build_type: pull-request node_type: "gpu-v100-latest-1" @@ -101,9 +138,10 @@ jobs: build_type: pull-request script: ci/build_wheel_cuspatial.sh wheel-tests-cuspatial: - needs: wheel-build-cuspatial + needs: [wheel-build-cuspatial, changed-files] secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.12 + if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python with: build_type: pull-request script: ci/test_wheel_cuspatial.sh @@ -115,9 +153,10 @@ jobs: build_type: pull-request script: ci/build_wheel_cuproj.sh wheel-tests-cuproj: - needs: [wheel-build-cuspatial, wheel-build-cuproj] + needs: [wheel-build-cuspatial, wheel-build-cuproj, changed-files] secrets: inherit uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.12 + if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python with: build_type: pull-request script: ci/test_wheel_cuproj.sh From 724d170a2105441a3533b5eaf9ee82ddcfc49be0 Mon Sep 17 00:00:00 2001 From: Jordan Jacobelli Date: Mon, 28 Oct 2024 14:01:50 +0100 Subject: [PATCH 25/28] devcontainer: replace `VAULT_HOST` with `AWS_ROLE_ARN` (#1478) This PR is replacing the `VAULT_HOST` variable with `AWS_ROLE_ARN`. This is required to use the new token service to get AWS credentials. Authors: - Jordan Jacobelli (https://github.com/jjacobelli) Approvers: - Paul Taylor (https://github.com/trxcllnt) - Bradley Dice (https://github.com/bdice) - Jake Awe (https://github.com/AyodeAwe) URL: https://github.com/rapidsai/cuspatial/pull/1478 --- .devcontainer/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile index 974aea1ee..95763953e 100644 --- a/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -31,5 +31,5 @@ ENV PYTHONDONTWRITEBYTECODE="1" ENV SCCACHE_REGION="us-east-2" ENV SCCACHE_BUCKET="rapids-sccache-devs" -ENV VAULT_HOST="https://vault.ops.k8s.rapids.ai" +ENV AWS_ROLE_ARN="arn:aws:iam::279114543810:role/nv-gha-token-sccache-devs" ENV HISTFILE="/home/coder/.cache/._bash_history" From 98fda4cb5251492569630f1e3ec61b3c1a1aff15 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Wed, 13 Nov 2024 21:32:11 -0600 Subject: [PATCH 26/28] enforce wheel size limits, README formatting in CI (#1482) Contributes to https://github.com/rapidsai/build-planning/issues/110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Mark Harris (https://github.com/harrism) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cuspatial/pull/1482 --- ci/build_wheel_cuproj.sh | 5 ++++- ci/build_wheel_cuspatial.sh | 5 ++++- ci/build_wheel_libcuspatial.sh | 4 +++- ci/validate_wheel.sh | 21 +++++++++++++++++++++ python/cuproj/pyproject.toml | 8 ++++++++ python/cuspatial/pyproject.toml | 8 ++++++++ python/libcuspatial/pyproject.toml | 8 ++++++++ 7 files changed, 56 insertions(+), 3 deletions(-) create mode 100755 ci/validate_wheel.sh diff --git a/ci/build_wheel_cuproj.sh b/ci/build_wheel_cuproj.sh index 6001e44e9..4b29fbb46 100755 --- a/ci/build_wheel_cuproj.sh +++ b/ci/build_wheel_cuproj.sh @@ -3,4 +3,7 @@ set -euo pipefail -ci/build_wheel.sh cuproj python/cuproj python +package_dir="python/cuproj" + +ci/build_wheel.sh cuproj ${package_dir} python +ci/validate_wheel.sh ${package_dir} final_dist diff --git a/ci/build_wheel_cuspatial.sh b/ci/build_wheel_cuspatial.sh index e581a3e76..780ac0710 100755 --- a/ci/build_wheel_cuspatial.sh +++ b/ci/build_wheel_cuspatial.sh @@ -3,6 +3,8 @@ set -euo pipefail +package_dir="python/cuspatial" + RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})" # Downloads libcuspatial wheel from this current build, @@ -14,4 +16,5 @@ RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-whe echo "libcuspatial-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo /tmp/libcuspatial_dist/libcuspatial_*.whl)" > /tmp/constraints.txt export PIP_CONSTRAINT="/tmp/constraints.txt" -ci/build_wheel.sh cuspatial python/cuspatial python +ci/build_wheel.sh cuspatial ${package_dir} python +ci/validate_wheel.sh ${package_dir} final_dist diff --git a/ci/build_wheel_libcuspatial.sh b/ci/build_wheel_libcuspatial.sh index ed3822207..b6582d87f 100755 --- a/ci/build_wheel_libcuspatial.sh +++ b/ci/build_wheel_libcuspatial.sh @@ -3,6 +3,7 @@ set -euo pipefail +package_dir="python/libcuspatial" package_name="libcuspatial" rapids-logger "Generating build requirements" @@ -25,4 +26,5 @@ python -m pip install \ # 0 really means "add --no-build-isolation" (ref: https://github.com/pypa/pip/issues/5735) export PIP_NO_BUILD_ISOLATION=0 -ci/build_wheel.sh "${package_name}" python/libcuspatial cpp +ci/build_wheel.sh "${package_name}" ${package_dir} cpp +ci/validate_wheel.sh ${package_dir} final_dist diff --git a/ci/validate_wheel.sh b/ci/validate_wheel.sh new file mode 100755 index 000000000..5910a5c59 --- /dev/null +++ b/ci/validate_wheel.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# Copyright (c) 2024, NVIDIA CORPORATION. + +set -euo pipefail + +package_dir=$1 +wheel_dir_relative_path=$2 + +cd "${package_dir}" + +rapids-logger "validate packages with 'pydistcheck'" + +pydistcheck \ + --inspect \ + "$(echo ${wheel_dir_relative_path}/*.whl)" + +rapids-logger "validate packages with 'twine'" + +twine check \ + --strict \ + "$(echo ${wheel_dir_relative_path}/*.whl)" diff --git a/python/cuproj/pyproject.toml b/python/cuproj/pyproject.toml index 35a95c728..c8049eadb 100644 --- a/python/cuproj/pyproject.toml +++ b/python/cuproj/pyproject.toml @@ -126,3 +126,11 @@ requires = [ "ninja", "wheel", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. + +[tool.pydistcheck] +select = [ + "distro-too-large-compressed", +] + +# PyPI limit is 100 MiB, fail CI before we get too close to that +max_allowed_size_compressed = '75M' diff --git a/python/cuspatial/pyproject.toml b/python/cuspatial/pyproject.toml index c497434c4..662648337 100644 --- a/python/cuspatial/pyproject.toml +++ b/python/cuspatial/pyproject.toml @@ -145,3 +145,11 @@ requires = [ "rmm==24.12.*,>=0.0.0a0", "wheel", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. + +[tool.pydistcheck] +select = [ + "distro-too-large-compressed", +] + +# PyPI limit is 100 MiB, fail CI before we get too close to that +max_allowed_size_compressed = '75M' diff --git a/python/libcuspatial/pyproject.toml b/python/libcuspatial/pyproject.toml index 1de8df8c1..725d2d882 100644 --- a/python/libcuspatial/pyproject.toml +++ b/python/libcuspatial/pyproject.toml @@ -75,3 +75,11 @@ requires = [ "ninja", "wheel", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. + +[tool.pydistcheck] +select = [ + "distro-too-large-compressed", +] + +# PyPI limit is 100 MiB, fail CI before we get too close to that +max_allowed_size_compressed = '75M' From 20abaaa5f5efa9a476fc875b5de5e94d72bfe394 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Thu, 14 Nov 2024 16:02:17 -0600 Subject: [PATCH 27/28] prefer wheel-provided libcuspatial.so in load_library(), use RTLD_LOCAL (#1483) Contributes to https://github.com/rapidsai/build-planning/issues/118 Modifies `libcuspatial.load_library()` in the following ways: * prefer wheel-provided `libcuspatial.so` to system installation * expose environment variable `RAPIDS_LIBCUSPATIAL_PREFER_SYSTEM_LIBRARY` for switching that preference * load `libcuspatial.so` with `RTLD_LOCAL`, to prevent adding symbols to the global namespace ([dlopen docs](https://linux.die.net/man/3/dlopen)) ## Notes for Reviewers ### How I tested this See "How I tested this" in https://github.com/rapidsai/cudf/pull/17316 Also opened this PR originally pulling in built packages from https://github.com/rapidsai/cudf/pull/17316 # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cuspatial/pull/1483 --- python/libcuspatial/libcuspatial/load.py | 75 ++++++++++++++++-------- 1 file changed, 51 insertions(+), 24 deletions(-) diff --git a/python/libcuspatial/libcuspatial/load.py b/python/libcuspatial/libcuspatial/load.py index 7842de9a0..78a8590ef 100644 --- a/python/libcuspatial/libcuspatial/load.py +++ b/python/libcuspatial/libcuspatial/load.py @@ -16,8 +16,37 @@ import ctypes import os +# Loading with RTLD_LOCAL adds the library itself to the loader's +# loaded library cache without loading any symbols into the global +# namespace. This allows libraries that express a dependency on +# this library to be loaded later and successfully satisfy this dependency +# without polluting the global symbol table with symbols from +# libcuspatial that could conflict with symbols from other DSOs. +PREFERRED_LOAD_FLAG = ctypes.RTLD_LOCAL + + +def _load_system_installation(soname: str): + """Try to dlopen() the library indicated by ``soname`` + + Raises ``OSError`` if library cannot be loaded. + """ + return ctypes.CDLL(soname, PREFERRED_LOAD_FLAG) + + +def _load_wheel_installation(soname: str): + """Try to dlopen() the library indicated by ``soname`` + + Returns ``None`` if the library cannot be loaded. + """ + if os.path.isfile( + lib := os.path.join(os.path.dirname(__file__), "lib64", soname) + ): + return ctypes.CDLL(lib, PREFERRED_LOAD_FLAG) + return None + def load_library(): + """Dynamically load libcuspatial.so and its dependencies""" try: # libcudf must be loaded before libcuspatial because libcuspatial # references its symbols @@ -33,32 +62,30 @@ def load_library(): # the loader can find it. pass - # Dynamically load libcuspatial.so. Prefer a system library if one is - # present to avoid clobbering symbols that other packages might expect, - # but if no other library is present use the one in the wheel. + prefer_system_installation = ( + os.getenv("RAPIDS_LIBCUSPATIAL_PREFER_SYSTEM_LIBRARY", "false").lower() + != "false" + ) + + soname = "libcuspatial.so" libcuspatial_lib = None - try: - libcuspatial_lib = ctypes.CDLL("libcuspatial.so", ctypes.RTLD_GLOBAL) - except OSError: - # If neither of these directories contain the library, we assume we are - # in an environment where the C++ library is already installed - # somewhere else and the CMake build of the libcuspatial Python package - # was a no-op. - # - # Note that this approach won't work for real editable installs of the - # libcuspatial package. scikit-build-core has limited support for - # importlib.resources so there isn't a clean way to support that case - # yet. - for lib_dir in ("lib", "lib64"): - if os.path.isfile( - lib := os.path.join( - os.path.dirname(__file__), lib_dir, "libcuspatial.so" - ) - ): - libcuspatial_lib = ctypes.CDLL(lib, ctypes.RTLD_GLOBAL) - break + if prefer_system_installation: + # Prefer a system library if one is present to + # avoid clobbering symbols that other packages might expect, but if no + # other library is present use the one in the wheel. + try: + libcuspatial_lib = _load_system_installation(soname) + except OSError: + libcuspatial_lib = _load_wheel_installation(soname) + else: + # Prefer the libraries bundled in this package. If they aren't found + # (which might be the case in builds where the library was prebuilt + # before packaging the wheel), look for a system installation. + libcuspatial_lib = _load_wheel_installation(soname) + if libcuspatial_lib is None: + libcuspatial_lib = _load_system_installation(soname) # The caller almost never needs to do anything with this library, but no # harm in offering the option since this object at least provides a handle - # to inspect where the library was loaded from. + # to inspect where the libcuspatial was loaded from. return libcuspatial_lib From 0a794562fb6385e9dd97bae5c9090f1a41327319 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Fri, 15 Nov 2024 15:29:19 -0600 Subject: [PATCH 28/28] fix library-loading issues in editable installs (#1484) Contributes to https://github.com/rapidsai/build-planning/issues/118 The pattern introduced in #1483 breaks editable installs in devcontainers. In that type of build, `libcuspatial.so` is built outside of the wheel but **not installed**, so it can't be found by `ld`. Extension modules in `cuspatial` are able to find it via RPATHs instead. This proposes: * try-catching the entire library-loading attempt, to silently do nothing in cases like that * ~adding an import of the `cuspatial` Python library in the `devcontainers` CI job, as a smoke test to catch issues like this in the future~ *(edit: removed those, [`devcontainer` builds run on CPU nodes](https://github.com/rapidsai/shared-workflows/blob/4e84062f333ce5649bc65029d3979569e2d0a045/.github/workflows/build-in-devcontainer.yaml#L19))* ## Notes for Reviewers ### How I tested this Tested this approach on https://github.com/rapidsai/kvikio/pull/553 # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cuspatial/pull/1484 --- python/libcuspatial/libcuspatial/load.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/python/libcuspatial/libcuspatial/load.py b/python/libcuspatial/libcuspatial/load.py index 78a8590ef..977586669 100644 --- a/python/libcuspatial/libcuspatial/load.py +++ b/python/libcuspatial/libcuspatial/load.py @@ -81,11 +81,17 @@ def load_library(): # Prefer the libraries bundled in this package. If they aren't found # (which might be the case in builds where the library was prebuilt # before packaging the wheel), look for a system installation. - libcuspatial_lib = _load_wheel_installation(soname) - if libcuspatial_lib is None: - libcuspatial_lib = _load_system_installation(soname) + try: + libcuspatial_lib = _load_wheel_installation(soname) + if libcuspatial_lib is None: + libcuspatial_lib = _load_system_installation(soname) + except OSError: + # If none of the searches above succeed, just silently return None + # and rely on other mechanisms (like RPATHs on other DSOs) to + # help the loader find the library. + pass # The caller almost never needs to do anything with this library, but no # harm in offering the option since this object at least provides a handle - # to inspect where the libcuspatial was loaded from. + # to inspect where libcuspatial was loaded from. return libcuspatial_lib