From 6dd5689d123bdb68be849fd15ff4cb6348535c72 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Tue, 10 Sep 2024 13:13:32 -0500 Subject: [PATCH] use libkvikio wheels in wheel builds (#16778) Follow-up to #15483. Contributes to https://github.com/rapidsai/build-planning/issues/33. Adds a build-time dependency on `libkvikio` wheels for `libcudf` wheels (per https://github.com/rapidsai/cudf/pull/15483#discussion_r1583969365). With this change, CPM is no longer used to download and install the kvikio headers. Before: ```text -- Found cuFile: /usr/local/cuda/lib64/libcufile.so -- CPM: Adding package KvikIO@24.10 (branch-24.10) ``` ([recent build link from branch-24.10](https://github.com/rapidsai/cudf/actions/runs/10780576194/job/29896649202#step:9:7673)) After: ```text -- KvikIO: Found cuFile Batch API: TRUE -- KvikIO: Found cuFile Stream API: TRUE -- CPM: Using local package KvikIO@24.10.0 ``` ([build link from this PR](https://github.com/rapidsai/cudf/actions/runs/10780504202/job/29896555443?pr=16778#step:9:7754)) ## Notes for Reviewers ### This removes kvikio headers/CMake files from libcudf wheels Cuts around 0.8 MB (23 files) out of `libcudf` wheels. As of this PR, these would no longer be vendored in `libcudf` wheels: ```text 0 09-08-2024 06:17 libcudf/include/kvikio/ 0 09-08-2024 06:17 libcudf/include/kvikio/shim/ 6356 09-08-2024 06:17 libcudf/include/kvikio/batch.hpp 3812 09-08-2024 06:17 libcudf/include/kvikio/buffer.hpp 10499 09-08-2024 06:17 libcudf/include/kvikio/utils.hpp 1399 09-08-2024 06:17 libcudf/include/kvikio/cufile_config.hpp 33385 09-08-2024 06:17 libcudf/include/kvikio/file_handle.hpp 7299 09-08-2024 06:17 libcudf/include/kvikio/driver.hpp 9678 09-08-2024 06:17 libcudf/include/kvikio/defaults.hpp 5352 09-08-2024 06:17 libcudf/include/kvikio/stream.hpp 6002 09-08-2024 06:17 libcudf/include/kvikio/error.hpp 4501 09-08-2024 06:17 libcudf/include/kvikio/bounce_buffer.hpp 3197 09-08-2024 06:17 libcudf/include/kvikio/parallel_operation.hpp 9864 09-08-2024 06:17 libcudf/include/kvikio/posix_io.hpp 717 09-08-2024 06:17 libcudf/include/kvikio/version_config.hpp 4529 09-08-2024 06:17 libcudf/include/kvikio/shim/cuda.hpp 3331 09-08-2024 06:17 libcudf/include/kvikio/shim/utils.hpp 4055 09-08-2024 06:17 libcudf/include/kvikio/shim/cufile_h_wrapper.hpp 2242 09-08-2024 06:17 libcudf/include/kvikio/shim/cuda_h_wrapper.hpp 7510 09-08-2024 06:17 libcudf/include/kvikio/shim/cufile.hpp 0 09-08-2024 06:17 libcudf/lib64/cmake/kvikio/ 5031 09-08-2024 06:17 libcudf/lib64/cmake/kvikio/kvikio-targets.cmake 3681 09-08-2024 06:17 libcudf/lib64/cmake/kvikio/kvikio-config-version.cmake 6915 09-08-2024 06:17 libcudf/lib64/cmake/kvikio/kvikio-config.cmake 1529 09-08-2024 06:17 libcudf/lib64/cmake/kvikio/kvikio-dependencies.cmake 3851 09-08-2024 06:17 libcudf/lib64/cmake/kvikio/FindcuFile.cmake ``` This is safe because kvikio is a PRIVATE dependency of `libcudf`. https://github.com/rapidsai/cudf/blob/150f1b10ed9c702d5283216b746df685e1708716/cpp/CMakeLists.txt#L796-L802 # Authors: - James Lamb (https://github.com/jameslamb) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/16778 --- dependencies.yaml | 29 +++++++++++++++++++++++++++-- python/libcudf/pyproject.toml | 1 + 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/dependencies.yaml b/dependencies.yaml index 32c1d7a0845..483335c02ff 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -13,6 +13,7 @@ files: - cuda - cuda_version - depends_on_cupy + - depends_on_libkvikio - depends_on_librmm - depends_on_rmm - develop @@ -135,6 +136,7 @@ files: includes: - build_base - build_cpp + - depends_on_libkvikio - depends_on_librmm py_build_pylibcudf: output: pyproject @@ -349,8 +351,6 @@ dependencies: - output_types: conda packages: - fmt>=10.1.1,<11 - - librmm==24.10.*,>=0.0.0a0 - - libkvikio==24.10.*,>=0.0.0a0 - flatbuffers==24.3.25 - librdkafka>=2.5.0,<2.6.0a0 # Align nvcomp version with rapids-cmake @@ -889,6 +889,31 @@ dependencies: packages: &cupy_packages_cu11 - cupy-cuda11x>=12.0.0 - {matrix: null, packages: *cupy_packages_cu11} + depends_on_libkvikio: + common: + - output_types: conda + packages: + - &libkvikio_unsuffixed libkvikio==24.10.*,>=0.0.0a0 + - output_types: requirements + packages: + - --extra-index-url=https://pypi.nvidia.com + - --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple + specific: + - output_types: [requirements, pyproject] + matrices: + - matrix: + cuda: "12.*" + cuda_suffixed: "true" + packages: + - libkvikio-cu12==24.10.*,>=0.0.0a0 + - matrix: + cuda: "11.*" + cuda_suffixed: "true" + packages: + - libkvikio-cu11==24.10.*,>=0.0.0a0 + - matrix: + packages: + - *libkvikio_unsuffixed depends_on_librmm: common: - output_types: conda diff --git a/python/libcudf/pyproject.toml b/python/libcudf/pyproject.toml index 5f4b9957fd0..2c98b97eddf 100644 --- a/python/libcudf/pyproject.toml +++ b/python/libcudf/pyproject.toml @@ -66,6 +66,7 @@ dependencies-file = "../../dependencies.yaml" matrix-entry = "cuda_suffixed=true" requires = [ "cmake>=3.26.4,!=3.30.0", + "libkvikio==24.10.*,>=0.0.0a0", "librmm==24.10.*,>=0.0.0a0", "ninja", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.