Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use libkvikio wheels in wheel builds #16778

Merged
merged 3 commits into from
Sep 10, 2024
Merged

use libkvikio wheels in wheel builds #16778

merged 3 commits into from
Sep 10, 2024

Conversation

jameslamb
Copy link
Member

@jameslamb jameslamb commented Sep 9, 2024 •
edited
Loading

Description

Follow-up to #15483.
Contributes to rapidsai/build-planning#33.

Adds a build-time dependency on libkvikio wheels for libcudf wheels (per #15483 (comment)).

With this change, CPM is no longer used to download and install the kvikio headers.

Before:

  -- Found cuFile: /usr/local/cuda/lib64/libcufile.so
  -- CPM: Adding package [email protected] (branch-24.10)

(recent build link from branch-24.10)

After:

  -- KvikIO: Found cuFile Batch API: TRUE
  -- KvikIO: Found cuFile Stream API: TRUE
  -- CPM: Using local package [email protected]

(build link from this PR)

Notes for Reviewers

This removes kvikio headers/CMake files from libcudf wheels

Cuts around 0.8 MB (23 files) out of libcudf wheels.

As of this PR, these would no longer be vendored in libcudf wheels:

    0  09-08-2024 06:17   libcudf/include/kvikio/
    0  09-08-2024 06:17   libcudf/include/kvikio/shim/
 6356  09-08-2024 06:17   libcudf/include/kvikio/batch.hpp
 3812  09-08-2024 06:17   libcudf/include/kvikio/buffer.hpp
10499  09-08-2024 06:17   libcudf/include/kvikio/utils.hpp
 1399  09-08-2024 06:17   libcudf/include/kvikio/cufile_config.hpp
33385  09-08-2024 06:17   libcudf/include/kvikio/file_handle.hpp
 7299  09-08-2024 06:17   libcudf/include/kvikio/driver.hpp
 9678  09-08-2024 06:17   libcudf/include/kvikio/defaults.hpp
 5352  09-08-2024 06:17   libcudf/include/kvikio/stream.hpp
 6002  09-08-2024 06:17   libcudf/include/kvikio/error.hpp
 4501  09-08-2024 06:17   libcudf/include/kvikio/bounce_buffer.hpp
 3197  09-08-2024 06:17   libcudf/include/kvikio/parallel_operation.hpp
 9864  09-08-2024 06:17   libcudf/include/kvikio/posix_io.hpp
  717  09-08-2024 06:17   libcudf/include/kvikio/version_config.hpp
 4529  09-08-2024 06:17   libcudf/include/kvikio/shim/cuda.hpp
 3331  09-08-2024 06:17   libcudf/include/kvikio/shim/utils.hpp
 4055  09-08-2024 06:17   libcudf/include/kvikio/shim/cufile_h_wrapper.hpp
 2242  09-08-2024 06:17   libcudf/include/kvikio/shim/cuda_h_wrapper.hpp
 7510  09-08-2024 06:17   libcudf/include/kvikio/shim/cufile.hpp
    0  09-08-2024 06:17   libcudf/lib64/cmake/kvikio/
 5031  09-08-2024 06:17   libcudf/lib64/cmake/kvikio/kvikio-targets.cmake
 3681  09-08-2024 06:17   libcudf/lib64/cmake/kvikio/kvikio-config-version.cmake
 6915  09-08-2024 06:17   libcudf/lib64/cmake/kvikio/kvikio-config.cmake
 1529  09-08-2024 06:17   libcudf/lib64/cmake/kvikio/kvikio-dependencies.cmake
 3851  09-08-2024 06:17   libcudf/lib64/cmake/kvikio/FindcuFile.cmake

This is safe because kvikio is a PRIVATE dependency of libcudf.

cudf/cpp/CMakeLists.txt

Lines 796 to 802 in 150f1b1

# Specify the target module library dependencies
target_link_libraries(
cudf
PUBLIC CCCL::CCCL rmm::rmm $<BUILD_LOCAL_INTERFACE:BS::thread_pool>
PRIVATE $<BUILD_LOCAL_INTERFACE:nvtx3::nvtx3-cpp> cuco::cuco ZLIB::ZLIB nvcomp::nvcomp
kvikio::kvikio $<TARGET_NAME_IF_EXISTS:cuFile_interface> nanoarrow
)

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@jameslamb jameslamb added 2 - In Progress Currently a work in progress improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Sep 9, 2024
@jameslamb jameslamb changed the title WIP: use libkvikio in wheel builds WIP: use libkvikio wheels in wheel builds Sep 9, 2024
@github-actions github-actions bot added the Python Affects Python cuDF API. label Sep 9, 2024
@jameslamb jameslamb mentioned this pull request Sep 9, 2024
Merged
3 tasks
@jameslamb jameslamb changed the title WIP: use libkvikio wheels in wheel builds use libkvikio wheels in wheel builds Sep 9, 2024
@jameslamb jameslamb marked this pull request as ready for review September 9, 2024 22:11
@jameslamb jameslamb requested a review from a team as a code owner September 9, 2024 22:11
@jameslamb jameslamb requested review from bdice and msarahan September 9, 2024 22:11
@jameslamb jameslamb added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Sep 10, 2024
bdice
bdice requested changes Sep 10, 2024
View reviewed changes
Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caught something on further inspection. I'll push a fix.

dependencies.yaml Outdated Show resolved Hide resolved
@bdice
Copy link
Contributor

bdice commented Sep 10, 2024

/merge

@rapids-bot rapids-bot bot merged commit 6dd5689 into rapidsai:branch-24.10 Sep 10, 2024
97 checks passed
@jameslamb jameslamb deleted the use-libkvikio-wheels branch September 11, 2024 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bdice bdice bdice approved these changes

@msarahan msarahan Awaiting requested review from msarahan

Assignees
No one assigned
Labels
3 - Ready for Review Ready for review by team improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jameslamb @bdice