Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Rework
read_csv
IO to avoid reading whole input with a single `host…
…_read` (#16826) Issue #13797 The CSV reader ingests all input data with single call to host_read. This is a problem for a few reasons: 1. With `cudaHostRegister` we cannot reliably copy from the mapped region to the GPU without issues with mixing registered and unregistered areas. The reader can't know the datasource implementation details needed to avoid this issue. 2. Currently the reader performs the H2D copies manually, so there's no multi-threaded or pinned memory optimizations. Using `device_read` has the potential to outperform manual copies. This PR changes `read_csv` IO to perform small `host_read`s to get the data like BOM and first row. Most of the data is then read in chunks using `device_read` calls. We can further remove host_reads by moving some of the host processing to the GPU. No significant changes in performance. We are likely to get performance improvements from future changes like increasing the kvikIO thread pool size. Authors: - Vukasin Milovanovic (https://github.com/vuule) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - MithunR (https://github.com/mythrocks) - Karthikeyan (https://github.com/karthikeyann) URL: #16826
- Loading branch information