Fixes #16625 This PR fixes a slow implementation of the centroid merging step during the tdigest merge aggregation. Previously it was doing a linear march over the individual tdigests per group and merging them one by one. This led to terrible performance for large numbers of groups. In principle though, all this really was doing was a segmented sort of centroid values. So that's what this PR changes it to. Speedup for 1,000,000 input tidests with 1,000,000 individual groups is ~1000x, ``` Old --------------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations --------------------------------------------------------------------------------------------------------------- TDigest/many_tiny_groups/1000000/1/1/10000/iterations:8/manual_time 7473 ms 7472 ms 8 TDigest/many_tiny_groups2/1000000/1/1/1000/iterations:8/manual_time 7433 ms 7431 ms 8 ``` ``` New --------------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations --------------------------------------------------------------------------------------------------------------- TDigest/many_tiny_groups/1000000/1/1/10000/iterations:8/manual_time 6.72 ms 6.79 ms 8 TDigest/many_tiny_groups2/1000000/1/1/1000/iterations:8/manual_time 1.24 ms 1.32 ms 8 ``` Authors: - https://github.com/nvdbaranec - Muhammad Haseeb (https://github.com/mhaseeb123) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Muhammad Haseeb (https://github.com/mhaseeb123) - Nghia Truong (https://github.com/ttnghia) - Mike Wilson (https://github.com/hyperbolic2346) URL: #16780

Commits

Commits on Sep 25, 2024

Pin polars for 24.10 and update polars test suite xfail list (#16886)

Build cudf-polars with build.sh (#16898)

Fix DataFrame.drop(columns=cudf.Series/Index, axis=1) (#16712)

[DOC] Update Pylibcudf doc strings (#16810)

Optimization of tdigest merge aggregation. (#16780)

Display deltas for cudf.pandas test summary (#16864)

JSON tree algorithms refactor I: CSR data structure for column tree (#15979)

Pin polars for 24.10 and update polars test suite xfail list (#16886 )

Build `cudf-polars` with `build.sh` (#16898 )

Fix DataFrame.drop(columns=cudf.Series/Index, axis=1) (#16712 )

[DOC] Update Pylibcudf doc strings (#16810 )

Optimization of tdigest merge aggregation. (#16780 )

Display deltas for `cudf.pandas` test summary (#16864 )

JSON tree algorithms refactor I: CSR data structure for column tree (#15979 )