Skip to content

Commit

Permalink
Fix for lead/lag window test failures. (#11823)
Browse files Browse the repository at this point in the history
Fixes #11807.

`test_lead_lag_for_structs_with_arrays` in `window_function_test` fails intermittently because of
non-deterministic data ordering.
Window function tests are sensitive to data ordering.  With certain values of DATAGEN_SEED,
there are repeated values of partitioning/ordering keys, causing the window function
to return different values on CPU and GPU.

This commit fixes the test so that the ordering is deterministic.

Signed-off-by: MithunR <[email protected]>
  • Loading branch information
mythrocks authored Dec 9, 2024
1 parent 3449c8a commit 45cdac3
Showing 1 changed file with 4 additions and 6 deletions.
10 changes: 4 additions & 6 deletions integration_tests/src/main/python/window_function_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -971,14 +971,12 @@ def do_it(spark):
def test_lead_lag_for_structs_with_arrays(a_b_gen, struct_gen):
data_gen = [
('a', RepeatSeqGen(a_b_gen, length=20)),
('b', IntegerGen(nullable=False, special_cases=[])),
('b', UniqueLongGen(nullable=False)),
('c', struct_gen)]
# By default for many operations a range of unbounded to unbounded is used
# This will not work until https://github.com/NVIDIA/spark-rapids/issues/216
# is fixed.
# For many operations, a range of unbounded to unbounded is used by default.

# Ordering needs to include c because with nulls and especially on booleans
# it is possible to get a different ordering when it is ambiguous.
# Ordering needs to include `b` because with nulls and especially on booleans,
# it is possible to get a different result when the ordering is ambiguous.
base_window_spec = Window.partitionBy('a').orderBy('b')

def do_it(spark):
Expand Down

0 comments on commit 45cdac3

Please sign in to comment.