[BUG] Failed test_multi_tier_ast[DATAGEN_SEED=1701445668] on CI

[cindyyuanjiang]

Describe the bug
Failed integration test. Pipeline: rapids_it-MT-egx06-standalone

Details:

FAILED ../../src/main/python/ast_test.py::test_multi_tier_ast[DATAGEN_SEED=1701445668]

[2023-12-01T22:00:53.723Z] --- CPU OUTPUT

[2023-12-01T22:00:53.723Z] +++ GPU OUTPUT

[2023-12-01T22:00:53.723Z] @@ -1,4 +1,3 @@

[2023-12-01T22:00:53.723Z] -Row(((id < x) = (id < (id + x)))=False)

[2023-12-01T22:00:53.723Z]  Row(((id < x) = (id < (id + x)))=False)

[2023-12-01T22:00:53.723Z]  Row(((id < x) = (id < (id + x)))=False)

[2023-12-01T22:00:53.723Z]  Row(((id < x) = (id < (id + x)))=False)

[2023-12-01T22:00:53.723Z] @@ -8,3 +7,4 @@

[2023-12-01T22:00:53.723Z]  Row(((id < x) = (id < (id + x)))=False)

[2023-12-01T22:00:53.723Z]  Row(((id < x) = (id < (id + x)))=False)

[2023-12-01T22:00:53.723Z]  Row(((id < x) = (id < (id + x)))=False)

[2023-12-01T22:00:53.723Z] +Row(((id < x) = (id < (id + x)))=False)

Steps/Code to reproduce bug
Integration test

Expected behavior
Should pass

[gerashegalov]

It does not easily reproduce for me in the local mode. To reproduce with Spark 3.5.0 locally, run against a local-cluster with at least 2 executors using the following command presuming you have installed pytest-repeat because it may take several iterations

DATAGEN_SEED=1701445707 \
NUM_LOCAL_EXECS=2 \
SPARK_HOME=~/dist/spark-3.5.0-bin-hadoop3 \
TEST_PARALLEL=0 \
  ./integration_tests/run_pyspark_from_build.sh -k test_multi_tier_ast --count=10

Reran the above 3 times: 4 (pass) / 6 (fail), 3 (pass) / 7 (fail), 5 (pass) / 5 (fail)

[jlowe]

The problem is one of ordering. It becomes very apparent when updating the projection to include the original input value and increasing the number of executors.

[revans2]

I think it has to be because of the repartition(1) in the test. A project should maintain the order, but a repartition has races in it. Why do we have that in it?