[BUG] json_test failed on "NameError: name 'TimestampNTZType' is not defined"

[NvTimLiu]

Describe the bug
json_test.py tests failed on "NameError: name 'TimestampNTZType' is not defined" for 23.10:

 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 ../../src/main/python/json_test.py:223: in json_ts_formats_round_trip_ntz
     data_gen = TimestampGen(tzinfo=None if timestamp_type == "TIMESTAMP_NTZ" else timezone.utc)
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 
 self = <[AttributeError("'TimestampGen' object has no attribute 'nullable'") raised in repr()] TimestampGen object at 
 start = None, end = None, nullable = True, tzinfo = None
 
     def __init__(self, start=None, end=None, nullable=True, tzinfo=timezone.utc):
 >       super().__init__(TimestampNTZType() if tzinfo==None else TimestampType(), nullable=nullable)
 E       NameError: name 'TimestampNTZType' is not defined
 
 ../../src/main/python/data_gen.py:576: NameError
 _ test_json_ts_formats_round_trip_ntz_v2[TIMESTAMP_NTZ-dd/MM/yyyy-'T'HH:mm:ss[.SSS]] _

[NvTimLiu]

Similar timestamp failure issue #9325, cc @andygrove

[andygrove]

I can understand why this error is occurring. TimestampNTZType is new as of Spark 3.4.0. However, I have been unable to reproduce the issue locally, even after downgrading to pyspark 3.1.1.

@NvTimLiu Do you have any more information about which environment / Spark version you saw this on? The tests in CI passed for the PR that added this, so I assume they are all using a pyspark version which is at least 3.4.0.

[NvTimLiu]

yes, seems json_tests xx_v1/v2 only failed on spark 3.4.0, and PASS on other spark versions like 3.4.1/3.5.0

@andygrove I can repro on both my local and with Blossom jobs, let me share you with slack msg the Blossom job and my local test CLIs

it’s more like a bug from spark-3.4.0 release

[andygrove]

I confirmed that this is a known bug in 3.4.0 that was fixed in 3.4.1

https://issues.apache.org/jira/browse/SPARK-43759

Perhaps we need to skip some tests for 3.4.0 and only run for 3.4.1 and later

[pxLi]

this is still reproducible, please help make the decision to skip or only run on 3.4.1+ thanks