Make serialization key checks uniform for Hive Text file read/write.

[mythrocks]

This is a minor change, to bring the serialization format checks on the Read and Write sides of Hive text files in line with each other.

The read side would check if the serialization-key is set to empty (""):

spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/hive/rapids/HiveProviderImpl.scala

Lines 155 to 161 in aa2da41

	val serializationFormat = storage.properties.getOrElse(serializationKey, "")
	if (serializationFormat != ctrlASeparatedFormat) {
	willNotWorkOnGpu(s"unsupported serialization format found: " +
	s"$serializationFormat, " +
	s"only \'^A\' separated text input (i.e. serialization.format=1) " +
	s"is currently supported")
	}

The write side was checking for the expected default: "1":

spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/hive/rapids/GpuHiveFileFormat.scala

Lines 130 to 136 in aa2da41

	val serializationFormat = storage.properties.getOrElse(serializationKey, "1")
	if (serializationFormat != ctrlASeparatedFormat) {
	meta.willNotWorkOnGpu(s"unsupported serialization format found: " +
	s"$serializationFormat, " +
	s"only \'^A\' separated text output (i.e. serialization.format=1) " +
	s"is currently supported")
	}

This caused confusion in behaviour, as voiced in #11803.

The change in this commit makes the checks uniformly conservative.

[mythrocks]

Build

[revans2]

Do you know if this fixes #11803?

[mythrocks]

Do you know if this fixes #11803?

I'm afraid this doesn't fix #11803. :/ That error is on the read path, while this fix affects (only the defaults in) the write path.

[mythrocks]

Looks like I spoke too soon about making these checks the same. Debugging now.

[mythrocks]

I've had to add an xfail case for 3.2, where creating a table via CTAS doesn't not seem to set the "serialization.format" key in the storage properties. That bug seems to have been fixed in subsequent Spark versions.

Note that this is only in the test setup.

[mythrocks]

Build

[mythrocks]

It now looks like there is no guarantee of serialization.format being set correctly, when constructed via CTAS. The CI failure seems to be with Spark 3.3.0. I was also able to repro this from 3.3.4.

I'll look into setting this programmatically.

[mythrocks]

Ok, I don't think this is the right approach.

If the output table exists prior to the write, there's a guarantee that the serialization.format would have been set.

However, if the table is created via CTAS, there is no guarantee that STORAGE PROPERTIES would be set until the write has completed. From all tracing, it appears that the properties are empty until after the write.

I'm inclined to document this difference in code, and not change the defaults. I'm closing this change.

[mythrocks]

I have documented the need for the different settings over at #11856.