You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
It would be nice if we could support spark.sql.legacy.parquet.int96RebaseModeInWrite=LEGACY as some customers use this and its default on Databricks
Note the definition of this config:
When LEGACY, Spark will rebase INT96 timestamps from Proleptic Gregorian calendar to " +
"the legacy hybrid (Julian + Gregorian) calendar when writing Parquet files. " +
"When CORRECTED, Spark will not do rebase and write the timestamps as it is. " +
"When EXCEPTION, which is the default, Spark will fail the writing if it sees ancient " +
"timestamps that are ambiguous between the two calendars."
Also note this config was renamed and used to be in Spark 3.1: spark.sql.legacy.parquet.int96RebaseModeInWrite. 3.2 changed to use the new conf spark.sql.parquet.int96RebaseModeInWrite. The old configuration will still work but is deprecated
The text was updated successfully, but these errors were encountered:
tgravescs
changed the title
[FEA]Support spark.sql.legacy.parquet.int96RebaseModeInWrite = LEGACY
[FEA]Support spark.sql.parquet.int96RebaseModeInWrite= LEGACY
Aug 16, 2023
sameerz
changed the title
[FEA]Support spark.sql.parquet.int96RebaseModeInWrite= LEGACY
[FEA] Support spark.sql.parquet.int96RebaseModeInWrite= LEGACY
Dec 4, 2023
Is your feature request related to a problem? Please describe.
It would be nice if we could support spark.sql.legacy.parquet.int96RebaseModeInWrite=LEGACY as some customers use this and its default on Databricks
Note the definition of this config:
Also note this config was renamed and used to be in Spark 3.1: spark.sql.legacy.parquet.int96RebaseModeInWrite. 3.2 changed to use the new conf spark.sql.parquet.int96RebaseModeInWrite. The old configuration will still work but is deprecated
The text was updated successfully, but these errors were encountered: