Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Support spark.sql.parquet.int96RebaseModeInWrite= LEGACY #9037

Closed
tgravescs opened this issue Aug 14, 2023 · 1 comment · Fixed by #9658
Closed

[FEA] Support spark.sql.parquet.int96RebaseModeInWrite= LEGACY #9037

tgravescs opened this issue Aug 14, 2023 · 1 comment · Fixed by #9658
Assignees
Labels
feature request New feature or request

Comments

@tgravescs
Copy link
Collaborator

tgravescs commented Aug 14, 2023

Is your feature request related to a problem? Please describe.
It would be nice if we could support spark.sql.legacy.parquet.int96RebaseModeInWrite=LEGACY as some customers use this and its default on Databricks

Note the definition of this config:

When LEGACY, Spark will rebase INT96 timestamps from Proleptic Gregorian calendar to " +
        "the legacy hybrid (Julian + Gregorian) calendar when writing Parquet files. " +
        "When CORRECTED, Spark will not do rebase and write the timestamps as it is. " +
        "When EXCEPTION, which is the default, Spark will fail the writing if it sees ancient " +
        "timestamps that are ambiguous between the two calendars."

Also note this config was renamed and used to be in Spark 3.1: spark.sql.legacy.parquet.int96RebaseModeInWrite. 3.2 changed to use the new conf spark.sql.parquet.int96RebaseModeInWrite. The old configuration will still work but is deprecated

@tgravescs tgravescs added feature request New feature or request ? - Needs Triage Need team to review and classify labels Aug 14, 2023
@tgravescs
Copy link
Collaborator Author

Note we have issues for datetimeRebase #8596

@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Aug 15, 2023
@tgravescs tgravescs changed the title [FEA]Support spark.sql.legacy.parquet.int96RebaseModeInWrite = LEGACY [FEA]Support spark.sql.parquet.int96RebaseModeInWrite= LEGACY Aug 16, 2023
@ttnghia ttnghia self-assigned this Aug 16, 2023
@sameerz sameerz changed the title [FEA]Support spark.sql.parquet.int96RebaseModeInWrite= LEGACY [FEA] Support spark.sql.parquet.int96RebaseModeInWrite= LEGACY Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants