Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support yyyyMMdd in GetTimestamp operator for LEGACY mode [databricks] #11449

Merged
merged 3 commits into from
Sep 23, 2024
Merged

Support yyyyMMdd in GetTimestamp operator for LEGACY mode [databricks] #11449

merged 3 commits into from
Sep 23, 2024

Conversation

res-life
Copy link
Collaborator

@res-life res-life commented Sep 9, 2024 •
edited
Loading

closes #11445

This PR supports yyyyMMdd in GetTimestamp operator for LEGACY mode
Note: this PR introduced inconsistent behavior compared to Spark:

Spark behavior:

  "20240101"  =>  2024-01-01
  "202401 01" =>  2024-01-01
  "2024 0101" =>  invalid
  "202411"    =>  invalid
  "2024101"   =>  2024-10-01  // Note: 10-01 not 01-01
  "202411"    =>  invalid

GPU only support 8 digit strings.

@res-life res-life force-pushed the yyyyMMdd-GetTimestamp branch from c7f0cc6 to adbbefd Compare September 9, 2024 09:35
@res-life
Copy link
Collaborator Author

res-life commented Sep 9, 2024

build

@res-life res-life changed the title Support yyyyMMdd in GetTimestamp operator for LEGACY mode Support yyyyMMdd in GetTimestamp operator for LEGACY mode [databricks] Sep 9, 2024
@res-life res-life marked this pull request as ready for review September 9, 2024 09:57
revans2
revans2 previously approved these changes Sep 9, 2024
View reviewed changes
@res-life
Copy link
Collaborator Author

Building failed due to #11436

Errors:
[2024-09-09T10:14:38.020Z] - extract mortgage data *** FAILED ***
[2024-09-09T10:14:38.021Z] 0 did not equal 10000 (MortgageSparkSuite.scala:65)

@firestarman
Copy link
Collaborator

firestarman commented Sep 10, 2024 •
edited
Loading

Looks like this change will also enable yyyyMMdd support for all the relevant date time expressions, e.g. from_unix, data_format ...
We may need tests for them too.

@res-life
Copy link
Collaborator Author

build

@sameerz sameerz added the feature request New feature or request label Sep 15, 2024
@res-life
Copy link
Collaborator Author

Looks like this change will also enable yyyyMMdd support for all the relevant date time expressions, e.g. from_unix, data_format ... We may need tests for them too.

Updated cases.

@res-life
Copy link
Collaborator Author

build

gerashegalov
gerashegalov reviewed Sep 19, 2024
View reviewed changes
integration_tests/src/main/python/date_time_test.py Outdated
assert_gpu_and_cpu_are_equal_sql(
lambda spark : unary_op_df(spark, gen),
"tab",
"select unix_timestamp(a, 'yyyyMMdd'), from_unixtime(unix_timestamp(a, 'yyyyMMdd'), 'yyyyMMdd'), date_format(to_timestamp(a, 'yyyyMMdd'), 'yyyyMMdd') from tab",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider using multi-line strings

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@firestarman
Copy link
Collaborator

Looks good to me.

@res-life
Copy link
Collaborator Author

build

@res-life
Copy link
Collaborator Author

Spark 400 building failure is related to #11479

@res-life res-life merged commit 0f5d510 into NVIDIA:branch-24.10 Sep 23, 2024
43 of 45 checks passed
@res-life res-life deleted the yyyyMMdd-GetTimestamp branch September 23, 2024 01:56
@res-life res-life mentioned this pull request Sep 24, 2024
Merged
@pxLi pxLi mentioned this pull request Sep 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@firestarman firestarman firestarman left review comments

@gerashegalov gerashegalov gerashegalov approved these changes

@revans2 revans2 revans2 left review comments

@winningsix winningsix Awaiting requested review from winningsix

Assignees
No one assigned
Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Support format 'yyyyMMdd' in GetTimestamp operator
5 participants
@res-life @firestarman @gerashegalov @revans2 @sameerz