Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(integration_tests): use timestamptz for MySQL, TiDB, Doris and StarRocks #13641

Merged
merged 1 commit into from
Nov 29, 2023

Conversation

xiangjinwu
Copy link
Contributor

@xiangjinwu xiangjinwu commented Nov 24, 2023

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Sequel to #13616.

https://buildkite.com/risingwavelabs/integration-tests/builds/414 (RW_IMAGE_TAG=nightly-20231127)

In short:

  • PG/RW timestamptz ~ MySQL/TiDB timestamp
  • PG/RW timestamp ~ MySQL/TiDB datetime ~ Doris/StarRocks datetime
  • Event time (instants of physical timeline) should always be timestamptz. Local/civil time can be ambiguous during Daylight Saving jumpback and is subject to arbitrary and unpredictable law changes.
  • The best input syntax for instants is 2006-01-02T22:04:05Z, which is both ISO 8601 and RFC 3339 compliant.

Changes made:

  • mysql-sink
    • old: rw TIMESTAMP -> mysql TIMESTAMP -> rw TIMESTAMPTZ
    • new: rw TIMESTAMP -> mysql DATETIME -> rw TIMESTAMP and rw TIMESTAMPTZ -> mysql TIMESTAMP -> rw TIMESTAMPTZ
  • tidb-cdc-sink
    • old: tidb timestamp -> rw timestamp -> tidb timestamp
    • new: tidb timestamp/datetime -> rw timestamp (#13682) and rw timestamptz -> tidb timestamp
  • doris-sink / starrocks-sink
    • old: rw timestamp -> datetime
    • new: rw timestamptz AT TIME ZONE -> datetime

https://dev.mysql.com/doc/refman/8.0/en/datetime.html
https://docs.pingcap.com/tidb/stable/data-type-date-and-time

The DATETIME type is used for values that contain both date and time parts. MySQL retrieves and displays DATETIME values in 'YYYY-MM-DD hh:mm:ss' format. The supported range is '1000-01-01 00:00:00' to '9999-12-31 23:59:59'.

The TIMESTAMP data type is used for values that contain both date and time parts. TIMESTAMP has a range of '1970-01-01 00:00:01' UTC to '2038-01-19 03:14:07' UTC.

MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval. (This does not occur for other types such as DATETIME.)

In MySQL 8.0.19 and later, you can specify a time zone offset when inserting a TIMESTAMP or DATETIME value into a table.

When TIMESTAMP is to be stored, TiDB converts the TIMESTAMP value from the current time zone to UTC time zone. When TIMESTAMP is to be retrieved, TiDB converts the stored TIMESTAMP value from UTC time zone to the current time zone (Note: DATETIME is not handled in this way).

https://doris.apache.org/docs/dev/advanced/time-zone/
https://docs.starrocks.io/en-us/2.4/administration/timezone

nor does it affect the display of values stored as DATE/DATETIME type.

Functions affected by time zone:

  • FROM_UNIXTIME: Given a UTC timestamp, return the date and time of the specified time zone, such as FROM_UNIXTIME(0), return the CST time zone: 1970-01-01 08:00:00.
  • UNIX_TIMESTAMP: Given a specified time zone date and time, return UTC timestamp, such as CST time zone UNIX_TIMESTAMP('1970-01-01 08:00:00'), return 0.
  • CURTIME: Returns the datetime of specified time zone.
  • NOW: Returns the specified date and time of specified time zone.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added test labels as necessary. See details.
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

@github-actions github-actions bot added the type/fix Bug fix label Nov 24, 2023
Copy link

codecov bot commented Nov 24, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (2ddab6b) 68.27% compared to head (21fb46c) 68.27%.
Report is 8 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #13641   +/-   ##
=======================================
  Coverage   68.27%   68.27%           
=======================================
  Files        1523     1523           
  Lines      261953   261953           
=======================================
+ Hits       178843   178846    +3     
+ Misses      83110    83107    -3     
Flag Coverage Δ
rust 68.27% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@xiangjinwu

This comment was marked as outdated.

@xiangjinwu xiangjinwu marked this pull request as draft November 28, 2023 03:54
@xiangjinwu xiangjinwu force-pushed the fix-integration-tests-timestamptz-1 branch from ae847f0 to c00022a Compare November 28, 2023 07:30
@xiangjinwu xiangjinwu changed the title fix(integration_tests): use timestamptz for MySQL, TiDB, Doris, StarRocks and ClickHouse fix(integration_tests): use timestamptz for MySQL, TiDB, Doris and StarRocks Nov 28, 2023
@xiangjinwu xiangjinwu force-pushed the fix-integration-tests-timestamptz-1 branch from c00022a to 293d5f8 Compare November 28, 2023 07:44
@xiangjinwu xiangjinwu force-pushed the fix-integration-tests-timestamptz-1 branch from 293d5f8 to 21fb46c Compare November 28, 2023 08:37
@xiangjinwu xiangjinwu marked this pull request as ready for review November 28, 2023 09:00
Copy link
Contributor

@StrikeW StrikeW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiangjinwu xiangjinwu added this pull request to the merge queue Nov 29, 2023
Merged via the queue into main with commit 83e9586 Nov 29, 2023
36 of 37 checks passed
@xiangjinwu xiangjinwu deleted the fix-integration-tests-timestamptz-1 branch November 29, 2023 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/fix Bug fix
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants