Add creationTime in mysql column

[jiang95-dev]

Summary

Add creationTime in mysql column. This column will be used in multiple cases, such as Orion inventory, dual-writer HI data collection, and stats-collection job monitoring.

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

Manually insert a row without creationTime:

mysql> select * from user_table_row;
+-------------+----------+---------+-------------------+--------------+---------------+---------------------+----------------------------+
| database_id | table_id | version | metadata_location | storage_type | creation_time | last_modified_time  | ETL_TS                     |
+-------------+----------+---------+-------------------+--------------+---------------+---------------------+----------------------------+
| d1          | t1       |       0 | /data/v1          | hdfs         |             0 | 2024-11-05 23:42:09 | 2024-11-05 23:42:09.745050 |
+-------------+----------+---------+-------------------+--------------+---------------+---------------------+----------------------------+

Create a table:

scala> spark.sql("create table d1.t2 (col string)")
res0: org.apache.spark.sql.DataFrame = []

scala> spark.sql("insert into d1.t2 values ('a')")
res1: org.apache.spark.sql.DataFrame = []                                       

scala> spark.sql("select * from d1.t2").show
+---+                                                                           
|col|
+---+
|  a|
+---+

scala> spark.sql("show tables in d1").show
+---------+---------+
|namespace|tableName|
+---------+---------+
|       d1|       t1|
+---------+---------+

scala> spark.sql("show databases").show
+---------+
|namespace|
+---------+
|       d1|
+---------+

After creating a table:

mysql> select * from user_table_row;
+-------------+----------+---------+---------------------------------------------------------------------------------------------------------------------+--------------+---------------+---------------------+----------------------------+
| database_id | table_id | version | metadata_location                                                                                                   | storage_type | creation_time | last_modified_time  | ETL_TS                     |
+-------------+----------+---------+---------------------------------------------------------------------------------------------------------------------+--------------+---------------+---------------------+----------------------------+
| d1          | t1       |       0 | /data/v1                                                                                                            | hdfs         |             0 | 2024-11-05 23:42:09 | 2024-11-05 23:42:09.745050 |
| d1          | t2       |       0 | /data/openhouse/d1/t2-43cb50ee-f1f7-4383-a262-4b0bf507d02e/00000-5724f70d-a526-48d1-8620-9d3c513ae3e2.metadata.json | hdfs         | 1730850353924 | 2024-11-05 23:45:56 | 2024-11-05 23:45:56.030361 |
+-------------+----------+---------+---------------------------------------------------------------------------------------------------------------------+--------------+---------------+---------------------+----------------------------+

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

[autumnust]

no major concern - can we keep track of deployment / backfilling steps ? This needs to be handled very carefully

[jiang95-dev]

no major concern - can we keep track of deployment / backfilling steps ? This needs to be handled very carefully

It is captured in the ticket.

[jiang95-dev]

Discussed offline. BigInt has intrinsic advantages over Timestamp where it may not be supported everywhere (such as H2). We choose to rely on Java instead of mysql for this value and let users to interpret it. Inconsistency is not favored but for the sake of time, we will leave the column lastModifiedTime along and look for better schema evolution strategy.