feat(sink): add config auto_create for BigQuery
#17393
Conversation
jetjinser
added
the
user-facing-changes
| _ => {} | ||
| } | ||
| } | ||
|
|
||
| let mut rs = client |
There was a problem hiding this comment.
When auto_create_table=true, it seems that there is no need to query the schema again
| DataType::Time => TableFieldSchema::time(&rw_field.name), | ||
| DataType::Timestamp => TableFieldSchema::date_time(&rw_field.name), | ||
| DataType::Timestamptz => TableFieldSchema::timestamp(&rw_field.name), | ||
| DataType::Interval => { |
There was a problem hiding this comment.
It seems to be supported, If it is not possible, please also set what is written below as not supported
There was a problem hiding this comment.
It should indeed be unsupported for now: https://cloud.google.com/bigquery/docs/schemas#standard_sql_data_types. (no interval here, in pre-ga)
What needs to be added in the error message?
There was a problem hiding this comment.
I tested it before and it works, you can retest it, if it doesn't work, please disable interval in the code elsewhere
doc is in https://cloud.google.com/bigquery/docs/write-api?hl=zh-cn
There was a problem hiding this comment.
It does support it, but it is not a stable feature yet.
In addition, the sdk FieldType have no Interval. It's used to construct Table that table().create() would use.
I think we can let it keep the unsupported error? Panic when validate failed.
There was a problem hiding this comment.
Forgot to change it in get_string_and_check_support_from_datatype?
There was a problem hiding this comment.
About Interval?
Unlike the case of map_field, which is used to create a BigQuery table, we can avoid actively using unstable type Interval; the return value of get_string_and_check_support_from_datatype is used to validate types of BigQuery table columns. If a column is indeed an Interval type, I don't think we have any reason to refuse?
There was a problem hiding this comment.
Got it, thanks for the explanation. Would you mind adding the explanations here into the code comment?
jetjinser
force-pushed
the
jinser/bigquery-auto-create-table
branch
from
8b8bac2 to
72b4250
Compare
Co-authored-by: Tao Wu <[email protected]>
jetjinser
changed the title
auto_create_table for BigQueryauto_create for BigQuery
jetjinser
force-pushed
the
jinser/bigquery-auto-create-table
branch
from
c418061 to
bbdc227
Compare
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
resolve #15586.
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
New config
bigquery.auto_create_tableorauto_create_table, defaults tofalse.If enabled, a new table will be automatically created in BigQuery when the specified table is not found.
The datatype mapping from RisingWave to BigQuery:
^1: Pre-GA: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#interval_type