feat: specifying table schema when creating source/table from avro/protobuf encode whose full schema is externally defined

[st1page]

Currently, we do not allow user to define schema in column clause

CREATE SOURCE(a int, b int) FORMAT PLAIN ENCODE PROTOBUF;
 ERROR: ExecuteError: Protocol error: User-defined schema is not allowed with FORMAT PLAIN ENCODE PROTOBUF

But some user needs it to prune columns, especially when they are creating a table with the connector, the selected columns determine how many columns will be materialized in storage. Also, casting the source data into the expected datatype is needed too.

Another issue is that if the user want to define a generated column, he must specify columns in the create source/table statement. We might need to introduce another syntax #12209 fixed

• avro
• protobuf
• json with schema registry

[fuyufjh]

How can user specifies a subset of schema when using schema registry?

It feels like this is not necessary. They may just ignore these columns when creating MVs on this source.

How to use generated column when using schema registry?

+1 for the syntax of your proposed.

[st1page]

How can user specifies a subset of schema when using schema registry?

It feels like this is not necessary. They may just ignore these columns when creating MVs on this source.

But when they create tables with primary key, the specified columns influence which columns will be materialized in storage.

[hzxa21]

How can user specifies a subset of schema when using schema registry?

It feels like this is not necessary. They may just ignore these columns when creating MVs on this source.

One thing semi-related to this issue is #10949. If user can specify a subset of columns, we may be able to filter out unnecessary changes (new row == old row) in the Table's materialize executor during the conflict check.

[tabVersion]

How can user specifies a subset of schema when using schema registry?

It feels like this is not necessary. They may just ignore these columns when creating MVs on this source.

But when they create tables with primary key, the specified columns influence which columns will be materialized in storage.

The feature seems not a pain point. But still have some concerns about the compatibility when updating schema.
Let's keep this open and make it a ramp-up task.

[tabVersion]

we can allow user-defined parts only when both name and type are matched with the ones mapped from avro/pb. Let's take another look, cc @st1page

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.