Discussion: the way to maintain direct CDC Backfill with existing direct CDC

[StrikeW]

Here are my thoughts on the way to maintain the CDC backfill feature and the existing direct cdc feature:

1. The original way of creating a single table cdc by create table t (…) with ("connector"="xxx-cdc") (interface 1) is still retained , mainly to support the connector without backfill capability, such as citus-cdc and possible new upstreams in the future.

2. For connectors that already support backfill (e.g. mysql-cdc), only tables created with the new syntax create table t (...) from <source> (interface 2) will enable backfill. Single table cdc created by the interface 1 won't enable backfill.

3. For connectors that support backfill (e.g. mysql-cdc), we will gradually unify the implementation of interface 1 to the table-on-source implementation, and it is necessary to ensure that the single-table cdc job created by the user previously will still work after the we unified the implementation. (Low priority)

[StrikeW]

@hzxa21 @fuyufjh @neverchanje PTAL whether the above strategy is ok to you.

[hzxa21]

LGTM. Just to make sure we are all on the same page, let me reiterate the plan:

• For upstream that supports cdc backfill (mysql and pg as for now), we will recommend user to use CREATE SOURCE + CREATE TABLE FROM ... after cdc backfill is stable and won't support CREATE TABLE WITH connector for newly created source. In the future, when multi-statement DDL transaction is supported, we can then treat CREATE TABLE WITH connector as a syntax suger.
• For upstream that doesn't support cdc backfill (mongo, citus), we only support CREATE TABLE WITH connector.

[fuyufjh]

Agree with @hzxa21.

In the future, when multi-statement DDL transaction is supported, we can then treat CREATE TABLE WITH connector as a syntax suger.

For this, after a double thought, I prefer to simply deprecate the old syntax (create table with connector) and only keep the new syntax (create source then create table from source) e.g. raise an error to notice users to move to the new syntax when they used. Syntax sugar is not that straight-forward especially when user want to create a second table from an existing source.

For backward compatibility, we shall retain the old code path for a long time, and eventually remove these code in some major version update in the future.

[hzxa21]

For this, after a double thought, I prefer to simply deprecate the old syntax (create table with connector) and only keep the new syntax (create source then create table from source) e.g. raise an error to notice users to move to the new syntax when they used. Syntax sugar is not that straight-forward especially when user want to create a second table from an existing source.

+1.

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.