Discussion: End-to-end schema change

[neverchanje]

This proposal aims to explore the feasibility of automating the schema changes of sources, tables and sinks when a schema change happened in the upstream.

Let's consider a case where a schema change (add column) is made to a PG table, from which a RisingWave CDC table is set up. In order to land the table change through table and sink and finally onto the destination, a StarRocks table, we have to execute 3 consecutive ALTER ADD COLUMN commands.

1. ALTER TABLE tbl ADD COUMN in RW
2. ALTER SINK snk ADD COUMN in RW, which it's not yet supported.
3. ALTER TABLE tbl ADD COLUMN in SR.

Based on the breakdown of the schema change, it's apparent that we can introduce some automation to table and sink to eliminate these tedious works. That is,

• Automatically apply schema changes on CDC tables when the upstream changes schema.
  Here's the internal doc that details the requirement.
• Automatically apply schema changes on sinks when the parent table changes schema. #16839