Investigate sustainable ways to ensure backwards compatibility

[kwannoel]

Currently we try to support backwards compatibility from 1.0 to the latest main.

The way this is done is by ensuring metadata remains backwards compatible, i.e. old meta-data will still need to be maintained.

Additionally the code which handles this metadata will also need to be kept compatible.

Long term, this can lead to some messy code.

This issue tracks the investigation of alternative ways we can support backwards compatibility, which minimize the technical debt.

Additional motivation is from @stdrc who shared the painpoint.

[stdrc]

I just realized that 0020507 this commit, which is #13376, though seems so harmless, also causes serious incompatibility. In plan pb generated by older version of frontend, there's no order_columns, in the new backend executor, it uses the order_columns in plan node instead of derive it on its own. If the plan node is from an older frontend, non-append-only min/max calls will stop working!

This implies that, for keys/schemas/... that used to be derived by backend executors, and not passed via protobuf from frontend (examples that were passed from fe include operators' pk_indices and schema), we can never remove the derivation logic from executors. Instead, we can only add a new branch that uses what frontend passed in.

[kwannoel]

We can support migration of metadata I suppose.

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.