improve: output stream_key of Dedup should be upstream's stream_key instead of all columns

[fuyufjh]

Output stream_key of Dedup should be upstream's stream_key instead of all columns, because it's strictly better.

By the way, a related problem: now AppendOnlyDedup uses PK as its dedup key.

[st1page]

Currently, we use the dedup_cols as the stream key which means the keys in the distinct on clause.

risingwave/src/frontend/src/optimizer/plan_node/generic/dedup.rs

Line 53 in bc05743

fn stream_key(&self) -> Option<Vec<usize>> {

what do you mean about the "all columns"?

[fuyufjh]

Currently, we use the dedup_cols as the stream key which means the keys in the distinct on clause.

risingwave/src/frontend/src/optimizer/plan_node/generic/dedup.rs

Line 53 in bc05743

fn stream_key(&self) -> Option<Vec<usize>> {

what do you mean about the "all columns"?

I mean for select distinct (without on (<columns>)), all the columns will be used as stream key.

[fuyufjh]

Yesterday I read some code around this part, and I am not sure whether it's good.

1. For distinct on (<columns>), it seems reasonable to use the specified distinct columns as stream key. Usually in this cases the distinct columns are selected by users, only including the useful ones.
2. Additionally, using upstream stream key is not 100% "strictly better" considering hidden key row_id. It makes the work become a bit tricky.

Perhaps I should close this issue first.

[st1page]

Sadly, there are two invariants on the stream key

1. It should be a unique key
2. The dist key must be a subset of the stream key

So if we want to change the stream_key of the Dedup, we have to also change its to_stream implementation to modify its distribution enforcement logic.