feat: reduce the size of stored streaming metadata to avoid too large request to etcd

[yezizp2012]

Is your feature request related to a problem? Please describe.

Related #7728

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[BugenZhao]

It looks like the large request is mainly caused by duplication of the NodeBody for a fragment with too many parallelisms (actors). I guess this can be resolved by introducing a (de)compression middleware for the protobuf values actually stored.

---

Can a Protobuf message begin with a gzip magic number?

[yezizp2012]

It looks like the large request is mainly caused by duplication of the NodeBody for a fragment with too many parallelisms (actors). I guess this can be resolved by introducing a (de)compression middleware for the protobuf values actually stored.

Yes exactly. Previously I'm thinking of introducing a new field in TableFragment that stores only one copy of the NodeBody data and necessary distribution info, which can be used to generate the real running StreamActor. By simply adding some transform logic in meta, the duplicate part of NodeBody can be reduced. Add non-related to this issue, this will also pay off for switching to sql backend later on. WDUT?

[yezizp2012]

The issue will not exist after SQL-backend meta store. Keep it open for record.

[yezizp2012]

It looks like the large request is mainly caused by duplication of the NodeBody for a fragment with too many parallelisms (actors). I guess this can be resolved by introducing a (de)compression middleware for the protobuf values actually stored.

Yes exactly. Previously I'm thinking of introducing a new field in TableFragment that stores only one copy of the NodeBody data and necessary distribution info, which can be used to generate the real running StreamActor. By simply adding some transform logic in meta, the duplicate part of NodeBody can be reduced. Add non-related to this issue, this will also pay off for switching to sql backend later on. WDUT?

@shanicky is working on integration the transform logic to current TableFragments implementation, which was introduced for sql backends. The PR is otw.

[shanicky]

I submitted a simple PR #13598 to implement compression for PbTableFragments to reduce redundant stream nodes in RPC processes and etcd storage.

[yezizp2012]

Done by #17315 ?

[BugenZhao]

Done by #17315 ?

It's a quick solution, but I believe it should be enough for the goal in this issue. 😄 Feel free to close it.