CloudPubSubConnector Message Aggregation

[yaronworldwide]

Hi ,

I would like to tune KafkaConnect so it will aggregate multiple records in a single pubsub msg.
Is this requirement currently supported ?

Regards,

[kamalaboulhosn]

Do you mean a single Pub/Sub request, i.e., messages are batched together and sent as a single batch to the service or do you mean they are aggregated into a single PubSubMessage? If the former, yes, this is supported. You can use the maxBufferSize, maxBufferBytes, and maxDelayThresholdMs properties to control the batching. If you mean the latter, then no, that is not supported.

[yaronworldwide]

Hey ,
I mean aggregated into a single PubSubMessage .

[kamalaboulhosn]

What is the use case for this aggregation and what would you expect it to look like? The PubSubMessage is just bytes. How would the data be encoded in that message?