gRPC Connectors' deadlock + Ack de-duplication

[maha-hajja]

The deadlock case:

If the server fails before sending acks to the client, and restarts, then it will start waiting to receive records from the client, while the client is waiting to receive acks from the server.

Solution:
If the client loses connection to the server while waiting for acks, it should resend all the records that it didn’t receive an ack for, when the connection is reestablished.

BUT, if the server manages to send an ack for the old batch and the records that were resent, then we have another problem, which is duplicated acks received in the client!

Ack De-duplication
If the client resent records after losing and restoring connection with the server, then the server might have the same records twice, and send duplicated acks.
The client needs to do some kind of assertion on the received acks to make sure the server received the right records, but we need a way for the client to know which ack it received (the records that were sent before the connection was lost, or the record that was resent)

Solution:
Client will prepend an index to the position before sending the record to the server, this will be a monotonically increasing sequence number that tells the order of the records sent by the client.

• Client prepends a monotonically increasing sequence number to the record’s position before sending it to the server, reserving the first 4 bytes(uint32) of the position for the index.
  Note: the case where the 4bytes value wraps around back to zero should be handled.

• Server needs to send the ack to the client with its sequence number, but also needs to strip the sequence number out of the position before sending the record to its destination.

• The acknowledgments that are flowing back to the source connector (server) are always going to be in the same order as the records that are produced by the client.

• The client needs to assert that:

  1. The sequence number for the ack it got back is higher than the last one it received.
  2. The number of acks received is equal to or less than the total number of sent records, and greater or equal to the batch size.
  3. The acknowledgments are received in the right order and no acknowledgments are skipped.