fix: Close connections in gRPC server on shutdown

[chubei]

No description provided.

[Jesse-Bakker]

What is the difference between using the custom AsyncWrite implementation and serve_with_shutdown?

[chubei]

AddrStream is not affected by the signal passed to serve_with_shutdown, which results in nasty bugs in live. After dozer is stopped, any connection to the stopped server is still alive and the HTTP requests over it are served. The client sees old data from last run.

Our custom implementation ShutdownAddrStream will close the connection (by calling AddrStream::pool_shutdown) when shutdown signal is received. And serve_with_shutdown will actually wait for connection being disconnected on shutdown, thus the shutdown is complete and graceful.

[Jesse-Bakker]

Okay, I dove way too deep into the source code of tonic and hyper. The way we used to do it with an AbortHandle is definitely wrong, as you noticed, because hyper will spawn a new task for every connection and those tasks will just keep running.

serve_with_shutdown and serve_with_incoming_shutdown should (according to their implementation) gracefully shutdown connections (they pass a channel sender to the task and await the receiver closing) and only complete its future after all connections are drained. If that's not working, it's a bug in hyper.

[chubei]

It works correctly, but we can't wait for all connections to go down. We need to close the connection from server side.

[Jesse-Bakker]

Alright, I get it now 👍