Feature/SK-592 | Refactor gRPC server

[Wrede]

There is a lot of redundancy in the server logic of the combiner. Also logging is messy and hard to understand, introduce logger for different verbosity . Some functions/endpoints can be merged to improve readability and reduce complexity.

Update:

gRPC endpoint Report has been removed and replaced with ListActiveClients. The combiner interfaces (interfaces.py) has list_active_clients(queue_name) which will return the number of "online" clients attached to the specific combiner. The Queue (also reffered as channel in the logic) is either MODEL_UPDATE_REQUESTS or MODEL_VALIDATION_REQUESTS.
Obeserve when the controller loadbalancer LeatPacked calls list_active_clients() it defaults to MODEL_UPDATE_REQUESTS, which means it the loadbalancer is based on the number of online "training" clients, and does not consider "validation" clients.

The way client status is managed has changed. The combiner (gRPC server) now also keeps track of the "online"/"offline" status of clients using the state dict. A client is considered online if it's gRPC context (stream) has been active for the last 10 seconds, or if the client has sent a hearbeat within 10 seconds. The status of clients are then pushed to the statestore DB, bulk update (mongodb update_many) using a deamon process, this push occur every 10 seconds. This should be made configurable in the future. It's also possible to use the gRPC endpoint ListActiveClients, this will aslo push the status updates to statestore DB once called.
The decision for this approach is:

1. For scalability we want to avoid sending to many DB updates, now it's based on bulk updates and only clients that have changed state are updated. Also it should be configurable (the timeout for the deamon or completely disable it and solely use the ListActiveClients endpoint to update the DB)
2. There should be a way to solely do request based updates, this is what ListActiveClients enable.

[ahellander]

Looks good imo. I agree that we should refactor the streams, there should only be one connection for update-messages. But why not first finish these improvements to error handling and logging and try to merge those?

[ahellander]

Awesome, go go go :-)