Monitoring heartbeat status

[MichaelScofield]

What type of enhancement is this?

Tech debt reduction

What does the enhancement do?

Heartbeat plays a critical role in our system. However, we are lacking of the ability to monitor the status of it. It's useful because through the lens of heartbeat monitoring, one could tell whether the network between Datanode/Frontend and Metasrv is healthy, or understand the actions Metasrv does by deriving from dealing with the heartbeat's payload data. Very helpful for database maintainer.

So I propose adding the following prometheus metrics:

• Heartbeat request send count in Datanode and Frontend
• Heartbeat response receive count in Datanode and Frontend, labelled with response types
• Metasrv heartbeat receive count, labelled with pusher key

Implementation challenges

No response

[killme2008]

We need some docs to describe how to add these metrics and where to add them. @MichaelScofield