GRPC Call Performance issue and lead to node not function

[liam-icheng-lai]

Summary of Bug

While doing the load testing, I found out the grpc call make the whole system slow down and cant run in parallel.
After digging into it, the grpc can use multiple cpu cores but not disk. Disk become bottleneck if some grpc call is heavily read/write disk.

Also the more delegations the slower it is. This is matching how disk performs as well.

And while doing this load testing, the babylon node can't receive any block either response peer. I guess all the operations involve reading disk are impacted.

Nov 15 00:48:25 babylon-rpc-0 babylond[15275]: 12:48AM ERR failed to write response err="write tcp 45.32.122.69:26657->18.162.253.87:59633: write: broken pipe" module=rpc-server
Nov 15 00:48:28 babylon-rpc-0 babylond[15275]: 12:48AM ERR failed to write response err="write tcp 45.32.122.69:26657->139.180.191.142:29214: write: connection reset by peer" module=rpc-server
Nov 15 00:48:28 babylon-rpc-0 babylond[15275]: 12:48AM ERR failed to write response err="write tcp 45.32.122.69:26657->139.180.191.142:29232: write: broken pipe" module=rpc-server

Version

v0.16.0

Steps to Reproduce

grpc call:

 grpcurl -plaintext -d '{ "status": 1 }' $ENDPOINT babylon.btcstaking.v1.Query.BTCDelegations

Experiment

Simultaneously Requests Test on Devnet-5: (93983 delegations)
1 request: 6.460s
2 request: 14.892s
4 request: 28.479s
8 request: 52.562s

Simultaneously Requests Test on Devnet-6: (No delegations)
1 request: 1.487s
2 request: 1.601s
4 request: 2.966s
8 request: 3.103s

CPU and Disk Resource Consumption

2 requests

time grpcurl -plaintext -d '{ "status": 1 }' $ENDPOINT babylon.btcstaking.v1.Query.BTCDelegations >> /tmp/a &
time grpcurl -plaintext -d '{ "status": 2 }' $ENDPOINT babylon.btcstaking.v1.Query.BTCDelegations >> /tmp/a

CPU Utilization

Disk Utilization

vmstat 1

8 requests

time grpcurl -plaintext -d '{ "status": 1 }' $ENDPOINT babylon.btcstaking.v1.Query.BTCDelegations >> /tmp/a &
time grpcurl -plaintext -d '{ "status": 2 }' $ENDPOINT babylon.btcstaking.v1.Query.BTCDelegations >> /tmp/a &
time grpcurl -plaintext -d '{ "status": 3 }' $ENDPOINT babylon.btcstaking.v1.Query.BTCDelegations >> /tmp/a &
time grpcurl -plaintext -d '{ "status": 4 }' $ENDPOINT babylon.btcstaking.v1.Query.BTCDelegations >> /tmp/a &
time grpcurl -plaintext -d '{ "status": 4 }' $ENDPOINT babylon.btcstaking.v1.Query.BTCDelegations >> /tmp/a &
time grpcurl -plaintext -d '{ "status": 4 }' $ENDPOINT babylon.btcstaking.v1.Query.BTCDelegations >> /tmp/a &
time grpcurl -plaintext -d '{ "status": 4 }' $ENDPOINT babylon.btcstaking.v1.Query.BTCDelegations >> /tmp/a &
time grpcurl -plaintext -d '{ "status": 4 }' $ENDPOINT babylon.btcstaking.v1.Query.BTCDelegations >> /tmp/a

CPU Utilization

Disk Utilization

vmstat 1