Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rpc-server get halted when run on GCP Cloud run #586

Open
ausias-armesto opened this issue Nov 22, 2023 · 2 comments
Open

Rpc-server get halted when run on GCP Cloud run #586

ausias-armesto opened this issue Nov 22, 2023 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@ausias-armesto
Copy link
Contributor

ausias-armesto commented Nov 22, 2023

As part of the development of the new latency monitor some few changes need to be done on the rpc server

  • Investigate the error:
2023-11-21T23:01:58.406Z rpch:sdk:error error sending segment segment[rId: a2d1e6ee-be39-4dba-91ad-4b67f0d8468d, nr: 0, total: 3] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11) {
  cause: _SocketError: other side closed
      at Socket.onSocketEnd (node:internal/deps/undici/undici:7998:26)
      at Socket.emit (node:events:529:35)
      at endReadableNT (node:internal/streams/readable:1368:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    code: 'UND_ERR_SOCKET',
    socket: {
      localAddress: '169.254.8.1',
      localPort: 37162,
      remoteAddress: undefined,
      remotePort: undefined,
      remoteFamily: undefined,
      timeout: undefined,
      bytesWritten: 2606,
      bytesRead: 1543
      }
    }
  }
...
2023-11-21T22:59:32.439Z rpch:sdk:nodes-collector:verbose segmentFailed segment[rId: ce0ae86c-5227-46fd-be4d-af06191f785c, nr: 0, total: 3] .uZ2U[ping: 503ms, seg: 24(702ms)/28+2, msgs: 35(1445ms)/36, 2x: .RjnT[8(4034ms)/8+1], .hndt[1(4264ms)/1]]
2023-11-21T22:59:32.440Z rpch:sdk:nodes-collector:verbose requestFailed req[ce0ae86c-5227-46fd-be4d-af06191f785c,.uZ2U>.RjnT,https://gnosis-provider.rpch.tech] .uZ2U[ping: 503ms, seg: 24(702ms)/28+2, msgs: 35(1445ms)/36, 2x: .RjnT[8(4034ms)/9], .hndt[1(4264ms)/1]]
2023-11-21T22:59:32.539Z rpch:sdk:nodes-collector:verbose found fallback route pair .7M3R > .95K9 (via best message retrieval latency)
2023-11-21T22:59:36.048Z rpch:sdk:nodes-collector:verbose requestStarted req[deac2fdd-0d7d-4485-9544-4182f169db4b,.7M3R>.95K9,https://gnosis-provider.rpch.tech] .7M3R[ping: 311ms, seg: 0, msgs: 2(38ms)/2, 1x: .95K9[0+1]]
2023-11-21T22:59:36.048Z rpch:sdk resending request NaN for original ce0ae86c-5227-46fd-be4d-af06191f785c

Which is halting the server and not recovering
https://console.cloud.google.com/run/detail/europe-west3/latency-monitor/logs?project=rpch-375921
At 2023-11-21 23:59:32.439 CET

  • Create a /health GET http endpoint where an external service can check for the rpch-server health based on a 200 response.
@ausias-armesto ausias-armesto added the bug Something isn't working label Nov 22, 2023
@ausias-armesto ausias-armesto moved this from none to in progress in Sprint Planning Nov 22, 2023
@esterlus
Copy link
Member

Does GCP block the HTTP connection to the nodes because they are not HTTPS??

@ausias-armesto
Copy link
Contributor Author

Does GCP block the HTTP connection to the nodes because they are not HTTPS??

No. When the latency-monitor starts it works, but after some requests then it fails and crashes without restarting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants