502 causes stack restart #1033

jim-counter · 2022-05-17T19:46:52Z

jim-counter
May 17, 2022

I've noticed this happen a few times:

coordinator_service  | 2022-05-17T15:28:08.287Z <network> INFO retry to send transactions
coordinator_service  | 2022-05-17T15:33:04.798Z <network> INFO retry to send transactions
coordinator_service  | 2022-05-17T15:33:07.187Z <contract> ERROR failed to get indexing status for project: QmYR8xQgAXuCXMPGPVxxR91L4VtKZsozCM7Qsa5oAbyaQ3
coordinator_service  | 2022-05-17T15:33:13.603Z <transaction> INFO report project status: PROCESSING | project QmSzPQ9f4U1GERv | block height: 12725273 | mmrRoot: 0x6d3d9b9960de2b8ad59188d64ea8dd38d8e1c4c4cf90547e57f1088ec4702aa1
coordinator_service  | 2022-05-17T15:33:53.353Z <transaction> INFO report project status: SUCCEED
coordinator_service  | 2022-05-17T16:33:05.096Z <network> INFO retry to send transactions
coordinator_service  | 2022-05-17T16:33:43.456Z <transaction> INFO report project status: PROCESSING | project QmSzPQ9f4U1GERv | block height: 12725872 | mmrRoot: 0x4cb400181aa6c9acb731751509e5aec8edf568d23fc4bc93f5b88a7cc4295cc7
coordinator_service  | /usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/logger/lib/index.js:233
coordinator_service  |         var error = new Error(message);
coordinator_service  |                     ^
coordinator_service  |
coordinator_service  | Error: bad response (status=502, headers={"content-type":"text/plain; charset=utf-8","content-length":"93","connection":"close","date":"Tue, 17 May 2022 16:34:22 GMT","x-kong-upstream-latency":"2645","x-kong-proxy-latency":"0","via":"kong/2.1.4"}, body="i/o timeout. The gateway cannot get a response, please try again or contact the administrator", requestBody="{\"method\":\"eth_getTransactionByHash\",\"params\":[\"0x0a4e957280d66565b1a946bd79cf46f5aba6a8482029d7d07b4e1c438b279d0b\"],\"id\":16175,\"jsonrpc\":\"2.0\"}", requestMethod="POST", url="https://sqtn.api.onfinality.io/public", code=SERVER_ERROR, version=web/5.6.0)
coordinator_service  |     at Logger.makeError (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/logger/lib/index.js:233:21)
coordinator_service  |     at Logger.throwError (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/logger/lib/index.js:242:20)
coordinator_service  |     at /usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:267:36
coordinator_service  |     at step (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:33:23)
coordinator_service  |     at Object.next (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:14:53)
coordinator_service  |     at fulfilled (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:5:58)
coordinator_service  |     at runMicrotasks (<anonymous>)
coordinator_service  |     at processTicksAndRejections (node:internal/process/task_queues:96:5) {
coordinator_service  |   reason: 'bad response',
coordinator_service  |   code: 'SERVER_ERROR',
coordinator_service  |   status: 502,
coordinator_service  |   headers: {
coordinator_service  |     'content-type': 'text/plain; charset=utf-8',
coordinator_service  |     'content-length': '93',
coordinator_service  |     connection: 'close',
coordinator_service  |     date: 'Tue, 17 May 2022 16:34:22 GMT',
coordinator_service  |     'x-kong-upstream-latency': '2645',
coordinator_service  |     'x-kong-proxy-latency': '0',
coordinator_service  |     via: 'kong/2.1.4'
coordinator_service  |   },
coordinator_service  |   body: 'i/o timeout. The gateway cannot get a response, please try again or contact the administrator',
coordinator_service  |   requestBody: '{"method":"eth_getTransactionByHash","params":["0x0a4e957280d66565b1a946bd79cf46f5aba6a8482029d7d07b4e1c438b279d0b"],"id":16175,"jsonrpc":"2.0"}',
coordinator_service  |   requestMethod: 'POST',
coordinator_service  |   url: 'https://sqtn.api.onfinality.io/public'
coordinator_service  | }
coordinator_service  | 2022-05-17T16:34:29.156Z <nestjs> INFO Starting Nest application...
coordinator_service  | 2022-05-17T16:34:29.255Z <nestjs> INFO TypeOrmModule dependencies initialized
coordinator_service  | 2022-05-17T16:34:29.255Z <nestjs> INFO ConfigureModule dependencies initialized
coordinator_service  | 2022-05-17T16:34:29.255Z <nestjs> INFO AppModule dependencies initialized
coordinator_service  | 2022-05-17T16:34:29.256Z <nestjs> INFO ServeStaticModule dependencies initialized
coordinator_service  | 2022-05-17T16:34:29.256Z <nestjs> INFO GraphQLSchemaBuilderModule dependencies initialized
coordinator_service  | 2022-05-17T16:34:29.257Z <nestjs> INFO GraphQLModule dependencies initialized
coordinator_service  | 2022-05-17T16:34:29.492Z <nestjs> INFO TypeOrmCoreModule dependencies initialized
coordinator_service  | 2022-05-17T16:34:29.493Z <nestjs> INFO TypeOrmModule dependencies initialized
coordinator_service  | 2022-05-17T16:34:29.493Z <nestjs> INFO TypeOrmModule dependencies initialized
coordinator_service  | 2022-05-17T16:34:29.496Z <nestjs> INFO AccountModule dependencies initialized
coordinator_service  | 2022-05-17T16:34:29.501Z <nestjs> INFO ProjectModule dependencies initialized
coordinator_service  | 2022-05-17T16:34:29.551Z <nestjs> INFO AdminController {/}:
coordinator_service  | 2022-05-17T16:34:29.554Z <nestjs> INFO Mapped {/env.js, GET} route
coordinator_service  | 2022-05-17T16:34:29.864Z <nestjs> INFO Nest application successfully started
coordinator_service  | 2022-05-17T16:34:29.872Z <indexer-coordinator> INFO coordinator service started
coordinator_service  | 2022-05-17T16:34:29.873Z <indexer-admin> INFO indexer admin app started
coordinator_service  | 2022-05-17T16:34:46.509Z <network> INFO transaction interval: 300000

It looks like a 502 when calling eth_getTransactionByHash causes a restart.

I also noticed that my project nodes were not happy afterwards (complaining of errors). A restart of the whole stack seemed to fix everything.

Is this by design? Is there scope to add a retry mechanism or is that dangerous in this scenario?

mzxyz · 2022-05-18T01:53:44Z

mzxyz
May 18, 2022
Maintainer

@jim-counter Thanks for report this issue, a couple questions for this issue:

What's the version for the running coordinator service? v0.1.9 or v0.1.10.
What kind of error for the project nodes and what's the subql-node image version for the project?

Basing on the provided log, the failed transaction make the coordinator restart which is a feature other than a bug, we need to make sure the service can keep running even got crash issue.

However, this failed issue should be handled properly, will have an investigation for it.

1 reply

jim-counter May 18, 2022
Author

Hi, at the time I saw this I was running onfinality/subql-coordinator:v0.1.9 and onfinality/subql-node:v1.0.0. I've now upgraded to onfinality/subql-coordinator:v0.1.10 and will report back if/when I see it again.

Query node output.

Polkadot

2022-05-17T16:01:29.409Z <health> ERROR Indexer is not healthy
2022-05-17T16:01:35.751Z <benchmark> INFO NaN bps, target: #10346053, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:01:39.430Z <health> ERROR Indexer is not healthy
2022-05-17T16:01:49.454Z <health> ERROR Indexer is not healthy
2022-05-17T16:01:50.751Z <benchmark> INFO NaN bps, target: #10346055, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:01:59.474Z <health> ERROR Indexer is not healthy
2022-05-17T16:02:05.751Z <benchmark> INFO NaN bps, target: #10346058, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:02:09.485Z <health> ERROR Indexer is not healthy
2022-05-17T16:02:19.505Z <health> ERROR Indexer is not healthy
2022-05-17T16:02:20.752Z <benchmark> INFO NaN bps, target: #10346061, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:02:29.520Z <health> ERROR Indexer is not healthy
2022-05-17T16:02:35.751Z <benchmark> INFO NaN bps, target: #10346063, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:02:39.534Z <health> ERROR Indexer is not healthy
2022-05-17T16:02:49.545Z <health> ERROR Indexer is not healthy
2022-05-17T16:02:50.752Z <benchmark> INFO NaN bps, target: #10346065, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:02:59.552Z <health> ERROR Indexer is not healthy
2022-05-17T16:03:05.751Z <benchmark> INFO NaN bps, target: #10346068, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:03:09.564Z <health> ERROR Indexer is not healthy
2022-05-17T16:03:19.573Z <health> ERROR Indexer is not healthy
2022-05-17T16:03:20.752Z <benchmark> INFO NaN bps, target: #10346070, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:03:29.584Z <health> ERROR Indexer is not healthy
2022-05-17T16:03:35.752Z <benchmark> INFO NaN bps, target: #10346073, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:03:39.595Z <health> ERROR Indexer is not healthy
2022-05-17T16:03:49.611Z <health> ERROR Indexer is not healthy
2022-05-17T16:03:50.752Z <benchmark> INFO NaN bps, target: #10346075, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:03:59.621Z <health> ERROR Indexer is not healthy
2022-05-17T16:04:05.752Z <benchmark> INFO NaN bps, target: #10346076, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:04:09.637Z <health> ERROR Indexer is not healthy
2022-05-17T16:04:19.645Z <health> ERROR Indexer is not healthy
2022-05-17T16:04:20.753Z <benchmark> INFO NaN bps, target: #10346080, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:04:29.664Z <health> ERROR Indexer is not healthy
2022-05-17T16:04:35.752Z <benchmark> INFO NaN bps, target: #10346083, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:04:39.674Z <health> ERROR Indexer is not healthy
2022-05-17T16:04:49.689Z <health> ERROR Indexer is not healthy
2022-05-17T16:04:50.752Z <benchmark> INFO NaN bps, target: #10346085, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:04:59.705Z <health> ERROR Indexer is not healthy
2022-05-17T16:05:05.753Z <benchmark> INFO NaN bps, target: #10346088, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:05:09.716Z <health> ERROR Indexer is not healthy
2022-05-17T16:05:19.728Z <health> ERROR Indexer is not healthy
2022-05-17T16:05:20.752Z <benchmark> INFO NaN bps, target: #10346089, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:05:29.746Z <health> ERROR Indexer is not healthy
2022-05-17T16:05:35.753Z <benchmark> INFO NaN bps, target: #10346092, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:05:39.759Z <health> ERROR Indexer is not healthy
2022-05-17T16:05:49.769Z <health> ERROR Indexer is not healthy
2022-05-17T16:05:50.754Z <benchmark> INFO NaN bps, target: #10346095, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:05:59.783Z <health> ERROR Indexer is not healthy
2022-05-17T16:06:05.755Z <benchmark> INFO NaN bps, target: #10346098, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:06:09.793Z <health> ERROR Indexer is not healthy
2022-05-17T16:06:19.810Z <health> ERROR Indexer is not healthy
2022-05-17T16:06:20.754Z <benchmark> INFO NaN bps, target: #10346101, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:06:29.828Z <health> ERROR Indexer is not healthy
2022-05-17T16:06:35.755Z <benchmark> INFO NaN bps, target: #10346103, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:06:39.844Z <health> ERROR Indexer is not healthy
2022-05-17T16:06:49.862Z <health> ERROR Indexer is not healthy
2022-05-17T16:06:50.755Z <benchmark> INFO NaN bps, target: #10346106, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:06:59.876Z <health> ERROR Indexer is not healthy
2022-05-17T16:07:05.755Z <benchmark> INFO NaN bps, target: #10346108, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:07:09.894Z <health> ERROR Indexer is not healthy
2022-05-17T16:07:19.910Z <health> ERROR Indexer is not healthy
2022-05-17T16:07:20.755Z <benchmark> INFO NaN bps, target: #10346109, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:07:29.918Z <health> ERROR Indexer is not healthy
2022-05-17T16:07:35.755Z <benchmark> INFO NaN bps, target: #10346113, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:07:39.936Z <health> ERROR Indexer is not healthy
2022-05-17T16:07:49.950Z <health> ERROR Indexer is not healthy
2022-05-17T16:07:50.761Z <benchmark> INFO NaN bps, target: #10346115, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:07:59.966Z <health> ERROR Indexer is not healthy
2022-05-17T16:08:05.757Z <benchmark> INFO NaN bps, target: #10346117, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:08:09.976Z <health> ERROR Indexer is not healthy
2022-05-17T16:08:19.984Z <health> ERROR Indexer is not healthy
2022-05-17T16:08:20.756Z <benchmark> INFO NaN bps, target: #10346121, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:08:30.002Z <health> ERROR Indexer is not healthy
2022-05-17T16:08:35.756Z <benchmark> INFO NaN bps, target: #10346123, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:08:40.015Z <health> ERROR Indexer is not healthy
2022-05-17T16:08:50.035Z <health> ERROR Indexer is not healthy
2022-05-17T16:08:50.764Z <benchmark> INFO NaN bps, target: #10346126, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:09:00.055Z <health> ERROR Indexer is not healthy
2022-05-17T16:09:05.764Z <benchmark> INFO NaN bps, target: #10346128, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:09:10.079Z <health> ERROR Indexer is not healthy
2022-05-17T16:09:20.103Z <health> ERROR Indexer is not healthy
2022-05-17T16:09:20.765Z <benchmark> INFO NaN bps, target: #10346130, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:09:30.119Z <health> ERROR Indexer is not healthy
2022-05-17T16:09:35.767Z <benchmark> INFO NaN bps, target: #10346133, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:09:40.135Z <health> ERROR Indexer is not healthy
2022-05-17T16:09:50.151Z <health> ERROR Indexer is not healthy
2022-05-17T16:09:50.765Z <benchmark> INFO NaN bps, target: #10346136, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:10:00.160Z <health> ERROR Indexer is not healthy
2022-05-17T16:10:05.794Z <benchmark> INFO NaN bps, target: #10346138, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:10:10.185Z <health> ERROR Indexer is not healthy
2022-05-17T16:10:20.206Z <health> ERROR Indexer is not healthy
2022-05-17T16:10:20.794Z <benchmark> INFO NaN bps, target: #10346141, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:10:30.225Z <health> ERROR Indexer is not healthy
2022-05-17T16:10:35.795Z <benchmark> INFO NaN bps, target: #10346142, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:10:40.238Z <health> ERROR Indexer is not healthy
2022-05-17T16:10:50.257Z <health> ERROR Indexer is not healthy
2022-05-17T16:10:50.794Z <benchmark> INFO NaN bps, target: #10346146, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:11:00.272Z <health> ERROR Indexer is not healthy
2022-05-17T16:11:05.810Z <benchmark> INFO NaN bps, target: #10346148, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:11:10.279Z <health> ERROR Indexer is not healthy
2022-05-17T16:11:20.286Z <health> ERROR Indexer is not healthy
2022-05-17T16:11:20.811Z <benchmark> INFO NaN bps, target: #10346150, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:11:30.307Z <health> ERROR Indexer is not healthy
2022-05-17T16:11:35.811Z <benchmark> INFO NaN bps, target: #10346152, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:11:40.315Z <health> ERROR Indexer is not healthy
2022-05-17T16:11:50.328Z <health> ERROR Indexer is not healthy
2022-05-17T16:11:50.810Z <benchmark> INFO NaN bps, target: #10346156, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:12:00.347Z <health> ERROR Indexer is not healthy
2022-05-17T16:12:05.810Z <benchmark> INFO NaN bps, target: #10346158, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:12:10.365Z <health> ERROR Indexer is not healthy
2022-05-17T16:12:20.375Z <health> ERROR Indexer is not healthy
2022-05-17T16:12:20.810Z <benchmark> INFO NaN bps, target: #10346161, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:12:30.391Z <health> ERROR Indexer is not healthy
2022-05-17T16:12:35.811Z <benchmark> INFO NaN bps, target: #10346163, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:12:40.405Z <health> ERROR Indexer is not healthy
2022-05-17T16:12:50.423Z <health> ERROR Indexer is not healthy
2022-05-17T16:12:50.810Z <benchmark> INFO NaN bps, target: #10346165, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:13:00.443Z <health> ERROR Indexer is not healthy
2022-05-17T16:13:05.811Z <benchmark> INFO NaN bps, target: #10346168, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:13:10.456Z <health> ERROR Indexer is not healthy
2022-05-17T16:13:20.466Z <health> ERROR Indexer is not healthy
2022-05-17T16:13:20.811Z <benchmark> INFO NaN bps, target: #10346170, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:13:30.477Z <health> ERROR Indexer is not healthy
2022-05-17T16:13:35.811Z <benchmark> INFO NaN bps, target: #10346173, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:13:40.486Z <health> ERROR Indexer is not healthy
2022-05-17T16:13:50.509Z <health> ERROR Indexer is not healthy
2022-05-17T16:13:50.811Z <benchmark> INFO NaN bps, target: #10346175, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:14:00.525Z <health> ERROR Indexer is not healthy
2022-05-17T16:14:05.811Z <benchmark> INFO NaN bps, target: #10346178, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:14:10.538Z <health> ERROR Indexer is not healthy
2022-05-17T16:14:20.550Z <health> ERROR Indexer is not healthy
2022-05-17T16:14:20.812Z <benchmark> INFO NaN bps, target: #10346180, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:14:30.567Z <health> ERROR Indexer is not healthy
2022-05-17T16:14:35.811Z <benchmark> INFO NaN bps, target: #10346182, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:14:40.580Z <health> ERROR Indexer is not healthy
2022-05-17T16:14:50.588Z <health> ERROR Indexer is not healthy
2022-05-17T16:14:50.812Z <benchmark> INFO NaN bps, target: #10346186, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:15:00.604Z <health> ERROR Indexer is not healthy
2022-05-17T16:15:05.811Z <benchmark> INFO NaN bps, target: #10346189, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:15:10.622Z <health> ERROR Indexer is not healthy
2022-05-17T16:15:20.638Z <health> ERROR Indexer is not healthy
2022-05-17T16:15:20.811Z <benchmark> INFO NaN bps, target: #10346190, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:15:30.658Z <health> ERROR Indexer is not healthy
2022-05-17T16:15:35.812Z <benchmark> INFO NaN bps, target: #10346194, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:15:40.665Z <health> ERROR Indexer is not healthy
2022-05-17T16:15:50.673Z <health> ERROR Indexer is not healthy
2022-05-17T16:15:50.812Z <benchmark> INFO NaN bps, target: #10346196, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:16:00.690Z <health> ERROR Indexer is not healthy
2022-05-17T16:16:05.819Z <benchmark> INFO NaN bps, target: #10346198, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:16:10.712Z <health> ERROR Indexer is not healthy
2022-05-17T16:16:20.729Z <health> ERROR Indexer is not healthy
2022-05-17T16:16:20.819Z <benchmark> INFO NaN bps, target: #10346201, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:16:30.744Z <health> ERROR Indexer is not healthy
2022-05-17T16:16:35.819Z <benchmark> INFO NaN bps, target: #10346203, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:16:40.775Z <health> ERROR Indexer is not healthy
2022-05-17T16:16:50.797Z <health> ERROR Indexer is not healthy
2022-05-17T16:16:50.820Z <benchmark> INFO NaN bps, target: #10346206, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:17:00.814Z <health> ERROR Indexer is not healthy
2022-05-17T16:17:05.820Z <benchmark> INFO NaN bps, target: #10346208, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:17:10.824Z <health> ERROR Indexer is not healthy
2022-05-17T16:17:20.820Z <benchmark> INFO NaN bps, target: #10346211, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:17:20.841Z <health> ERROR Indexer is not healthy
2022-05-17T16:17:30.859Z <health> ERROR Indexer is not healthy
2022-05-17T16:17:35.820Z <benchmark> INFO NaN bps, target: #10346212, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:17:40.874Z <health> ERROR Indexer is not healthy
2022-05-17T16:17:50.819Z <benchmark> INFO NaN bps, target: #10346216, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:17:50.887Z <health> ERROR Indexer is not healthy
2022-05-17T16:18:00.906Z <health> ERROR Indexer is not healthy
2022-05-17T16:18:05.819Z <benchmark> INFO NaN bps, target: #10346217, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:18:10.919Z <health> ERROR Indexer is not healthy
2022-05-17T16:18:20.820Z <benchmark> INFO NaN bps, target: #10346221, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:18:20.927Z <health> ERROR Indexer is not healthy
2022-05-17T16:18:30.945Z <health> ERROR Indexer is not healthy
2022-05-17T16:18:35.819Z <benchmark> INFO NaN bps, target: #10346222, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:18:40.967Z <health> ERROR Indexer is not healthy
2022-05-17T16:18:50.820Z <benchmark> INFO NaN bps, target: #10346225, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:18:50.979Z <health> ERROR Indexer is not healthy
2022-05-17T16:19:00.996Z <health> ERROR Indexer is not healthy
2022-05-17T16:19:05.821Z <benchmark> INFO NaN bps, target: #10346228, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:19:11.018Z <health> ERROR Indexer is not healthy
2022-05-17T16:19:20.821Z <benchmark> INFO NaN bps, target: #10346231, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:19:21.030Z <health> ERROR Indexer is not healthy
2022-05-17T16:19:31.046Z <health> ERROR Indexer is not healthy
2022-05-17T16:19:35.820Z <benchmark> INFO NaN bps, target: #10346231, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:19:41.054Z <health> ERROR Indexer is not healthy
2022-05-17T16:19:50.820Z <benchmark> INFO NaN bps, target: #10346235, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:19:51.069Z <health> ERROR Indexer is not healthy
2022-05-17T16:20:01.089Z <health> ERROR Indexer is not healthy
2022-05-17T16:20:05.820Z <benchmark> INFO NaN bps, target: #10346237, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:20:11.104Z <health> ERROR Indexer is not healthy
2022-05-17T16:20:20.820Z <benchmark> INFO NaN bps, target: #10346239, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:20:21.122Z <health> ERROR Indexer is not healthy
2022-05-17T16:20:31.133Z <health> ERROR Indexer is not healthy
2022-05-17T16:20:35.820Z <benchmark> INFO NaN bps, target: #10346242, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:20:41.139Z <health> ERROR Indexer is not healthy
2022-05-17T16:20:50.820Z <benchmark> INFO NaN bps, target: #10346244, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:20:51.156Z <health> ERROR Indexer is not healthy
2022-05-17T16:21:01.176Z <health> ERROR Indexer is not healthy
2022-05-17T16:21:05.820Z <benchmark> INFO NaN bps, target: #10346248, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:21:11.193Z <health> ERROR Indexer is not healthy
2022-05-17T16:21:20.820Z <benchmark> INFO NaN bps, target: #10346251, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:21:21.203Z <health> ERROR Indexer is not healthy
2022-05-17T16:21:31.219Z <health> ERROR Indexer is not healthy
2022-05-17T16:21:35.820Z <benchmark> INFO NaN bps, target: #10346252, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:21:41.228Z <health> ERROR Indexer is not healthy
2022-05-17T16:21:50.820Z <benchmark> INFO NaN bps, target: #10346255, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:21:51.234Z <health> ERROR Indexer is not healthy
2022-05-17T16:22:01.250Z <health> ERROR Indexer is not healthy
2022-05-17T16:22:05.820Z <benchmark> INFO NaN bps, target: #10346258, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:22:11.267Z <health> ERROR Indexer is not healthy
2022-05-17T16:22:20.821Z <benchmark> INFO NaN bps, target: #10346259, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:22:21.275Z <health> ERROR Indexer is not healthy
2022-05-17T16:22:31.295Z <health> ERROR Indexer is not healthy
2022-05-17T16:22:35.820Z <benchmark> INFO NaN bps, target: #10346263, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:22:41.307Z <health> ERROR Indexer is not healthy
2022-05-17T16:22:50.821Z <benchmark> INFO NaN bps, target: #10346265, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:22:51.324Z <health> ERROR Indexer is not healthy
2022-05-17T16:23:01.341Z <health> ERROR Indexer is not healthy
2022-05-17T16:23:05.820Z <benchmark> INFO NaN bps, target: #10346268, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:23:11.361Z <health> ERROR Indexer is not healthy
2022-05-17T16:23:20.821Z <benchmark> INFO NaN bps, target: #10346271, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:23:21.373Z <health> ERROR Indexer is not healthy
2022-05-17T16:23:31.386Z <health> ERROR Indexer is not healthy
2022-05-17T16:23:35.821Z <benchmark> INFO NaN bps, target: #10346272, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:23:41.397Z <health> ERROR Indexer is not healthy
2022-05-17T16:23:50.821Z <benchmark> INFO NaN bps, target: #10346276, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:23:51.412Z <health> ERROR Indexer is not healthy
2022-05-17T16:24:01.431Z <health> ERROR Indexer is not healthy
2022-05-17T16:24:05.821Z <benchmark> INFO NaN bps, target: #10346277, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:24:11.437Z <health> ERROR Indexer is not healthy
2022-05-17T16:24:20.821Z <benchmark> INFO NaN bps, target: #10346281, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:24:21.455Z <health> ERROR Indexer is not healthy
2022-05-17T16:24:31.466Z <health> ERROR Indexer is not healthy
2022-05-17T16:24:35.821Z <benchmark> INFO NaN bps, target: #10346283, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:24:41.482Z <health> ERROR Indexer is not healthy
2022-05-17T16:24:50.822Z <benchmark> INFO NaN bps, target: #10346285, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:24:51.498Z <health> ERROR Indexer is not healthy
2022-05-17T16:25:01.522Z <health> ERROR Indexer is not healthy
2022-05-17T16:25:05.822Z <benchmark> INFO NaN bps, target: #10346288, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:25:11.538Z <health> ERROR Indexer is not healthy
2022-05-17T16:25:20.825Z <benchmark> INFO NaN bps, target: #10346291, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:25:21.557Z <health> ERROR Indexer is not healthy
2022-05-17T16:25:31.579Z <health> ERROR Indexer is not healthy
2022-05-17T16:25:35.826Z <benchmark> INFO NaN bps, target: #10346293, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:25:41.599Z <health> ERROR Indexer is not healthy
2022-05-17T16:25:50.826Z <benchmark> INFO NaN bps, target: #10346295, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:25:51.619Z <health> ERROR Indexer is not healthy
2022-05-17T16:26:01.636Z <health> ERROR Indexer is not healthy
2022-05-17T16:26:05.826Z <benchmark> INFO NaN bps, target: #10346298, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:26:11.654Z <health> ERROR Indexer is not healthy
2022-05-17T16:26:20.827Z <benchmark> INFO NaN bps, target: #10346300, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:26:21.662Z <health> ERROR Indexer is not healthy
2022-05-17T16:26:31.675Z <health> ERROR Indexer is not healthy
2022-05-17T16:26:35.827Z <benchmark> INFO NaN bps, target: #10346303, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:26:41.697Z <health> ERROR Indexer is not healthy
2022-05-17T16:26:50.827Z <benchmark> INFO NaN bps, target: #10346305, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:26:51.719Z <health> ERROR Indexer is not healthy
2022-05-17T16:27:01.733Z <health> ERROR Indexer is not healthy
2022-05-17T16:27:05.826Z <benchmark> INFO NaN bps, target: #10346308, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:27:11.750Z <health> ERROR Indexer is not healthy
2022-05-17T16:27:20.826Z <benchmark> INFO NaN bps, target: #10346311, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:27:21.760Z <health> ERROR Indexer is not healthy
2022-05-17T16:27:31.776Z <health> ERROR Indexer is not healthy
2022-05-17T16:27:35.826Z <benchmark> INFO NaN bps, target: #10346312, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:27:41.784Z <health> ERROR Indexer is not healthy
2022-05-17T16:27:50.826Z <benchmark> INFO NaN bps, target: #10346316, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:27:51.810Z <health> ERROR Indexer is not healthy
2022-05-17T16:28:01.828Z <health> ERROR Indexer is not healthy
2022-05-17T16:28:05.827Z <benchmark> INFO NaN bps, target: #10346318, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:28:11.843Z <health> ERROR Indexer is not healthy
2022-05-17T16:28:20.827Z <benchmark> INFO NaN bps, target: #10346320, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:28:21.860Z <health> ERROR Indexer is not healthy
2022-05-17T16:28:31.876Z <health> ERROR Indexer is not healthy
2022-05-17T16:28:35.826Z <benchmark> INFO NaN bps, target: #10346324, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:28:41.892Z <health> ERROR Indexer is not healthy
2022-05-17T16:28:50.827Z <benchmark> INFO NaN bps, target: #10346326, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:28:51.910Z <health> ERROR Indexer is not healthy
2022-05-17T16:29:01.923Z <health> ERROR Indexer is not healthy
2022-05-17T16:29:05.826Z <benchmark> INFO NaN bps, target: #10346326, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:29:11.943Z <health> ERROR Indexer is not healthy
2022-05-17T16:29:20.827Z <benchmark> INFO NaN bps, target: #10346330, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:29:21.961Z <health> ERROR Indexer is not healthy
2022-05-17T16:29:31.969Z <health> ERROR Indexer is not healthy
2022-05-17T16:29:35.828Z <benchmark> INFO NaN bps, target: #10346333, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:29:41.985Z <health> ERROR Indexer is not healthy
2022-05-17T16:29:50.827Z <benchmark> INFO NaN bps, target: #10346334, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:29:52.006Z <health> ERROR Indexer is not healthy
2022-05-17T16:30:02.019Z <health> ERROR Indexer is not healthy
2022-05-17T16:30:05.827Z <benchmark> INFO NaN bps, target: #10346338, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:30:12.038Z <health> ERROR Indexer is not healthy
2022-05-17T16:30:20.827Z <benchmark> INFO NaN bps, target: #10346341, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:30:22.048Z <health> ERROR Indexer is not healthy
2022-05-17T16:30:32.073Z <health> ERROR Indexer is not healthy
2022-05-17T16:30:35.827Z <benchmark> INFO NaN bps, target: #10346343, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:30:42.083Z <health> ERROR Indexer is not healthy
2022-05-17T16:30:50.828Z <benchmark> INFO NaN bps, target: #10346346, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:30:52.100Z <health> ERROR Indexer is not healthy
2022-05-17T16:31:02.113Z <health> ERROR Indexer is not healthy
2022-05-17T16:31:05.828Z <benchmark> INFO NaN bps, target: #10346347, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:31:12.119Z <health> ERROR Indexer is not healthy
2022-05-17T16:31:20.828Z <benchmark> INFO NaN bps, target: #10346351, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:31:22.132Z <health> ERROR Indexer is not healthy
2022-05-17T16:31:32.149Z <health> ERROR Indexer is not healthy
2022-05-17T16:31:35.829Z <benchmark> INFO NaN bps, target: #10346353, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:31:42.167Z <health> ERROR Indexer is not healthy
2022-05-17T16:31:50.830Z <benchmark> INFO NaN bps, target: #10346355, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:31:52.181Z <health> ERROR Indexer is not healthy
2022-05-17T16:32:02.197Z <health> ERROR Indexer is not healthy
2022-05-17T16:32:05.830Z <benchmark> INFO NaN bps, target: #10346359, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:32:12.218Z <health> ERROR Indexer is not healthy
2022-05-17T16:32:20.830Z <benchmark> INFO NaN bps, target: #10346361, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:32:22.236Z <health> ERROR Indexer is not healthy
2022-05-17T16:32:32.255Z <health> ERROR Indexer is not healthy
2022-05-17T16:32:35.831Z <benchmark> INFO NaN bps, target: #10346361, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:32:42.272Z <health> ERROR Indexer is not healthy
2022-05-17T16:32:50.830Z <benchmark> INFO NaN bps, target: #10346366, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:32:52.283Z <health> ERROR Indexer is not healthy
2022-05-17T16:33:02.292Z <health> ERROR Indexer is not healthy
2022-05-17T16:33:05.830Z <benchmark> INFO NaN bps, target: #10346368, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:33:12.299Z <health> ERROR Indexer is not healthy
2022-05-17T16:33:20.831Z <benchmark> INFO NaN bps, target: #10346370, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:33:22.308Z <health> ERROR Indexer is not healthy
2022-05-17T16:33:32.325Z <health> ERROR Indexer is not healthy
2022-05-17T16:33:35.831Z <benchmark> INFO NaN bps, target: #10346373, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:33:42.342Z <health> ERROR Indexer is not healthy
2022-05-17T16:33:50.830Z <benchmark> INFO NaN bps, target: #10346375, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:33:52.352Z <health> ERROR Indexer is not healthy
2022-05-17T16:34:02.365Z <health> ERROR Indexer is not healthy
2022-05-17T16:34:05.830Z <benchmark> INFO NaN bps, target: #10346378, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:34:12.373Z <health> ERROR Indexer is not healthy
2022-05-17T16:34:20.830Z <benchmark> INFO NaN bps, target: #10346379, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:34:22.393Z <health> ERROR Indexer is not healthy
2022-05-17T16:34:32.417Z <health> ERROR Indexer is not healthy
2022-05-17T16:34:35.831Z <benchmark> INFO NaN bps, target: #10346382, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:34:42.433Z <health> ERROR Indexer is not healthy
2022-05-17T16:34:50.831Z <benchmark> INFO NaN bps, target: #10346385, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:34:52.451Z <health> ERROR Indexer is not healthy
2022-05-17T16:35:02.470Z <health> ERROR Indexer is not healthy
2022-05-17T16:35:05.831Z <benchmark> INFO NaN bps, target: #10346388, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:35:12.488Z <health> ERROR Indexer is not healthy
2022-05-17T16:35:20.832Z <benchmark> INFO NaN bps, target: #10346390, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:35:22.501Z <health> ERROR Indexer is not healthy
2022-05-17T16:35:32.513Z <health> ERROR Indexer is not healthy
2022-05-17T16:35:35.831Z <benchmark> INFO NaN bps, target: #10346392, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:35:42.528Z <health> ERROR Indexer is not healthy
2022-05-17T16:35:50.831Z <benchmark> INFO NaN bps, target: #10346395, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:35:52.542Z <health> ERROR Indexer is not healthy
2022-05-17T16:36:02.560Z <health> ERROR Indexer is not healthy
2022-05-17T16:36:05.832Z <benchmark> INFO NaN bps, target: #10346396, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:36:12.572Z <health> ERROR Indexer is not healthy
2022-05-17T16:36:20.832Z <benchmark> INFO NaN bps, target: #10346399, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:36:22.585Z <health> ERROR Indexer is not healthy
2022-05-17T16:36:32.603Z <health> ERROR Indexer is not healthy
2022-05-17T16:36:35.834Z <benchmark> INFO NaN bps, target: #10346402, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:36:42.622Z <health> ERROR Indexer is not healthy
2022-05-17T16:36:50.833Z <benchmark> INFO NaN bps, target: #10346405, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:36:52.637Z <health> ERROR Indexer is not healthy
2022-05-17T16:37:02.653Z <health> ERROR Indexer is not healthy
2022-05-17T16:37:05.833Z <benchmark> INFO NaN bps, target: #10346407, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:37:12.669Z <health> ERROR Indexer is not healthy
2022-05-17T16:37:20.834Z <benchmark> INFO NaN bps, target: #10346408, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:37:22.684Z <health> ERROR Indexer is not healthy
2022-05-17T16:37:32.698Z <health> ERROR Indexer is not healthy
2022-05-17T16:37:35.834Z <benchmark> INFO NaN bps, target: #10346412, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:37:42.704Z <health> ERROR Indexer is not healthy
2022-05-17T16:37:50.836Z <benchmark> INFO NaN bps, target: #10346415, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:37:52.709Z <health> ERROR Indexer is not healthy
2022-05-17T16:38:02.728Z <health> ERROR Indexer is not healthy
2022-05-17T16:38:05.837Z <benchmark> INFO NaN bps, target: #10346416, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:38:12.744Z <health> ERROR Indexer is not healthy
2022-05-17T16:38:20.838Z <benchmark> INFO NaN bps, target: #10346420, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:38:22.767Z <health> ERROR Indexer is not healthy
2022-05-17T16:38:32.786Z <health> ERROR Indexer is not healthy
2022-05-17T16:38:35.837Z <benchmark> INFO NaN bps, target: #10346422, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:38:42.803Z <health> ERROR Indexer is not healthy
2022-05-17T16:38:50.837Z <benchmark> INFO NaN bps, target: #10346424, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:38:52.811Z <health> ERROR Indexer is not healthy
2022-05-17T16:39:02.823Z <health> ERROR Indexer is not healthy
2022-05-17T16:39:05.837Z <benchmark> INFO NaN bps, target: #10346427, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:39:12.834Z <health> ERROR Indexer is not healthy
2022-05-17T16:39:20.838Z <benchmark> INFO NaN bps, target: #10346429, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:39:22.858Z <health> ERROR Indexer is not healthy
2022-05-17T16:39:32.878Z <health> ERROR Indexer is not healthy
2022-05-17T16:39:35.837Z <benchmark> INFO NaN bps, target: #10346432, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:39:42.905Z <health> ERROR Indexer is not healthy
2022-05-17T16:39:50.837Z <benchmark> INFO NaN bps, target: #10346435, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:39:52.928Z <health> ERROR Indexer is not healthy
2022-05-17T16:40:02.950Z <health> ERROR Indexer is not healthy
2022-05-17T16:40:05.838Z <benchmark> INFO NaN bps, target: #10346436, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:40:12.965Z <health> ERROR Indexer is not healthy
2022-05-17T16:40:20.837Z <benchmark> INFO NaN bps, target: #10346440, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:40:22.980Z <health> ERROR Indexer is not healthy
2022-05-17T16:40:32.988Z <health> ERROR Indexer is not healthy
2022-05-17T16:40:35.838Z <benchmark> INFO NaN bps, target: #10346442, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:40:43.003Z <health> ERROR Indexer is not healthy
2022-05-17T16:40:50.837Z <benchmark> INFO NaN bps, target: #10346444, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:40:53.015Z <health> ERROR Indexer is not healthy
2022-05-17T16:41:03.031Z <health> ERROR Indexer is not healthy
2022-05-17T16:41:05.837Z <benchmark> INFO NaN bps, target: #10346447, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:41:13.038Z <health> ERROR Indexer is not healthy
2022-05-17T16:41:20.837Z <benchmark> INFO NaN bps, target: #10346450, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:41:23.053Z <health> ERROR Indexer is not healthy
2022-05-17T16:41:33.067Z <health> ERROR Indexer is not healthy
2022-05-17T16:41:35.838Z <benchmark> INFO NaN bps, target: #10346452, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:41:43.093Z <health> ERROR Indexer is not healthy
2022-05-17T16:41:50.837Z <benchmark> INFO NaN bps, target: #10346455, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:41:53.112Z <health> ERROR Indexer is not healthy
2022-05-17T16:42:03.130Z <health> ERROR Indexer is not healthy
2022-05-17T16:42:05.838Z <benchmark> INFO NaN bps, target: #10346456, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:42:13.146Z <health> ERROR Indexer is not healthy
2022-05-17T16:42:20.839Z <benchmark> INFO NaN bps, target: #10346458, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:42:23.159Z <health> ERROR Indexer is not healthy
2022-05-17T16:42:33.168Z <health> ERROR Indexer is not healthy
2022-05-17T16:42:35.839Z <benchmark> INFO NaN bps, target: #10346462, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:42:43.183Z <health> ERROR Indexer is not healthy
2022-05-17T16:42:50.839Z <benchmark> INFO NaN bps, target: #10346463, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:42:53.200Z <health> ERROR Indexer is not healthy
2022-05-17T16:43:03.208Z <health> ERROR Indexer is not healthy
2022-05-17T16:43:05.839Z <benchmark> INFO NaN bps, target: #10346465, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:43:13.223Z <health> ERROR Indexer is not healthy
2022-05-17T16:43:20.839Z <benchmark> INFO NaN bps, target: #10346468, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:43:23.235Z <health> ERROR Indexer is not healthy
2022-05-17T16:43:33.249Z <health> ERROR Indexer is not healthy
2022-05-17T16:43:35.839Z <benchmark> INFO NaN bps, target: #10346471, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:43:43.258Z <health> ERROR Indexer is not healthy
2022-05-17T16:43:50.839Z <benchmark> INFO NaN bps, target: #10346474, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:43:53.273Z <health> ERROR Indexer is not healthy
2022-05-17T16:44:03.287Z <health> ERROR Indexer is not healthy
2022-05-17T16:44:05.839Z <benchmark> INFO NaN bps, target: #10346476, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:44:13.305Z <health> ERROR Indexer is not healthy
2022-05-17T16:44:20.840Z <benchmark> INFO NaN bps, target: #10346478, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:44:23.320Z <health> ERROR Indexer is not healthy
2022-05-17T16:44:33.333Z <health> ERROR Indexer is not healthy
2022-05-17T16:44:35.841Z <benchmark> INFO NaN bps, target: #10346480, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:44:43.342Z <health> ERROR Indexer is not healthy
2022-05-17T16:44:50.840Z <benchmark> INFO NaN bps, target: #10346484, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:44:53.357Z <health> ERROR Indexer is not healthy
2022-05-17T16:45:03.374Z <health> ERROR Indexer is not healthy
2022-05-17T16:45:05.840Z <benchmark> INFO NaN bps, target: #10346486, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:45:13.386Z <health> ERROR Indexer is not healthy
2022-05-17T16:45:20.841Z <benchmark> INFO NaN bps, target: #10346489, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:45:23.404Z <health> ERROR Indexer is not healthy
2022-05-17T16:45:33.417Z <health> ERROR Indexer is not healthy
2022-05-17T16:45:35.841Z <benchmark> INFO NaN bps, target: #10346490, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:45:43.425Z <health> ERROR Indexer is not healthy
2022-05-17T16:45:50.841Z <benchmark> INFO NaN bps, target: #10346493, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:45:53.434Z <health> ERROR Indexer is not healthy
2022-05-17T16:46:03.449Z <health> ERROR Indexer is not healthy
2022-05-17T16:46:05.842Z <benchmark> INFO NaN bps, target: #10346493, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:46:13.466Z <health> ERROR Indexer is not healthy
2022-05-17T16:46:20.842Z <benchmark> INFO NaN bps, target: #10346499, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:46:23.472Z <health> ERROR Indexer is not healthy
2022-05-17T16:46:33.488Z <health> ERROR Indexer is not healthy
2022-05-17T16:46:35.842Z <benchmark> INFO NaN bps, target: #10346499, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:46:43.502Z <health> ERROR Indexer is not healthy
2022-05-17T16:46:50.843Z <benchmark> INFO NaN bps, target: #10346503, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:46:53.510Z <health> ERROR Indexer is not healthy
2022-05-17T16:47:03.516Z <health> ERROR Indexer is not healthy
2022-05-17T16:47:05.842Z <benchmark> INFO NaN bps, target: #10346505, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:47:13.529Z <health> ERROR Indexer is not healthy
2022-05-17T16:47:20.842Z <benchmark> INFO NaN bps, target: #10346506, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:47:23.542Z <health> ERROR Indexer is not healthy
2022-05-17T16:47:33.552Z <health> ERROR Indexer is not healthy
2022-05-17T16:47:35.842Z <benchmark> INFO NaN bps, target: #10346511, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:47:43.565Z <health> ERROR Indexer is not healthy
2022-05-17T16:47:50.842Z <benchmark> INFO NaN bps, target: #10346513, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:47:53.574Z <health> ERROR Indexer is not healthy
2022-05-17T16:48:03.587Z <health> ERROR Indexer is not healthy
2022-05-17T16:48:05.843Z <benchmark> INFO NaN bps, target: #10346513, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:48:13.605Z <health> ERROR Indexer is not healthy
2022-05-17T16:48:20.843Z <benchmark> INFO NaN bps, target: #10346517, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:48:23.617Z <health> ERROR Indexer is not healthy
2022-05-17T16:48:33.628Z <health> ERROR Indexer is not healthy
2022-05-17T16:48:35.842Z <benchmark> INFO NaN bps, target: #10346519, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:48:43.641Z <health> ERROR Indexer is not healthy
2022-05-17T16:48:50.842Z <benchmark> INFO NaN bps, target: #10346521, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:48:53.655Z <health> ERROR Indexer is not healthy
2022-05-17T16:49:03.661Z <health> ERROR Indexer is not healthy
2022-05-17T16:49:05.843Z <benchmark> INFO NaN bps, target: #10346525, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:49:13.668Z <health> ERROR Indexer is not healthy
2022-05-17T16:49:20.843Z <benchmark> INFO NaN bps, target: #10346527, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:49:23.679Z <health> ERROR Indexer is not healthy
2022-05-17T16:49:33.686Z <health> ERROR Indexer is not healthy
2022-05-17T16:49:35.843Z <benchmark> INFO NaN bps, target: #10346529, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:49:43.692Z <health> ERROR Indexer is not healthy
2022-05-17T16:49:50.843Z <benchmark> INFO NaN bps, target: #10346531, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:49:53.704Z <health> ERROR Indexer is not healthy
2022-05-17T16:50:03.724Z <health> ERROR Indexer is not healthy
2022-05-17T16:50:05.843Z <benchmark> INFO NaN bps, target: #10346535, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:50:13.742Z <health> ERROR Indexer is not healthy
2022-05-17T16:50:20.851Z <benchmark> INFO NaN bps, target: #10346537, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:50:23.760Z <health> ERROR Indexer is not healthy
2022-05-17T16:50:33.773Z <health> ERROR Indexer is not healthy
2022-05-17T16:50:35.852Z <benchmark> INFO NaN bps, target: #10346540, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:50:43.781Z <health> ERROR Indexer is not healthy
2022-05-17T16:50:50.853Z <benchmark> INFO NaN bps, target: #10346542, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:50:53.792Z <health> ERROR Indexer is not healthy
2022-05-17T16:51:03.811Z <health> ERROR Indexer is not healthy
2022-05-17T16:51:05.852Z <benchmark> INFO NaN bps, target: #10346545, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:51:13.827Z <health> ERROR Indexer is not healthy
2022-05-17T16:51:20.853Z <benchmark> INFO NaN bps, target: #10346548, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:51:23.845Z <health> ERROR Indexer is not healthy
2022-05-17T16:51:33.861Z <health> ERROR Indexer is not healthy
2022-05-17T16:51:35.852Z <benchmark> INFO NaN bps, target: #10346549, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:51:43.876Z <health> ERROR Indexer is not healthy
2022-05-17T16:51:50.852Z <benchmark> INFO NaN bps, target: #10346553, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:51:53.890Z <health> ERROR Indexer is not healthy
2022-05-17T16:52:03.898Z <health> ERROR Indexer is not healthy
2022-05-17T16:52:05.852Z <benchmark> INFO NaN bps, target: #10346555, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:52:13.920Z <health> ERROR Indexer is not healthy
2022-05-17T16:52:20.852Z <benchmark> INFO NaN bps, target: #10346557, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:52:23.938Z <health> ERROR Indexer is not healthy
2022-05-17T16:52:33.955Z <health> ERROR Indexer is not healthy
2022-05-17T16:52:35.852Z <benchmark> INFO NaN bps, target: #10346559, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:52:43.971Z <health> ERROR Indexer is not healthy
2022-05-17T16:52:50.852Z <benchmark> INFO NaN bps, target: #10346561, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:52:53.985Z <health> ERROR Indexer is not healthy
2022-05-17T16:53:04.004Z <health> ERROR Indexer is not healthy
2022-05-17T16:53:05.853Z <benchmark> INFO NaN bps, target: #10346564, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:53:14.022Z <health> ERROR Indexer is not healthy
2022-05-17T16:53:19.105Z <indexer> ERROR failed to sync poi to mmr SequelizeConnectionError: getaddrinfo EAI_AGAIN postgres
2022-05-17T16:53:29.883Z <indexer> INFO set blockoffset to 2391
2022-05-17T16:53:29.908Z <subql-node> INFO Node started on port: 3008
2022-05-17T16:53:29.922Z <mmr> INFO Reset mmr start from block height 10336300
2022-05-17T16:53:29.922Z <mmr> INFO file based database MMR start with block height 10336300
2022-05-17T16:53:29.945Z <mmr> INFO CHECKING : Poi block height 10336300, Poi mmr is same as file based mmr
2022-05-17T16:53:31.763Z <fetch> INFO fetch block [10338671,10345806], total 4 blocks
2022-05-17T16:53:42.424Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:53:42.898Z <fetch> INFO fetch block [10346570,10346570], total 1 blocks
2022-05-17T16:53:48.209Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:53:53.581Z <fetch> INFO fetch block [10346571,10346571], total 1 blocks
2022-05-17T16:53:59.610Z <benchmark> INFO 0.09 bps, target: #10346573, current: #10346571, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:54:06.513Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:54:06.870Z <fetch> INFO fetch block [10346574,10346574], total 1 blocks
2022-05-17T16:54:14.609Z <benchmark> INFO 0.23 bps, target: #10346575, current: #10346574, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:54:22.030Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:54:23.110Z <fetch> INFO fetch block [10346577,10346577], total 1 blocks
2022-05-17T16:54:26.934Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:54:27.325Z <fetch> INFO fetch block [10346578,10346578], total 1 blocks
2022-05-17T16:54:29.626Z <benchmark> INFO 0.20 bps, target: #10346578, current: #10346578, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:54:41.701Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:54:43.726Z <fetch> INFO fetch block [10346579,10346581], total 3 blocks
2022-05-17T16:54:44.624Z <benchmark> INFO 0.18 bps, target: #10346581, current: #10346581, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:54:51.493Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:54:51.956Z <fetch> INFO fetch block [10346582,10346582], total 1 blocks
2022-05-17T16:54:59.623Z <benchmark> INFO 0.12 bps, target: #10346583, current: #10346582, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:55:14.623Z <benchmark> INFO NaN bps, target: #10346585, current: #10346582, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:55:22.378Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:55:23.773Z <fetch> INFO fetch block [10346587,10346587], total 1 blocks
2022-05-17T16:55:29.623Z <benchmark> INFO 0.16 bps, target: #10346587, current: #10346587, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:55:36.978Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:55:38.259Z <fetch> INFO fetch block [10346589,10346590], total 2 blocks
2022-05-17T16:55:44.624Z <benchmark> INFO 0.21 bps, target: #10346590, current: #10346590, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:55:46.778Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:55:47.488Z <fetch> INFO fetch block [10346591,10346591], total 1 blocks
2022-05-17T16:55:51.559Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:55:52.708Z <fetch> INFO fetch block [10346592,10346592], total 1 blocks
2022-05-17T16:55:59.624Z <benchmark> INFO 0.14 bps, target: #10346592, current: #10346592, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:56:12.053Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:56:12.969Z <fetch> INFO fetch block [10346595,10346595], total 1 blocks
2022-05-17T16:56:14.623Z <benchmark> INFO 0.15 bps, target: #10346595, current: #10346595, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:56:17.887Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:56:18.279Z <fetch> INFO fetch block [10346596,10346596], total 1 blocks
2022-05-17T16:56:21.695Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:56:22.510Z <fetch> INFO fetch block [10346597,10346597], total 1 blocks
2022-05-17T16:56:29.624Z <benchmark> INFO 0.21 bps, target: #10346597, current: #10346597, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:56:42.224Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:56:44.623Z <benchmark> INFO NaN bps, target: #10346600, current: #10346597, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:56:44.781Z <fetch> INFO fetch block [10346600,10346600], total 1 blocks
2022-05-17T16:56:56.744Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:56:57.151Z <fetch> INFO fetch block [10346602,10346603], total 2 blocks
2022-05-17T16:56:59.624Z <benchmark> INFO 0.17 bps, target: #10346603, current: #10346603, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:57:14.625Z <benchmark> INFO NaN bps, target: #10346605, current: #10346603, estimate time: NaN days NaN hours NaN mins
2022-05-17T16:57:17.049Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:57:17.665Z <fetch> INFO fetch block [10346606,10346606], total 1 blocks
2022-05-17T16:57:21.813Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:57:23.991Z <fetch> INFO fetch block [10346607,10346607], total 1 blocks
2022-05-17T16:57:27.624Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:57:29.625Z <benchmark> INFO 0.15 bps, target: #10346608, current: #10346607, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:57:34.886Z <fetch> INFO fetch block [10346608,10346608], total 1 blocks
2022-05-17T16:57:35.430Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:57:36.682Z <fetch> INFO fetch block [10346609,10346609], total 1 blocks
2022-05-17T16:57:42.196Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:57:42.934Z <fetch> INFO fetch block [10346610,10346610], total 1 blocks
2022-05-17T16:57:44.626Z <benchmark> INFO 0.16 bps, target: #10346610, current: #10346610, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:57:51.723Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:57:53.073Z <fetch> INFO fetch block [10346612,10346612], total 1 blocks
2022-05-17T16:57:57.487Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:57:58.372Z <fetch> INFO fetch block [10346613,10346613], total 1 blocks
2022-05-17T16:57:59.626Z <benchmark> INFO 0.19 bps, target: #10346613, current: #10346613, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:58:07.248Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:58:07.573Z <fetch> INFO fetch block [10346614,10346614], total 1 blocks
2022-05-17T16:58:14.625Z <benchmark> INFO 0.11 bps, target: #10346615, current: #10346614, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:58:21.749Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:58:23.601Z <fetch> INFO fetch block [10346616,10346617], total 2 blocks
2022-05-17T16:58:29.625Z <benchmark> INFO 0.19 bps, target: #10346617, current: #10346617, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:58:31.524Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:58:31.812Z <fetch> INFO fetch block [10346618,10346618], total 1 blocks
2022-05-17T16:58:37.262Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:58:38.036Z <fetch> INFO fetch block [10346619,10346619], total 1 blocks
2022-05-17T16:58:44.625Z <benchmark> INFO 0.14 bps, target: #10346620, current: #10346619, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:58:46.782Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:58:47.259Z <fetch> INFO fetch block [10346621,10346621], total 1 blocks
2022-05-17T16:58:52.551Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:58:54.389Z <fetch> INFO fetch block [10346622,10346622], total 1 blocks
2022-05-17T16:58:57.346Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:58:59.626Z <benchmark> INFO 0.18 bps, target: #10346623, current: #10346622, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:59:02.393Z <fetch> INFO fetch block [10346623,10346623], total 1 blocks
2022-05-17T16:59:14.626Z <benchmark> INFO 0.12 bps, target: #10346625, current: #10346623, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:59:16.651Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:59:17.618Z <fetch> INFO fetch block [10346626,10346626], total 1 blocks
2022-05-17T16:59:22.396Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:59:22.833Z <fetch> INFO fetch block [10346627,10346627], total 1 blocks
2022-05-17T16:59:29.627Z <benchmark> INFO 0.20 bps, target: #10346627, current: #10346627, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:59:32.179Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:59:33.230Z <fetch> INFO fetch block [10346628,10346629], total 2 blocks
2022-05-17T16:59:44.626Z <benchmark> INFO 0.19 bps, target: #10346629, current: #10346629, estimate time: 0 days 00 hours 00 mins
2022-05-17T16:59:47.737Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:59:49.189Z <fetch> INFO fetch block [10346631,10346631], total 1 blocks
2022-05-17T16:59:51.490Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T16:59:52.433Z <fetch> INFO fetch block [10346632,10346632], total 1 blocks
2022-05-17T16:59:59.626Z <benchmark> INFO 0.16 bps, target: #10346632, current: #10346632, estimate time: 0 days 00 hours 00 mins

It does look like something made the node unhappy around 2022-05-17T01:48:26.201Z

2022-05-17T01:32:18.980Z <benchmark> INFO NaN bps, target: #10337446, current: #10337442, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:32:25.083Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T01:32:25.630Z <fetch> INFO fetch block [10337447,10337447], total 1 blocks
2022-05-17T01:32:29.850Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T01:32:30.886Z <fetch> INFO fetch block [10337448,10337448], total 1 blocks
2022-05-17T01:32:33.981Z <benchmark> INFO 0.15 bps, target: #10337448, current: #10337448, estimate time: 0 days 00 hours 00 mins
2022-05-17T01:32:48.981Z <benchmark> INFO NaN bps, target: #10337451, current: #10337448, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:32:54.099Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T01:32:55.177Z <fetch> INFO fetch block [10337452,10337452], total 1 blocks
2022-05-17T01:33:03.980Z <benchmark> INFO 0.16 bps, target: #10337453, current: #10337452, estimate time: 0 days 00 hours 00 mins
2022-05-17T01:33:14.414Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T01:33:18.981Z <benchmark> INFO NaN bps, target: #10337455, current: #10337452, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:33:22.440Z <fetch> INFO fetch block [10337455,10337455], total 1 blocks
2022-05-17T01:33:33.980Z <benchmark> INFO 0.11 bps, target: #10337458, current: #10337455, estimate time: 0 days 00 hours 00 mins
2022-05-17T01:33:48.981Z <benchmark> INFO NaN bps, target: #10337460, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:34:03.981Z <benchmark> INFO NaN bps, target: #10337462, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:34:18.981Z <benchmark> INFO NaN bps, target: #10337465, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:34:33.982Z <benchmark> INFO NaN bps, target: #10337468, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:34:48.982Z <benchmark> INFO NaN bps, target: #10337469, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:35:03.983Z <benchmark> INFO NaN bps, target: #10337472, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:35:18.983Z <benchmark> INFO NaN bps, target: #10337476, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:35:33.983Z <benchmark> INFO NaN bps, target: #10337477, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:35:48.982Z <benchmark> INFO NaN bps, target: #10337480, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:36:03.983Z <benchmark> INFO NaN bps, target: #10337483, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:36:18.984Z <benchmark> INFO NaN bps, target: #10337485, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:36:33.985Z <benchmark> INFO NaN bps, target: #10337488, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:36:48.984Z <benchmark> INFO NaN bps, target: #10337491, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:37:03.984Z <benchmark> INFO NaN bps, target: #10337491, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:37:18.985Z <benchmark> INFO NaN bps, target: #10337495, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:37:33.984Z <benchmark> INFO NaN bps, target: #10337497, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:37:48.985Z <benchmark> INFO NaN bps, target: #10337500, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:38:03.985Z <benchmark> INFO NaN bps, target: #10337502, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:38:18.986Z <benchmark> INFO NaN bps, target: #10337504, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:38:33.996Z <benchmark> INFO NaN bps, target: #10337508, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:38:48.997Z <benchmark> INFO NaN bps, target: #10337511, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:39:03.998Z <benchmark> INFO NaN bps, target: #10337512, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:39:18.999Z <benchmark> INFO NaN bps, target: #10337514, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:39:33.999Z <benchmark> INFO NaN bps, target: #10337516, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:39:48.998Z <benchmark> INFO NaN bps, target: #10337520, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:40:03.999Z <benchmark> INFO NaN bps, target: #10337523, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:40:19.000Z <benchmark> INFO NaN bps, target: #10337525, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:40:34.000Z <benchmark> INFO NaN bps, target: #10337527, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:40:48.999Z <benchmark> INFO NaN bps, target: #10337530, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:41:03.999Z <benchmark> INFO NaN bps, target: #10337533, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:41:18.999Z <benchmark> INFO NaN bps, target: #10337534, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:41:33.999Z <benchmark> INFO NaN bps, target: #10337538, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:41:49.000Z <benchmark> INFO NaN bps, target: #10337540, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:42:03.999Z <benchmark> INFO NaN bps, target: #10337542, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:42:18.999Z <benchmark> INFO NaN bps, target: #10337546, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:42:33.999Z <benchmark> INFO NaN bps, target: #10337547, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:42:48.999Z <benchmark> INFO NaN bps, target: #10337551, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:43:04.000Z <benchmark> INFO NaN bps, target: #10337552, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:43:18.999Z <benchmark> INFO NaN bps, target: #10337555, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:43:34.000Z <benchmark> INFO NaN bps, target: #10337558, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:43:49.001Z <benchmark> INFO NaN bps, target: #10337560, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:44:04.000Z <benchmark> INFO NaN bps, target: #10337563, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:44:19.003Z <benchmark> INFO NaN bps, target: #10337565, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:44:34.002Z <benchmark> INFO NaN bps, target: #10337567, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:44:49.002Z <benchmark> INFO NaN bps, target: #10337570, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:45:04.003Z <benchmark> INFO NaN bps, target: #10337572, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:45:19.003Z <benchmark> INFO NaN bps, target: #10337575, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:45:34.002Z <benchmark> INFO NaN bps, target: #10337578, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:45:49.002Z <benchmark> INFO NaN bps, target: #10337579, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:46:04.003Z <benchmark> INFO NaN bps, target: #10337583, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:46:19.004Z <benchmark> INFO NaN bps, target: #10337586, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:46:34.005Z <benchmark> INFO NaN bps, target: #10337586, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:46:49.004Z <benchmark> INFO NaN bps, target: #10337591, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:47:04.004Z <benchmark> INFO NaN bps, target: #10337593, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:47:19.004Z <benchmark> INFO NaN bps, target: #10337593, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:47:34.005Z <benchmark> INFO NaN bps, target: #10337598, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:47:49.005Z <benchmark> INFO NaN bps, target: #10337600, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:48:04.004Z <benchmark> INFO NaN bps, target: #10337603, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:48:19.004Z <benchmark> INFO NaN bps, target: #10337604, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:48:26.201Z <health> ERROR Indexer is not healthy
2022-05-17T01:48:34.004Z <benchmark> INFO NaN bps, target: #10337606, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:48:36.299Z <health> ERROR Indexer is not healthy
2022-05-17T01:48:46.317Z <health> ERROR Indexer is not healthy
2022-05-17T01:48:49.004Z <benchmark> INFO NaN bps, target: #10337610, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:48:56.335Z <health> ERROR Indexer is not healthy
2022-05-17T01:49:04.004Z <benchmark> INFO NaN bps, target: #10337613, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:49:06.344Z <health> ERROR Indexer is not healthy
2022-05-17T01:49:16.363Z <health> ERROR Indexer is not healthy
2022-05-17T01:49:19.005Z <benchmark> INFO NaN bps, target: #10337613, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:49:26.375Z <health> ERROR Indexer is not healthy
2022-05-17T01:49:34.006Z <benchmark> INFO NaN bps, target: #10337618, current: #10337455, estimate time: NaN days NaN hours NaN mins
2022-05-17T01:49:36.391Z <health> ERROR Indexer is not healthy
2022-05-17T01:49:46.405Z <health> ERROR Indexer is not healthy

Kusama

2022-05-17T21:16:04.111Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T21:16:04.987Z <fetch> INFO fetch block [12728734,12728734], total 1 blocks
2022-05-17T21:16:07.420Z <benchmark> INFO 0.14 bps, target: #12728734, current: #12728734, estimate time: 0 days 00 hours 00 mins
2022-05-17T21:16:13.842Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T21:16:14.232Z <fetch> INFO fetch block [12728735,12728735], total 1 blocks
2022-05-17T21:16:19.642Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T21:16:20.619Z <fetch> INFO fetch block [12728736,12728737], total 2 blocks
2022-05-17T21:16:22.420Z <benchmark> INFO 0.19 bps, target: #12728737, current: #12728737, estimate time: 0 days 00 hours 00 mins
2022-05-17T21:16:29.370Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T21:16:29.873Z <fetch> INFO fetch block [12728738,12728738], total 1 blocks
2022-05-17T21:16:34.121Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T21:16:35.200Z <fetch> INFO fetch block [12728739,12728739], total 1 blocks
2022-05-17T21:16:37.421Z <benchmark> INFO 0.14 bps, target: #12728739, current: #12728739, estimate time: 0 days 00 hours 00 mins
2022-05-17T21:16:43.895Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T21:16:44.712Z <fetch> INFO fetch block [12728740,12728741], total 2 blocks
2022-05-17T21:16:52.422Z <benchmark> INFO 0.21 bps, target: #12728741, current: #12728741, estimate time: 0 days 00 hours 00 mins
2022-05-17T21:16:54.638Z <fetch> WARN Dictionary indexed block is behind current indexing block height
2022-05-17T21:16:55.035Z <fetch> INFO fetch block [12728742,12728742], total 1 blocks
2022-05-17T21:16:59.350Z <fetch> WARN Dictionary indexed block is behind current indexing block height

jim-counter · 2022-05-19T09:17:32Z

jim-counter
May 19, 2022
Author

Happened again yesterday.

coordinator_service  | 2022-05-18T12:28:39.269Z <transaction> INFO report project status: PROCESSING | project QmSzPQ9f4U1GERv | block height: 12737819 | mmrRoot: 0xc77e647e8ae7305c929893ac77c85ca07fe216e4f914c2aac029c350915fad2b
coordinator_service  | /usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/logger/lib/index.js:233
coordinator_service  |         var error = new Error(message);
coordinator_service  |                     ^
coordinator_service  |
coordinator_service  | Error: bad response (status=502, headers={"content-type":"text/plain; charset=utf-8","content-length":"93","connection":"close","date":"Wed, 18 May 2022 12:29:16 GMT","x-kong-upstream-latency":"267","x-kong-proxy-latency":"1","via":"kong/2.1.4"}, body="i/o timeout. The gateway cannot get a response, please try again or contact the administrator", requestBody="{\"method\":\"eth_getTransactionByHash\",\"params\":[\"0x359a6556ef47466b7ba22be2e461319c31cb7b01da6a39a03aeba17e0c7640f7\"],\"id\":4792,\"jsonrpc\":\"2.0\"}", requestMethod="POST", url="https://sqtn.api.onfinality.io/public", code=SERVER_ERROR, version=web/5.6.0)
coordinator_service  |     at Logger.makeError (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/logger/lib/index.js:233:21)
coordinator_service  |     at Logger.throwError (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/logger/lib/index.js:242:20)
coordinator_service  |     at /usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:267:36
coordinator_service  |     at step (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:33:23)
coordinator_service  |     at Object.next (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:14:53)
coordinator_service  |     at fulfilled (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:5:58)
coordinator_service  |     at runMicrotasks (<anonymous>)
coordinator_service  |     at processTicksAndRejections (node:internal/process/task_queues:96:5) {
coordinator_service  |   reason: 'bad response',
coordinator_service  |   code: 'SERVER_ERROR',
coordinator_service  |   status: 502,
coordinator_service  |   headers: {
coordinator_service  |     'content-type': 'text/plain; charset=utf-8',
coordinator_service  |     'content-length': '93',
coordinator_service  |     connection: 'close',
coordinator_service  |     date: 'Wed, 18 May 2022 12:29:16 GMT',
coordinator_service  |     'x-kong-upstream-latency': '267',
coordinator_service  |     'x-kong-proxy-latency': '1',
coordinator_service  |     via: 'kong/2.1.4'
coordinator_service  |   },
coordinator_service  |   body: 'i/o timeout. The gateway cannot get a response, please try again or contact the administrator',
coordinator_service  |   requestBody: '{"method":"eth_getTransactionByHash","params":["0x359a6556ef47466b7ba22be2e461319c31cb7b01da6a39a03aeba17e0c7640f7"],"id":4792,"jsonrpc":"2.0"}',
coordinator_service  |   requestMethod: 'POST',
coordinator_service  |   url: 'https://sqtn.api.onfinality.io/public'
coordinator_service  | }
coordinator_service  | 2022-05-18T12:29:23.836Z <nestjs> INFO Starting Nest application...
coordinator_service  | 2022-05-18T12:29:23.982Z <nestjs> INFO TypeOrmModule dependencies initialized
coordinator_service  | 2022-05-18T12:29:23.983Z <nestjs> INFO ConfigureModule dependencies initialized
coordinator_service  | 2022-05-18T12:29:23.983Z <nestjs> INFO AppModule dependencies initialized
coordinator_service  | 2022-05-18T12:29:23.983Z <nestjs> INFO ServeStaticModule dependencies initialized
coordinator_service  | 2022-05-18T12:29:23.984Z <nestjs> INFO GraphQLSchemaBuilderModule dependencies initialized
coordinator_service  | 2022-05-18T12:29:23.984Z <nestjs> INFO GraphQLModule dependencies initialized
coordinator_service  | 2022-05-18T12:29:24.544Z <nestjs> INFO TypeOrmCoreModule dependencies initialized

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

502 causes stack restart #1033

{{title}}

Replies: 2 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

502 causes stack restart #1033

jim-counter May 17, 2022

Replies: 2 comments · 1 reply

mzxyz May 18, 2022 Maintainer

jim-counter May 18, 2022 Author

jim-counter May 19, 2022 Author

jim-counter
May 17, 2022

Replies: 2 comments 1 reply

mzxyz
May 18, 2022
Maintainer

jim-counter May 18, 2022
Author

jim-counter
May 19, 2022
Author