Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log WARN for failed RPC response #8138

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

lu-pinto
Copy link
Contributor

@lu-pinto lu-pinto commented Jan 18, 2025

PR description

While debugging an acceptance test on the verkle branch I noticed that the cause of failure was an RPC method that was not added to the collection of supported RPC methods here.

However, without enabling TRACE logging it was impossible to see what was wrong as looking for WARN or ERROR was the first thing that came to mind. I think errors like this should not be hidden within TRACE and logging at least a warning when an RPC call fails is a must have for troubleshooting.

I don't think I'm missing any case where an RpcResponseType.ERROR might be legit, if so I can try limiting it to RpcErrorType.METHOD_NOT_ENABLED.

Before

2025-01-18 00:57:11.700+00:00 | pool-8-thread-1 | INFO  | SubProcessLog | 2025-01-18 00:57:11.700+00:00 | vert.x-worker-thread-6 | TRACE | AbstractJsonRpcExecutor | {"jsonrpc":"2.0","id":67,"error":{"code":-32604,"message":"Method not enabled"}}

After

2025-01-18 00:30:31.885+00:00 | pool-8-thread-1 | INFO  | SubProcessLog | 2025-01-18 00:30:31.885+00:00 | vert.x-worker-thread-6 | WARN  | AbstractJsonRpcExecutor | RPC error: {"jsonrpc":"2.0","id":67,"error":{"code":-32604,"message":"Method not enabled"}}

Thanks for sending a pull request! Have you done the following?

  • Checked out our contribution guidelines?
  • Considered documentation and added the doc-change-required label to this PR if updates are required.
  • Considered the changelog and included an update if required.
  • For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

  • unit tests: ./gradlew build
  • acceptance tests: ./gradlew acceptanceTest
  • integration tests: ./gradlew integrationTest
  • reference tests: ./gradlew ethereum:referenceTests:referenceTests

@lu-pinto lu-pinto force-pushed the add-log-failed-rpc-response branch from 72788c3 to fd4db28 Compare January 18, 2025 00:45
@lu-pinto lu-pinto changed the title Add log failed rpc response Log WARN for failed RPC response Jan 18, 2025
lazyTraceLogger(() -> getJsonObjectMapper().writeValueAsString(jsonRpcResponse));
} else {
LOG.warn("RPC error: {}", getJsonObjectMapper().writeValueAsString(jsonRpcResponse));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be pretty heavy on RPC nodes, I’m thinking of “Block not found” errors for example.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, this could potentially blow up the logs for a lot of users, even if they are genuine issues.
Maybe we need some sort of log throttling mechanism to go with it?

Would favour the more specific METHOD_NOT_FOUND but unless we also have throttling, I think we'd still need some confidence about the impact of this on a variety of usecases/networks.

Copy link
Contributor

@siladu siladu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the change in principle but it's a potentially noisy, maybe even dangerous change (e.g. disk issues due to over-logging) for existing poorly configured systems.

Btw, I think this may fix the root cause of the RPC method not being enabled
#8136

lazyTraceLogger(() -> getJsonObjectMapper().writeValueAsString(jsonRpcResponse));
} else {
LOG.warn("RPC error: {}", getJsonObjectMapper().writeValueAsString(jsonRpcResponse));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, this could potentially blow up the logs for a lot of users, even if they are genuine issues.
Maybe we need some sort of log throttling mechanism to go with it?

Would favour the more specific METHOD_NOT_FOUND but unless we also have throttling, I think we'd still need some confidence about the impact of this on a variety of usecases/networks.

@lu-pinto
Copy link
Contributor Author

Btw, I think this may fix the root cause of the RPC method not being enabled #8136

Hmm, but isn't the problem altogether about missing a definition that should be there? I don't understand how this could solve as it just moves the potential problem to a missing definition in the milestones now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants