Enhance json parser error logging to better track Istio Proxy error message #15176
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… would greatly help tracing to the root of error in case istio proxy return unexpected error messages during meeting connection issues.
cryptoe
requested changes
Oct 19, 2023
server/src/main/java/org/apache/druid/client/JsonParserIterator.java
Outdated
Show resolved
Hide resolved
|
Thanks for the PR @kaisun2000 . Please add test cases for this change. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
server/src/test/java/org/apache/druid/client/JsonParserIteratorTest.java
Fixed
Show fixed
Hide fixed
cryptoe
requested changes
Oct 23, 2023
cryptoe
reviewed
Oct 24, 2023
server/src/test/java/org/apache/druid/client/JsonParserIteratorTest.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/apache/druid/client/JsonParserIteratorTest.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/apache/druid/client/JsonParserIteratorTest.java
Outdated
Show resolved
Hide resolved
| public static class IAEExceptionConversionTest | ||
| { | ||
| @Rule | ||
| public ExpectedException expectedException = ExpectedException.none(); |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation Note test
cryptoe
approved these changes
Oct 27, 2023
CaseyPan
pushed a commit
to CaseyPan/druid
that referenced
this pull request
Nov 17, 2023
…essage (apache#15176) Currently the inter Druid communication via rest endpoints is based on json formatted payload. Upon parsing error, there is only a generic exception stating expected json token type and current json token type. There is no detailed error log about the content of the payload causing the violation. In the micro-service world, the trend is to deploy the Druid servers in k8 with the mesh network. Often the istio proxy or other proxies is used to intercept the network connection between Druid servers. The proxy may give error messages for various reasons. These error messages are not expected by the json parser. The generic error message from Druid can be very misleading as the user may think the message is based on the response from the other Druid server. For example, this is an example of mysterious error message QueryInterruptedException{msg=Next token wasn't a START_ARRAY, was[VALUE_STRING] from url[http://xxxxx:8088/druid/v2/], code=Unknown exception, class=org.apache.druid.java.util.common.IAE, host=xxxxx:8088}" While the context of the message is the following from the proxy when it can't tunnel the network connection. pstream connect error or disconnect/reset before header So this very simple PR is just to enhance the logging and get the real underlying message printed out. This would save a lot of head scratching time if Druid is deployed with mesh network. Co-authored-by: Kai Sun <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Currently the inter Druid communication via rest endpoints is based on json formatted payload. Upon parsing error, there is only a generic exception stating expected json token type and current json token type. There is no detailed error log about the content of the payload causing the violation.
In the micro-service world, the trend is to deploy the Druid servers in k8 with the mesh network. Often the istio proxy or other proxies is used to intercept the network connection between Druid servers. The proxy may give error messages for various reasons. These error messages are not expected by the json parser. The generic error message from Druid can be very misleading as the user may think the message is based on the response from the other Druid server.
For example, this is an example of mysterious error message
While the context of the message is the following from the proxy when it can't tunnel the network connection.
So this very simple PR is just to enhance the logging and get the real underlying message printed out. This would save a lot of head scratching time if Druid is deployed with mesh network.
Release note
Improved: enhanced the json parser unexpected token logging with the context of the expected VALUE_STRING token. This way, it is easier to track to mesh/proxy network error messages, instead misleading user to track the Druid server rest endpoint responses.
This PR has: