Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CLOB-1001] Add custom error value formatter to zerolog logging library #31

Closed
wants to merge 2 commits into from

Conversation

jonfung-dydx
Copy link

For datadog error tracking, we want each error-based log field to have:

error: { stack: <stacktrace>, kind: <error_kind>, message: <error_message>} 

In order to do this, we introduce a custom error formatter to the zerolog logger that is used by cometbft, cosmos-sdk, and v4-chain.

This formatter will run on any log event Key:Value tag value of type errors and massage it to be of the above form.
Thus, any error logs that have "error": error will be automatically datadog error tracked.

Note that cosmos-sdk emits error logs with the key "err" instead of "error". Zerolog logging library is not powerful enough to support custom error event logging for stack traces in their custom ErrorStackMarshallingFunc here. We could PR into the library, but an easier solution we have done is to set up a datadog log pipeline that remaps all err keys into error keys. That way, we can datadog error track cosmos-sdk errors.

TL;DR all error and err keys are now properly datadog error tracked.

Copy link

linear bot commented Dec 7, 2023

Copy link

github-actions bot commented Dec 7, 2023

@jonfung-dydx your pull request is missing a changelog!

Copy link

@lcwik lcwik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this change could be done within the SDK without needing to modify the Cosmos fork by updating root.go inside the PersistentPreRunE to set zerolog.ErrorMarshalFunc since it is a global variable.

@@ -197,9 +199,39 @@ func CreateSDKLogger(ctx *Context, out io.Writer) (log.Logger, error) {
// Check if the CometBFT flag for trace logging is set and enable stack traces if so.
opts = append(opts, log.TraceOption(ctx.Viper.GetBool("trace"))) // cmtcli.TraceFlag

// Error fields should be set under error object
zerolog.ErrorFieldName = "error"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the default in the library so we don't need to do it.
https://pkg.go.dev/github.com/rs/zerolog#section-readme

zerolog.ErrorFieldName = "error"

// Add the kind and message field
zerolog.ErrorMarshalFunc = func(err error) interface{} {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should guard this behind a flag to allow people to opt out of the custom marshalling function in case they aren't using datadog.

}
objectToReturn := DatadogErrorTrackingObject{
// Discard common prefix stack traces from zerolog call sites
Stack: stackArr[5:],
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should have a test for this to ensure that we are discarding exactly how much we want in case the stack depth changes due to a change in the implementation of this code or the underlying library.

@jonfung-dydx
Copy link
Author

Closing this. Did this in the protocol.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants