Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[gear] Change deadlock logs from warning to info #14251

Merged
merged 1 commit into from
Feb 6, 2024

Conversation

daniel-goldstein
Copy link
Contributor

@daniel-goldstein daniel-goldstein commented Feb 5, 2024

As discussed in #14240, we emit warnings on database deadlocks, which there are enough of to trigger noisy alerts. Since there's nothing to be done operationally (and there's no current work underway to get rid of them), these alerts only contribute to alert fatigue and hide potential problems in the system that could be addressed. This demotes a deadlock to the info level so we can still see how often they occur but are not alerted by them. In the future when we resolve the current deadlock we can re-escalate this error so that we can catch new deadlocks that are introduced.

jigold
jigold previously requested changes Feb 5, 2024
Copy link
Contributor

@jigold jigold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you put what the error code is in the comment?

@daniel-goldstein
Copy link
Contributor Author

I think GitHub doesn't show that far up from the diff but there is a comment describing the error code already on line 27. I tried to keep them close together.

@danking danking merged commit 14c9457 into hail-is:main Feb 6, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants