Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mimir.rules.kubernetes: Don't retry unrecoverable errors #616

Merged
merged 5 commits into from
Apr 23, 2024

Conversation

56quarters
Copy link
Contributor

@56quarters 56quarters commented Apr 19, 2024

PR Description

Change event processing to immediately stop applying changes to Mimir in response to events that cause HTTP 4XX errors. These errors indicate the request is malformed and will never succeed so there's no point retrying it.

Which issue(s) this PR fixes

Fixes #610

Notes to the Reviewer

This PR probably conflicts with #158 and will need to be rebased . Done

PR Checklist

  • CHANGELOG.md updated
  • Tests updated

@56quarters 56quarters changed the title mimir.rules.kubernetes: Don't retry non-recoverable errors mimir.rules.kubernetes: Don't retry unrecoverable errors Apr 19, 2024
@56quarters 56quarters force-pushed the 56quarters/event-retry branch 2 times, most recently from e19514a to 1d5b73a Compare April 19, 2024 19:42
@56quarters 56quarters marked this pull request as ready for review April 19, 2024 19:46
@56quarters 56quarters force-pushed the 56quarters/event-retry branch from 58fe912 to 233bc0f Compare April 22, 2024 16:16
@56quarters 56quarters requested review from Logiraptor and a team April 22, 2024 19:08
@rfratto rfratto added the backport-to-agent PR should be backported to the agent repo. label Apr 23, 2024
Copy link
Contributor

@wildum wildum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

internal/component/mimir/rules/kubernetes/events.go Outdated Show resolved Hide resolved
Change event processing to immediately stop applying changes to Mimir
in response to events that cause HTTP 4XX errors. These errors indicate
the request is malformed and will never succeed so there's no point
retrying it.

Fixes #610

Signed-off-by: Nick Pillitteri <[email protected]>
Signed-off-by: Nick Pillitteri <[email protected]>
Signed-off-by: Nick Pillitteri <[email protected]>
@56quarters 56quarters force-pushed the 56quarters/event-retry branch from a57241f to f65a49d Compare April 23, 2024 13:46
Signed-off-by: Nick Pillitteri <[email protected]>
@56quarters 56quarters requested a review from rfratto April 23, 2024 14:07
Copy link
Member

@rfratto rfratto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Nick! I'll merge as soon as the CI is green.

@rfratto rfratto merged commit 596fade into main Apr 23, 2024
13 checks passed
@rfratto rfratto deleted the 56quarters/event-retry branch April 23, 2024 14:09
hainenber pushed a commit to hainenber/alloy that referenced this pull request May 1, 2024
Change event processing to immediately stop applying changes to Mimir
in response to events that cause HTTP 4XX errors. These errors indicate
the request is malformed and will never succeed so there's no point
retrying it.

Fixes grafana#610

Signed-off-by: Nick Pillitteri <[email protected]>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 26, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backport-to-agent PR should be backported to the agent repo. frozen-due-to-age
Projects
None yet
Development

Successfully merging this pull request may close these issues.

mimir.rules.kubernetes: Ruler API calls blindly retried
4 participants