br: add retry for raw kv client put #58963
Conversation
ti-chi-bot
bot
added
release-note-none
|
Hi @Tristan1900. Thanks for your PR. PRs from untrusted users cannot be marked as trusted with I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
ti-chi-bot
bot
added
release-note
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #58963 +/- ##
================================================
+ Coverage 73.0261% 74.6823% +1.6561%
================================================
Files 1684 1731 +47
Lines 465659 475625 +9966
================================================
+ Hits 340053 355208 +15155
+ Misses 104692 97961 -6731
- Partials 20914 22456 +1542
Flags with carried forward coverage won't be shown. Click here to find out more.
|
br/pkg/restore/log_client/client.go
Outdated
|
|
||
| func PutRawKvWithRetry(ctx context.Context, client *rawkv.RawKVBatchClient, key, value []byte, originTs uint64) error { | ||
| retry := utils.InitialRetryState(rawKVMaxRetries, rawKVInitialRetryInterval, rawKVMaxRetryInterval) | ||
| for retry.ShouldRetry() { |
There was a problem hiding this comment.
you can use utils.WithRetry if you define the InitialRetryState.
Tristan1900
force-pushed
the
retry-on-idle-conn
branch
from
5e698b5 to
563cabd
Compare
Signed-off-by: Wenqi Mou <[email protected]>
Tristan1900
force-pushed
the
retry-on-idle-conn
branch
from
563cabd to
362a04f
Compare
Signed-off-by: Wenqi Mou <[email protected]>
|
/retest |
|
@BornChanger: Cannot trigger testing until a trusted user reviews the PR and leaves an In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/retest |
|
@BornChanger: Cannot trigger testing until a trusted user reviews the PR and leaves an In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Signed-off-by: Wenqi Mou <[email protected]>
|
@BornChanger @3pointer please take another look when you get a chance, thanks! |
ti-chi-bot
bot
added
approved
needs-1-more-lgtm
| @@ -53,6 +53,10 @@ const ( | |||
| recoveryMaxAttempts = 16 | |||
| recoveryDelayTime = 30 * time.Second | |||
| recoveryMaxDelayTime = 4 * time.Minute | |||
|
|
|||
| rawClientMaxAttempts = 5 | |||
There was a problem hiding this comment.
is 5 times(12~13s) retry enough for the special workload?
prefer make it longer
There was a problem hiding this comment.
it's mainly for retrying on removing the idle connection and create a new connection, I think it should be done within seconds, setting total retry to be >10s should be safe.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: 3pointer, BornChanger The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ti-chi-bot
bot
added
lgtm
and removed
needs-1-more-lgtm
[LGTM Timeline notifier]Timeline:
|
BornChanger
added
needs-cherry-pick-release-6.5
Signed-off-by: ti-chi-bot <[email protected]>
|
In response to a cherrypick label: new pull request created to branch |
|
In response to a cherrypick label: new pull request created to branch |
Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: ti-chi-bot <[email protected]>
|
In response to a cherrypick label: new pull request created to branch |
Signed-off-by: ti-chi-bot <[email protected]>
|
In response to a cherrypick label: new pull request created to branch |
Signed-off-by: ti-chi-bot <[email protected]>
|
In response to a cherrypick label: new pull request created to branch |
What problem does this PR solve?
Issue Number: close #58845
Problem Summary:
What changed and how does it work?
Adds a retry logic in raw kv client to mask temporary connection error from the client_go library
Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.