Retry requests when rate limited

[anekkanti]

What was changed

The api client will retry when it runs into rate limiting.

Why?

This capability was accidentally removed in PR-121
This change adds the middleware back on the client.

Checklist

1. Closes
   [Bug] Rate limit errors are not retried #142

2. How was this tested:

• CI
• Manual

[jlacefie]

TY Abhinav. Reviewing this, it appears I removed a bit too much when replacing the previous manual dial with the Go client and we lost the backoff retry options, yes?

I'll pull down and test now and validate this resolves the issue. For context, I ran into this earlier this week and believe it can be recreated easily.

[anekkanti]

TY Abhinav. Reviewing this, it appears I removed a bit too much when replacing the previous manual dial with the Go client and we lost the backoff retry options, yes?

Yes that is my understanding. I don't think the SDK we use provides this out of the box.

I'll pull down and test now and validate this resolves the issue. For context, I ran into this earlier this week and believe it can be recreated easily.

Thank you, I am also planning to do some tests on my end. More for my experience.

[jlacefie]

Thanks. Did you happen to ask if the Go SDK provides backoff support, similar to what's supported for WFs and Activities?

[jlacefie]

TY. Looks good. This resolved the pervious errors experienced locally.

As a future item, we may want to explore if the SDK already supports retry policies for the Cloud Ops API and/or make a request to that repo and then refactor to use the SDK.