[nexus][sled-agent] Try more to load firewall rules during service bringup #5054
Conversation
| // Load as many services as we can, and don't exit immediately | ||
| // upon failure... | ||
| let load_services_result = | ||
| self.inner.services.load_services().await.map_err(|err| { | ||
| BackoffError::transient(Error::from(err)) | ||
| }); | ||
|
|
||
| // ... and request firewall rule updates for as many services as | ||
| // we can. Note that we still make this request even if we only | ||
| // partially load some services. | ||
| let firewall_result = self | ||
| .request_firewall_update() | ||
| .await | ||
| .map_err(|err| BackoffError::transient(err)) | ||
| .map_err(|err| BackoffError::transient(err)); | ||
|
|
||
| // Only complete if we have loaded all services and firewall | ||
| // rules successfully. | ||
| load_services_result.and(firewall_result) |
There was a problem hiding this comment.
By doing this operations back-to-back, we basically "load as many firewall rules as we can" for a set of services.
I could make this trigger more individually for "services which need firewall rules" (e.g., the request is only made after adding a Nexus / External DNS / boundary NTP zone), but felt that this was more "obviously not missing anything".
I could push down the triggers into the ServiceManager if folks felt strongly about this.
There was a problem hiding this comment.
Hmm, like this we'll continue to hit the endpoint if we get stuck in the disk error loop right? Wonder if cause any issues constantly trying to update fw rules. Oh but there's the backoff in the retry loop. So, should be fine
| // Load as many services as we can, and don't exit immediately | ||
| // upon failure... | ||
| let load_services_result = | ||
| self.inner.services.load_services().await.map_err(|err| { | ||
| BackoffError::transient(Error::from(err)) | ||
| }); | ||
|
|
||
| // ... and request firewall rule updates for as many services as | ||
| // we can. Note that we still make this request even if we only | ||
| // partially load some services. | ||
| let firewall_result = self | ||
| .request_firewall_update() | ||
| .await | ||
| .map_err(|err| BackoffError::transient(err)) | ||
| .map_err(|err| BackoffError::transient(err)); | ||
|
|
||
| // Only complete if we have loaded all services and firewall | ||
| // rules successfully. | ||
| load_services_result.and(firewall_result) |
There was a problem hiding this comment.
Hmm, like this we'll continue to hit the endpoint if we get stuck in the disk error loop right? Wonder if cause any issues constantly trying to update fw rules. Oh but there's the backoff in the retry loop. So, should be fine
Fixes #5053 , with this solution implemented:
I used a bit of a brute-force solution here -- "just do it each time we try to launch a set of zones". This could certainly be more fine-grained, but it gets the job done.