Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Notifying nexus of live repair progress updates eats way too much CPU because of creating reqwest clients #1316

Closed
faithanalog opened this issue May 25, 2024 · 1 comment · Fixed by #1317
Assignees

Comments

@faithanalog
Copy link
Contributor

image

image

This rack is very distressed. Some disks have gone into live-repair. That distress is being made even worse because 8% of the rack's CPU is being spent notifying nexus of live-repair status, and most of that is being spent creating Reqwest clients. this is the same class of problem as oxidecomputer/omicron#5717 which was fixed by oxidecomputer/omicron#5750, but we are having it over here too.

Fixing this requires caching the Reqwest client, but it does not require caching the nexus client. This is an important distinction because caching the nexus client would cause problems when nexus moves around. But caching the reqwest client is fine, because it's just the thing that does HTTP requests, it doesn't care where they're going.

In short, we want to be calling nexus_client::Client::new_with_client() instead of nexus_client::Client::new() here

crucible/upstairs/src/lib.rs

Lines 1816 to 1819 in da333e9

Some(nexus_client::Client::new(
&format!("http://{}", nexus_address),
log.clone(),
))

A Reqwest client is internally Arc'd, so we just need one laying around that we can clone every time we make a nexus client

@iliana tagging you to make sure I'm explaining this right

@iliana
Copy link

iliana commented May 26, 2024

Yep that's right.

@jmpesp jmpesp self-assigned this May 27, 2024
jmpesp added a commit to jmpesp/crucible that referenced this issue May 27, 2024
Creating Reqwest clients is expensive, and creating them every time
Nexus is notified is therefore an awful idea. Reuse a single Reqwest
client instead.

Fixes oxidecomputer#1316.
@jmpesp jmpesp closed this as completed in 0c1a877 May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants