document a pattern for making Sled-Agent-owned config to be Nexus-owned instead

[davepacheco]

Recent and upcoming work have prompted discussions about how we take existing Sled-Agent-owned configuration and make it Nexus-owned instead. Here's my attempt to document this pattern for future reference. I'm afraid it can't be too prescriptive because each of the cases we've encountered is a little different. But I'm hopeful this can be a useful starting point when we go do things like add an artifact/image id to Omicron zone configs.

I may have gone overboard with Mermaid again.

[smklein]

So, in a case like adding the artifact/image ID, we're also adding the requirement that sled agent needs to learn about this value explicitly, right?

As an example -- "release 1" of the sled agent might launch instances using /opt/oxide/propolis-server.tar.gz -- and use whatever file that is, regardless of "version".

Just wanted to call that out, as "part of release 2 may involve making sled agent explicitly aware of what state it happens to be managing", since we've declared it should be non-optional in inventory.

(also, to be clear, it's non-optional in the sled agent API call to Nexus, not the inventory database structure, right? Otherwise, old inventory collections would be invalidated)

[davepacheco]

So, in a case like adding the artifact/image ID, we're also adding the requirement that sled agent needs to learn about this value explicitly, right?

As an example -- "release 1" of the sled agent might launch instances using /opt/oxide/propolis-server.tar.gz -- and use whatever file that is, regardless of "version".

Just wanted to call that out, as "part of release 2 may involve making sled agent explicitly aware of what state it happens to be managing", since we've declared it should be non-optional in inventory.

The code needs to be aware that this value, which may previously have been hardcoded or even absent altogether, is now a variable to be reported in inventory (this item) and controlled by Nexus (item 4). Is that what you mean by "learn about this value explicitly"?

(also, to be clear, it's non-optional in the sled agent API call to Nexus, not the inventory database structure, right? Otherwise, old inventory collections would be invalidated)

Good point! I will update this to be more clear that (1) the inventory API structure can make this required but the inventory database structures should have this value optional, and (2) in release 3, this can probably be made required in the inventory database structures, too. (This is annoyingly tricky: we might still have old collections at this point but it seems unlikely.)

[smklein]

The code needs to be aware that this value, which may previously have been hardcoded or even absent altogether, is now a variable to be reported in inventory (this item) and controlled by Nexus (item 4). Is that what you mean by "learn about this value explicitly"?

Yeah, basically just calling this out -- the phrasing "In this use case, Sled Agent always knows the current value of this field" is true, but if it was an "implicit" value before, Sled Agent might need to do some legwork to get an explicit value that can be reported.

Re: Inventory stuff

Yeah, totally - another spot where it might be useful to somehow provide a barrier. If we could ensure that all inventory collections are "from this release or the one prior, and no older", that could be a useful way to gradually make these values non-optional.

[jgallagher]

(This is annoyingly tricky: we might still have old collections at this point but it seems unlikely.)

Is this almost identical (or even exactly identical) to "changing an optional field to required in a blueprint requires dropping old blueprints", with the exception that inventory already rotates old collections out with relatively high frequency except in the case of errors (where we keep around the last good collection, which might be months old on a system like dogfood)? Do you think we #7278 should explicitly include inventory collections too? (It already implicitly covers them to some extent, since saving the reconfigurator state already includes them, but doesn't mention deleting old ones if they're still around.)

[davepacheco]

(This is annoyingly tricky: we might still have old collections at this point but it seems unlikely.)

Is this almost identical (or even exactly identical) to "changing an optional field to required in a blueprint requires dropping old blueprints", with the exception that inventory already rotates old collections out with relatively high frequency except in the case of errors (where we keep around the last good collection, which might be months old on a system like dogfood)? Do you think we #7278 should explicitly include inventory collections too? (It already implicitly covers them to some extent, since saving the reconfigurator state already includes them, but doesn't mention deleting old ones if they're still around.)

Yes, it's analogous. Sure -- I'll update that issue.

[smklein]

I definitely agree checking for the blueprint + ledgers is probably an ad-hoc operation that we need a human to do, but getting this standard process of:

• Collect inventory
• Make a new blueprint, check the diff, make sure it looks good
• Set it as target, execute it

As a part of each release seems like it would be very useful for us to rely on.

(I think it would be nice to be able to assume "if you have a rack running on release N, part of that upgrade involved creating a blueprint at release N and making sure it could run")

[smklein]

This is another spot where "if we know we added a blueprint during upgrade", it also might give us the leniency to "delete all older blueprints as the final step of an upgrade". Or perhaps as one of the first steps of upgrading to N+1

[jgallagher]

FWIW since you mentioned going overboard on Mermaid: I found this diagram and the one below it very helpful. (I didn't find the ones above as helpful, personally, but I don't think they detract anything from the doc, either.)