[nexus] add basic support for expunged sled policy and decommissioned sled state

[sunshowers]

This PR does a few things:

• Migrates our current sled_provision_state to a new sled_policy enum, which has more information (per RFD 457). This PR implements the expunged state, not the graceful removal state.
• Adds a sled_state enum, which describes Nexus's view of the sled. This PR adds the active and decommissioned states.
• Adds internal code to move around between valid states.
• Makes the blueprint execution code aware of sleds eligible for discretionary services.
• Adds tests for all of this new stuff, as well as valid and invalid state transitions -- and also makes sure that if we do end up in an invalid state, things don't break down.

Not done here, but in future PRs (to try and keep this PR a manageable size):

• We'll add the endpoint to mark the sled as expunged (this is an irreversible operation and will need the appropriate warnings): Define public endpoint for marking sled as expunged #5134
• We'll add blueprint code to start removing sleds: nexus expungement: assign an old Nexus instance's sagas to another Nexus instance #5136
• We'll also remove the sled time_deleted because it has a lifecycle too complicated to be described that way -- instead, we'll add a time_decommissioned field: Replace sled's time_deleted with time_decommissioned #5131

Questions

• Should we make sled_list hide decommissioned sleds by default?
• Do we need any unique indexes on the sled table -- maybe something like "each non-decommissioned sled ID should have a unique serial number"?

TODO:

• State transition tests.
• Blueprint tests.
• Update all the other places that need the policy.
• Region allocation tests.
• Ensure that sled state isn't decommissioned while trying to change sled policy.
• DB migrations + tests to ensure migrations happen successfully.

[sunshowers]

Almost done here, just need to finish up the region allocation tests. (I'll do the db migrations after acceptance.)

[andrewjstone]

This looks fantastic @sunshowers! Great work. Once you add the DB migrations I'll take a quick look and approve.

[sunshowers]

All right, finally ready for review. I had to break up the migration into two, where the second one just drops the column and deletes the type. I believe this is required for idempotence reasons.

Would love it if folks had thoughts on the questions as well, but those aren't critical, and otherwise this should be ready to go. Thanks for your patience, I ended up spending a lot of time refactoring and chasing down bugs.

[jgallagher]

LGTM! Just some minor questions and nits.

[davepacheco]

Very nice! Thanks for doing this. It's a big chunk of work.

[davepacheco]

Why do we have two schema versions in this PR?

Why do we use the micro number? I thought schema versions were always supposed to bump the major.

[sunshowers]

The reason we need two schema versions is that we have to make two separate transactions to achieve an idempotent result.

The order of operations we need to perform is:

1. Add the new sled_policy and sled_state enum types.
2. Add the new sled_policy and sled_state columns.
3. Update sled_policy to the new value based on the provision_state column.
4. Remove the provision_state column.
5. Remove the sled_provision_state enum types.

1 and 2 are inherently idempotent, so those are fine.

3 by itself is also fine to repeat.

But consider what happens if step 4 occurs, then the process fails, and we go back to 1 again. We'd repeat 3, but the corresponding column no longer exists so it will fail.

Solving this can be done in one of two ways:

1. Find a way to make 3 only perform the update if the provision_state column exists. This is possible, but somewhat complicated.
2. Add a transaction commit between 3 and 4. This is the approach I ended up using.

The reason it's 37.0.0 and 37.0.1 is because both updates are part of the same PR. This follows the pattern in #4261.