Reconfigurator: Move external IP allocation/deallocation into a transaction conditional on the current blueprint

[jgallagher]

See #6239 and #6207 (comment) for additional context, but briefly summarizing:

• Prior to this PR, blueprint execution would deallocate and allocate external IP and NIC records independently
• If two Nexus instances are trying to realize two different blueprints, this could result in them stomping on each other

Andrew suggested that we put the external IP resource set under a generation number, similar to how we mark generations for zones and DNS config. However, in both of those cases, there's an external entity (sled-agent or the DNS zone, respectively) that's ultimately responsible for rejecting updates with an old generation number. In this external networking case, there is no other entity - other parts of Nexus consume this information directly from CRDB. In chat, we discussed that we think it's equivalent to putting the set of allocation/deallocation operations inside a transaction that's conditional on the blueprint being executing still being the current target.

Most of this PR is moving the external_networking module from nexus-reconfigurator-execution into nexus-db-queries where it can be executed inside the new blueprint_ensure_external_networking_resources() method. This method opens a transaction, performs a SELECT ... FOR UPDATE to confirm that the blueprint we're ensuring is the current target, and then does the entire set of deallocation / allocation operations in that transaction. I added a test with some kinda gnarly #[cfg(test)] bits to control execution flow to show how the SELECT ... FOR UPDATE works: if the blueprint isn't the target, we immediately fail; if we are the target, any attempt to change the target will be queued by CRDB until we've finished our transaction.

Fixes #6239

[andrewjstone]

Great stuff @jgallagher

[andrewjstone]

The booleans at the call sites are somewhat confusing. I think it's probably overkill to make this two separate functions, even though that's an option. How about doing what we do elsewhere, and instead of passing true or false directly, create a local variable named select_for_update and pass that in.

[jgallagher]

Yeah I was already borderline on the call site booleans; thanks for the nudge. I don't love the local variable thing, so I replaced this with a two variant enum in 989c5db - how does that read?

[andrewjstone]

Great test!

I usually do not like to add test only code to production methods. However, in this case, I think ensuring that CRDB transaction serialization works as we expect warrants it. This also serves as a useful regression in case that behavior changes.

[andrewjstone]

I wonder if select_for_update, which should take a write lock on the target table, or at least target row, also blocks reading from that table. I'm not sure if that would be a problem, and don't think it warrants a test. I'm just kinda curious at this point.

[jgallagher]

I was also curious, and it was easy enough to expand this test to see! It does indeed block reads: 7fee264