[nexus] Make most transactions automatically retry

[smklein]

Integrates automatic transaction retry into Nexus for most transactions.

Additionally, this PR provides a "RetryHelper" object to help standardize how transaction retry is performed.
Currently, after a short randomized wait (up to an upper bound), we retry unconditionally, emitting each
attempt to Oximeter for further analysis.

• Depends on Add function for automatic transaction retry async-bb8-diesel#58
• As noted in Add function for automatic transaction retry async-bb8-diesel#58, this will require customizing CRDB session variables to work correctly. (Edit: this is done on each transaction)

Part of https://github.com/oxidecomputer/customer-support/issues/46
Part of #3814

[bnaecker]

I focused on the Oximeter integration portions in transaction_retry.rs. I've a few suggestions and comments, but overall this looks pretty solid. Thanks for wrangling this massive diff!

[luqmana]

Took a look with a focus on some of the parts I've been in more (rack & network_interface) and looks good overall. A couple of comments but not blockers

[luqmana]

Hmm, the retry aside for a moment, the only caller for nexus_external_addresses is Nexus::silo_create which later calls Datastore::silo_create that does its own transaction. Feels like these queries should just be part of that transaction?

[smklein]

I agree with you that these could become a single transaction, but I'm not 100% sure they should be -- there's logic to construct DNS updates via DnsVersionUpdateBuilder in between these two transactions, and by combining them, we're forcing that logic to happen in a transactional context.