fix phased startup of zones when sled agent starts up

[davepacheco]

When sled agent starts up, it starts up the zones it's responsible for in a specific order:

• internal DNS
• NTP (starting boundary NTP zones implicitly requires that internal DNS be up)
• (wait for time sync)
• start everything else

At least, that's what the comments say, and that appears to be what it should do. But the first call to start internal DNS also starts the NTP zones today. This appears to work by accident because the DNS zones happen to get processed first.

This was discussed on #4466:
#4466 (comment)
#4466 (comment)

This change fixes that.

[davepacheco]

This tiny change came along for the ride. This is not an error condition. It happens during normal operation.

[jgallagher]

Thanks!

[citrus-it]

NTP (starting boundary NTP zones implicitly requires that internal DNS be up)

What's the dependency here, being able to configure the boundary services for external network access?
(Chrony itself is not dependent on internal DNS, hence the question).

[davepacheco]

NTP (starting boundary NTP zones implicitly requires that internal DNS be up)

What's the dependency here, being able to configure the boundary services for external network access? (Chrony itself is not dependent on internal DNS, hence the question).

It's a dependency in the sled agent code that starts the boundary NTP zones, not the zones themselves. The comment from the code explains:

omicron/sled-agent/src/services.rs

Lines 839 to 843 in e3887d5

	// Initialize internal DNS only first: we need it to look up the
	// boundary switch addresses. This dependency is implicit: when we call
	// `ensure_all_omicron_zones` below, we eventually land in
	// `opte_ports_needed()`, which for some service types (including Ntp
	// but _not_ including InternalDns), we perform internal DNS lookups.

[citrus-it]

It's a dependency in the sled agent code that starts the boundary NTP zones, not the zones themselves. The comment from the code explains:

Got it thanks, it's a dependency needed for being able to find boundary services.

[davepacheco]

Besides CI, I tested the tip of this branch (3320239) on madrid:

• mupdate all sleds
• deploy Omicron (svcs -Zxv clear, timestamps up to date, all sleds on the right version, expected zones present)
• provision instance
• ssh into instance
• cold boot (uadmin 2 1 on all four sleds)
• wait for it to come up (svcs -Zxv clear, timestamps up to date, expected zones present)
• provision instance
• ssh into instance

That all worked.

[davepacheco]

This also happens during normal operation prior to the ledgers being initially written (during initial setup, if nothing else) so I found it misleading to make these warnings.