[sled-agent] Create an "Executor", which intercepts requests through std::process::Command

[smklein]

Goal

• Make it possible to test the Sled Agent's interface with the underlying Host OS
  • In particular, make it possible to test the commands in illumos-utils, which largely rely on std::process::Command.
  • Eventually, I'd like to expand the Executor introduced in this PR to emulate "all aspects of the host we expect Sled Agent to manipulate", which may include abstraction around FFI calls as well.

Implementation

• Replaces illumos_utils::execute with illumos_utils::host::Executor, which provides an interface to take a std::process::Command as input, execute it, and provide some output. This provides the following advantages:
  • All invocations made through this interface are logged consistently, for both input and output, clearly showing the equivalent command that could be made through a shell.
  • All invocations made through this interface have the opportunity to be intercepted by a FakeExecutor, so they can be explicitly tested. This allows us to reduce our usage on mocks, and also create tests that more clearly align to our system interface.
• Additional tests using these facilities exist in...
  • illumos_utils/src/dladm.rs
  • illumos_utils/src/zone.rs
  • sled-agent/src/services.rs
• Removes mockall dependency, as well as testing feature flag for illumos-utils.

Follow-up work:

• Cope with library interfaces (e.g., the smf and zone crates) which also issue std::process::Command, and ensure they also can be "faked" (edit: DONE, see 0.2.2: Split running from returning commands, add pfexec smf#4 and 0.3.0: Provide access to zone commands instead of running them directly zone#22)
• Expand the Sled Agent test suite to take advantage of these fakes. (edit: Done, but more work can be done here too!)

Fixes #2422

[smklein]

Thanks for the excellent feedback, @jgallagher and @gjcolombo !

[smklein]

I went ahead and split this implementation into different crates, according to the following diagram:

These crates exist in the top level helios directory, under helios/fusion, helios/tokamak, and helios/protostar. All callers depending on helios/tokamak currently only do so via a dev-dependency.

[jordanhendricks]

We chatted some about this PR offline. After some reflection, I have a few higher-level concerns about the broader goal of doing host-level emulation as an abstraction to enable testing in sled agent. Since this is such a big re-architecture of the sled agent, and I am going to be taking on more work in this area shortly, I would really like a chance to get to understand these changes (and maybe discuss more the goals here).

I would appreciate getting some time to chat more about this when I am back from PTO (after Labor Day). And I am also happy to write down my thoughts here as well, and ideas for other designs I would consider.

[smklein]

We chatted some about this PR offline. After some reflection, I have a few higher-level concerns about the broader goal of doing host-level emulation as an abstraction to enable testing in sled agent. Since this is such a big re-architecture of the sled agent, and I am going to be taking on more work in this area shortly, I would really like a chance to get to understand these changes (and maybe discuss more the goals here).

I would appreciate getting some time to chat more about this when I am back from PTO (after Labor Day). And I am also happy to write down my thoughts here as well, and ideas for other designs I would consider.

I would really like to move forward on this PR if I'm allowed to. Please let me know what I can do to unblock this.

[jordanhendricks]

For posterity: I had some questions about the design and goals of the PR and was not attempting to allow or disallow anything; as I said before, I wanted to understand the approach and goals better. @smklein and I chatted offline and have some concrete action items to take to address my concerns, but I don't have any objections to merging this PR.