CmdError error messages should provide more information

[papertigers]

Fixes: #4419

This commit is a little invasive at various call sites so I would appreciate some feedback.

Before this commit you would see:

sled-agent: Failed to delete all XDE devices

After this commit we now see:

sled-agent: Failed to delete all XDE devices

Caused by:
    0: Failure interacting with the OPTE ioctl(2) interface: command ListPorts failed: BadApiVersion { user: 25, kernel: 23 }
    1: command ListPorts failed: BadApiVersion { user: 25, kernel: 23 }

[papertigers]

CI failed because I neglected the non illumos cfg sections. Is there a good way to make sure one doesn't miss these in the future before CI runs?

[jgallagher]

Thanks! These feel like kinda tedious changes, but better error messages are a huge 👍 for everybody.

[jgallagher]

CI failed because I neglected the non illumos cfg sections. Is there a good way to make sure one doesn't miss these in the future before CI runs?

In theory one can do something like cargo check --target x86_64-unknown-linux-gnu, but I suspect that won't work for annoying reasons (OpenSSL for one). If I'm working in cfg-controlled areas like this I end up just building on both platforms locally, which isn't super satisfying. Would be happy to hear from others if there's a better option.

[citrus-it]

Thanks! These feel like kinda tedious changes, but better error messages are a huge 👍 for everybody.

Speaking as someone who hit exactly this today and only managed to resolve it quickly because I was chatting to @papertigers - most definitely!

[papertigers]

Thanks! These feel like kinda tedious changes, but better error messages are a huge 👍 for everybody.

Speaking as someone who hit exactly this today and only managed to resolve it quickly because I was chatting to @papertigers - most definitely!

And because we were speaking I realized that ./tools/install_prerequisites.sh gets you a working env on macOS making the cfg illumos bits easier to spot locally. Thanks!

[papertigers]

Sorry for all the commit noise on this. I used a macro to fix up all of the e.into() -> anyhow!(e) so that we are consistent everywhere. I also applied all of your other suggested fixes @jgallagher. Hopefully this is in better shape now -- I will try to do one more manual run on a bench gimlet before merging this.

[jgallagher]

Looks great - thanks!

[papertigers]

With the latest patchset things still look good:

sled-agent: Failed to delete all XDE devices

Caused by:
    0: Failure interacting with the OPTE ioctl(2) interface: command ListPorts failed: BadApiVersion { user: 25, kernel: 23 }
    1: command ListPorts failed: BadApiVersion { user: 25, kernel: 23 }

[jordanhendricks]

Thanks for jumping on this so quickly! Will be so nice to have when someone hits this again.