[xtask] Convert ci_download bash scripts to Rust

[smklein]

Implements cargo xtask download, which provides options to replace the existing scripts:

• ci_download_clickhouse
• ci_download_cockroachdb
• ci_download_console
• ci_download_dendrite_openapi
• ci_download_dendrite_stub
• ci_download_maghemite_mgd
• ci_download_maghemite_openapi
• ci_download_softnpu_machinery
• ci_download_thundermuffin
• ci_download_transceiver_control

This PR additionally introduces the cargo xtask download all option, which attempts to download all artifacts concurrently.

Somewhat related to #3939

[smklein]

Example usage:

 $ cargo xtask download --help
    Finished dev [unoptimized + debuginfo] target(s) in 0.80s
     Running `target/debug/xtask download --help`
Download binaries, OpenAPI specs, and other out-of-repo utilities

Usage: xtask download [OPTIONS] <TARGETS>...

Arguments:
  <TARGETS>...
          The targets to be downloaded. This list is additive

          Possible values:
          - all:                 Download all targets
          - clickhouse:          Clickhouse binary
          - cockroach:           CockroachDB binary
          - console:             Web console assets
          - dendrite-openapi:    Dendrite OpenAPI spec
          - dendrite-stub:       Stub Dendrite binary tarball
          - maghemite-mgd:       Maghemite mgd binary
          - maghemite-openapi:   Maghemite OpenAPI spec
          - softnpu:             SoftNPU, an admin program (scadm) and a pre-compiled P4 program
          - thundermuffin:       Thundermuffin binary, for sending non-stop buffers between devices
          - transceiver-control: Transceiver Control binary

Options:
      --output-dir <OUTPUT_DIR>
          The path to the "out" directory of omicron
          
          [default: out]

      --versions-dir <VERSIONS_DIR>
          The path to the versions and checksums directory
          
          [default: tools]

  -h, --help
          Print help (see a summary with '-h')

[david-crespo]

@rcgoodfellow did you say maghemite openapi might not be necessary anymore?

[rcgoodfellow]

@rcgoodfellow did you say maghemite openapi might not be necessary anymore?

It's still necessary to download ddmd and mgd from maghemite CI. But I do not believe downloading the openapi spec should be needed anymore.

[smklein]

CI failing looks like test flake, filed #5498

[smklein]

@rcgoodfellow did you say maghemite openapi might not be necessary anymore?

It's still necessary to download ddmd and mgd from maghemite CI. But I do not believe downloading the openapi spec should be needed anymore.

Removed this in the rust implementation in 0cd628c

[smklein]

The follow-up to remove the bash scripts is in: #5482

[rcgoodfellow]

Using the rev value from Cargo.toml is fine here, and we can probably get rid of the maghemite_mg_openapi_version file.

[smklein]

I agree that we probably should do this, but it's slightly sticky because the flake.nix file also depends on having this as a rev.

I agree that we should minimize version dups - but if it's okay with you, I'm going to punt this to a different PR, since my goal is "parity with the bash scripts" before changing this download process too much.

[hawkw]

The Nix flake could be rewritten to also parse the rev from Cargo.toml, FWIW, but I would love to have advance warning so I can make sure to actually do that. :)

[rcgoodfellow]

SGTM

[iliana]

LGTM given the goal of "parity with the bash scripts" until we remove those.

The additional deps in xtask increase the build time from 30s to 40s on my machine, but I don't think that's enough to warrant this being an external xtask.