Long runtimes for integration tests

[simonsan]

https://github.com/iqlusioninc/abscissa/blob/main/cli/template/tests/acceptance.rs.hbs#L31

I built an integration tests with some stages, unfortunately the default configuration of the CmdRunner invokes cargo each time which rebuilds/checks the binary each time as well. There was a huge performance penalty because of that. When I initialized the CmdRunner with:

pub static RUNNER: Lazy<CmdRunner> = Lazy::new(|| {
    let mut runner = CmdRunner::new(env!("CARGO_BIN_EXE_RUSTIC"));
    runner.exclusive().capture_stdout();
    runner
});

the gain was huge, tests that ran 90 seconds on my device ran in 1 second.

src.: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-crates

Is there any downsides about this, that I don't understand/know? If not, would it be reasonable to adopt that as a default option?

[simonsan]

TL;DR: this is a worthwhile change, but I'd like to see it on the framework level (i.e. in abscissa) rather than fix up every user individually.

I've compared this against the main branch, measuring the second run of the tests (once the crates.io index cloning is done, which takes up the vast majority of the time), and it takes 50 seconds regardless of these changes.

With an up-to-date crates.io index, running cargo clean; cargo test --release --all-features --no-run and then measuring cargo test --release --all-features shows the new approach to be faster, because cargo run is invoked without --release and genuinely needs compile the debug build instead of using the release build that's already been compiled.

Originally posted by @Shnatsel in rustsec/rustsec#883 (comment)

Shall I make a PR and add a with_bin method on CmdRunner utilizing this?

[Shnatsel]

Does abscissa know the name of the binary? If so, it should just use it internally in CmdRunner::default()

If not, we'll have to just update documentation and examples instead to explain this superior approach.