Support pass-through for all arguments that cargo test supports

[humb1t]

Right now if you pass test-thread argument to cargo test, cucumber will fail:

     Running `/home/humb1t/workspace/soramitsu/iroha/target/debug/deps/cucumber-09561164203f68fb --test-threads=1 --skip=cucumber`
error: Found argument '--test-threads' which wasn't expected, or isn't valid in this context

Command: cargo test --workspace --verbose -- --test-threads=1

It will be great to support this feature or at list ignore it instead of failure.

[bbqsrc]

If anybody wishes to take on this issue, I'd be happy to mentor.

[fhntvv]

@bbqsrc hello!
I would like to take this issue. what should I start with?

[bbqsrc]

Hi @privalou :)

If we take an ordinary Rust crate and run cargo test -- --help, we'll get what I've pasted in this gist.

I think the absolute first step to take would be to modify this code so that unsupported flags are captured in a "catch all" and printed as warnings. We are currently using clap for parsing command line input. You could experiment with enabling TrailingVarArg and see if we can at least capture unhandled flags.

If this isn't feasible, or you're struggling, group back here and we'll chat again. 😄

[fhntvv]

Thanks for your guidance! But to be honest it is not quite clear to me how to implement "catch all" logic at cli.rs

[bbqsrc]

@privalou the premise is that if you have a mechanism that can catch all "unknown arguments" into a vector, you can handle them separately. If that list isn't empty, then it means some unknown arguments were parsed, but not handled.

An example could be a hypothetical app called ./runner. It accepts the flags -a and --yes, but nothing else. We called ./runner -a --foo --yes. What we would expect to happen in this circumstance, is that -a and --yes would be handled, and --foo added to the unhandled vector in the parsing tool we use. If we passed ./runner -a --yes --foo, we'd expect the same behaviour. If we passed ./runner some other -a --yes --foo stuff, we'd expect ["some", "other", "--foo", "stuff"] to be captured in our unhandled vector.

Why do this? Well, unhandled flags are blocking our test runner from continuing right now, so at least catching them and ignoring them would allow the tests to run! Later we can work out how to handle them properly.

It is absolutely possible that clap doesn't support this behaviour at all, so I linked you to the one feature I think might work that way. If it turns out it doesn't, we investigate other options. 😄 One other might be the crate gumdrop.

[fhntvv]

@bbqsrc
this helped me a lot :)
but one more stupid question: should I test my changes at another project using cucumer_rust as lib or I can test it directly in this project?

[bbqsrc]

It might be easier to test it with another project, and use a path dependency to your modified cucumber crate.

You can do that with the following in your Cargo.toml in place of the usual test dependency:

cucumber = { path = "path/to/cucumber-rust" }

The path can be relative from your testing crate, or absolute.

And the question isn't stupid. These things are not intuitive when you're new to Rust, and test runners like this are even less intuitive in the current cargo model. :)

[fhntvv]

it seems like it is not failing when we are passing parameters like this
cargo test cucumber -- --test-threads=1
I will try to dive more into clap to ignore unrecognized params

[humb1t]

For @privalou, some additional information about Rust + Cargo testing:

• https://doc.rust-lang.org/book/ch11-02-running-tests.html
• https://doc.rust-lang.org/cargo/commands/cargo-test.html

[humb1t]

Answer to @privalou question about dependency between cli.rs and test arguments.

Basically cucumber.rs::main method is running by cargo while will run cucumber tests inside.

cli.rs will pass arguments for Cargo and Rust and may use them in cucumber runner (https://github.com/bbqsrc/cucumber-rust/blob/main/src/runner.rs).

Here you can see that cli options are used to filter scenarios for example.

[fhntvv]

@humb1t
Thank you for your response, but this part is not quite clear to me

Here you can see that cli options are used to filter scenarios for example.

I don't see where cli(self) method is called.

[humb1t]

@privalou https://github.com/bbqsrc/cucumber-rust/blob/2d5de3261fe9ec2368cee96eaed36d11cb786287/src/cucumber.rs#L131

[fhntvv]

@bbqsrc
Hello! it seems like it might be reasonable to get rid of this cli-lib "clap" because it is not used anywhere.
Btw, cargo test --test cucumber --verbose -- -nocapture --test-threads=1 command is working fine for me with an example from README

[tyranron]

@ilslv this should be paired with #134

[ilslv]

@tyranron this feature can't really be implemented, as those options are implemented by default test harness, which we are turning off with harness = false.

But #144 supports all possible similar options and introduces ability to add custom CLI options. For --test-threads see #144 (comment)

All possible options supported by default test harness to be sure that I didn't miss anything important:

Options:
        --include-ignored 
                        Run ignored and not ignored tests
        --ignored       Run only ignored tests
        --force-run-in-process 
                        Forces tests to run in-process when panic=abort
        --exclude-should-panic 
                        Excludes tests marked as should_panic
        --test          Run tests and not benchmarks
        --bench         Run benchmarks instead of tests
        --list          List all tests and benchmarks
    -h, --help          Display this message
        --logfile PATH  Write logs to the specified file
        --nocapture     don't capture stdout/stderr of each task, allow
                        printing directly
        --test-threads n_threads
                        Number of threads used for running tests in parallel
        --skip FILTER   Skip tests whose names contain FILTER (this flag can
                        be used multiple times)
    -q, --quiet         Display one character per test instead of one line.
                        Alias to --format=terse
        --exact         Exactly match filters rather than by substring
        --color auto|always|never
                        Configure coloring of output:
                        auto = colorize if stdout is a tty and tests are run
                        on serially (default);
                        always = always colorize output;
                        never = never colorize output;
        --format pretty|terse|json|junit
                        Configure formatting of output:
                        pretty = Print verbose output;
                        terse = Display one character per test;
                        json = Output a json document;
                        junit = Output a JUnit document
        --show-output   Show captured stdout of successful tests
    -Z unstable-options Enable nightly-only flags:
                        unstable-options = Allow use of experimental features
        --report-time [plain|colored]
                        Show execution time of each test. Available values:
                        plain = do not colorize the execution time (default);
                        colored = colorize output according to the `color`
                        parameter value;
                        Threshold values for colorized output can be
                        configured via
                        `RUST_TEST_TIME_UNIT`, `RUST_TEST_TIME_INTEGRATION`
                        and
                        `RUST_TEST_TIME_DOCTEST` environment variables.
                        Expected format of environment variable is
                        `VARIABLE=WARN_TIME,CRITICAL_TIME`.
                        Durations must be specified in milliseconds, e.g.
                        `500,2000` means that the warn time
                        is 0.5 seconds, and the critical time is 2 seconds.
                        Not available for --format=terse
        --ensure-time   Treat excess of the test execution time limit as
                        error.
                        Threshold values for this option can be configured via
                        `RUST_TEST_TIME_UNIT`, `RUST_TEST_TIME_INTEGRATION`
                        and
                        `RUST_TEST_TIME_DOCTEST` environment variables.
                        Expected format of environment variable is
                        `VARIABLE=WARN_TIME,CRITICAL_TIME`.
                        `CRITICAL_TIME` here means the limit that should not
                        be exceeded by test.