Skip to content

Commit

Permalink
site: progress on page about the test pattern DSL
Browse files Browse the repository at this point in the history
  • Loading branch information
bobbobbio committed Jan 12, 2024
1 parent aedb5e3 commit 75ceb7b
Show file tree
Hide file tree
Showing 4 changed files with 196 additions and 6 deletions.
3 changes: 2 additions & 1 deletion site/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@
- [Running Tests](./cargo_metest/running_tests.md)
- [Filtering Tests](./cargo_metest/filtering_tests.md)
- [Test Pattern DSL](./cargo_metest/test_pattern_dsl.md)
- [`-i` and `-x` Flags](./cargo_metest/i_and_x_flags.md)
- [Test Pattern DSL BNF](./cargo_metest/test_pattern_dsl/bnf.md)
- [`--include` and `--exclude` Flags](./cargo_metest/include_and_exclude_flags.md)
- [Listing](./cargo_metest/listing.md)
- [Configuration](./cargo_metest/configuration.md)
- [Execution Environment](./cargo_metest/execution_environment.md)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
# `-i` and `-x` Flags
# `--include` and `--exclude` Flags

These flags are about filtering which tests `cargo-metest` runs.

The `-i` and `-x` flags accept a snippet of the [Test Pattern
DSL](./test_pattern_dsl.md). The `-i` flag includes any tests matching the
pattern and the `-x` flag excludes any test which matches the pattern. They are
both able to be provided multiple times via the command line.
The `--include` and `--exclude` flags (shorted as `-i` and `-x`) accept a
snippet of the [Test Pattern DSL](./test_pattern_dsl.md). The `-i` flag includes
any tests matching the pattern and the `-x` flag excludes any test which matches
the pattern. They are both able to be provided multiple times via the command
line.

The tests that are ran are the set which matches any of the `-i` flag patterns
after subtracting the set which matches any of the `-x` flag patterns. To put
Expand Down
151 changes: 151 additions & 0 deletions site/src/cargo_metest/test_pattern_dsl.md
Original file line number Diff line number Diff line change
@@ -1 +1,152 @@
# Test Pattern DSL

This domain-specific language has been designed specifically for `cargo-metest`
to let users easily describe a set of tests to run.

If you are a fan of formal explanations check out the
[BNF](./test_pattern_dsl/bnf.md). Otherwise, this page will attempt to give a
more informal explanation of how the language works.

## Selectors

Conceptually every test has a variety of properties about it. Selecting tests
via a pattern is the process of describing something of these properties.

We call the usage of these properties in expressions "selectors". Here is a
listing of them.

- **name**: Test function name with its module path prepended (e.g. mod1::test1)
- **package**: Package name the test is a part of
- **binary / test / benchmark / example**: Which binary is the test
a part of. See [Selecting Executables](#selecting-executables)

The simplest usage of the language is to craft an expression which expresses
something about one of these properties.

## Matchers
In order to express something about one of the test properties we need to use a
matcher. Here is a listing of those

- **equals**: true only if the property matches exactly
- **contains**: true only if the property contains the given sub-string
- **starts_with**: true only if the property starts with the given sub-string
- **ends_with**: true only if the property ends with the given sub-string
- **matches**: true only if the property matches the given regular expression
- **globs**: true only if the property matches the given glob expression

## Crafting a Simple Expression
Using a property and a matcher we can craft a simple expression. For example, if
we want to only select tests which have the exact name "my_mod::my_test" we do
something like.

```
name.equals(my_mod::my_test)
```

Or if we want to select all the tests in the package called foobar, we would do

```
package.equals(foobar)
```

## Compound Expressions
To create more interesting expression, simple expressions can be combined using
the following operators

- **`!`, `not`**: logical not
- **`&&`, `and`**: logical and
- **`||`, `or`**: logical or
- **`\`, `-`, `minus`**: same as && !(...)

Here are some examples

- `!name.equals(foobar)` only tests not named foobar
- `package.equals(foo) && name.equals(bar)` test(s) named bar in package foo
- `package.equals(foo) - name.equals(bar)` tests in package foo except ones named bar

When combining multiple operators, parenthesis are allowed to control the
precedence.

To select tests named foo or bar in package baz, you can do
```
(name.equals(foo) || name.equals(bar)) && package.equals(baz)
```

To select tests named bar in package baz or tests named foo from any package
```
name.equals(foo) || (name.equals(bar) && package.equals(baz))
```

## Selecting Executables
A package can contain a few different types of executables. When selecting tests
from an executable the type of executable can be selected using the following
selectors.

- **benchmark**: a benchmark
- **binary**: a binary
- **example**: an example
- **library**: a library
- **test**: an integration test

For more details about these different types of targets, check out the [cargo
documentation](https://doc.rust-lang.org/cargo/reference/cargo-targets.html).

These selectors make complete expression in and of themselves, but also they can
be used to match the name of the executable (with the exception of `library`)

- `benchmark.equals(my_benchmark)` test the benchmark called my_benchmark
- `name.equals(cli_test) && binary` test tests called cli_test and inside binaries
- `binary.equals(cli_a) || binary.equals(cli_b)` tests from binaries cli_a, cli_b
- `library` test only the tests found in libraries

Note that you can't do something like `library.equals(foo)`, this is not allowed
in favor of doing `package.equals(foo) && library`. This is because a package is
only allowed one library.

## Special Expressions
These other expressions are useful occasionally

- **`true`, `all`, `any`**: selects all tests
- **`false`, `none`**: selects no tests

When you provide no filter to `cargo-metest` it acts as if typed `cargo metest
-i all`

## Abbreviations

The matchers and selectors can be shortened as long as it doesn't create some
ambiguity

For example, the following are all the same
```
name.equals(foo)
name.eq(foo)
n.eq(foo)
```

We can abbreviate `name` to `n` since no other selector starts with "n", but we
can't abbreviate `equals` to `e` because there is another selector `ends_with`
which also starts with an "e".

## Using Regex and Glob Matchers

The regular expression and glob matchers work how you would expect. They allow
you to pass a regular expression or glob pattern and it uses that to evaluate if
the expression matches some test.

Before showing some examples, first lets talk about the parenthesis we've used
when writing expressions. Matchers actually support other punctuation also

- **`(` and `)`**
- **`[` and `]`**
- **`{` and `}`**
- **`<` and `>`**
- **`/` and `/`**

This feature is useful for matchers which accept strings that have potentially
the same punctuation as part of it

Here are some examples of regular expressions and glob patterns

- `name.matches/foo::.*/` tests starting with foo::
- `name.globs(bar::*)` tests starting with bar::
37 changes: 37 additions & 0 deletions site/src/cargo_metest/test_pattern_dsl/bnf.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Test Pattern DSL BNF

Included on this page is the [Backus-Naur
form](https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form) notation for the
DSL

```BNF
pattern := or-expression
or-expression := and-expression
| or-expression or-operator and-expression
or-operator := "|" | "||" | "or"
and-expression := not-expression
| and-expression and-operator not-expression
| and-expression diff-operator not-expression
and-operator := "&" | "&&" | "and" | "+"
diff-operator := "\" | "-" | "minus"
not-expression := simple-expression
| not-operator not-expression
not-operator := "!" | "~" | "not"
simple-expression := "(" or-expression ")"
| simple-selector
| compound-selector
simple-selector := simple-selector-name
| simple-selector-name "(" ")"
simple-selector-name := "all" | "any" | "true"
| "none" | "false"
| "library"
| compound-selector-name
compound-selector := compound-selector-name "." matcher-name
matcher-parameter
compound-selector-name := "name" | "binary" | "benchmark" | "example" |
"test" | "package"
matcher-name := "equals" | "contains" | "starts_with" | "ends_with" |
"matches" | "globs"
matcher-parameter := <punctuation mark followed by characters followed by
matching punctuation mark>
```

0 comments on commit 75ceb7b

Please sign in to comment.