Skip to content

Latest commit

 

History

History
166 lines (129 loc) · 7.23 KB

TestingCombinations.md

File metadata and controls

166 lines (129 loc) · 7.23 KB

Testing Combinations

Contents

When to use Combinations

You have a function that takes, for example, 3 parameters, and you want to test its behaviour with a bunch of different values for each of those parameters.

If you have only one parameter that you want to vary, check out How to Test a Variety of Values for One Input.

Steps

  1. Copy this starter text, and adjust for the number of inputs that you have.

TEST_CASE("CombinationsStartingPoint")
{
    std::vector<std::string> inputs1{"input1.value1", "input1.value2"};
    std::vector<std::string> inputs2{"input2.value1", "input2.value2", "input2.value3"};
    ApprovalTests::CombinationApprovals::verifyAllCombinations(
        "TITLE",
        [&](auto /*input1*/, auto /*input2*/) { return "placeholder"; },
        inputs1,
        inputs2);
}

snippet source | anchor

  1. Modify each input container for your chosen values.
  2. Make sure each input type can be converted to a string (See To String)
  3. Run it, and make sure that you have your inputs wired up correctly.

If they are wired up correctly, you will see a file that looks like this: it is the left hand side of the file that matters at this point: all combinations of your own input values should be listed:

TITLE


(input1.value1, input2.value1) => placeholder
(input1.value1, input2.value2) => placeholder
(input1.value1, input2.value3) => placeholder
(input1.value2, input2.value1) => placeholder
(input1.value2, input2.value2) => placeholder
(input1.value2, input2.value3) => placeholder

snippet source | anchor

  1. Implement the body of your lambda
  2. Make sure that your lambda's return value also has an ostream operator<<
  3. Change the TITLE to something meaningful
  4. Run it, and approve the output.

The Basics

You can use CombinationApprovals::verifyAllCombinations to test the content of multiple containers.

This makes a kind of approval test matrix, automatically testing all combinations of a set of inputs. It's a powerful way to quickly get very good test coverage.

In this small example, all combinations of {"hello", "world"} and {1, 2, 3} are being used:

TEST_CASE("YouCanVerifyCombinationsOf2")
{
    std::vector<std::string> v{"hello", "world"};
    std::vector<int> numbers{1, 2, 3};
    ApprovalTests::CombinationApprovals::verifyAllCombinations(
        [](std::string s, int i) {
            return std::string("(") + s + ", " + std::to_string(i) + ")";
        },
        v,
        numbers);
}

snippet source | anchor

The format is carefully chosen to show both inputs and outputs, to make the test results easy to interpret. The output looks like this:

(hello, 1) => (hello, 1)
(hello, 2) => (hello, 2)
(hello, 3) => (hello, 3)
(world, 1) => (world, 1)
(world, 2) => (world, 2)
(world, 3) => (world, 3)

snippet source | anchor

For advice on effective formatting, see Tips for Designing Strings. As you write out larger volumes of data in your approval files, experience has shown that the choice of layout of text in approval files can make a big difference to maintainability of tests, when failures occur.

Passing in a Reporter

Note: Over releases, the position of the optional Reporter parameter to verifyAllCombinations has changed, as the code has evolved:

Release Position of optional Reporter argument
Before v.6.0.0 The optional Reporter argument goes after all the inputs
In v.6.0.0 The optional Reporter argument should be the second argument.
After v.6.0.0 The optional Reporter argument should be the first argument.
After v.8.7.0 Reporter is now stored in Options, which should be the first argument.

See:

C++ Language Versions

If you are using C++11, you will need to supply the exact parameter types to your lambda:

ApprovalTests::CombinationApprovals::verifyAllCombinations(
    [](const std::string& input1, const int input2, const double input3) {
        return functionThatReturnsSomethingOutputStreamable(input1, input2, input3);
    }, // This is the converter function
    listOfInput1s,
    listOfInput2s,
    listOfInput3s);

snippet source | anchor

If you are using C++14 or above, you can simplify this by using auto or auto& for the lambda parameters:

ApprovalTests::CombinationApprovals::verifyAllCombinations(
    [](auto& input1, auto& input2, auto& input3) {
        return functionThatReturnsSomethingOutputStreamable(input1, input2, input3);
    }, // This is the converter function
    listOfInput1s,
    listOfInput2s,
    listOfInput3s);

snippet source | anchor


Back to User Guide