lest – lest errors escape testing

This tiny C++11 test framework is based on ideas and examples by Kevlin Henney [1,2] and on ideas found in the CATCH test framework by Phil Nash [3].

Let writing tests become irresistibly easy and attractive.

Contents

Example usage
In a nutshell
License
Dependencies
Installation
Usage
Synopsis
Variants of lest
Features of lest
Reported to work with
Building tests and examples
Contributions to lest
Other test frameworks
Notes and references

Example usage

#include "lest.hpp"

using namespace std;

const lest::test specification[] =
{
    CASE( "Empty string has length zero (succeed)" )
    {
        EXPECT( 0 == string(  ).length() );
        EXPECT( 0 == string("").length() );
    },

    CASE( "Text compares lexically (fail)" )
    {
        EXPECT( string("hello") > string("world") );
    },

    CASE( "Unexpected exception is reported" )
    {
        EXPECT( (throw std::runtime_error("surprise!"), true) );
    },

    CASE( "Unspecified expected exception is captured" )
    {
        EXPECT_THROWS( throw std::runtime_error("surprise!") );
    },

    CASE( "Specified expected exception is captured" )
    {
        EXPECT_THROWS_AS( throw std::bad_alloc(), std::bad_alloc );
    },

    CASE( "Expected exception is reported missing" )
    {
        EXPECT_THROWS( true );
    },

    CASE( "Specific expected exception is reported missing" )
    {
        EXPECT_THROWS_AS( true, std::runtime_error );
    },
};

int main( int argc, char * argv[] )
{
    return lest::run( specification, argc, argv );
}

Compile and run

prompt>g++ -Wall -Wextra -std=c++11 -I../include/lest -o 05_select.exe 05_select.cpp && 05_select.exe
05_select.cpp:17: failed: Text compares lexically (fail): string("hello") > string("world") for "hello" > "world"
05_select.cpp:22: failed: got unexpected exception with message "surprise!": Unexpected exception is reported: (throw std::runtime_error("surprise!"), true)
05_select.cpp:37: failed: didn't get exception: Expected exception is reported missing: true
05_select.cpp:42: failed: didn't get exception of type std::runtime_error: Specific expected exception is reported missing: true
4 out of 7 selected tests failed.

With Buck:

prompt> buck run example/:05_select
...

In a nutshell

lest is a small C++11 test framework for unit testing, regression testing, Test-driven development (TDD) and Behaviour-driven design (BDD). It replicates innovative ideas in C++ testing from the Catch test framework such as function-level fixtures and expression-decomposing assertion macros in a form that is compact enough to read in five minutes. The lest_cpp03 variant provides similar capabilities to use with C++98/03 compilers.

Features and properties of lest are ease of installation (single header), no boilerplate code, traditional unit test cases and BDD style scenarios, strings as test names, function-level fixtures, expression-decomposing assertion macros, support for floating point comparison, test selection from commandline, test duration timing, test randomisation and sorting, display of passing tests, colourised output (compile-time option), C++11 code and a C++98/03 variant with comparable features (compilable with VC6 and as C++11),

Not provided are things present in other test frameworks, such as suites of tests, parameterised tests, templated tests, test data generators, built-in hamcrest matchers (see variants of lest), customisable reporting, easy logging of extra information, breaking into a debugger, concurrent execution of tests, shielded execution of tests, Visual Studio Test Adapter.

License

lest uses the Boost Software License.

Dependencies

lest has no other dependencies than the C++ standard library.

Installation

lest is a single-file header-only library. Put lest.hpp, or a variant of it such as lest_cpp03.hpp directly into the project source tree or somewhere reachable from your project.

Usage

Making a case – example
Asserting for success – example
Asserting for failure – example
Asserting for exceptions – example
Asserting for a specific exception – example
Comparing floating point numbers – example.
Auto test registration – single-file example
Auto test registration with multiple source files – example part 1, 2, 3.
Non-auto test registration – single-file example
Non-auto test registration with multiple source files – example part 1, 2, 3.
Using fixtures – example.
Using lest assertions in a (reusable) user-defined function or lambda – function, templated function, lambda.
Writing the test main function – single-file auto, non-auto – multi-file auto part 1, 2, 3, non-auto part 1, 2, 3.
Integrating lest with Trompeloeil – example.
Reporting a user-defined type – example.
Running tests

Synopsis

Contents

Command line
Test case macro
Fixture macros
Assertion macros
BDD style macros
Module registration macro
Other macros
Namespace
Tests
Main
Main (Trompeloeil)
Floating point comparison
Reporting a user-defined type

Command line

Usage: test [options] [test-spec ...]

Options:

-h, --help, this help message
-a, --abort, abort at first failure
-c, --count, count selected tests
-g, --list-tags, list tags of selected tests
-l, --list-tests, list selected tests
-p, --pass, also report passing tests
-t, --time, list duration of selected tests
--order=declared, use source code test order (default)
--order=lexical, use lexical sort test order
--order=random, use random test order
--random-seed=n, use n for random generator seed
--random-seed=time, use time for random generator seed
--repeat=n, repeat selected tests n times (-1: indefinite)
--version, report lest version and compiler used
--, end options

Test specification:

"@", "*": all tests, unless excluded
empty: all tests, unless tagged [hide] or [.optional-name]
"text": select tests that contain text (case insensitive)
"!text": omit tests that contain text (case insensitive)

Test descriptions can contain tags such as [option], [hide] and [.integration]. Tests that contain the tag [hide] or a tag that starts with [. in their description are skipped, unless they are specifically selected by specifying "@", "*", or by specifying (part of) the tag.

Test specifications can be combined and are evaluated left-to-right. For example: a !ab abc selects all tests that contain 'a', except those that contain 'ab', but include those that contain 'abc'.

When regular expression selection has been enabled (and works), test specifications can use the regular expression syntax of std::regex_search(). See also lest_FEATURE_REGEX_SEARCH in section Other Macros.

Test case macro

A lest test specification can consist of a) one or more arrays of test cases that use lambdas, or b) auto-registered test cases that use free functions. See also macro lest_FEATURE_AUTO_REGISTER.

CASE( "proposition", ...) { code } (array of cases)
Describe the expected behaviour to test for and specify the actions and expectations. After the description you can add a lambda capture list to refer to symbols in the enclosing scope. See also section Module registration macro – Single-file code example – Multi-file code example part 1, 2, 3.

lest_CASE( specification, "proposition" ) { code } (auto-registered cases)
Provide the collection of test cases, describe the expected behaviour to test for and specify the actions and expectations. Consider defining macro CASE(proposition) to hide the collection of test cases and define it in terms of lest_CASE(...) – Single-file code example – Multi-file code example part 1, 2, 3.

Fixture macros

lest provides function-level fixtures. Fixtures are stack-based and their setup and teardown occurs at the block scope of SETUP and (nested) SECTIONs – Code example.

SETUP( "context" ) { code }
Describe and setup the context to use afresh in each enclosed section.

SECTION( "proposition" ) { code }
Describe the expected behaviour to test for using the enclosing context and specify the actions and expectations. The objects in the enclosing setup or section come into existence and go out of scope for each section. A section must be enclosed in setup or in another section.

Assertion macros

lest has expression-decomposing assertion macros. An expression with strings such as hello > world may be reported with code and expansion as hello > world ("hello" > "world"). As a consequence, only a few assertion macro variants are needed – Code example.

EXPECT( expr )
Evaluate the expression and report failure. If an exception is thrown it is caught, reported and counted as a failure.

EXPECT_NOT( expr )
Evaluate the expression, record the logical not of its result and report failure. If an exception is thrown it is caught, reported and counted as a failure. This macro is a workaround to circumvent ! prefixed expressions as these cannot be decomposed.

EXPECT_NO_THROW( expr )
Expect that no exception (of any type) is thrown during evaluation of the expression.

EXPECT_THROWS( expr )
Expect that an exception (of any type) is thrown during evaluation of the expression.

EXPECT_THROWS_AS( expr, exception )
Expect that an exception of the specified type is thrown during evaluation of the expression.

If an assertion fails, the remainder of the test that assertion is part of is skipped.

BDD style macros

lest provides several macros to write Behaviour-Driven Design (BDD) style scenarios – Code example.

SCENARIO( "sketch", ...) { code }

GIVEN( "context" ) { code }

WHEN( "action" ) { code }

THEN( "result" ) { code }

AND_WHEN( "action" ) { code }

AND_THEN( "result" ) { code }

These macros simply map to macros CASE(), SETUP() and SECTION().

Module registration macro

When using arrays of test cases written across multiple files, you can use macro MODULE() to add a module's test cases to the overall specification – Code example part 1, 2, 3.

MODULE( overall-specification, module-specification )
Register this module's test specification with the overall specification.

Note that with lest using auto test case registration there's no need for macro MODULE(), see the auto-registration example part 1, 2, 3. The same holds for lest_cpp03, see cpp03 example part 1, 2, 3.

Other macros

-Dlest_NO_SHORT_MACRO_NAMES
-Dlest_NO_SHORT_ASSERTION_NAMES (deprecated)
All public API macros of lest exist as lest_MACRO and shorthand MACRO variant. Define this macro to omit the shorthand macros.

-Dlest_FEATURE_AUTO_REGISTER=0
Define this to 1 to enable auto registration of test cases. Default is 0.

Namespace

namespace lest { }
Types and functions are located in namespace lest.

Tests

struct env { };

struct test
{
std::string name;
std::function<void( env & )> behaviour;
};

You'll need type env and variable lest_env when you have a test case that calls a user-defined function or lambda that uses lest assertions like EXPECT() – Call reusable function, Call reusable templated function, and Call reusable lambda.

Main

A typical main() function for lest may look as follows:

#include "lest.hpp"

const lest::test specification[] = { CASE("..."){} };

int main( int argc, char *argv[] )
{
    if ( int failures = lest::run( specification, argc, argv ) )
        return failures;
    
    return std::cout << "All tests passed\n", EXIT_SUCCESS;
}

Compile and run:

prompt>g++ -std=c++11 -o main.exe -I./include/lest main.cpp && main.exe
All tests passed

Or, if feedback on success is moved to the command line:

#include "lest.hpp"

const lest::test specification[] = { CASE("..."){} };

int main( int argc, char *argv[] )
{
    return lest::run( specification, argc, argv );
}

Compile and run with feedback on success:

prompt>g++ -std=c++11 -o main.exe -I./include/lest main.cpp && main.exe && echo All tests passed
All tests passed

You can use the following variants of lest's run() function in main.

inline
int run( std::vector<test> specification, std::vector<std::string> arguments, std::ostream & os = std::cout );

inline
int run( std::vector<test> specification, int argc, char * argv[], std::ostream & os = std::cout );

template<std::size_t N>
int run( test const (& specification )[N], std::ostream & os = std::cout );

template<std::size_t N>
int run( test const (& specification )[N], std::vector<std::string> arguments, std::ostream & os = std::cout );

template<std::size_t N>
int run( test const (& specification )[N], int argc, char * argv[], std::ostream & os = std::cout );

specification - collection of tests
arguments - options and arguments to select and omit tests
argc, arcv - options and arguments to select and omit tests
os - stream to report to
returns number of failing tests

Main (Trompeloeil)

You can integrate the Trompeloeil mocking framework with lest by providing a reporter for Trompeloeil – Code example.

#include "lest.hpp"
#include "trompeloeil.hpp"

int main( int argc, char * argv[] )
{
    std::ostream & stream = std::cout;
    
    trompeloeil::set_reporter(
        [&stream]( trompeloeil::severity severity, const char * file, unsigned long line, std::string const & msg )
    {
        if ( severity == trompeloeil::severity::fatal )
        {
            throw lest::message{"", lest::location{ line ? file : "[file/line unavailable]", int(line) }, "", msg };
        }
        else
        {   
            stream << lest::location{ line ? file : "[file/line unavailable]", int(line) } << ": " << msg;
        }
    });

    return lest::run( specification, argc, argv, stream );
}

Floating point comparison

lest provides class approx to compare floating point values – Code example.

class approx { };

Use approx as follows:

EXPECT( 1.23 == approx( 1.23 ) );
EXPECT( 1.23 != approx( 1.24 ) );

EXPECT( 1.23 != approx( 1.231 ) );
EXPECT( 1.23 == approx( 1.231 ).epsilon( 0.1 ) );

approx custom = approx::custom().epsilon( 0.1 );

EXPECT( approx( 1.231 ) != 1.23 );
EXPECT( custom( 1.231 ) == 1.23 );

Class approx also provides less-than or equal and greater-than or equal operators.

Reporting a user-defined type

lest allows you to report a user-defined type via operator<<() – Code example.

To report a type not yet supported by lest, define a streaming function for it:

namespace ns {
struct user-defined-type { ... };
std::ostream & operator<< ( std::ostream & os, user-defined-type const & type )
{
using lest::to_string;
return os << ... ;
}
}

In it, stream the constituent parts of the type via lest's to_string() conversion functions.

Variants of lest

Various variants of lest are kept here. The simple ones, such as lest_basic and lest_decompose provide an easy read into the techniques used and remain the tiny test frameworks that are a good fit to include with small projects.

You are encouraged to take it from here and change and expand it as you see fit and publish your variant. If you do, I'd much appreciate to hear from you!

lest.hpp - lest's latest development, this project.
lest_basic.hpp - lest at its very basic, this project.
lest_decompose.hpp - lest with expression decomposition, this project.
lest_cpp03.hpp - lest with expression decomposition for C++03, this project.
hamlest - matchers for lest.
lest with groups - Pavel Medvedev

Features of lest

Feature / variant	latest	cpp03	decompose	basic
Assert expressions	+	+	+	+
Assert exceptions	+	+	+	+
Assert abortion (death)	contrib	contrib	-	-
Assert assertions (death)	contrib	contrib	-	-
Expression decomposition	+	modest	modest	-
Literal suffix u, l, f	+	-	-	-
Colourised output	+	+	-	-
BDD style scenarios	+	+	-	-
Fixtures (sections)	+	+	-	-
Floating point comparison, approx	+	+	-	-
Floating point comparison, ulp	-	-	-	-
Test selection (include/omit)	+	+	-	-
Test selection (regexp)	+	+	-	-
Help screen	+	+	-	-
Abort at first failure	+	+	-	-
Count selected tests	+	+	-	-
List tags of selected tests	+	+	-	-
List selected tests	+	+	-	-
Report passing tests	+	+	-	-
Time duration of tests	+	+	-	-
Control order of tests	+	+	-	-
Repeat tests	+	+	-	-
Auto registration of tests	+	+	-	-
Modules of tests	+	+	-	-

Suites of tests	-	-	-	-
Parameterised tests	-	-	-	-
Templated tests	-	-	-	-
Test data generators	-	-	-	-
Hamcrest matchers	+/-	-	-	-
Logging facility	-	-	-	-
Break into debugger	-	-	-	-
Concurrent execution of tests	-	-	-	-
Shielded execution of tests	-	-	-	-

Reported to work with

The table below mentions the lowest version of a compiler lest is reported to work with.

Variant / compiler	clang	GCC	VC
lest (latest)	3.2	4.8.1	12
lest_basic	3.2	4.6	12
lest_decompose	3.2	4.6	12
lest_cpp03 (decompose)	?	?	6 SP6

Note: I've made a few concessions to enable compilation of lest.hpp with Visual C++:

Prevent error C2797: replace braced member initialisation with C++98 style initialisation.
Prevent error C2144: use enum{ value } instead of static constexpr bool in struct is_container (for VC only).

Building tests and examples

Tests and examples can be build via Makefiles or using CMake.

To build the tests and examples with CMake you need:

Buck or CMake version 2.8 or later to be installed and in your PATH.
A suitable compiler.

The following steps assume that the lest source code has been cloned into a directory named lest.

Buck

lest> buck run test:test_lest_basic
lest> buck run test:test_lest_cpp03
lest> buck run test:test_lest_decompose
lest> buck run test:test_lest

CMake

Create a directory for the build outputs for a particular architecture.
Here we use lest/build.
```
 lest> mkdir build && cd build
```
Configure CMake to use the compiler of your choice (run cmake --help for a list) and build the tests for lest and the examples.
```
 lest/build> cmake -G "Unix Makefiles" [see 3. below] ..
```
Optional. You can control above configuration through the following options:
- -DLEST_BUILD_TEST=ON: build the tests for lest, default on
- -DLEST_BUILD_EXAMPLE=ON: build the examples, default on
- -DLEST_BUILD_CONTRIB=OFF: build the contrib folder, default off
Build the test suite.
```
 lest/build> cmake --build .
```
Run the test suite.
```
 lest/build> ctest -V
```

All tests should pass, indicating your platform is supported and you are ready to use lest. Note that quite some examples fail. They do so to demonstrate the usage of things.

Contributions to lest

Folder contrib contains extensions to lest. These extensions are not part of lest itself because for example they use non-standard behaviour, they are considered to be for a too-specific use case, or they are considered not yet ripe for inclusion in lest and we first like to gain more experience with them.

Other test frameworks

Catch - A modern, C++-native, header-only, framework for unit-tests, TDD and BDD.
doctest - The lightest feature-rich C++ single-header testing framework for unit tests and TDD. Catch/lest-inspired tests that accompany code.
stf - Simple TDD Framework. Catch/lest-inspired with an emphasis on floating point precision testing.
cute - C++ unit test executor.
bandit - Human friendly unit testing for C++11.
igloo - BDD style unit testing for C++.
xUnit++ - A unit testing platform for C++ inspired by xUnit.net.
Search for C++11 test frameworks on GitHub
Search for C++14 test frameworks on GitHub
List of C++ unit testing frameworks at Wikipedia

Notes and references

[1] Kevlin Henney on Rethinking Unit Testing in C++ (Video).

[2] Martin Moene. Elefant C++11 test setup on the ACCU mailing list accu-general (requires login). It mentions the C++11 test appoach Andrzej Krzemieński uses for Optional. A library for optional (nullable) objects for C++11.

[3] Phil Nash. CATCH, an automated test framework for C, C++ and Objective-C.

[4] A more technically informed name: lest - lambda engaged small tester.

Name		Name	Last commit message	Last commit date
Latest commit History 499 Commits
contrib		contrib
example		example
include/lest		include/lest
test		test
.buckconfig		.buckconfig
.gitattributes		.gitattributes
.gitignore		.gitignore
.travis.yml		.travis.yml
BUCK		BUCK
CHANGES.txt		CHANGES.txt
CMakeLists.txt		CMakeLists.txt
LICENSE_1_0.txt		LICENSE_1_0.txt
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

lest – lest errors escape testing

Example usage

Compile and run

In a nutshell

License

Dependencies

Installation

Usage

Synopsis

Command line

Test case macro

Fixture macros

Assertion macros

BDD style macros

Module registration macro

Other macros

Namespace

Tests

Main

Main (Trompeloeil)

Floating point comparison

Reporting a user-defined type

Variants of lest

Features of lest

Reported to work with

Building tests and examples

Buck

CMake

Contributions to lest

Other test frameworks

Notes and references

About

Releases

Packages

Languages

License

lucckb/lest

Folders and files

Latest commit

History

Repository files navigation

lest – lest errors escape testing

Example usage

Compile and run

In a nutshell

License

Dependencies

Installation

Usage

Synopsis

Command line

Test case macro

Fixture macros

Assertion macros

BDD style macros

Module registration macro

Other macros

Namespace

Tests

Main

Main (Trompeloeil)

Floating point comparison

Reporting a user-defined type

Variants of lest

Features of lest

Reported to work with

Building tests and examples

Buck

CMake

Contributions to lest

Other test frameworks

Notes and references

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages