Skip to content

Unit testing tools benchmark and DQMH example

Jump to bottom
Peter edited this page Sep 13, 2019 · 5 revisions

Unit testing tools benchmark and DQMH example

This page compares the execution times of the most popular unit testing tools for LabVIEW based on a DQMH example.

Further links and prerequisites

Source of the LabVIEW project on GitHub: link. The LabVIEW project is implemented in LabVIEW 2018.

The unit testing tools included in this benchmark:

The unrestricted execution of the benchmark project requires the installation of the DQMH framework, VI Tester, Caraya, Unit Test Framework (UTF) and InstaCoverage Pro.

The benchmark project

DQMH is a popular framework for creating distributed applications in LabVIEW. We augmented the built-in Thermal chamber project. We manually created 8 genuine unit tests and executed them once, 5, 10, 50 and 100 times and measured the time of execution with each unit testing tool. Each of the 8 unit tests is implemented in all unit testing tools, which means 4 versions for each unit test.

Main structure of the benchmark project

The unit tests are located in virtual folders named after the corresponding unit test tool, e.g., UTF unit tests. The VIs needed for running the benchmark are located in Time Measurement. For each tool we implemented a runner VI, e.g., UTF Time Measurement Runner.vi. The runner VI uses the tool's API for the programmatic iteration and execution of the unit tests. Since there is no API for iterating through a set of VIs in Caraya, we hard-coded the list of unit tests in the runner VI belonging to Caraya.

The benchmark results

The following chart displays the results of the benchmark. The benchmark used the latest available version of each tool (as of September 1, 2019) and was executed on an avarage desktop machine running Windows.

Conclusion:

  • InstaCoverage is the fastest unit testing tool with test coverage support. InstaCoverage outperforms UTF significantly (e.g., 17.9 seconds vs 24.2 minutes on 800 unit tests), the only other tool supporting test coverage measurement.
  • InstaCoverage is faster on most experiments among all tools supporting the programmatic execution of unit tests (UTF, VI Tester, InstaCoverage). This is true even given that VI Tester does not measure test coverage during execution.
  • Caraya is the fastest unit testing tools given the following limitations:
    • Caraya does not support the programmatic execution of unit tests.
    • Caraya cannot measure test coverage.
Clone this wiki locally