Test Models

Home > Model Development Topics > Test Models

This topic contains detailed information about the OpenM++ test_models utility. test_models builds, runs, and compares results for different versions of the same model, or for the same model built in different ways or on different platforms.

Topic contents

• Introduction and overview
• Windows Quick start
• Linux or MacOS Quick start
• Concepts
• Arguments and Options
• Control - Files
• Output - Report
• Output - Files
• Example 1: Preparing the default run
• Example 2: A new OpenM++ release
• Example 3: A model code change (under construction)
• Example 4: When results differ (under construction)
• Example 5: Exercising a model in Debug (under construction)
• Example 6: A performance comparison
• Example 7: Using event trace output (under construction, some content present)

Introduction and overview

The test_models utility builds, runs, and compares results for different versions of the same model, or for the same model built on different platforms. It can process multiple models in a single invocation and can compare OpenM++ and Modgen versions of cross-compatible models. It can play a useful role during incremental model development, and for quality assurance.

test_models is efficient enough that it can be used routinely in model development workflow cycles, even for large models (using a small sized Default simulation). even multiple times per hour or more during active model development.

When used routinely to assess the effects of incremental model changes, test_models can help identify bugs in model logic. It highlights when model changes have unexpected effects in model outputs, by comparing all cells of all output tables between a Current run and a Reference run.

test_models can be used to compare outputs of ompp model versions across platforms. Such differences are rare, and are typically caused by errors in model logic (often due to bad C++ code in models, eg use of uninitialized variables, or accessing memory beyond array limits using index variables with bad values).

test_models works with the standard model folder structure illustrated in the sample models in the OpenM++ distribution. It organizes results herarchically in a subfolder test_models of the model folder. test_models organizes comparisons in two dimensions. The first dimension is {Reference, Current}. A user can tell test_models to move Current results to Reference results to create a new point of reference for subsequent output comparisons. The second dimension of comparison is the platform ‘flavour’ {modgen, ompp-win, ompp-linux, ompp-macos}. Arguments to test_models determine where model results will be stored and which comparisons will be performed after test_models does the requested runs. For speed and ease of use, each model output table is converted to three forms: full precision, rounded precision, and a summary 'digest' of the file contents. The use of rounded precision eliminates most (but not all) spurious differences, eg differences at 6-7th decimal place. The table digest makes it possible for test_models to determine whether two versions of a table differ at very high speed, without comparing the two tables cell-by-cell.

test_models can compare results of Modgen and ompp versions of models, despite large underlying differences in output storage schema and technology. It reads the database of each and converts output tables into a common comparable csv format.

test_models can be very useful for cross-compatible model conversion. One can break cross-compatible model conversion into atomic steps and use test_models after each atomic step to verify that Modgen results have not changed, even before an OpenM++ version of the model is buildable. The comparisons relevant during cross-compatible model conversion are

1. Current(modgen) vs. Reference(modgen) - Modgen results should remain identical at each atomic change to model source, when build in Modgen.
2. Current(ompp) vs. Current(modgen) – After the model code is cross-compatible, ie when the ompp version can be built with no compiler errors, the model can be run in both versions and model results should be identical, because the model specifications are identical (same model source code and same Default scenario). Sometimes they are not, in which case the tables which differ can sometimes provide a clue.

When tracking down the cause of an unexpected difference in two runs, it can cometimes be useful to simplify the Default scenario and run test_models again, to see which area of the model might responsible for the difference. For example, if all tables differ, but when one turns off immigration in the Default scenario all differences disappear, one knows that the immigration code is somehow responsible. It can also be useful to turn on event tracking in the model with a small simulated population. If test_models indicates that the two event tracking files are identical, but table results are different, then the cause could be a Modgen tabulation bug.

test_models is also useful for model development. One can break model development into small atomic steps to help identify unexpected changes in model outputs. If possible, one can structure model changes so that Default parameters and code changes should have no effect on outputs, eg by turning a new model option off in Default parameters. That can identify whether the new model code, when disabled using the new parameters, has unexpected effects on other parts of the model (it should not). The test_models comparisons typically used for model development are

1. Current(ompp) vs. Reference(ompp) - Model results from incremental changes should affect only new tables or expected interactions among model components. Once verified, one can tell test_models to copy Current results to Reference results (using the --newref option) to create a new Reference for the next atomic set of changes to model code.

If one prefers to work in the Modgen environment for model development, one can instead use

1. Current(modgen) vs. Reference(modgen) - See notes above.
2. Current(modgen) vs. Current(ompp) – From time to time, build the ompp version to identify if non cross-compatible code has crept into the model source, and rectify if so.

test_models also standardizes the detailed event trace outputs (if activated) in ompp and Modgen models so that they are comparable and readable. This can identify precisely when a simulation diverges in the Modgen and ompp versions of a model, or in (for example) two ompp versions of a model. This is useful for unexpected and hard-to-understand differences in simulations. If the time and event of earliest divergence in the two runs is insufficient to understand the cause in and of itself, that information provides what’s required to set conditional breakpoints for a parallel debugging session of the two runs, stepping through the simulation in each of the two debugging sessions, to identify the precise code location responsible for the divergence in the simulation in the two versions.

test_models also notes some tombstone information about each set of outputs, eg the modgen version. It also keeps a copy of the build log output and Default run log output. test_models can also report the elpased time of various steps, including model build and model run. A count of compiler warnings is also reported.

test_models can run and process a single model, or multiple models in subfolders of a parent folder. That can be useful for testing mutliple scenarios of a single model, using mutliple git clones inside a parent folder. It can also be useful to bulk test multiple models routinely, for example after a change in the OpenM++ version.

[back to topic contents]

Windows Quick Start

1. Verify installation of test_models (Windows)

A 64-bit Windows executable version of test_models is distributed with OpenM++ at OM_ROOT/bin/test_models.exe, where OM_ROOT stands for the OpenM++ installation directory. A 32-bit version is at OM_ROOT/bin/test_models32.exe. The 32-bit version may be required for the Modgen flavour to work successfully using the executable version of test_models. To test installation and operation of test_models, open a command prompt, change the current directory to OM_ROOT/bin, and type the command

test_models -v

Output should be similar to the following:

test_models version 2.1

test_models is written in the Perl language, and distributed with OpenM++ at OM_ROOT/Perl/test_models.pl. Most examples in this topic invoke test_models using the Perl interpreter from the OM_ROOT/Perl directory, eg

perl test_models.pl -v

On Windows, unless you have Perl and the required Perl components installed, invoke the executable version of test_models from the OM_ROOT/bin directory with a command like

test_models -v

or

test_models32 -v

2. Display test_models options (Windows)

From the OM_ROOT/bin directory, type the command

test_models -h

Output should be similar to the following:

test_models [-hmv] [long options...] model...
        -m STR --models_root STR    directory containing models (default is .)
        --newref                    replace Reference results with Current
                                    results
        --noompp                    skip OpenM++ build and run
        --nomodgen                  skip Modgen build and run
        --nocomp                    skip flavour comparison
        --allfiles                  report all different and orphaned files
        --timing                    report elapsed time of steps
        --nosteps                   skip reporting which step is being
                                    performed
        --config STR                build configuration: debug or
                                    release(default)
        --mpi_processes INT         build MPI version and run with n
                                    processes (default 0, means no MPI)
        --gencode                   keep a copy of the generated C++ code
        --ini STR                   OpenM++ model ini file to pass to model
                                    (in model root, default is
                                    test_models.ini if present)
        --clean                     remove all build files after run
        --significant_digits INT    significant digits (default 6)
        --nocells                   disable fallback cell-by-cell
                                    verification of differing tables and copy
                                    of original data
        -h --help                   report usage message and exit
        -v --version                report test_models version and exit
        --windows_platform STR      Windows platform: x64(default) or Win32
        --modgen_platform STR       Modgen platform: Win32(default) or x64on of unrounded versions
                                    of tables

3. Run test_models on the RiskPaths model (Windows)

From the OM_ROOT/bin directory, type the command

test_models -m ../models RiskPaths

If Modgen is installed, use test_models32 instead of test_models. test_models will build and run both the OpenM++ and the Modgen versions of RiskPaths, and compare their results. Output should be similar to the following:

             =========================
              test_models 2.0
             =========================

             Testing: RiskPaths
             modgen settings: version=12,1,3,0 (2019-12-19 20:31 GMT) platform=Win32 configuration=release
             ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release

             RiskPaths: modgen: Build model and prepare Default scenario
             RiskPaths: modgen: Run model using RiskPaths.ini
             RiskPaths: modgen: Convert outputs (7 digits of precision)
             RiskPaths: modgen: Create digests of current outputs
             RiskPaths: modgen: No Reference outputs - create using Current outputs
             RiskPaths: modgen: Current vs. Reference:   9 the same (of 9)
             RiskPaths:
             RiskPaths: ompp-win: Build and publish model and Default scenario
  warning => RiskPaths: ompp-win: 10 build warning(s) - see RiskPaths/test_models/current/ompp-win/logs/build.log
             RiskPaths: ompp-win: Run model using RiskPaths.ini
             RiskPaths: ompp-win: Convert outputs (7 digits of precision)
             RiskPaths: ompp-win: Create digests of current outputs
             RiskPaths: ompp-win: No Reference outputs - create using Current outputs
             RiskPaths: ompp-win: Current vs. Reference:   9 the same (of 9)
             RiskPaths:
             RiskPaths:     Flavour comparisons:
             RiskPaths:
             RiskPaths: ompp-win vs. modgen:      Reference:   9 the same (of 9)
             RiskPaths:
             RiskPaths: ompp-win vs. modgen:      Current:     9 the same (of 9)

If Modgen is not installed, output will not include the Modgen portion nor the flavour comparison portions of the report.

test_models uses the Microsoft application msbuild.exe to build models. msbuild.exe is normally installed as part of Visual Studio installation, but the install location can vary. test_models attempts to determine the location of msbuild.exe but may not always succeed. If you encounter issues running test_models which seem related to msbuild.exe you can provide test_models the location explicitly using the environment variable MSBUILD_EXE, for example:

set MSBUILD_EXE=C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\msbuild.exe

in the command window before invoking test_models.

[back to topic contents]

Linux or MacOS Quick Start

1. Verify installation of test_models (Linux, MacOS)

test_models is a Perl script distributed with OpenM++ at OM_ROOT/Perl/test_models.pl, where OM_ROOT stands for the OpenM++ installation directory. To test installation and operation of test_models, open a command prompt, change the current directory to OM_ROOT\Perl, and type the command

perl test_models.pl -v

Output should be similar to the following:

test_models version 2.1

Depending on your operating system version and installation history, Perl may ask you to install missing Perl modules required by test_models.pl. If so, it will name them explicitly when you invoke test_models.pl. We do recommend to use cpanm for Perl modules installation. Typical scenario is:

cpan App::cpanminus              # initialize cpanm, if not done before

cpanm Getopt::Long::Descriptive
cpanm Capture::Tiny
cpanm Config::Tiny
cpanm File::Copy::Recursive
cpanm File::Which

Above list of modules can be different and depends on your current Perl configuration, and on the version of test_models.

2. Display test_models options (Linux, MacOS)

From the OM_ROOT/Perl directory, type the command

perl test_models.pl -h

Output should be similar to the following:

test_models [-hmv] [long options...] model...
        -m STR --models_root STR    directory containing models (default is .)
        --newref                    replace Reference results with Current
                                    results
        --noompp                    skip OpenM++ build and run
        --nomodgen                  skip Modgen build and run
        --nocomp                    skip flavour comparison
        --allfiles                  report all different and orphaned files
        --timing                    report elapsed time of steps
        --nosteps                   skip reporting which step is being
                                    performed
        --config STR                build configuration: debug or
                                    release(default)
        --mpi_processes INT         build MPI version and run with n
                                    processes (default 0, means no MPI)
        --gencode                   keep a copy of the generated C++ code
        --ini STR                   OpenM++ model ini file to pass to model
                                    (in model root, default is
                                    test_models.ini if present)
        --clean                     remove all build files after run
        --significant_digits INT    significant digits (default 6)
        --nocells                   disable fallback cell-by-cell
                                    verification of differing tables and copy
                                    of original data
        -h --help                   report usage message and exit
        -v --version                report test_models version and exit
        --windows_platform STR      Windows platform: x64(default) or Win32
        --modgen_platform STR       Modgen platform: Win32(default) or x64

3. Run test_models on the RiskPaths model (Linux, MacOS)

From the OM_ROOT/Perl directory, type the command

perl test_models.pl -m ../models RiskPaths

Output should be similar to the following:

             =========================
              test_models 2.0 
             =========================
              
             Testing: RiskPaths
             ompp-linux settings: compiler=omc (2021-05-29 16:47 GMT) configuration=release
              
             RiskPaths: ompp-linux: Build and publish model and Default scenario
             RiskPaths: ompp-linux: Run model using RiskPaths.ini
             RiskPaths: ompp-linux: Convert outputs (7 digits of precision)
             RiskPaths: ompp-linux: Create digests of current outputs
             RiskPaths: ompp-linux: No Reference outputs - create using Current outputs.
             RiskPaths: ompp-linux: Current vs. Reference:   9 the same (of 9)

[back to topic contents]

Concepts

This subtopic describes key underlying concepts used in test_models.

• Run version: Current or Reference
• Run flavour: Windows, Linux, MacOS, Modgen
• Output comparison:
• Build options:
• Run options:

[back to topic contents]

Run version

=== under construction ===

• Current or Reference

[back to concepts]
[back to topic contents]

Run flavour

=== under construction ===

• Windows, Linux, MacOS, Modgen

[back to concepts]
[back to topic contents]

Output comparison

=== under construction ===

[back to concepts]
[back to topic contents]

Build options

=== under construction ===

[back to concepts]
[back to topic contents]

Run options

=== under construction ===

[back to concepts]
[back to topic contents]

Arguments and Options

This subtopic describes the command line options and arguments of test_models, organized into sections. It also describes default test_models behaviour for each section.

• Syntax: The syntax of test_models arguments
• Models: The models to process
• Actions: The actions to perform on a model
• Verbosity: The level of report detail
• Build: How a model is built
• Run: How a model is run
• Comparison: How result are compared
• Informational: Informational options
• EventTrace: EventTrace options

A complete list of options is displayed by issuing the command

perl test_models.pl -h

test_models [-hmv] [long options...] model...
        -m STR --models_root STR    directory containing models (default is .)
        --newref                    replace Reference results with Current
                                    results
        --noompp                    skip OpenM++ build and run
        --nomodgen                  skip Modgen build and run
        --nocomp                    skip flavour comparison
        --allfiles                  report all different and orphaned files
        --timing                    report elapsed time of steps
        --nosteps                   skip reporting which step is being
                                    performed
        --config STR                build configuration: debug or
                                    release(default)
        --mpi_processes INT         build MPI version and run with n
                                    processes (default 0, means no MPI)
        --gencode                   keep a copy of the generated C++ code
        --ini STR                   OpenM++ model ini file to pass to model
                                    (in model root, default is
                                    test_models.ini if present)
        --clean                     remove all build files after run
        --significant_digits INT    significant digits (default 6)
        --zero_fuzz NUM             zero fuzz value (default 1e-15)
        --nocells                   disable fallback cell-by-cell
                                    verification of differing tables and copy
                                    of original data
        -h --help                   report usage message and exit
        -v --version                report test_models version and exit
        --time_format STR           time format for event trace (default
                                    13.6f)
        --modgen_id_offset INT      offset adjustment to Modgen IDs (default
                                    0)
        --windows_platform STR      Windows platform: x64(default) or Win32
        --modgen_platform STR       Modgen platform: Win32(default) or x64

[back to arguments and options]
[back to topic contents]

Syntax

Arguments to test_models consist of a series of options followed by a list of models to process. Either or both may be empty, in which case default values are used. An option starts with -- followed immediately by the option name. Some options may be followed by an option value. Some options have a synonymous short form consisting of a single - followed imediately by a single letter.

[back to arguments and options]
[back to topic contents]

Models

The --models_root option (short form -m) specifies the parent folder of one or more folders each of which contains a model. If not specified, the models root is the current working directory. Consider the following command, issued after setting the current working directory to OM_ROOT/Perl:

perl test_models.pl -m ../models NewCaseBased RiskPaths

On Windows without Perl installed, a similar command would be issued after setting the current working directory to OM_ROOT/bin:

test_models -m ../models NewCaseBased RiskPaths

The command instructs test_models to set the models root to the sister directory models in the OpenM++ distribution. That folder is OM_ROOT/models and contains several sample models distributed with OpenM++, each in its own subfolder, eg OM_ROOT/models/RiskPaths. The final two arguments instruct test_models to process the two models in the subfolders OM_ROOT/models/NewCaseBased and OM_ROOT/models/RiskPaths.

On Linux, output is similar to:

             =========================
              test_models 2.0 
             =========================
              
             Testing: NewCaseBased, RiskPaths
             ompp-linux settings: compiler=omc (2021-05-29 16:47 GMT) configuration=release
              
             NewCaseBased: ompp-linux: Build and publish model and Default scenario
             NewCaseBased: ompp-linux: Run model using test_models.ini
             NewCaseBased: ompp-linux: Convert outputs (7 digits of precision)
             NewCaseBased: ompp-linux: Create digests of current outputs
             NewCaseBased: ompp-linux: No Reference outputs - create using Current outputs
             NewCaseBased: ompp-linux: Current vs. Reference:   3 the same (of 3)
              
             RiskPaths: ompp-linux: Build and publish model and Default scenario
             RiskPaths: ompp-linux: Run model using RiskPaths.ini
             RiskPaths: ompp-linux: Convert outputs (7 digits of precision)
             RiskPaths: ompp-linux: Create digests of current outputs
             RiskPaths: ompp-linux: No Reference outputs - create using Current outputs
             RiskPaths: ompp-linux: Current vs. Reference:   9 the same (of 9)

On Windows with Modgen installed, output is similar to:

             =========================
              test_models 2.0
             =========================

             Testing: NewCaseBased, RiskPaths
             modgen settings: version=12,1,3,0 (2019-12-19 20:31 GMT) platform=Win32 configuration=release
             ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release

             NewCaseBased: modgen: Build model and prepare Default scenario
             NewCaseBased: modgen: Run model using test_models.ini
             NewCaseBased: modgen: Convert outputs (7 digits of precision)
             NewCaseBased: modgen: Create digests of current outputs
             NewCaseBased: modgen: No Reference outputs - create using Current outputs
             NewCaseBased: modgen: Current vs. Reference:   3 the same (of 3)
             NewCaseBased:
             NewCaseBased: ompp-win: Build and publish model and Default scenario
  warning => NewCaseBased: ompp-win: 3 build warning(s) - see NewCaseBased/test_models/current/ompp-win/logs/build.log
             NewCaseBased: ompp-win: Run model using test_models.ini
             NewCaseBased: ompp-win: Convert outputs (7 digits of precision)
             NewCaseBased: ompp-win: Create digests of current outputs
             NewCaseBased: ompp-win: No Reference outputs - create using Current outputs
             NewCaseBased: ompp-win: Current vs. Reference:   3 the same (of 3)
             NewCaseBased:
             NewCaseBased:     Flavour comparisons:
             NewCaseBased:
             NewCaseBased: ompp-win vs. modgen:      Reference:   3 the same (of 3)
             NewCaseBased:
             NewCaseBased: ompp-win vs. modgen:      Current:     3 the same (of 3)

             RiskPaths: modgen: Build model and prepare Default scenario
             RiskPaths: modgen: Run model using RiskPaths.ini
             RiskPaths: modgen: Convert outputs (7 digits of precision)
             RiskPaths: modgen: Create digests of current outputs
             RiskPaths: modgen: No Reference outputs - create using Current outputs
             RiskPaths: modgen: Current vs. Reference:   9 the same (of 9)
             RiskPaths:
             RiskPaths: ompp-win: Build and publish model and Default scenario
  warning => RiskPaths: ompp-win: 10 build warning(s) - see RiskPaths/test_models/current/ompp-win/logs/build.log
             RiskPaths: ompp-win: Run model using RiskPaths.ini
             RiskPaths: ompp-win: Convert outputs (7 digits of precision)
             RiskPaths: ompp-win: Create digests of current outputs
             RiskPaths: ompp-win: No Reference outputs - create using Current outputs
             RiskPaths: ompp-win: Current vs. Reference:   9 the same (of 9)
             RiskPaths:
             RiskPaths:     Flavour comparisons:
             RiskPaths:
             RiskPaths: ompp-win vs. modgen:      Reference:   9 the same (of 9)
             RiskPaths:
             RiskPaths: ompp-win vs. modgen:      Current:     9 the same (of 9)

If no models are specified on the command line, test_models will process All subfolders of the models root.

[back to arguments and options]
[back to topic contents]

Actions

Unless instructed otherwise, test_models builds and runs all available 'flavours' of a model on the current operating system. On Windows, two flavours are available: OpenM++ and Modgen (if installed). On Linux and MacOS, the only available flavour is OpenM++. After building and running each flavour, test_models compares Current results to existing Reference results. If there are no Reference results, test_models creates them by copying Current results. After completing all flavours for a model, test_models compares results for other flavours (if present) to results for the OpenM++ flavour on the current operating system.

This default behaviour can be modified by the following command options:

• --newref For each flavour in this invocation, discard all Reference results and replace them with Current results
• --noompp Suppress processing the OpenM++ flavour
• --nomodgen Suppress processing the Modgen flavour (only applies if invoked on Windows)
• --nocomp Suppress reporting on differences between flavours

For example, on Windows, the command

test_models -m ../models --noompp --nomodgen RiskPaths

would process the RiskPaths model, but skip build and run of both Windows flavours (OpenM++ and Modgen). test_models would still compare results for any previously run flavours to OpenM++ Windows results (if present). Output might be similar to the following (or could be empty, if no other flavours were run previously):

             =========================
              test_models 2.0
             =========================

             Testing: RiskPaths

             RiskPaths:
             RiskPaths:     Flavour comparisons:
             RiskPaths:
             RiskPaths: ompp-win vs. modgen:      Reference:   9 the same (of 9)
             RiskPaths:
             RiskPaths: ompp-win vs. modgen:      Current:     9 the same (of 9)
             RiskPaths:
             RiskPaths: ompp-win vs. ompp-linux:  Reference:   9 the same (of 9)
             RiskPaths:
             RiskPaths: ompp-win vs. ompp-linux:  Current:     9 the same (of 9)

In this invocation, the RiskPaths model was neither built nor run. test_models detected the presence of Current and Reference results for two other flavours (Modgen and OpenM++ on Linux), and compared them to results from OpenM++ on Windows which were present from a previous invocation of test_models. Incidentally, because test_models uses digests to compare results, comparing all results between two flavours is almost instantaneous, even for models with many tables or with very large tables.

[back to arguments and options]
[back to topic contents]

Verbosity

Unless instructed otherwise, test_models reports which step it is processing, It does not report the elapsed time of each step. When comparing results, test_models reports file counts and reports by name the first five differing or orphaned files.

This default behaviour can be modified by the following command options:

• --allfiles Report each differing and orphaned file by name, not just the first five
• --timing Also report the elapsed time for each step
• --nosteps Don't report which step is being processed

For example, on Windows, the command

perl test_models.pl -m ../models --nomodgen --nocomp --timing --allfiles --significant_digits 8 OzProj

instructs test_models to build and run just the OpenM++ version of OzProj, suppress the flavour comparison output, report elapsed time of each step, and report every differing file by name, not just the first five. For illustrative purposes, this invocation also changes the number of significant digits used to construct digests from 7 to 8 to deliberately produce differences between Current and Reference OzProj results. Output is similar to the following:

             =========================
              test_models 2.0
             =========================

             Testing: OzProj
             ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release

             OzProj: ompp-win: Build and publish model and Default scenario
             OzProj: ompp-win: Build time 0m 7s
  warning => OzProj: ompp-win: 63 build warning(s) - see OzProj/test_models/current/ompp-win/logs/build.log
             OzProj: ompp-win: Run model using OzProj.ini
             OzProj: ompp-win: Run time 0m 2s
             OzProj: ompp-win: Convert outputs (8 digits of precision)
             OzProj: ompp-win: Convert time 0m 0s
             OzProj: ompp-win: Create digests of current outputs
             OzProj: ompp-win: Digest time 0m 0s
             OzProj: ompp-win: Current vs. Reference:  13 the same (of 21)
DIFFERS ===> OzProj: ompp-win: Current vs. Reference:   8 differ
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS:           0_SIGNIFICANT_DIGITS.txt
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS:           Experiment1.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS:           Experiment2.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS:           Experiment3.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS:           Experiment4.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS:           Experiment5.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS:           Experiment6.csv
DIFFERS ===> OzProj: ompp-win: Current vs. Reference: DIFFERS:           PersonYearsLived.csv

The specially-named 'result' file 0_SIGNIFICANT_DIGITS.txt is described elswhere in this topic.

[back to arguments and options]
[back to topic contents]

Build

Unless instructed otherwise, test_models builds a Release version of a model, without MPI multi-processing capability, and does not save copy of generated C++ code.

This default behaviour can be modified by the following command options:

• --config Build configuration: debug or release(default)
• --mpi_processes Build MPI version (default 0, means no MPI)
• --gencode Save a copy of the generated C++ code in subfolder generated_code

The Debug version of a model is typically considerably slower than the Release version. The Debug version can, however, perform run-time checks which are absent in the Release version. This is particularly true if C++ model code uses assert↗ to verify that expected logical conditions are satisfied at run time (in Debug mode only).

Supply a non-zero value to --mpi_processes to build an MPI-enabled model (OpenM++ only). The value will be used to launch that number of MPI instances when the model is subsequently run as described in Run. The suffix _mpi will be appended to the name of the model executable to distinguish it from the normal non-MPI version.

The specialized --gencode option creates a copy of the C++ code generated by the omc compiler (or by the Modgen compiler for the modgen flavour). If this option is activated, the src temporary build folder is copied to the output folder generated_code. The --gencode option can be used to compare the C++ code generated by different versions of the OpenM++ compiler.

Incidentally, models are capable of multi-threading independent of MPI.

[back to arguments and options]
[back to topic contents]

Run

Unless instructed otherwise, test_models builds and runs a non-MPI enabled model using the Default scenario. It retains all files from the build, including the model executable and database after the model is run. Also by default, test_models looks in the model root for a model ini file to use to run the model. The search for the ini file proceeds as follows: If the file test_models.ini is present, it is used. If test_models.ini is not present, the file MODEL.ini is used if present, where MODEL stands for the name of the model. For more on how test_models uses a model ini file, see Control - Files.

This default behaviour can be modified by the following command options:

• --ini The name of the OpenM++ model ini used when the model is run
• --mpi_processes The number of mpi processes to launch
• --clean Remove all build files after the run

For example, if the file OM_ROOT/RiskPaths/Test.ini exists with content

[OpenM]
SubValues = 16
Threads = 4

[Parameter]
SimulationCases = 16000000

and the following command is issued

perl test_models.pl -m ../models --newref --nomodgen --nocomp --timing --mpi_processes 4 --ini Test.ini RiskPaths

an MPI-enabled version of RiskPaths is built and 4 instances are launched on the workstation under MPI control. The run consists of 16 replicates with 1,000,000 cases in each replicate, for a total of 16,000,000 cases. Each MPI instance will run with 4 threads.

Output is similar to the following:

             =========================
              test_models 2.0
             =========================

             Testing: RiskPaths
             ompp-win settings: compiler=omc.exe (2021-05-28 02:28 GMT) platform=x64 configuration=release mpi_processes=4

             RiskPaths: ompp-win: Deleting previous Reference information
             RiskPaths: ompp-win: Build and publish model and Default scenario
             RiskPaths: ompp-win: Build time 0m 5s
  warning => RiskPaths: ompp-win: 10 build warning(s) - see RiskPaths/test_models/current/ompp-win/logs/build.log
             RiskPaths: ompp-win: Run model using Test.ini
             RiskPaths: ompp-win: Run time 2m 15s
             RiskPaths: ompp-win: Convert outputs (7 digits of precision)
             RiskPaths: ompp-win: Convert time 0m 0s
             RiskPaths: ompp-win: Create digests of current outputs
             RiskPaths: ompp-win: Digest time 0m 0s
             RiskPaths: ompp-win: No Reference outputs - create using Current outputs
             RiskPaths: ompp-win: Current vs. Reference:   9 the same (of 9)

The truncated log file of the run is similar to the following:

2021-05-31 16:18:57.819 RiskPaths
2021-05-31 16:18:57.819 RiskPaths
2021-05-31 16:18:57.993 Parallel run of 4 modeling processes, 4 thread(s) each
2021-05-31 16:18:57.994 Model build    : Windows 64 bit Release 
2021-05-31 16:18:57.995 Prepare fixed and missing parameters
2021-05-31 16:18:58.012 Run: 102 Default
2021-05-31 16:18:58.012 Get scenario parameters for process
2021-05-31 16:18:58.013 Model build    : Windows 64 bit Release 
2021-05-31 16:18:58.013 Prepare fixed and missing parameters
2021-05-31 16:18:58.013 Run: 102 Default
2021-05-31 16:18:58.014 Get scenario parameters for process
2021-05-31 16:18:58.014 member=0 Bind scenario parameters
2021-05-31 16:18:58.014 member=0 Compute derived parameters
2021-05-31 16:18:58.014 member=0 Prepare for simulation
2021-05-31 16:18:58.015 member=1 Bind scenario parameters
2021-05-31 16:18:58.015 member=1 Compute derived parameters
2021-05-31 16:18:58.015 member=1 Prepare for simulation
2021-05-31 16:18:58.015 member=0 Simulation progress=0% cases=0
...

[back to arguments and options]
[back to topic contents]

Comparison

After running a model, test_models extracts each output table from the model database and saves a copy of the original data as well as a version rounded to 6 significant digits. test_models then computes and stores a digest of each rounded csv file which it uses for subsequent comparisons. When a rounded table differs, test_models verifies that at least one cell differs by more than one part per million using the original table data, before reporting the table as different.

This default behaviour can be modified by the following command options:

• --significant_digits The number of significant digits for table result comparison
• --zero_fuzz Set table cell values to zero if within this value of zero
• --nocells Disable fallback cell-by-cell verification of differing tables and copy of original data

The number of significant digits used for rounding can be modified using the --significant_digits option. The default value is 6 (one part in one million). This level of precision can help reduce the number of differing tables reported by test_models while still identifying differences of substantive or logical significance.

Rounding reduces the number of spurious differences which would otherwise be reported by test_models. It is not unusual for different model versions to produce slightly different results, often at the level of numerical precision (15-16 digits of precision). This can occur due to different C++ compiler optimizations, or by logically equivalent but slightly different algorithms in models. Rounding to less precision reduces the number of spurious differences reported by test_models.

Most spurious differences are eliminated by rounding, but not all. For example, two numbers can differ only at the 7th digit of precision but round to different values at 6 digits of precision if the two numbers happen to fall on different sides of a 'rounding boundary' (last digit 5). test_models eliminates these false positives by verifying differing rounded tables using the original unrounded data. In practice, this verification is rapid. This verification can be disabled using the --nocells command line option. The --nocells option also disables the creation of original (unrounded) copies of model output tables.

The value of a table cell can be undefined, eg the mean value of an attribute for a cell with no observations. This is not the same as a table cell value of zero, eg a mean value of 0.0 calculated from one or more observations. test_models considers an undefined value to be different from the value 0.

Table measures using subtraction can sometimes produce values very close to zero as an artifact of floating-point computations. The presence of such values can produce false positives: tables which test_models considers different but which are not different substantively. test_models uses a zero-fuzz value to handle this situation. If the absolute value of a table cell is within zero_fuzz of 0.0, the table cell will be set to 0.0. The default zero_fuzz value is 1e-15. The zero_fuzz value can be set to 0 to disable zero_fuzz altogether.

test_models computes digests using the MD5 cryptographic algorithm. For example, the MD5 digest of the RiskPaths table T04_FertilityRatesByAgeGroup in the Default run, rounded to 6 digits and normalized to csv is the hexadecimal value 993ead71da6eed5dde398342278f629a. The MD5 digest is always 32 hexadecimal digits in length, independent of the size of the input file. Files with different digests are guaranteed to be different, and it is extremely unlikely (read almost impossible) for different files to have the same digest.

[back to arguments and options]
[back to topic contents]

Informational

The following options are informational.

• --help report usage message and exit
• --version report test_models version and exit

They have short form versions -h and -v.

If either option is on the command line, test_models will display the requested information and immediately exit.

[back to arguments and options]
[back to topic contents]

EventTrace

The following options affect the normalization of event trace output, if present..

• --time_format STR time format for event trace (default 13.6f)
• --modgen_id_offset INT offset adjustment to Modgen IDs (default 0)

The --time_format option changes the precision of time in the normalized event trace report. For example, specifying --time_format 19.12f increases the precision from 6 to 12 digits after the decimal point, producing output like

  1969.960131010721 BirthdayEvent (Person 29)
                   1970.960131010721 timeBirthdayEvent (Person 29)

The --modgen_id_offset option corrects the entity ID in the Modgen normalized event trace report. This is helpful if the run reproduces a single case from a larger run using a case seed. For such runs, Modgen assigns entity IDs incorrectly, causing spurious differences with ompp event trace output. To fix the issue, the entity IDs in the Modgen event trace report need to be offset by the number of the sub they came from. That sub # may already be known, but if not it can be calculated from the case seed by dividing the case seed by 2^31 and taking the integer part of the result. For example, to determine the sub # of the case seed 11636581014, divide it by 2147483648 giving 5.4187052948.... The sub # of the case is 5. Specifying --modgen_id_offset 5 will offset all entity IDs in the Modgen event trace report by 5, correcting the error and bringing the Modgen event trace report into alignment.

[back to arguments and options]
[back to topic contents]

Deprecated

• --windows_platform Windows platform: x64(default) or Win32
• --modgen_platform Modgen platform: Win32(default) or x64

If desired, a 32-bit OpenM++ version of a model can be built instead of a 64-bit version, on Windows only, using the --windows_platform option.

Possibly, a 64-bit version of a Modgen model can be built instead of a 32-bit version using the --modgen_platform option. The build or run may fail, and run functionality may be limited or compromised.

[back to arguments and options]
[back to topic contents]

Control - Files

=== under construction ===

test_model operation is controlled mainly by command-line options, but also by the presence and contents of optional specially-named files in the model folder.

OpenM++ model ini files are described here. test_models always enables Model development options by automatically specifying the -OpenM.IniAnyKey to the model when a model ini file is used.

[back to topic contents]

Output - Report

=== under construction ===

test_models writes a report line by line to the command window where it was invoked as it carries out operations.

[back to topic contents]

Output - Files

test_models produces two kinds of output: A report it writes to the command window where it was invoked, and files which record and persist information about the model runs it performs. The files created by test_folders can be useful to probe details behind the summary information in a test_models report. For example, test_models reports a count of warnings during model build and also creates the file build.log containing more information about those warnings.

This subtopic contains the following sections.

• test_models folder: The test_models folder hierarchy and contents
• table outputs: The tables from a model run
• event trace: The normalized event trace from a model run

test_models folder

The output files generated by test_models are in the subfolder test_models of the model folder. The test_models subfolder and hierarchy are created automatically by test_models as needed. The test_models subfolder can be deleted at any time, and will be recreated on a subsequent invocation of test_models.

The following diagram shows the hierarchical structure of the test_models folder for the RiskPaths model, with some other subfolders of RiskPaths shown for context. In this example, test_models was previously invoked for the three flavours ompp-win, ompp-linux, and modgen.

RiskPaths
├───code
├───modgen
│   ├───bin
│   └───src
├───ompp
│   ├───bin
│   └───src
├───ompp-linux
│   ├───bin
│   └───src
├───parameters
│   └───Default
└───test_models
    ├───current
    │   ├───modgen
    │   │   ├───logs
    │   │   └───outputs
    │   │       └───original
    │   ├───ompp-linux
    │   │   ├───logs
    │   │   └───outputs
    │   │       └───original
    │   └───ompp-win
    │       ├───logs
    │       └───outputs
    │           └───original
    └───reference
        ├───modgen
        │   ├───logs
        │   └───outputs
        │       └───original
        ├───ompp-linux
        │   ├───logs
        │   └───outputs
        │       └───original
        └───ompp-win
            ├───logs
            └───outputs
                └───original

test_models organizes information using two folder hierarchy levels, as illustrated. For example, RiskPaths results for the Reference run with the ompp-win flavour are in the folder RiskPaths/test_models/reference/ompp-win.

Below is a list of all files produced by test_models within each flavour folder, eg within RiskPaths/current/ompp-win:

• tombstone.txt - Contains basic information about the compiler, members, ini file, etc. used to build and run the model. test_models reports if this file differs between Current and Reference.
• logs/build.log - A copy of captured console output from all stages of the build process.
• logs/run.log - A copy of the log file produced by the model when run.
• outputs/trace.txt - A normalized version of the event trace file, if produced by the model.
• outputs/0_MODEL_INI.txt - The contents of the model ini file used to run the model, if any.
• outputs/0_SIGNIFICANT_DIGITS.txt - Contains the number of significant digits used to round model output tables before computing digests.
• outputs/digests.txt - Contains the name and digest value of each file in the outputs folder (excluding subfolders).
• outputs/*.csv - Model output tables, rounded.
• outputs/original/*.csv - Model output tables, original (unrounded).

[back to output files]
[back to topic contents]

table outputs

Each table is saved in a flat csv format with a header line. Zero-based indexes identify the table cell. Each row consists of all classificatory dimension indices, followed by the single measure dimension index, followed by the cell value. The order of classificatory dimensions is the same as the order in the table statement which declared the table in model code. Classificatory dimensions will include an additional trailing level for the margin, if a margin is specified for the dimension in the table declaration. The measure index is always present even if there is only one measure in the measure dimension. The cell value can be empty. Here's an example extract:

Dim0,Dim1,Dim2,Dim3,Value
0,0,0,0,258.453
0,0,0,1,1119.96
0,0,0,2,933.714
0,0,0,3,86.1511
0,0,1,0,172.302
...

[back to output files]
[back to topic contents]

event trace

=== under construction ===

Note that times >= 32767 will be shown as 99999, as will +inf, to improve comparability between ompp and Modgen event trace files.

Note options which control normalized event trace operation.

Ompp models must set EventTrace.Style to modgen for event trace files to be normalized and comparable.

[back to output files]
[back to topic contents]

Example 1 Preparing the default run

This example illustrates how to arrange that the default test_models run is fast, and is easier to debug or investigate if run-time anomalies or unexpected outputs occur during model development. The underlying idea is to develop a model in small incremental steps, and use test_models to "fail fast, fail early", after each incremental change to the model. This approach is efficient only if the testing phase is fast. This example uses the OncoSimX model on the Windows platform.

Step 1 investigates how the model performs with a normal Default run. First, a fresh version of the model is created using git to clone from the git server to the temporary folder C:/Temp/OncoSimX. Next, test_models is run with the --timing option to evaluate the performance of the Default run.

C:\temp\OncoSimX>%OM_ROOT%\bin\test_models -m .. --nomodgen --nocomp --timing OncoSimX
             =========================
              test_models 2.1
             =========================

             Testing: OncoSimX
             ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release

             OncoSimX: ompp-win: Build and publish model and Default scenario
             OncoSimX: ompp-win: Build time 1m 47s
  warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
             OncoSimX: ompp-win: Run model using OncoSimX.ini
             OncoSimX: ompp-win: Run time 25m 3s
             OncoSimX: ompp-win: Convert outputs (6 digits of precision)
             OncoSimX: ompp-win: Convert time 1m 33s
             OncoSimX: ompp-win: Create digests of current outputs
             OncoSimX: ompp-win: Digest time 0m 0s
             OncoSimX: ompp-win: No Reference outputs - create using Current outputs
             OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
             OncoSimX: ompp-win: Compare time 0m 0s

The test_models report shows that the run itself took 25m, and that the run used the model ini file OncoSimX.ini. An ini file with the same name as the model is used by default if it exists. Here's the contents of that file:

[OpenM]
SubValues = 12
Threads = 6

[Parameter]
SimulationCases = 1000000
SimulationSeed = 16807

; Complete example of ini-file located at: props/model/ompp/Model-example.ini

The run had 1,000,000 cases divided among 12 members (aka replicates or sub-samples), and was accelerated by running 6 threads in parallel (the workstation for the run had 8 processor cores). As a test, 1,000,000 cases does explore many conditions in the model and can produce statistically meaningful results, but a run of that size with OncoSimX is not suitable for quick iteration during model development. Moreover, a multi-threaded run is much more difficult to debug than a single-threaded run, making it a poor choice for testing during development. Ignoring run time, test_models took about 3m 15s. If the run were modified to take 1m 45s of run time instead, with one thread of execution, the total time would be about 5m, which would be acceptable for rapid testing.

Step 2 creates a new model ini file named test_models.ini based on the original model ini file OncoSimX.ini with several modifications. The number of members is changed to 1, the number of threads to 1, and the the number of cases changed somewhat arbitrarily to 5,000 to see how fast that will turn out:

[OpenM]
SubValues = 1
Threads = 1

[Parameter]
SimulationSeed = 1
SimulationCases = 5000

The choice of file name for the modified model ini file was not arbitrary. test_models will use by default a model ini file named test_models.ini if it exists.

Step 3 runs test_models with the new model ini test_models.ini which it will use by default. The --newref option is used to replace the previous results to avoid reporting meaningless differences.

C:\temp\OncoSimX>%OM_ROOT%\bin\test_models -m .. --nomodgen --nocomp --timing --newref OncoSimX
             =========================
              test_models 2.1
             =========================

             Testing: OncoSimX
             ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release

             OncoSimX: ompp-win: Deleting previous Reference information
             OncoSimX: ompp-win: Build and publish model and Default scenario
             OncoSimX: ompp-win: Build time 1m 46s
  warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
             OncoSimX: ompp-win: Run model using test_models.ini
             OncoSimX: ompp-win: Run time 1m 41s
             OncoSimX: ompp-win: Convert outputs (6 digits of precision)
             OncoSimX: ompp-win: Convert time 1m 19s
             OncoSimX: ompp-win: Create digests of current outputs
             OncoSimX: ompp-win: Digest time 0m 1s
             OncoSimX: ompp-win: No Reference outputs - create using Current outputs
             OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
             OncoSimX: ompp-win: Compare time 0m 0s

The report shows that test_models.ini was used as intended. The total time for test_models to build, run, and compare results was 4m 47s, which is about what was targeted as a quick test for incremental model development.

Step 4 uses git to add test_models.ini to the local repository in C:/Temp/OncoSimX, and then push the change to the server repository to persist it and make it available to others on the modelling team.

Step 5 runs test_models a final time using test_models.ini to see how long the Modgen version takes.

C:\Development\X\ompp\Perl>perl test_models.pl -m ../../models --noompp --timing OncoSimX
             =========================
              test_models 2.1
             =========================

             Testing: OncoSimX
             modgen settings: version=12,1,3,0 (2019-12-19 20:31 GMT) platform=Win32 configuration=release

             OncoSimX: modgen: Build model and prepare Default scenario
             OncoSimX: modgen: Build time 0m 15s
  warning => OncoSimX: modgen: 1 build warnings - see OncoSimX/test_models/current/modgen/logs/build.log
             OncoSimX: modgen: Run model using test_models.ini
             OncoSimX: modgen: Run time 38m 54s
  warning => OncoSimX: modgen: 1 run warnings - see OncoSimX/test_models/current/modgen/logs/run.log
             OncoSimX: modgen: Convert outputs (6 digits of precision)
             OncoSimX: modgen: Convert time 12m 3s
             OncoSimX: modgen: Create digests of current outputs
             OncoSimX: modgen: Digest time 0m 1s
             OncoSimX: modgen: Current vs. Reference: 205 the same (of 205)
             OncoSimX: modgen: Compare time 0m 0s

Even with the small run in test_models.ini, the Modgen version takes 50m 13s, making it not suitable for quick development tests for the OncoSimmX model with all tables. The Modgen version simulates quickly, but writing all table outputs and converting those outputs takes a long time. The reason is that Modgen uses a Microsoft Access database for storing tables, and MS Access is slow.

The folder C:/Temp/OncoSimX is now deleted, having served its purpose.

[back to topic contents]

Example 2 A new OpenM++ release

In this example, a new OpenM++ release has been announced, and a model developer is asked to verify that the team's model successfully builds and runs with the new version and produces identical results. The model developer uses a Windows workstation.

Step 1 downloads and extracts the new version of OpenM++, released on 2021-06-29, to a new folder on the workstation. To simplify changing between OpenM++ versions, the model developer previously created a folder named C:/ompp_versions on the workstation. A new command prompt is opened and used to verify the contents of that folder after the download and extraction:

C:\ompp_versions>dir
 Volume in drive C is OS
 Volume Serial Number is 14E2-D15F

 Directory of C:\ompp_versions

2021-06-29  07:27 AM    <DIR>          .
2021-06-29  07:27 AM    <DIR>          ..
2021-06-12  09:49 PM    <DIR>          openmpp_win_20210505
2021-06-12  09:50 PM    <DIR>          openmpp_win_20210602
2021-06-12  09:50 PM    <DIR>          openmpp_win_20210612
2021-06-29  07:27 AM    <DIR>          openmpp_win_20210629
               0 File(s)              0 bytes
               6 Dir(s)  1,717,655,265,280 bytes free

A command is issued to verify the version of OpenM++ currently in use on the workstation:

C:\ompp_versions>echo %OM_ROOT%
C:\ompp_versions\openmpp_win_20210602

The version currently in use is 2021-06-02 which is two versions old because the development team decided to skip the mid-month OpenM++ bug-fix release because that bug was known not to affect the team's model.

Step 2 prepares the test. The OncoSimX model is used in this example and it's assumed that the model developer is working actively on the model and has unchecked code changes. To not interfere with that current work-in-progress, git is used to create a new temporary clone of the stable trunk version of OncoSimX in the temporary folder C:/Temp/OncoSimX.

Step 3 invokes test_models to create a Reference run using the team's working version (2021-06-02) of OpenM++:

C:\ompp_versions>cd C:\Temp\OncoSimX
C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp OncoSimX
             =========================
              test_models 2.0
             =========================

             Testing: OncoSimX
             ompp-win settings: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release

             OncoSimX: ompp-win: Build and publish model and Default scenario
  warning => OncoSimX: ompp-win: 1794 build warning(s) - see OncoSimX/test_models/current/ompp-win/logs/build.log
             OncoSimX: ompp-win: Run model using test_models.ini
             OncoSimX: ompp-win: Convert outputs (7 digits of precision)
             OncoSimX: ompp-win: Create digests of current outputs
             OncoSimX: ompp-win: No Reference outputs - create using Current outputs
             OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)

The -m .. option specifies that the folder containing models is the parent folder of the current folder.

Step 4 switches to the new OpenM++ release dated 2021-06-29 by changing the OM_ROOT environment variable:

C:\temp\OncoSimX>set OM_ROOT=C:\ompp_versions\openmpp_win_20210629

and invokes test_models again using the same arguments:

C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp OncoSimX
             =========================
              test_models 2.1
             =========================

             Testing: OncoSimX
             ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release

             OncoSimX: ompp-win: Build and publish model and Default scenario
  warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
             OncoSimX: ompp-win: Run model using test_models.ini
             OncoSimX: ompp-win: Convert outputs (6 digits of precision)
             OncoSimX: ompp-win: Create digests of current outputs
             OncoSimX: ompp-win: Current vs. Reference: tombstone info differs:
             OncoSimX: ompp-win:   Reference: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release members=1
             OncoSimX: ompp-win:   Current:   compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release members=1
             OncoSimX: ompp-win: Current vs. Reference: 214 the same (of 215)
DIFFERS ===> OncoSimX: ompp-win: Current vs. Reference:   1 differ
DIFFERS ===> OncoSimX: ompp-win: Current vs. Reference: DIFFERS:           0_SIGNIFICANT_DIGITS.txt

test_models reports that the OpenM++ compiler versions for the Reference and Current runs differ, as expected. Unexpectedly, it reported a single differing file. The name of that file and the two test_models reports pinpoint the cause: The default number of digits used by test_models was changed from 7 digits to 6 digits between the two OpenM++ releases. The change in the number of significant digits turned out not to affect the rounded versions of the model output tables in the two runs.

The test, at this point, has established that the model builds and runs successfully with the new version of OpenM++ and produces identical outputs for the small fast run used normally by test_models during OncoSimX development.

In step 5, the model developer decides to perform a more demanding verification by comparing results between the two OpenM++ releases using a larger run of 1,000,000 cases and 12 replicates. These are the settings in the model ini file OncoSimX/OncoSimX.ini:

[OpenM]
SubValues = 12
Threads = 6

[Parameter]
SimulationCases = 1000000
SimulationSeed = 16807

; Complete example of ini-file located at: props/model/ompp/Model-example.ini

The OpenM++ version is switched back to the older 2021-06-02 version

C:\temp\OncoSimX>set OM_ROOT=C:\ompp_versions\openmpp_win_20210602

and test_models is invoked, this time explicitly specifying the number of significant digits as well as the model ini file to use. The --newref option is used to create a new Reference run using the larger run specified in the ini file.

C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp --significant_digits 6 --ini OncoSimX.ini --newref OncoSimX
             =========================
              test_models 2.0
             =========================

             Testing: OncoSimX
             ompp-win settings: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release

             OncoSimX: ompp-win: Deleting previous Reference information
             OncoSimX: ompp-win: Build and publish model and Default scenario
  warning => OncoSimX: ompp-win: 1794 build warning(s) - see OncoSimX/test_models/current/ompp-win/logs/build.log
             OncoSimX: ompp-win: Run model using OncoSimX.ini
             OncoSimX: ompp-win: Convert outputs (6 digits of precision)
             OncoSimX: ompp-win: Create digests of current outputs
             OncoSimX: ompp-win: No Reference outputs - create using Current outputs
             OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)

The OpenM++ version is switched once again to the new release 2021-06-29

C:\temp\OncoSimX>set OM_ROOT=C:\ompp_versions\openmpp_win_20210629

and test_models is run again to produce a comparable Current run and compare it to the Reference run:

C:\temp\OncoSimX>perl %OM_ROOT%/perl/test_models.pl -m .. --nomodgen --nocomp --significant_digits 6 --ini OncoSimX.ini OncoSimX
             =========================
              test_models 2.1
             =========================

             Testing: OncoSimX
             ompp-win settings: compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release

             OncoSimX: ompp-win: Build and publish model and Default scenario
  warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
             OncoSimX: ompp-win: Run model using OncoSimX.ini
             OncoSimX: ompp-win: Convert outputs (6 digits of precision)
             OncoSimX: ompp-win: Create digests of current outputs
             OncoSimX: ompp-win: Current vs. Reference: tombstone info differs:
             OncoSimX: ompp-win:   Reference: compiler=omc.exe (2021-06-02 03:23 GMT) platform=x64 configuration=release members=12
             OncoSimX: ompp-win:   Current:   compiler=omc.exe (2021-06-29 06:31 GMT) platform=x64 configuration=release members=12
             OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)

The larger run executed successfully with the new OpenM++ release, and all output results were the same to 6 significant digits.

In this example, the OM_ROOT environment variable was changed using the set command (not the setx command) to select an OpenM++ version. The changed value of OM_ROOT applied only to commands issued in the command prompt window used for the test. The value of OM_ROOT in other command prompt windows or Visual Studio sessions was unaffected by the test.

[back to topic contents]

Example 3 A model code change

=== under construction ===

[back to topic contents]

Example 4 When results differ

=== under construction ===

[back to topic contents]

Example 5 Exercising a model in Debug

In this example, test_models is used to build and run Debug versions of a model, using both OpenM++ and Modgen versions. Debug versions run more slowly but contain more run-time checks which can identify errors, so it is a good idea to test the Debug versions from time to time. test_models1 makes that easy. This example uses the OncoSimX` model running on a Windows system.

In step 1, a fresh set of Reference runs are created:

The Modgen run took considerably longer than the OpenM++ run.

[back to topic contents]

Example 6 A performance comparison

In this example, test_models is used to evaluate the performance difference between two versions of the same model, holding other effects constant. Specifically, this example seeks to measure the performance cost of the run-time checks which OpenM++ performs to ensure that attributes are not modified directly or indirectly in event time functions. Although those runtime checks are core OpenM++ functionality, the OpenM++ option verify_attribute_modification can be used to remove them.

Step 1 sets up the experiment. The OncoSimX model is chosen because it is a large-scale production model with many events. The experiment is done on a secondary (older) Windows workstation which will run no other tasks during the experiment. The experiment is temporary, so a single git command is issued to create a new clone of OncoSimX in a convenient temporary location, which in this example is C:/Temp/OncoSimX. The OncoSimX clone contains no test_models results, which is appropriate for a model git repository. It does contain the model ini file test_models.ini in the model root.

Step 2 sets up the test_models model run to serve as a performance test. Normally, a model run for test_models is small and fast. This experiment requires a run which is large enough to produce reliable performance information. The OncoSimX model ini file test_models.ini produces small fast runs for rapid development. The contents are:

[OpenM]
SubValues = 1
Threads = 1

[Parameter]
SimulationSeed = 1
SimulationCases = 5000

[Tables]
;Retain = \
;	 TG01_Breast_Cancer_Tables

A single line is modified to increase the number of cases from 5,000 to 1,000,000 for the experiment.

SimulationCases = 1000000

Step 3 uses test_models to create a Reference run to serve as a basis for comparison. A command window is opened and test_models is invoked:

perl test_models.pl -m C:/Temp --timing --nomodgen OncoSimX

The temporary OncoSimX clone was created in the folder C:/Temp, so that folder is specified as the models folder using the -m option. Timing information is requested using the --timing option. The experiment is being done on a Windows workstation with Modgen installed, so the --nomodgen argument is used to suppress that flavour.

The output is

             =========================
              test_models 2.1
             =========================

             Testing: OncoSimX
             ompp-win settings: compiler=omc.exe (2021-07-01 12:27 GMT) platform=x64 configuration=release

             OncoSimX: ompp-win: Build and publish model and Default scenario
             OncoSimX: ompp-win: Build time 3m 58s
  warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
             OncoSimX: ompp-win: Run model using test_models.ini
             OncoSimX: ompp-win: Run time 98m 8s
             OncoSimX: ompp-win: Convert outputs (6 digits of precision)
             OncoSimX: ompp-win: Convert time 2m 10s
             OncoSimX: ompp-win: Create digests of current outputs
             OncoSimX: ompp-win: Digest time 0m 1s
             OncoSimX: ompp-win: No Reference outputs - create using Current outputs
             OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
             OncoSimX: ompp-win: Compare time 0m 0s
             OncoSimX: Flavour compare time 0m 0s

Note that test_models created Reference results from Current results because no Reference run was present. The reported run time was 98m 8s.

Step 4 modifies the model to remove the normal run-time checks for the experiment. Because this is a temporary clone of OncoSimX which will be discarded after the experiment, the source code can be modified freely with no consequences. The small source file TraceOptions.mpp was selected arbitrarily to implement the experiment, and the following line added:

options verify_attribute_modification = off;

Step 5 uses test_models to create a Current run which will be compared to the Reference run. The invocation of test_models is identical to the previous invocation.

perl test_models.pl -m C:/Temp --timing --nomodgen OncoSimX

The output is

             =========================
              test_models 2.1
             =========================

             Testing: OncoSimX
             ompp-win settings: compiler=omc.exe (2021-07-01 12:27 GMT) platform=x64 configuration=release

             OncoSimX: ompp-win: Build and publish model and Default scenario
             OncoSimX: ompp-win: Build time 3m 58s
  warning => OncoSimX: ompp-win: 1794 build warnings - see OncoSimX/test_models/current/ompp-win/logs/build.log
             OncoSimX: ompp-win: Run model using test_models.ini
             OncoSimX: ompp-win: Run time 95m 16s
  warning => OncoSimX: ompp-win: 1 run warnings - see OncoSimX/test_models/current/ompp-win/logs/run.log
             OncoSimX: ompp-win: Convert outputs (6 digits of precision)
             OncoSimX: ompp-win: Convert time 2m 12s
             OncoSimX: ompp-win: Create digests of current outputs
             OncoSimX: ompp-win: Digest time 0m 0s
             OncoSimX: ompp-win: Current vs. Reference: 215 the same (of 215)
             OncoSimX: ompp-win: Compare time 0m 0s
             OncoSimX: Flavour compare time 0m 0s

All model outputs are identical as expected. test_models reports one run-time warning which was absent in the first run. That's because OpenM++, by design, emits a warning on every run of a model which disables run-time checking of event times. test_models noted the warning and reported it when it analyzed the run log. The reported run time was 95m 16s.

Step 6 compares performance between the Reference and Current runs. The model run time in the Current run was 172s less than the Reference run, and shows that about 2.9% of the elapsed time of the Reference run was overhead associated with the run-time checks. That is a realistic assessment of the overhead of the run-time checks for OncoSimX when used in production.

The elapsed time reported by test_models includes the time required to read parameters and to assemble and write tables, which can be significant. It might also be interesting to assess the performance effect on the simulation time alone, removing the fixed cost of parameter reading and table assembly and writing. The model log files saved by test_models can be used to perform that calculation.

Here's a line extracted from the Reference log file, located at
C:/Temp/OncoSimX/test_models/reference/ompp-win/logs/run.log:

2021-07-01 10:16:52.928 member=0 Simulation summary: cases=1000000, events/case=1128.9, entities/case=5.4, elapsed=5693.747633s

and here's the corresponding line extracted from the Current log file, located at
C:/Temp/OncoSimX/test_models/current/ompp-win/logs/run.log.

2021-07-01 12:31:45.777 member=0 Simulation summary: cases=1000000, events/case=1128.9, entities/case=5.4, elapsed=5525.430919s

Conveniently, this line in the OpenM++ model log reports the elapsed time of the computational phase, in seconds. The simulation time was 5694s for the Reference run and 5525s for the Current run, for a difference of 168s. So about 3.0% of the computation time was overhead associated with the run-time checks. That's only slightly higher than the proportion computed using elapsed time because the time required to read parameters and to assemble and write tables (about 190s) is a relatively small portion of the elapsed time of an OncoSimX run of 1,000,000 (5525s).

[back to topic contents]

Example 7 Using event trace output

=== under construction ===

The following applies to a case-based model.

If test_models detects unexpected output differences which are not immediately explained by recent model code changes or by perusing differences in specific tables, it can be useful to determine whether the differences are due to a handful of different cases in the run, or to some other cause.

To find out, turn on the case checksum option by adding the statement

options case_checksum = on;

to model code. In the RiskPaths example model, this is done in a dedicated model source file named TraceOptions.mpp whcih is normally:

//LABEL(TraceOptions, EN) Control options for trace output

/* NOTE(TraceOptions, EN)
    The options must be set to off for normal use.
    If the options are on, very large trace output files may be produced by the model.
*/

options event_trace = off;
options case_checksum = off;

Turning case_checksum on will cause a trace file to be generated when test_models runs the model which contains one line for each case. For RiskPaths, the first lines of the trace file look like this:

Seed:               1 Sample:  0 Checksum: 4.4094661700e+02
Seed:       470583131 Sample:  0 Checksum: 4.1293225500e+02
Seed:      1715377866 Sample:  0 Checksum: 3.4860055300e+02
Seed:       430166727 Sample:  0 Checksum: 1.3482143400e+02
Seed:       743781969 Sample:  0 Checksum: 1.4831547800e+02
Seed:       594673803 Sample:  0 Checksum: 4.1803896100e+02
Seed:      2048386738 Sample:  0 Checksum: 3.5386231900e+02
Seed:      1681919254 Sample:  0 Checksum: 4.0119902000e+02
Seed:        78320563 Sample:  0 Checksum: 3.9439706500e+02
Seed:      1637697257 Sample:  0 Checksum: 4.9065368400e+02
Seed:       127732046 Sample:  0 Checksum: 9.8715092900e+02
Seed:      2114816392 Sample:  0 Checksum: 3.9576701300e+02
Seed:      1616242686 Sample:  0 Checksum: 1.2478963010e+03
Seed:      1222943222 Sample:  0 Checksum: 4.5861339400e+02
Seed:       170881636 Sample:  0 Checksum: 1.4320257000e+02

This file is noticed by test_models and will participate in test_models comparisons. Test_models saves the trace file under the name trace.txt in the same folder where it places csv versions of tables.

If test_models reports that the file trace.txt differs between the ompp and modgen versions, then one or more cases differ in the two flavours. Use a tool or text editor to compare the two versions of trace.txt to determine the seed of the first case which has a differing checksum in the two variants.

More to follow...

[back to topic contents]