Feature Request: flag to re-use a previous build

[hagertnl]

Proposed feature:

runtests.py --reuse-last-build [--no-match-runtag] [--runtag=myruntag]
OR
runtests.py --use-build-from=/path/to/build

In both of these cases, the Source directory will NOT be copied. Instead, a build_directory from a previous test or another build directory will be used. If no eligible build or directory is found, the harness will exit with a build error (ie, build_end exit status of 1).

The first option, runtests.py --reuse-last-build will copy the build_directory as-is from the previous build. By default, the harness will search backwards by test ID to find the first test that (1) completed the build stage successfully and (2) has the same runtag as the current run. Optionally, the second criteria can be voided by either passing --no-match-runtag, in which case the first build found will be used, or using --runtag=myruntag to specify a runtag from which to grab the build from. This option comes with this caveat -- any updates to your Source directory since that run won't be realized, including updates to things like environment scripts. This is not a good option if you hard-code module versions that you wish to test in the environment script. However, if you use the default modules and/or Lmod MODULERC files, this is a great option.

The second option is to directly specify a directory to use the build from. This can either be a harness-generated build, or it can be a custom build. However, it must be compatible with your test's expected build_directory. For example, if your job script does source ${BUILD_DIR}/env.sh, then your custom build must provide the env.sh file at the top level.

[hagertnl]

An option would also be to just use the build_directory sym-link to point to the reusable build, instead of physically copying a build. From a reproducibility standpoint, it's still perfectly valid (the files still exist, we're not anonymously redirecting), and makes it much more traceable, as opposed to grepping a file for a path.