ReFrame 2.16

ReFrame 2.16 Release Notes

New features and enhancements

• Add support for setting a minimum number of required tasks in tests with flexible task allocation (#652)
• Add command line option for increasing the verbosity level of output (#650)
• Add command line options for showing the system and environments configuration (#633)
• Print error information when a modules system operation fails (#628)
• Add support for units in performance references (#576)
• Add a new count_uniq() sanity function (#584)

Bug fixes

• Improve handling of bad tests to avoid frontend crashing (#623)
• Fix bug in variable names in Slurm backend (#639)
• Do not crash if Slurm node does not contain partition information (#621)
• Check if 'UnavailableNodes' are down to cancel job (#599)
• Fix deadlock in the flexible task allocation when the node list output is too large (#614)

Regression tests

• Port remaining tests to the new syntax (#656)
• Add Likwid benchmarks for cache, memory and cross-socket bandwidths (#647)
• Add new MPI_Init_thread() check (#620)
• Add benchmark measuring CUDA kernel launch latency (#646)
• Add HPX 1D stencil tests (#617)
• Add tests for SPEC-ACCEL benchmarks (#618)
• Add test for distributed training with TensorFlow and Horovod (#637)
• Add flexible HPCG test (#604)
• Add HPX hello world test (#616)
• Add allocation speed check (#597)
• Fail if test is not compiled with OpenACC (#615)
• Convert GpuBandwidthCheck to a flexible diagnostic check (#586)
• Testing the environment of the login nodes when accessed remotely over SSH (#583)
• Promote DGEMM to a multi-node check (#530)
• Enable NCO and CDO for the C2SM-RCM environments (#624)
• Make Alltoallv tests non-strict (#622)
• Add specific tag for MCH tests (#594)
• Remove ifdef USE_MPI from the MCH OpenACC/CUDA/MPI/C++ test (#585)
• Set exclusive_access for Kesch tests (#591)
• Adapt MCH regression tests to C2SM-RCM environments (#513)
• Adapt CP2K test (#596)

Other

• Make .pep8speaks.yml consistent with coding style (#649)
• Add option to disable CSCS CI tests (#648)
• Enable pep8speaks for code style checking in the repository (#626)
• Use integer comparison for ReFrame versions in deployment script (#600)
• Replace python 3.7-dev with 3.7 in .travis.yml (#588)
• Document verbosity option (#665)
• Add link for the automatic invites to Slack (#661)
• Update README with contact info (#642)
• Move SC18 presentation & demo to docs (#606)

ReFrame 2.15

ReFrame 2.15 Release Notes

New features and enhancements

• Update retries stage/output directory naming convention (#580)
• Full support for flexible task allocation (#522)
• Add new command-line option for more verbose test listing (#506)
• List only the checks that are supported on the current system (#487)
• Use different stage and output directories for different retries of the same test (#464)

Bug fixes

• Directly assign to num_tasks if --flex-alloc-tasks has an integer value (#558)
• Show also hidden partitions in Slurm, when trying to flexibly allocate tasks (#553)
• Fix redundant export statements (#448)
• Fix retry report when test fails before the setup phase (#535)
• Do not silently abort tasks in hold when all currently running tasks fail (#533)
• Remove the validation of the name of the test's base classes (#480)
• Remove time limit from build jobs (#474)

Regression tests

• Update NAMD test (#574)
• Run libsci_acc and libsci_resolve tests on compute nodes as well (#581)
• Update GROMACS test for Dom (#579)
• Ruby 2.2.2 NArray test (#575)
• Add test checking if CRAY_CPU_TARGET is set (#572)
• Update MAGMA tests (#571)
• Update CP2K test (#559)
• Adapt the GPU burn test for multi-gpu nodes (#564)
• Disable Trilinos tests which fail due to the corresponding module (#551)
• Adapt usertool tests to new syntax (#552)
• Update GROMACS performance numbers (#548)
• Remove Monch from supported systems (#537)
• Remove read-only files attribute from the CP2K check (#529)
• Remove wildcards from programming environment specification in tests (#534)
• Fix NetCDF test on Kesch (#523)
• Sync OpenMP versions for Dom/Daint in notool test (#519)
• Add MCH CUDA stress test (#514)
• Adapt boost tests to latest pe18.08 (#518)
• Adapt the TensorFlow module name for PE 18.08 (#516)
• Adapt Score-P modules to PE 18.08 (#515)
• Add MCH halo exchange benchmark (#509)
• Adapt IOR checks to build systems (#505)
• Add MCH alltoallv check (#347)
• Adjust LAMMPS performance values (#504)
• Adapt the LAMMPS test to PE18.08 (#502)
• Adapt QE test to PE18.08 (#501)
• Remove PrgEnv-gnu support from the Scalapack test on Kesch (#500)
• Adapt NetCDF test to Kesch (#482)
• New OpenACC/CUDA/C++ test from MCH (#342)
• Adapt Kesch tests to final CSCS prog. environments (#479)
• Publish MAGMA tests and adapt to PE 18.08 (#463)
• Remove the matrix file from version control (#484)
• Set required version for tests using build systems (#476)
• Add matrix file add20.rua in the src dir (#481)
• Adapt trilinos regression test to PE18.08 and build systems (#466)
• Adapt gpu direct test to build systems (#462)
• Adapt CUDA NVML test to build systems (#461)
• Increase GROMACS sanity check tolerance to 0.1% (#459)
• Replace FFLAGS with FCFLAGS in Makefiles (#457)

Other

• Add C2SM environments on Kesch (#512)
• Re-enable running modified tests on Daint (#517)
• Adapt Kesch settings (#477)
• Automate deployment and generation of release notes (#460)
• Document the --flex-alloc-tasks feature (#582)
• Add User Lab Day slides to documentation (#520)
• Update documentation on the topic of filtering tests by system (#492)

ReFrame 2.14

ReFrame 2.14 Release Notes

New features and enhancements

• Add support for Autotools-based projects (#452)
• Add support for CMake (#447)
• Add a new infrastructure for build systems (#340)
  • Setting compilation flags directly in the programming environment is now deprecated.
  • Passing arguments to the compile() method is now deprecated.
  • For more information on how to use the new build system feature, please refer to the tutorial and the reference guide.
• Evaluate and log all performance values before asserting them (#359)
• Allow purging modules before running regression tests (#399)

Bug fixes

• Fix reading of logdir setting from system config (#445)
• Convert message for inexistent test files to warning (#440)
• Do not load environments twice (#441)
• Add a persistent version of rmtree() (#437)
• Filter out empty strings before writing preamble (#410)
• Fix bug in unloading of conflicting modules (#405)
• Enable sphinx versioning only for releases (#361)
• Change the resources dir path (#358)
• Fix expected OpenMP versions for Dom (#350)
• Comment out Graylog handler from CSCS config (#352)
• Fix site_configuration in docs (#344)

Regression tests

• Adapt stream microbenchmark test to build systems (#420)
• Adapt prgenv test to build systems (#449)
• Fix sourcesdir for the scorep test (#455)
• Update OSU microbenchmark test to new syntax and build system (#442)
• Adapt GpuBandwidthTest to build_systems (#438)
• Adapt perftools to use build systems (#439)
• Adapt helloworld tests to build systems (#416)
• Adapt netcdf tests to build systems (#434)
• Adapt cdo test to PE 18.07 (#431)
• Adapt ddt regression test to PE 18.07 (#425)
• Adapt cuda_gdb to build systems (#428)
• Adapt notools check to build systems (#427)
• Adapt hdf5 tests to build systems (#422)
• Adapt libsci_resolve test to PE 18.07 (#424)
• Adapt scorep test to PE 18.07 (#426)
• Adapt cuda checks to build systems (#421)
• Adapt dgemm test to use build systems (#418)
• Adapt the scalapack tests to use build systems (#417)
• Adapt haswell_fma to buildsystems style (#415)
• Adapt nvprof test to PE 18.07 (#414)
• Adapt the nco test for PE 18.07 (#412)
• Adapt Tensorflow tests for PE 18.07 (#409)
• Adapt the Boost Python test for PE 18.07 (#408)
• Adapt Paraview check for PE 18.07 (#401)
• Adapt the petsc test for PE 18.07 (#404)
• Update PrgEnv-pgi definitions on Kesch (#348)
• New flexible CUDA memtest check (#346)

Other

• Add testing for python3.7 in travis (#356)
• Remove monch from the machinesList (#436)
• New programming environments for Kesch (#432)
• Temporarily disable running of checks on Daint (#406)
• Check added/modified regression tests w.r.t master (#353)
• Update documentation for the build systems feature (#433)

ReFrame 2.13

ReFrame 2.13 Release Notes

API Changes

• Enhanced syntax of regression tests using decorators that eliminate the need of boilerplate code (#264, #294)
• Support for specifying compatible ReFrame versions for tests (#287)
• Enhance behavior of pre_run and post_run attributes, allowing to specify multiple job steps (#333)

Framework

• Redesign of performance logging (#213)
  • Performance logging can now be configured from the configuration file
  • Two different backends: file logging, Graylog
• More logging attributes added for formatting log messages (#213, #337)
• Support for PBS job scheduler (#316)
• Support for TMod4 (#282)
• Include name of the regression test file to the list of checks (#288)
• Better filtering of tests (#319)
• Add a knob for controlling the 'sacct'/'squeue' ratio in Slurm 'sacct' backend (#289)
• Migrate to pytest (#231)
• Bug fixes: #293, #305, #307, #308, #312, #318, #323, #325

Regression tests

• New regression tests and enhancements: #281, #284, #286, #310, #311
• Bug fixes: #309, #330, #331, #332

ReFrame 2.12

ReFrame 2.12 Release Notes

API Changes

• post_run and pre_run attributes removed from the Job class
• Support for wildcards in valid_prog_environs

Framework

• Support for automatic retry of failing tests (new --max-retries option)
• Support for custom configuration files (new -C option)
• Poll rate of the asynchronous execution policy is now lowered
• Unique test names are now enforced
• Environment variables are expanded in the -c option
• Documentation fixes and updates
• Internal abstraction layer for treating ReFrame versions
• Bug fixes: #192, #207, #221, #266, #274, #279

Regression tests

• New regression test for TensorFlow
• New flexible MPI Hello World test
• Add numerical sanity checks for Amber, CP2K, CPMD, GROMACS, LAMMPS, NAMD
• Corrected use of scoped dict key updates in the multi-GPU CUDA test
• Bug fixes: #242

ReFrame 2.11

ReFrame 2.11 Release Notes

API Changes

None

Framework

• New feature: support for module mappings
• Submit jobs from the stage directory
• Allow thresholds >= 1.0 in assert_reference() sanity function
• Flexible allocation take into account the --exclude-nodes option
• Add prefix rfm_ to the job names of ReFrame tests
• Fix crash when an unknown modules system is set in the configuration
• Fix and update documentation on the check_info logging attribute
• Fix math domain error in the polling function of the asynchronous execution policy
• Fix failure of the asynchronous execution policy's unit tests
• Fix confusing error messages in serial execution policy
• Fix emission of Slurm's --exclusive flag
• Fix logging configuration in generic settings

Regression tests

• Add regression tests for GPU direct
• Add a G2G meteoswiss regression test
• Add ICON RRTMGP regression test
• Add numerical sanity check in quantum espresso test
• Enable the switches resource certain tests
• Fix deprecation warnings in regression tests

ReFrame 2.10

ReFrame 2.10 Release Notes

API Changes

• RegressionTest.sourcesdir is now always copied to the stage directory
• Support for specifying Git repositories in RegressionTest.sourcesdir
• Job.pre_run and Job.post_run fields are deprecated in favour of the new ones RegressionTest.pre_run and RegressionTest.post_run

Framework

• Support for the LMod modules system
• Support for module aliases by the internal modules system abstraction
• New more robust design of the regression test execution policies
• More robust and efficient error checking for the Tmod modules system backend
• Fix framework behaviour when trying to run interactive commands
• Fir random unit tests failures with Python 3.5
• Documentation fixes

ReFrame 2.9

ReFrame 2.9 Release Notes

API Changes

• Old sanity and performance patterns syntax completely removed
• New extended syntax of extra_resources
• Allow sourcesdir to be None even for compiled regression tests
• More advanced examples in documentation

Framework

• Redesign of the exception hierarchy
• Version information moved away from settings
• Support for flexible job allocations that span a full reservation
• Support for easy deprecation of fields
• Fix ungraceful exit when a job is blocked in in unrecoverable state
• Fix regex group references to be compatible with Python 3.5
• Fix random failures in unit tests
• Fix behaviour of --partition option to override any similar option passed in site configuration

ReFrame 2.8.1

ReFrame 2.8.1 Release Notes

API Changes

None.

Framework

• Suppress SKIP messages in the output
• Support of an squeue-only Slurm backend
• ReFrame version is now printed in the performance logs
• New sanity function wrappers for itertools.chain(), glob.glob() and glob.iglob() functions
• Sanity functions reading files now accept an encoding option
• Fix typos in documentation
• Fix broken images in documentation

ReFrame 2.8

ReFrame 2.8 Release Notes

API Changes

• New RegressionTest attribute, extra_resources, for specifying custom job resources
• New API for retrieving and setting the parallel program launchers
• New API for retrieving the job scheduler of a system partition

Framework

• Redesign of the modules system backend
• Tmod is no more required to run ReFrame
• New system configuration option for setting the modules system
• More detailed output in the performance log
• Redesign of the job scheduler and parallel program launcher backends
• Expand support of several new parallel program launchers (mpirun, mpiexec etc.)
• Documentation fixes