TST Framework for running functional tests on S3DF

[gadorlhiac]

Description

This PR provides a framework for running functional tests of workflows on S3DF.

Under tests/functional each test is given a sub-directory which must contain:

• config.yaml to specify the LUTE config YAML for running the test workflow. This YAML must specify an experiment and run. It cannot rely on retrieving this information from the eLog or the command-line.
• dag.yaml : A workflow specification in line with the dyanmic Airflow DAG YAML syntax for LUTE.

Each test sub-directory may also contain:

• A README.md file to explain the test.
• A SHOULD_FAIL file which is empty. This should be provided if the workflow is expected to fail and the test should be marked succesful if it does.

Checklist

• A set of test workflows
• run_functional.py script to run all functional tests.
• submit_run_functional.sh to supervise tests from a batch job.
• Provide documentation on testing

PR Type:

• Testing

Address issues:

Simplify functional testing.

Testing

Usage

Full Script Usage

There are two main scripts: run_functional.py which does the bulk of the work, and submit_run_functional.sh which is a thin wrapper to submit a SLURM job. As an alternative to a SLURM job, run_functional.py can be run in the background with output redirected to a file, or perhaps more securely using, e.g. screen. run_functional.py can be used like:

usage: run_functional.py [-h] [-a] [--git_pr_id GIT_PR_ID] [--git_tag GIT_TAG] [--no_delete] -r RUN_DIR [--run_tests RUN_TESTS] [--tests_dir TESTS_DIR] [--test_airflow] [--use_local_tests]

Run the LUTE functional test suite.

optional arguments:
  -h, --help            show this help message and exit
  -a, --admin           Run as Airflow admin. Requires permissions.
  --git_pr_id GIT_PR_ID
                        Check out a specific GitHub PR ID of LUTE to run (a PR branch).
  --git_tag GIT_TAG     Check out a specific git tag of LUTE to run (e.g. a release).
  --no_delete           If passed, do not delete output files when tests are finished.
  -r RUN_DIR, --run_dir RUN_DIR
                        Directory to install LUTE to.
  --run_tests RUN_TESTS
                        Provide a comma-separated string of tests to run. If provided, this script will only run those, rather than the default behaviour of running all tests. E.g: --run_these_tests test2,test5. Tests that do not exist are silently ignored.
  --tests_dir TESTS_DIR
                        Specify an alternative path to tests than those from the LUTE clone. Must have the same directory structure: $DIR/test1/... $DIR/test2/... If this flag and --use_local_tests are both passed, this one is used.
  --test_airflow        Use test Airflow instance.
  --use_local_tests     Use the tests from the installation of LUTE where this script is called, rather than those from the clone of LUTE which is run against, or another directory if passed. If this flag and --tests_dir are both passed, --tests_dir is used.

Refer to https://github.com/slac-lcls/lute for more information.

submit_run_functional.sh transparently passes parameters so has the same interface.

Example/walkthrough

• Check out this branch (until merging)
• Run this command on S3DF:

./lute/tests/submit_run_functional.sh --run_dir <$OUTPUT_DIR> --git_pr_id 64 --test_airflow

$OUTPUT_DIR can be replaced with any directory you would like to work in. The test script will create two folders in the directory:

OUTPUT_DIR
|
|------------ lute # (The clone of the repository)
|
|------------ lute_output # (All output files will be placed here)

For the time being you should include --git_pr_id 64 when running the command above. This ensures that LUTE gets cloned from this branch that actually has the tests to run. You can altenatively pass a directory which has been setup with a separate set of tests (see full usage above). After cloning this branch you could also use --use_local_tests.

--test_airflow should be used to use the test Airflow instance. This is to get the improvements to the dynamic Airflow DAG which are needed to mark failure of a test properly. This will be pushed to the production Airflow on merging of this PR. The tests can be run without the improvements to the DAG (i.e. using the production Airflow instance) now as well, but may be incorrectly marked as successful/unsuccessful.

The output folders are temporary (i.e. lute and lute_output) and everything will be deleted when the tests have finished. The parent OUTPUT_DIR will not be touched. It must exist before running the script and will not be created.

You can examine the slurm output file as the tests run, or at the end. All the logs from each step should be there, but to check on the results of the tests, grep and tail are sufficient.

E.g.

> grep "Test workflow" slurm-59804105.out 
INFO:Launch_Func_Tests:Test workflow test2 completed successfully.
INFO:Launch_Func_Tests:Test workflow test1 was unsuccessful but this is marked as intentional.
> tail -n 4 slurm-59804105.out 
INFO:Launch_Func_Tests:Ran 2 tests. 2 were successful.
DEBUG:Launch_Func_Tests:Removing duplicate Kerberos credentials.
INFO:Launch_Func_Tests:Cleaning up /sdf/scratch/users/d/dorlhiac/func_tests/lute
INFO:Launch_Func_Tests:Cleaning up /sdf/scratch/users/d/dorlhiac/func_tests/lute_output

Screenshots