Integration Test

Objective

The objective is to load data and perform some processing and to verify that pipeline processing products are as expected. The processing data flow picture shows a flow of data from Pawsey to the local NGAS server at Shanghai. The arrival of this data triggers loading and processing of the data to perform imaging and source finding tasks (depending on use case) and the generation of data products. These products are stored in the NGAS archive and made available via CASDA VO Tools.

Processing is dependent on the use case. The general approach is similar for any case and specific cases are identified and described below.

General Approach

In the first instance we will process a file currently stored in local NGAS instance.

The preference is to have the completion of data transfer from Pawsey to local NGAS trigger processing.

The test should run automatically and consists of the following main steps:

Transfer data from Pawsey to SHAO NGAS

• once file transfers are complete 'event' is fired to run NGAS plugin src/datamanagement/ngas-plugins/kickoff_source_extractor.py
• this calls a wrapper <test>_export_graph.py to initiate load data and process passing the relevant obs_ids.
• TopCat can be used to verify the files are in NGAS. See WP3

Load data and process

The <test>_export_graph.py script will:

• download the nominated file(s) from NGAS identified by the given obs_ids
• call processing task which will process the data files and archive results in NGAS
• wait for the result(s) to be produced, wait_for_file

Verification

• examine data products and compare with reference
• test succeeds if data products as expected, otherwise fails
• see separate use cases for details

Use cases

Listed in order of priority

Source Finding

Perform a source finding operation (see pipelines functional test notes).

Integration entry point src/integration/code/sf-test1-export-graph.py

Input

• a 512x512 pixel image with a single component at the centre, src/pipelines/Simple_Selavy_Test/singleSource.fits
• parset for configuration, src/pipelines/Simple_Selavy_Test/selavy-singleSource.in
• input files are specified in src/pipelines/Simple_Selavy_Test/selavy-fits-list.txt one file per line

Output

• components catalogue - text format
• components catalogue - xml/VO format
• island results also generated (not planning to use)

Expected result

• components catalogue - text format, as in src/pipelines/Simple_Selavy_Test/selavy-results.components.txt
• components catalogue - xml/VO format, as in src/pipelines/Simple_Selavy_Test/selavy-results.components.xml

Pass when

• The RA/Dec and flux values (flux_peak, flux_int) in the output files are the same as those in the expected result files (above).
• This can be examined by eye or include programatic comparison in test script.

Source Finding with NGAS interaction

Perform source-finding on an image retrieved from NGAS, and store the resulting components catalogue into NGAS. This can be used within the integration test to verify the result and (ultimately) display along with the image.

Integration Entry point

• TBC

Inputs

• Script is provided with an NGAS FileID of the image to be searched. See src/pipelines/datasets-for-tests.txt for example FileIDs.

Outputs

• Selavy should create a selavy-results.new.components.xml file, that should be uploaded to NGAS. This should be the FileID that appears in NGAS once it is uploaded.

Expected result

• As above, this catalogue should match the reference catalogue.

Pass when

• Apply same tests as first Selavy test suite, after downloading the created catalogue from NGAS.

Continuum Image ASKAP

Perform an imaging operation on askap data (see pipelines functional test notes).

Integration entry point src/integration/code/askap-test1-export-graph.py

Input

• Simulated visibilities for an ASKAP observation of a single-point-source field src/pipelines/askap_imaging/singleSource_Continuum.ms
• Generation script that creates parset and execution scripts src/pipelines/askap_imaging/generator_cimager.sh

Output

• Set of askapsoft images, named image.askap.test.fits, image.askap.test.restored.fits, psf.askap.test.fits, psf.image.askap.test.fits, residual.askap.test.fits, weights.askap.test.fits

Expected result

• image.askap.test.restored is the same as src/pipelines/askap_imaging/image.askap.reference.restored.fits

Pass when

• Difference between image.askap.test.restored and image.askap.reference.restored.fits is zero. There is a script available src/pipelines/askap_imaging/imageComparison.py that you run via "imageComparison.py --im1=image1.fits --im2=image2.fits" and it returns "PASS" if images are identical and "FAIL" otherwise.

MWA preparation (conversion/flagging) test

Integration entry point

• Integration script TBD
• Graph is run via ...

Input

• UVfits files at ...

Output

• Produces measurement sets as ...

Expected result

Pass when

Continuum Image MWA

Perform an imaging operation on MWA data.

Integration entry point TODO

Input

• TODO

Output

• TODO

Expected result

• TODO

Pass when

• TODO

Spectral Line MWA

Perform an imaging operation on MWA data.

Integration entry point TODO

Input

• TODO

Output

• TODO

Expected result

• TODO

Pass when

• TODO