Work Package 1 Data Transfer

Deliverable - MWA and ASKAP datasets are being transferred from Pawsey to SHAO in a streaming mode with user-friendly management consoles at a sustained rate of >400 MB/s.

General Approach

We split the deliverable into parts:

• Network testing - what throughput do we get from ASKAP to SHAO and MWA to SHAO via pure network methods (iperf).
• Network remediation - what changes can be made during the workshop to improve iperf results.
• NGAS transfer of ASKAP files via subscription to SHAO - this had already been established prior to the workshop
• NGAS transfer of MWA files via subscription to SHAO - create a new subscription filter plugin and create a new subscription on fe2 at Pawsey which will transfer specific observations (50 exactly - provided by Slava) to SHAO.
• NGAS trigger - when a new file is archived to SHAO, add metadata to the obscore database and trigger the appropriate pipeline, passing the relevant parameters.

Network testing

MWA

• iperf 1.95 Gbps (1 stream) up to 2.9 Gbps (12 streams)
• obsdownload (from SHAO) 2.08 Gbps (20 streams)

ASKAP

• iperf 1.04 Gbps, then after tuning 2.6 Gps (12 streams)

Network Remediation

SHAO

• Kernel Parameters / TCP windows were optimised for long haul transfers
• Multiple firewall rules were required to be changed to allow for testing with iperf

Pawsey / MWA

• Set optimised kernel parameters for TCP window size, etc. An update at Pawsey had inadvertently reset these values to default
• Retest

Pawsey / ASKAP

• Set Optimised kernel parameters for TCP window size, etc.
• Retest

NGAS subscription for MWA to SHAO:

• New plugin created: ngamsMWA_SHAO_FilterPlugin.py
• Subscription created on fe2.pawsey.org.au: curl "http://fe2.pawsey.org.au:7777/SUBSCRIBE?priority=1&url=http://159.226.233.198:7777/ARCHIVE&subscr_id=fe02_to_shao&filter_plug_in=ngamsMWA_SHAO_FilterPlugin&start_date=2013-01-01T12:00:00.000&concurrent_threads=12" --user x:x

NGAS Trigger:

• Trigger logic:

NGAS triggers archive event handler passing file_id and file_version
Determine if file is MWA or ASKAP

if mwa:
   Obtain obs_id from file_id (use first 10 characters) 
   Check local data structure for metadata info about obs_id
   if no info:
      get file list metadata from http://mwa-metadata01.pawsey.org.au/metadata/obs?obs_id=[obs_id]
      store file list metadata in local data structure
   mark file as being received in local data structure
   if this was the last file:
      get mwa obscore metadata from VO TAP service http://vo.mwatelescope.org:8000/mwa_asvo/tap
      insert obscore record into shaoska database
   trigger pipeline script (kevin's script) passing in obs_id and all filenames

elseif askap:
   Obtain obs_id from file_id (use "observation-xxx-...") 
   Check local data structure for metadata info about obs_id
   if no info:
      get obscore metadata from https://casda.csiro.au/casda_vo_tools/tap  (this also provides file_list info)
      store file list metadata in local data structure
      store obscore metadata in local data structure
   mark file as being received in local data structure
   if this was the last file:
      insert obscore record into shaoska database

Not Achieved in Workshop:

• NGAS trigger is not complete, but there are no blockers to finishing this. Just time and effort.
• Monitoring of transfer and getting statistics

Future work:

• Complete NGAS trigger
• Develop more useful NGAS MWA subscription trigger- currently only sending fixed list from Slava
• Look into ways of building in realtime monitoring into NGAS
• Analyse/test different techniques to transfer files via NGAS - e.g. bbcp, UDP, etc
• Perform testing in a more controlled environment to minimise confounding factors

See Also: Data-Transfer-performance-results