Work Package 3 Data Management
Deliverable - (a) An NGAS cluster is running inside SHAO cluster that manages all data - including archive, ingest cache, processing cache, and data subscriptions. (b) An IVOA compliant interface that allows users to access visibilities and images (cubes) using IVOA tools and Python API. e.g. MWA ASVO.
- NGAS: http://159.226.233.198:7777/STATUS
- CASDA VO Tools TAP Service: http://202.127.29.97:8888/casda_vo_tools/tap
NGAS is used to store files and track the status of these files. The NGAS interface is only available on the SHAO network including from the cluster.
The CASDA VO Tools package is used to provide a TAP service which allows access to the metadata for files in the archive. Each file is represented by a single row in the ivoa.obscore table. The access_url column provides a link to download the file from NGAS.
See the Interfaces Page for details on the interfaces between the data archive, the data transfer from Pawsey and the pipeline.
To interactively access data we recommend either Aladin or TopCat
In TopCat please follow these steps:
- Click on the tables icon
- Enter the TAP address above into the TAP URL field.
- Click the "Use Service" button
- In the top panel, select the ivoa.obscore table
- In the bottom panel, click Examples > Full Table
- CLick Run Query
- The main window will appear woioth a new tbale listed. Douoble click this table to see the file list.
- To download the data on the SHAO network, copy the access_url value for a row and download from that web address.
For automated access use the retrieve_data_product.py sample script to find and download a file.
usage: retrieve_data_product.py [-h] file_id destination_directory
Check for a deposited file_id in NGAS and download it if present
positional arguments:
file_id The id of the data product to be checked.
destination_directory
The directory where the resulting files will be stored
optional arguments:
-h, --help show this help message and exit
