avi-tutorial-jupyter-notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start using Gaia data in GAVIP\n",
    "\n",
    "### Background\n",
    "The Gaia satellite will create a catalog of more than 1 billion stars in our local galaxy, eventually forming an archive of more than 1 Petabyte in size.\n",
    "The volume of the Gaia dataproducts will be too large for most users to download.\n",
    "GAVIP allows users to create and submit reusable tools which can be run close to the data (allowing the user to run analysis without downloading the data to their machine). \n",
    "These tools are known as Added Value Interfaces or **AVIs**.\n",
    "\n",
    "### Objective\n",
    "In this tutorial we will perform some analysis of data retrieved from the Gaia archive (Gaia Data Release 1).\n",
    "We will first perform this analysis in a Jupyter notebook, then create a simple AVI which others can use in GAVIP.\n",
    "\n",
    "In this notebook, we will:\n",
    "* Use TAP+ to retrieve data from GACS\n",
    "* Temporarily store it as a votable\n",
    "* Parse the votable as a Pandas dataframe\n",
    "* Create a pandas profile report from the dataframe\n",
    "\n",
    "We will later package this up for others to use in GAVIP.\n",
    "\n",
    "**Note:** This notebook was made using Python 2. It is compatible with Python 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from astropy.io.votable import parse_single_table    \n",
    "from astropy.table import Table\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "import pandas_profiling\n",
    "import pandas as pd\n",
    "\n",
    "import tempfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The asynchronous TAP+ module\n",
    "GAVIP AVIs are built using the AVI framework. \n",
    "The framework handles authentication and asynchronous job processing to name a few.\n",
    "The framework also includes \"connectors\" and \"services\". \n",
    "Connectors provide implementations of protocols useful to AVIs, such as TAP+. Services provide blocks of reusable codes using connectors - these will be shown later in the tutorial.\n",
    "\n",
    "The Jupyter notebooks provided by GAVIP include the AVI framework, so we will use the **TAP+ connector** rather than writing the requests manually. We do this by importing the `AsyncJob` class from the `connectors.tapquery` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Asynchronous TAP+ class\n",
    "from connectors.tapquery import AsyncJob  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a function here using the `AsyncJob` class which will accept an ADQL query and a target TAP+ server.\n",
    "\n",
    "This function performs the following:\n",
    "1. Submit the ADQL query to the target TAP server \n",
    "2. Wait for the job to complete\n",
    "3. Store the result in a temporary file as a VOtable\n",
    "4. Parse the VOtable using the votable library from astropy\n",
    "5. Return the table to a Pandas dataframe\n",
    "\n",
    "**Note:** The AsyncJob class can be initialized with a `username` and `password` parameter. If your Cosmos login credentials are provided, your jobs will be recorded under your account. In this demo, we are making the TAP+ request anonymously.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_gaia_data(query, target):\n",
    "    \n",
    "    \"\"\"\n",
    "    Query a TAP service (designated by its tap_endpoint)\n",
    "    with a given ADQL query\n",
    "    \n",
    "    Query is performed asynchronously\n",
    "    \n",
    "    Return an AstroPy Table object\n",
    "    \n",
    "    This object is converted to a PandasDataframe\n",
    "    \"\"\"\n",
    "    \n",
    "    async_check_interval = 1\n",
    "    gacs_tap_conn = AsyncJob(target, query, poll_interval=async_check_interval)\n",
    "\n",
    "    # Run the job (start + wait + raise_exception)\n",
    "    gacs_tap_conn.run()\n",
    "\n",
    "    # Store the response\n",
    "    result = gacs_tap_conn.open_result()\n",
    "\n",
    "    tmp_vot = tempfile.NamedTemporaryFile(delete = False)\n",
    "    with open(tmp_vot.name, 'w') as f:\n",
    "        f.write(result.content.decode(\"utf-8\"))\n",
    "\n",
    "    table = parse_single_table(tmp_vot.name).to_table()\n",
    "\n",
    "    # finally delete temp files\n",
    "    os.unlink(tmp_vot.name)\n",
    "    \n",
    "    #returns the pandas dataframe\n",
    "    return pd.DataFrame(np.ma.filled(table.as_array()), columns=table.colnames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we specify our TAP server target address, and the ADQL query (taken from the ADQL Help page on the Gaia Archive web page.\n",
    "\n",
    "Once both are specified, we use the `get_gaia_data()` function defined above to obtain a Pandas dataframe of the resulting VOTable from the TAP server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: W50: /tmp/tmpORnvUX:15:0: W50: Invalid unit string 'Angle[deg]' [astropy.io.votable.exceptions]\n",
      "WARNING: W50: /tmp/tmpORnvUX:18:0: W50: Invalid unit string 'Angle[deg]' [astropy.io.votable.exceptions]\n",
      "WARNING: W50: /tmp/tmpORnvUX:21:0: W50: Invalid unit string 'Flux[e-/s]' [astropy.io.votable.exceptions]\n",
      "WARNING: W50: /tmp/tmpORnvUX:24:0: W50: Invalid unit string 'Magnitude[mag]' [astropy.io.votable.exceptions]\n"
     ]
    }
   ],
   "source": [
    "target = \"http://gea.esac.esa.int/tap-server/tap\"\n",
    "\n",
    "# sample query from https://gea.esac.esa.int/archive/\n",
    "query = \"\"\"\n",
    "        SELECT source_id, ra, dec, phot_g_mean_flux, phot_g_mean_mag,\n",
    "        DISTANCE(POINT('ICRS',ra,dec), POINT('ICRS',266.41683,-29.00781)) \n",
    "        AS dist FROM gaiadr1.gaia_source WHERE 1=CONTAINS(POINT('ICRS',ra,dec),\n",
    "        CIRCLE('ICRS',266.41683,-29.00781, 0.08333333))\n",
    "        \"\"\"\n",
    "\n",
    "df = get_gaia_data(query, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we specify column names of interest as an array to retrieve from the dataframe. Then we pass the new dataframe to the `ProfileReport()` function provided by pandas_profiling. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gaiamagcols=['dec', 'dist', 'phot_g_mean_flux', 'phot_g_mean_mag', 'ra', 'source_id']\n",
    "gaiadf = df[gaiamagcols]\n",
    "pandas_profiling.ProfileReport(gaiadf)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}