CLML-Datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CLML Datasets Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> (C) 2015 Mike Maul -- CC-BY-SA 3.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This document is part series of tutorials illustrating the use of CLML. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets, what and why"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CLML datasets are two dimensional tabular data structures. In CLML datasets are used for (not to sound recursive) storing datasets. Datasets may contain numerical and categorical data. Datasets also contain column metadata (`dimensions`) and also provide facilities for extracting columns, dataset cleaning and splitting. Datasets in CLML are similar to dataframes in R or Pandas.DataFrames in Python.\n",
    "\n",
    "Lets get started by loading the system necessary for this tutorial and creating a namespace to work in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To load \"clml.utility\":\n",
      "  Load 1 ASDF system:\n",
      "    clml.utility\n",
      "\n",
      "; Loading \"clml.utility\"\n",
      "....\n",
      "To load \"clml.hjs\":\n",
      "  Load 1 ASDF system:\n",
      "    clml.hjs\n",
      "\n",
      "; Loading \"clml.hjs\"\n",
      "\n",
      "To load \"iolib\":\n",
      "  Load 1 ASDF system:\n",
      "    iolib\n",
      "\n",
      "; Loading \"iolib\"\n",
      ".....\n",
      "To load \"clml.extras.eazy-gnuplot\":\n",
      "  Load 1 ASDF system:\n",
      "    clml.extras.eazy-gnuplot\n",
      "\n",
      "; Loading \"clml.extras.eazy-gnuplot\"\n",
      "\n",
      "To load \"eazy-gnuplot\":\n",
      "  Load 1 ASDF system:\n",
      "    eazy-gnuplot\n",
      "\n",
      "; Loading \"eazy-gnuplot\"\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(:CLML.UTILITY :CLML.HJS :IOLIB :CLML.EXTRAS.EAZY-GNUPLOT :EAZY-GNUPLOT)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(ql:quickload '(:clml.utility ; Need clml.utility.data to get data from the net\n",
    "                :clml.hjs ; Need clml.hjs.read-data for dataset\n",
    "                :iolib\n",
    "                :clml.extras.eazy-gnuplot\n",
    "                :eazy-gnuplot\n",
    "            ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#<PACKAGE \"DATASETS-TUTORIAL\">"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(defpackage #:datasets-tutorial\n",
    "  (:use #:cl\n",
    "        #:cl-jupyter-user ; Not needed unless using iPython notebook\n",
    "        #:clml.hjs.read-data\n",
    "        #:clml.hjs.meta ; util function\n",
    "        #:clml.extras.eazy-gnuplot))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#<PACKAGE \"DATASETS-TUTORIAL\">"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(in-package :datasets-tutorial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets load some data that we will use as we learn about datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DATASET"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(defparameter dataset (read-data-from-file \n",
    "                       (clml.utility.data:fetch \"https://mmaul.github.io/clml.data/sample/cars.csv\") \n",
    "                       :type :csv :csv-type-spec '(integer integer)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Data and Datasets\n",
    "\n",
    "CLML has a number of different specializations of datasets such as\n",
    "  - `unspecialized-dataset` untyped and unspecialized data\n",
    "  - `numeric-dataset` dataset containing numeric (`double-float`) data\n",
    "  - `category-dataset` dataset for categorical (`string`) data\n",
    "  - `numeric-and-category-dataset` dataset containing a mixture of numeric and categorical data\n",
    "  - `numeric-matrix-dataset` dataset where numeric values are stored as a matrix\n",
    "  - `numeric-matrix-and-category-dataset` dataset where numeric values are stored as a matrix as well as having categorical data\n",
    "\n",
    "###Data representation\n",
    "All datasets except the matrix datasets represent data as a vector of vectors. The inner vector contains the columns of each row. For datasets with categories, numeric and category data are stored in seperate vectors.\n",
    "\n",
    "We can see below how the data is represented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#(#(4 2) #(4 10) #(7 4) #(7 22) #(8 16) #(9 10) #(10 18) #(10 26) #(10 34)\n",
       "  #(11 17) #(11 28) #(12 14) #(12 20) #(12 24) #(12 28) #(13 26) #(13 34)\n",
       "  #(13 34) #(13 46) #(14 26) #(14 36) #(14 60) #(14 80) #(15 20) #(15 26)\n",
       "  #(15 54) #(16 32) #(16 40) #(17 32) #(17 40) #(17 50) #(18 42) #(18 56)\n",
       "  #(18 76) #(18 84) #(19 36) #(19 46) #(19 68) #(20 32) #(20 48) #(20 52)\n",
       "  #(20 56) #(20 64) #(22 66) #(23 54) #(24 70) #(24 92) #(24 93) #(24 120)\n",
       "  #(25 85))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(dataset-points dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It may not be convenient to display the whole dataset to take a look at is. We could have used `subseq` but there is a helper method called `head-points`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#(#(4 2) #(4 10) #(7 4) #(7 22) #(8 16))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(head-points dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###Dimensions\n",
    "All Datasets have the `dimensions` slot which contain the column metadata. The dimensions slot contains a list of `dimension` instances. Each dimension instance contains the following slots (accessor prefix is dimension):\n",
    "  - `name` column name\n",
    "  - `type` type of data in column (e.g. :category :numeric :unknown)\n",
    "  - `index` index on column vectors of column\n",
    "  - `metadata` - alist that CAN containing useful information, such as equality tests for category data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#(#<CLML.HJS.READ-DATA::DIMENSION NAME: speed, TYPE: UNKNOWN, INDEX: 0.>\n",
       "  #<CLML.HJS.READ-DATA::DIMENSION NAME: distance, TYPE: UNKNOWN, INDEX: 1.>)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(dataset-dimensions dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating datasets\n",
    "Datasets can be created directly or can be created by reading them from a file. Supported data formats or CSV and SEXP.\n",
    " Earlier we used the `read-data-from-file` function to read a dataset from a CSV file. The file in this case is a file that is obtained with the `fetch` from the `clml.utility` system, which downloads and caches a file in a location on a local files system or a URL. Datasets can also be created programatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: cat 1 | num 1\n",
       "TYPES:      CATEGORY | NUMERIC\n",
       "NUMBER OF DIMENSIONS: 2\n",
       "CATEGORY DATA POINTS: 2 POINTS\n",
       "NUMERIC DATA POINTS: 2 POINTS\n"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(make-numeric-and-category-dataset  \n",
    " '(\"cat 1\" \"num 1\")                           ; <-- Column names \n",
    " (vector (v2dvec #(1.0d0)) (v2dvec #(2.0d0))) ; <-- Numeric data \n",
    "          '(1)                                ; <-- Indexes of numeric column\n",
    "          #(#(\"a\") #(\"b\"))                    ; <-- Category Data\n",
    "          '(0)                                ; <-- Indexes of category data\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specializing datasets\n",
    "The dataset we loaded is currently unspecialized, we haven't told CLML much about it yet. We can use the `pick-and-specialize-data` method to fill in the details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#<NUMERIC-DATASET >\n",
       "DIMENSIONS: speed | distance\n",
       "TYPES:      NUMERIC | NUMERIC\n",
       "NUMBER OF DIMENSIONS: 2\n",
       "NUMERIC DATA POINTS: 50 POINTS\n"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pick-and-specialize-data dataset :data-types '(:numeric :numeric))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We can see `pick-and-specialize-data` returned a numeric dataset based on the supplied `:data-types` specification. `pick-and-specialize-data` has two parameters `:range` and :`except`. Both parameters deal with column selection `:range` specifies a range of columns (as a list) to use in our new dataset while `:except` specifies a list of columns to exclude from our new dataset. We had also mentioned the matrix datasets, `pick-and-specialize-data` can also change the representation from a vector of vectors to a matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "#<NUMERIC-MATRIX-DATASET >\n",
      "DIMENSIONS: speed | distance\n",
      "\n",
      "TYPES:      NUMERIC | NUMERIC\n",
      "\n",
      "NUMBER OF DIMENSIONS: 2\n",
      "\n",
      "NUMERIC-MATRIX DATA POINTS: 50 POINTS\n",
      " "
     ]
    },
    {
     "data": {
      "text/plain": [
       "#2A((4.0d0 2.0d0)\n",
       "    (4.0d0 10.0d0)\n",
       "    (7.0d0 4.0d0)\n",
       "    (7.0d0 22.0d0)\n",
       "    (8.0d0 16.0d0)\n",
       "    (9.0d0 10.0d0)\n",
       "    (10.0d0 18.0d0)\n",
       "    (10.0d0 26.0d0)\n",
       "    (10.0d0 34.0d0)\n",
       "    (11.0d0 17.0d0)\n",
       "    (11.0d0 28.0d0)\n",
       "    (12.0d0 14.0d0)\n",
       "    (12.0d0 20.0d0)\n",
       "    (12.0d0 24.0d0)\n",
       "    (12.0d0 28.0d0)\n",
       "    (13.0d0 26.0d0)\n",
       "    (13.0d0 34.0d0)\n",
       "    (13.0d0 34.0d0)\n",
       "    (13.0d0 46.0d0)\n",
       "    (14.0d0 26.0d0)\n",
       "    (14.0d0 36.0d0)\n",
       "    (14.0d0 60.0d0)\n",
       "    (14.0d0 80.0d0)\n",
       "    (15.0d0 20.0d0)\n",
       "    (15.0d0 26.0d0)\n",
       "    (15.0d0 54.0d0)\n",
       "    (16.0d0 32.0d0)\n",
       "    (16.0d0 40.0d0)\n",
       "    (17.0d0 32.0d0)\n",
       "    (17.0d0 40.0d0)\n",
       "    (17.0d0 50.0d0)\n",
       "    (18.0d0 42.0d0)\n",
       "    (18.0d0 56.0d0)\n",
       "    (18.0d0 76.0d0)\n",
       "    (18.0d0 84.0d0)\n",
       "    (19.0d0 36.0d0)\n",
       "    (19.0d0 46.0d0)\n",
       "    (19.0d0 68.0d0)\n",
       "    (20.0d0 32.0d0)\n",
       "    (20.0d0 48.0d0)\n",
       "    (20.0d0 52.0d0)\n",
       "    (20.0d0 56.0d0)\n",
       "    (20.0d0 64.0d0)\n",
       "    (22.0d0 66.0d0)\n",
       "    (23.0d0 54.0d0)\n",
       "    (24.0d0 70.0d0)\n",
       "    (24.0d0 92.0d0)\n",
       "    (24.0d0 93.0d0)\n",
       "    (24.0d0 120.0d0)\n",
       "    (25.0d0 85.0d0))"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(let ((ds (pick-and-specialize-data dataset :data-types '(:numeric :numeric) \n",
    "           :store-numeric-data-as-matrix t)))\n",
    "           (print ds)\n",
    "           (dataset-numeric-points ds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should also show an example of a dataset with categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: year season | UKgas\n",
       "TYPES:      CATEGORY | NUMERIC\n",
       "NUMBER OF DIMENSIONS: 2\n",
       "CATEGORY DATA POINTS: 108 POINTS\n",
       "NUMERIC DATA POINTS: 108 POINTS\n"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pick-and-specialize-data (read-data-from-file \n",
    "                           (clml.utility.data:fetch \"https://mmaul.github.io/clml.data/sample/UKgas.sexp\"))\n",
    "     :data-types '(:category :numeric))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Datasets can be created and combined. Generally the dataset creation methods take the form of `make-<dataset type>` and either take vectors containing data or other datasets and create a new dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Missing Values\n",
    "CLML datasets support missing values. Missing values are represented as follows in the dataset-points:\n",
    "  - category CLML.HJS.MISSING-VALUE:+C-NAN+\n",
    "  - numeric  CLML.HJS.MISSING-VALUE:+NAN+\n",
    "  - unspecialized :NA\n",
    "  \n",
    "There are also the following predicates available to detect missing values:\n",
    "  - `CLML.HJS.MISSING-VALUE:C-NAN-P`\n",
    "  -  `CLML.HJS.MISSING-VALUE:NAN-P`\n",
    "  \n",
    "The `read-data-from-file` also supports the mapping representations of missing values in data files to datasets.\n",
    "The `missing-values-list` keyword argument specifies the character sequences that will be recognized as missing values.\n",
    "\n",
    "To illustrate missing values support lets read in a CSV file containing the follow:\n",
    "\n",
    "    a,   b,   c\n",
    "    1.0, 2.0, x\n",
    "    NA,  3.0, NA\n",
    "    \n",
    "Here missing values are represented in the CSV file by *NA*. For the `read-data` function to recognize the missing values we must set the `:missing-values-list` parameter as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"#<UNSPECIALIZED-DATASET >\n",
       "DIMENSIONS: a | b | c\n",
       "TYPES:      UNKNOWN | UNKNOWN | UNKNOWN\n",
       "NUMBER OF DIMENSIONS: 3\n",
       "DATA POINTS: 2 POINTS\n",
       "\n",
       "#(#(1.0d0 2.0d0 x) #(NA 3.0d0 NA))\n",
       "\""
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(let ((ds (read-data-from-file \n",
    "            (clml.utility.data:fetch \"https://mmaul.github.io/clml.data/sample/simple1.csv\") \n",
    "            :type :csv \n",
    "            :csv-type-spec '(double-float double-float string) \n",
    "            :missing-values-list '(\"NA\")\n",
    "          )))\n",
    "            (format nil \"~A~%~A~%\" ds (dataset-points ds)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also see how missing values are represented in a specialized dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"#<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: a | b | c\n",
       "TYPES:      NUMERIC | NUMERIC | CATEGORY\n",
       "NUMBER OF DIMENSIONS: 3\n",
       "CATEGORY DATA POINTS: 2 POINTS\n",
       "NUMERIC DATA POINTS: 2 POINTS\n",
       "\n",
       "#(#(1.0d0 2.0d0) #(#<DOUBLE-FLOAT quiet NaN> 3.0d0))\n",
       "#(#(x) #(0))\n",
       "\""
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(let ((ds (pick-and-specialize-data\n",
    "             (read-data-from-file \n",
    "               (clml.utility.data:fetch \"https://mmaul.github.io/clml.data/sample/simple1.csv\") \n",
    "               :type :csv \n",
    "               :csv-type-spec '(double-float double-float string) \n",
    "               :missing-values-list '(\"NA\")\n",
    "              )\n",
    "              :data-types '(:numeric :numeric :category)\n",
    "           )))\n",
    "     (format nil \"~A~%~A~%~A~%\" ds  (dataset-numeric-points ds) (dataset-category-points ds))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dataset manipulation\n",
    "The following operations can be preformed on datasets:\n",
    "  - copying\n",
    "  - splitting and sampling\n",
    "  - cleaning\n",
    "  - shuffling\n",
    "  - deduplication\n",
    "  - storing\n",
    "  \n",
    "We will use the UK Gas dataset to illustrate these operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "UKGAS"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(defparameter ukgas (pick-and-specialize-data (read-data-from-file \n",
    "                           (clml.utility.data:fetch \"https://mmaul.github.io/clml.data/sample/UKgas.sexp\"))\n",
    "                           :data-types '(:category :numeric)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Copying\n",
    "The simplest operation is copying. `copy-dataset` makes a deep copy of the contents of a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: year season | UKgas\n",
       "TYPES:      CATEGORY | NUMERIC\n",
       "NUMBER OF DIMENSIONS: 2\n",
       "CATEGORY DATA POINTS: 108 POINTS\n",
       "NUMERIC DATA POINTS: 108 POINTS\n"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(copy-dataset ukgas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Splitting and Sampling\n",
    "Datasets can be subdivided by two similar methods `make-bootstrap-sample-dataset` and `divide-dataset`\n",
    "\n",
    "The `divide dataset` returns a dataset split into two parts based upon the `:divide-ratio` like `pick-and-specialize-data` `divide-dataset` can limit the section values accessed with the `:range` and `:except` parameters. It can also pull values in a pseudo-random manner values in to their new datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: year season | UKgas\n",
       "\n",
       "TYPES:      CATEGORY | NUMERIC\n",
       "\n",
       "NUMBER OF DIMENSIONS: 2\n",
       "\n",
       "CATEGORY DATA POINTS: 81 POINTS\n",
       "\n",
       "NUMERIC DATA POINTS: 81 POINTS\n",
       "\n",
       " #<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: year season | UKgas\n",
       "\n",
       "TYPES:      CATEGORY | NUMERIC\n",
       "\n",
       "NUMBER OF DIMENSIONS: 2\n",
       "\n",
       "CATEGORY DATA POINTS: 27 POINTS\n",
       "\n",
       "NUMERIC DATA POINTS: 27 POINTS\n",
       ")"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " (multiple-value-list (divide-dataset ukgas :divide-ratio '(3 1) :random t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`make-bootstrap-sample-datasets` on the other hand shuffles a dataset into a number of specified datasets of equal length to the original dataset. The `:number-of-datasets`parameter defaults to 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: year season | UKgas\n",
       "\n",
       "TYPES:      CATEGORY | NUMERIC\n",
       "\n",
       "NUMBER OF DIMENSIONS: 2\n",
       "\n",
       "CATEGORY DATA POINTS: 108 POINTS\n",
       "\n",
       "NUMERIC DATA POINTS: 108 POINTS\n",
       "\n",
       " #<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: year season | UKgas\n",
       "\n",
       "TYPES:      CATEGORY | NUMERIC\n",
       "\n",
       "NUMBER OF DIMENSIONS: 2\n",
       "\n",
       "CATEGORY DATA POINTS: 108 POINTS\n",
       "\n",
       "NUMERIC DATA POINTS: 108 POINTS\n",
       "\n",
       " #<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: year season | UKgas\n",
       "\n",
       "TYPES:      CATEGORY | NUMERIC\n",
       "\n",
       "NUMBER OF DIMENSIONS: 2\n",
       "\n",
       "CATEGORY DATA POINTS: 108 POINTS\n",
       "\n",
       "NUMERIC DATA POINTS: 108 POINTS\n",
       ")"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(make-bootstrap-sample-datasets ukgas :number-of-datasets 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Cleaning\n",
    "\n",
    "One nice features of CLML is the dataset cleaning capabilities. The `dataset-cleaning` method provides the following:\n",
    "  - outlier detection for numeric points\n",
    "    - standard deviation \n",
    "    - mean deviation \n",
    "    - user provided function\n",
    "    - smirnov-grubbs \n",
    "   - outlier detection for categorical points\n",
    "    - frequency based\n",
    "    - User provided function\n",
    "  - Outlier and missing value interpolation using:\n",
    "    - zero \n",
    "    - min \n",
    "    - max \n",
    "    - mean \n",
    "    - median \n",
    "    - mode \n",
    "    - spline\n",
    "    \n",
    "To illustrate we will preform dataset cleaning where outliers will be points that exceed 1 standard deviation and will be replaced by zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: year season | UKgas\n",
       "TYPES:      CATEGORY | NUMERIC\n",
       "NUMBER OF DIMENSIONS: 2\n",
       "CATEGORY DATA POINTS: 108 POINTS\n",
       "NUMERIC DATA POINTS: 108 POINTS\n"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " (dataset-cleaning ukgas :outlier-types-alist '((\"UKgas\" . :std-dev)) \n",
    "                         :outlier-values-alist '((:std-dev . 1)) \n",
    "                         :interp-types-alist '((\"UKgas\" . :zero)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####Adding dimensions and Concatenating Datasets\n",
    "\n",
    "###### Adding dimensions\n",
    "In some cases you may want to add a computed column or add a column to a dataset to hold the product of a computation on a dataset. The `add-dim` method can accomplish this easily. It can add an existing column of points with the :points parameter, it can also create a column with points filled with a initial value with the `:initial-value` parameter. The two mandatory parameters are the dataset to add the dimension to, the name of the new dimension and the type. If the dataset is either a category or numeric only dataset `add-dim` will create a numeric-and-category-dataset if a column of a different type is added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: year season | UKgas | mpg\n",
       "TYPES:      CATEGORY | NUMERIC | NUMERIC\n",
       "NUMBER OF DIMENSIONS: 3\n",
       "CATEGORY DATA POINTS: 108 POINTS\n",
       "NUMERIC DATA POINTS: 108 POINTS\n"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(add-dim ukgas \"mpg\" :numeric :initial-value 0.0d0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Concatenating datasets\n",
    "Two datasets with equal numbers of rows can be concatenated or glued together vertically. `concatenate-datasets` takes two datasets as parameters and return a dataset with the points of the first dataset stacked on top of the points of the second dataset. The dimension name names of the first one dataset are retained in the new dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#<NUMERIC-AND-CATEGORY-DATASET >\n",
       "DIMENSIONS: year season | UKgas\n",
       "TYPES:      CATEGORY | NUMERIC\n",
       "NUMBER OF DIMENSIONS: 2\n",
       "CATEGORY DATA POINTS: 216 POINTS\n",
       "NUMERIC DATA POINTS: 216 POINTS\n"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(concatenate-datasets ukgas ukgas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Deduplicating datasets\n",
    "Datasets can be deduplicated in place with the `dedup-dataset!` method. This functionality is currently only implemented for numeric, category and unspecialized datasets.\n",
    "\n",
    "##### Storing datasets\n",
    "Datasets can be saved to a file in csv format with the `write-dataset` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(CLML.UTILITY.CSV::OK)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " (write-dataset ukgas \"gasgas.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Working with dataset points\n",
    "\n",
    "Columns and values can be accessed and extracted from datasets using the `!!` macros. This macro returns the column name or list of column names as a vectors of vectors if multiple column names are specified or as a single vector if a single column name is specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#(160.1d0 129.7d0 84.8d0 120.1d0 160.1d0 124.9d0 84.8d0 116.9d0 169.7d0 140.9d0\n",
       "  89.7d0 123.3d0 187.3d0 144.1d0 92.9d0 120.1d0 176.1d0 147.3d0 89.7d0 123.3d0\n",
       "  185.7d0 155.3d0 99.3d0 131.3d0 200.1d0 161.7d0 102.5d0 136.1d0 204.9d0\n",
       "  176.1d0 112.1d0 140.9d0 227.3d0 195.3d0 115.3d0 142.5d0 244.9d0 214.5d0\n",
       "  118.5d0 153.7d0 244.9d0 216.1d0 188.9d0 142.5d0 301.0d0 196.9d0 136.1d0\n",
       "  267.3d0 317.0d0 230.5d0 152.1d0 336.2d0 371.4d0 240.1d0 158.5d0 355.4d0\n",
       "  449.9d0 286.6d0 179.3d0 403.4d0 491.5d0 321.8d0 177.7d0 409.8d0 593.9d0\n",
       "  329.8d0 176.1d0 483.5d0 584.3d0 395.4d0 187.3d0 485.1d0 669.2d0 421.0d0\n",
       "  216.1d0 509.1d0 827.7d0 467.5d0 209.7d0 542.7d0 840.5d0 414.6d0 217.7d0\n",
       "  670.8d0 848.5d0 437.0d0 209.7d0 701.2d0 925.3d0 443.4d0 214.5d0 683.6d0\n",
       "  917.3d0 515.5d0 224.1d0 694.8d0 989.4d0 477.1d0 233.7d0 730.0d0 1087.0d0\n",
       "  534.7d0 281.8d0 787.6d0 1163.9d0 613.1d0 347.4d0 782.8d0)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(!! ukgas \"UKgas\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset points can also be accessed with the slot accessor. Since category and numeric data are stored separately in heterogeneous datasets separate accessors are used to access the points.\n",
    "\n",
    "The list below shows which methods are applicable to the dataset type.\n",
    "- `dataset-points`: `unspecialized-dataset`\n",
    "- `dataset-numeric-points`: `numeric-dataset` `numeric-and-category-dataset` `numeric-matrix-dataset numeric-matrix-and-category-dataset`\n",
    "- `dataset-category-points`: `category-dataset` `numeric-and-category-dataset` `numeric-matrix-dataset` `numeric-matrix-and-category-dataset`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#(#(160.1d0) #(129.7d0) #(84.8d0) #(120.1d0) #(160.1d0) #(124.9d0) #(84.8d0)\n",
       "  #(116.9d0) #(169.7d0) #(140.9d0) #(89.7d0) #(123.3d0) #(187.3d0) #(144.1d0)\n",
       "  #(92.9d0) #(120.1d0) #(176.1d0) #(147.3d0) #(89.7d0) #(123.3d0) #(185.7d0)\n",
       "  #(155.3d0) #(99.3d0) #(131.3d0) #(200.1d0) #(161.7d0) #(102.5d0) #(136.1d0)\n",
       "  #(204.9d0) #(176.1d0) #(112.1d0) #(140.9d0) #(227.3d0) #(195.3d0) #(115.3d0)\n",
       "  #(142.5d0) #(244.9d0) #(214.5d0) #(118.5d0) #(153.7d0) #(244.9d0) #(216.1d0)\n",
       "  #(188.9d0) #(142.5d0) #(301.0d0) #(196.9d0) #(136.1d0) #(267.3d0) #(317.0d0)\n",
       "  #(230.5d0) #(152.1d0) #(336.2d0) #(371.4d0) #(240.1d0) #(158.5d0) #(355.4d0)\n",
       "  #(449.9d0) #(286.6d0) #(179.3d0) #(403.4d0) #(491.5d0) #(321.8d0) #(177.7d0)\n",
       "  #(409.8d0) #(593.9d0) #(329.8d0) #(176.1d0) #(483.5d0) #(584.3d0) #(395.4d0)\n",
       "  #(187.3d0) #(485.1d0) #(669.2d0) #(421.0d0) #(216.1d0) #(509.1d0) #(827.7d0)\n",
       "  #(467.5d0) #(209.7d0) #(542.7d0) #(840.5d0) #(414.6d0) #(217.7d0) #(670.8d0)\n",
       "  #(848.5d0) #(437.0d0) #(209.7d0) #(701.2d0) #(925.3d0) #(443.4d0) #(214.5d0)\n",
       "  #(683.6d0) #(917.3d0) #(515.5d0) #(224.1d0) #(694.8d0) #(989.4d0) #(477.1d0)\n",
       "  #(233.7d0) #(730.0d0) #(1087.0d0) #(534.7d0) #(281.8d0) #(787.6d0)\n",
       "  #(1163.9d0) #(613.1d0) #(347.4d0) #(782.8d0))"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(dataset-numeric-points ukgas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A little something extra, `R-datasets`\n",
    "\n",
    "One thing that I've always found handy in R is a standard, curated, extensive and documented series of datasets. Wouldn't it be nice to have access to these directly as datasets in CLML. The `R-datasets` system in `clml.extras` provides this capability. A particularly good use case for these datasets is to be able to follow along with examples and tutorials for R in CLML.\n",
    "\n",
    "#### Note\n",
    "The `clml.extras` systems are not currently part of quicklisp so if you are following along with this tutorial and are expecting just to `(quickload :clml.extras.Rdatasets)` you can't till you clone the clml.extras repository [http://github.com/mmaul/clml.extras.git](http://github.com/mmaul/clml.extras.git) into your `quicklisp/local-projects` directory\n",
    "\n",
    "### Using Rdatasets\n",
    "The Rdatasets package makes datasets included with the R language distribution available as clml datasets.\n",
    "R datasets are obtained csv files on Vincent Centarel's github repository.\n",
    "More information on these datasets can be found at <http://vincentarelbundock.github.com/Rdatasets>\n",
    "\n",
    "Because type information is not included it may be necessary to provide a `csv-type-spec`\n",
    "for the columns in the csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To load \"clml.r-datasets\":\n",
      "  Load 1 ASDF system:\n",
      "    clml.r-datasets\n",
      "\n",
      "; Loading \"clml.r-datasets\"\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(:CLML.R-DATASETS)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(ql:quickload :clml.r-datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "T"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(use-package :clml.r-datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DD"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(defparameter dd (get-r-dataset-directory))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Package                   Item                      Title                     \n",
       "------------------------- ------------------------- ------------------------- \n",
       "datasets                  AirPassengers             Monthly Airline Passenger Numbers 1949-1960 \n",
       "\n",
       "datasets                  BJsales                   Sales Data with Leading Indicator \n",
       "\n",
       "datasets                  BOD                       Biochemical Oxygen Demand \n",
       "\n",
       "datasets                  Formaldehyde              Determination of Formaldehyde\""
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(subseq (inventory dd :stream nil) 0 505)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\\"\n",
       "R: Biochemical Oxygen Demand\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "BODR Documentation\n",
       "\n",
       " Biochemical Oxygen Demand \n",
       "\n",
       "Description\n",
       "\n",
       "The BOD data frame has 6 rows and 2 columns giving the\n",
       "biochemical oxygen demand versus time in an eval\""
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(subseq (dataset-documentation  dd  \"datasets\" \"BOD\" :stream nil) 0 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BOD"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(defparameter bod (get-dataset dd \"datasets\" \"BOD\" :csv-type-spec '(double-float double-float double-float)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#<UNSPECIALIZED-DATASET >\n",
       "DIMENSIONS:  | Time | demand\n",
       "TYPES:      UNKNOWN | UNKNOWN | UNKNOWN\n",
       "NUMBER OF DIMENSIONS: 3\n",
       "DATA POINTS: 6 POINTS\n"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#<NUMERIC-DATASET >\n",
       "DIMENSIONS:  | Time | demand\n",
       "TYPES:      NUMERIC | NUMERIC | NUMERIC\n",
       "NUMBER OF DIMENSIONS: 3\n",
       "NUMERIC DATA POINTS: 6 POINTS\n"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " (pick-and-specialize-data bod :data-types '(:numeric :numeric :numeric))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Conclusion\n",
    "\n",
    "The iPython notebook and source for this tutorial can be found in the [clml.tutorials https://github.com/mmaul/clml.tutorials.git](https://github.com/mmaul/clml.tutorials.git) github repository.\n",
    "\n",
    "###Stay tuned to [clml.tutorials](https://mmaul.github.io/clml.tutorials/) blog or [RSS feed](https://mmaul.github.io/clml.tutorials/feed.xml) for more CLML tutorials..\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SBCL Lisp",
   "language": "lisp",
   "name": "lisp"
  },
  "language_info": {
   "name": "common-lisp",
   "version": "1.2.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}