simple_demo.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import random\n",
    "import dill"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 1: Save each multivariate time series (MTS) to its own numpy binary file (.npy):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.79067304, -0.08754497, -0.43380599, -1.13602887, -0.34579843],\n",
       "       [ 0.53905942, -0.37755445,  1.15316678, -0.65020959,  0.10962188],\n",
       "       [ 1.10066567,  0.01289209,  0.34136527, -0.26922331,  0.52186689]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate a multivariate time-series dataset for this example\n",
    "random.seed(42)\n",
    "M = 3 # 3 independent processes\n",
    "T = 100 # 100 samples per process\n",
    "dataset = np.random.randn(M,T) # generate our multivariate time-series\n",
    "dataset[:,:5] # Print the first five time points for each process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll generate three example datasets that are each stored as dictionary entries, then iterate over the dictioanry to save one .npy file per MTS dataset. Note that by default, each process is $z$-scored along the time domain, so no need to normalize the data beforehand -- although you can disable this functionality by setting `normalise=False` in line 108 of `distribute_jobs.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate dictionary of 3 MTS datasets as an example\n",
    "MTS_datasets = {\"Dataset_\"+str(i) : np.random.randn(M,T) for i in range(3)}\n",
    "\n",
    "# Save the datasets to files\n",
    "for i, dataset in enumerate(MTS_datasets):\n",
    "    np.save('example_data/multivariate_time_series_{}.npy'.format(i), dataset)\n",
    "\n",
    "# Define the YAML file\n",
    "yaml_file = \"example_data/sample.yaml\"\n",
    "\n",
    "# Use ps dimension order to indicate that processes are the rows while timepoints are the columns\n",
    "dim_order = \"ps\"\n",
    "\n",
    "# Iterate over the keys and values of the dictionary\n",
    "for key, value in MTS_datasets.items():\n",
    "    # Define template string and fill in variables\n",
    "    yaml_string = \"{{file: example_data/{key}.npy, name: {key}, dim_order: {dim_order}, labels: [{key}]}}\\n\"\n",
    "    yaml_string_formatted = f\"{yaml_string.format(key=key, dim_order=dim_order)}\"\n",
    "\n",
    "    # Append line to file\n",
    "    with open(yaml_file, \"a\") as f:\n",
    "        f.write(yaml_string_formatted)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that here we set the MTS name to e.g. \"Dataset_0\" as well as the labels, but you can use the `labels` argument to include any metadata about a given MTS that you wish.\n",
    "\n",
    "Now that we've saved our MTS datasets to .npy files and generated the configuration file, we're ready to submit PBS jobs through `pyspi-distribute`. Use the following as a guide:\n",
    "\n",
    "```\n",
    "cmd=\"python3 distribute_jobs.py --data_dir example_data/ --calc_file_name calc.pkl --compute_file pyspi_compute.py \\\n",
    "--template_pbs_file template.pbs --sample_yaml example_data/sample.yaml --pbs_notify a --email your_email_here \\\n",
    "--conda_env your_conda_env_here --queue your_PBS_queue_here --walltime_hrs 3 --cpu 2 --mem 20 --table_only\"\n",
    "\n",
    "echo $cmd\n",
    "$cmd\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in the above command line code, you can customize the name of the pickle file (here we use `calc.pkl` as a standard) and you should input your email, conda environment, and PBS queue name as appropriate. The walltime hours, # CPUs, and memory requests are all examples---you should also do a trial run or two with a minimal example dataset to get a sense of the time/memory requirements for your dataset before submitting all of the jobs. We also include the optional flag `--table_only` such that only the SPI results table is saved as opposed to the entire `Calculator` object, but you can omit this if you wish to save the whole object.\n",
    "\n",
    "Once an individual PBS job is completed, you will find a new folder in your data directory (`example_data/` here) with the corresponding sample name (e.g., `Dataset_0`) that contains job output information as well as the saved `pyspi` computation result in `calc.pkl`. Since we set the `--table_only` flag, we can read in `calc.pkl` to get the SPI results table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>process</th>\n",
       "      <th>proc-0</th>\n",
       "      <th>proc-1</th>\n",
       "      <th>proc-2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>proc-0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.098141</td>\n",
       "      <td>0.134365</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proc-1</th>\n",
       "      <td>-0.098141</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.087860</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proc-2</th>\n",
       "      <td>0.134365</td>\n",
       "      <td>0.087860</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "process    proc-0    proc-1    proc-2\n",
       "proc-0        NaN -0.098141  0.134365\n",
       "proc-1  -0.098141       NaN  0.087860\n",
       "proc-2   0.134365  0.087860       NaN"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('example_data/Dataset_0/calc.pkl', 'rb') as f:\n",
    "    Dataset_0_res = dill.load(f)\n",
    "\n",
    "# Print the results for the empirical covariance, which here is equivalent to the Pearson correlation\n",
    "Dataset_0_res['cov_EmpiricalCovariance']"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pyspi",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}