1 - First steps with Python and Ocelot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First steps with Python and Ocelot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import ocelot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import a `spold` (ecospold2) file. Normally we would import a directory with many datasets, but we start slowly and look at just one file for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filename = \"heat-cogeneration-glo.spold\"\n",
    "datafile = ocelot.io.extract_ecospold2.generic_extractor(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "However, the ecospold2 format is a tricky beast. One implementation detail says that it is OK to have multiple datasets within a file. We first check to make sure that we have only one dataset, and then select it by index.\n",
    "\n",
    "Python is *zero-indexed*. That means if I have a list:\n",
    "\n",
    "    a = ['cow', 'chicken', 'pig']\n",
    "    \n",
    "The first element of that list has index `0`:\n",
    "\n",
    "    In[0]: a[0]\n",
    "    Out[0]: 'cow'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(datafile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dataset = datafile[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can do anything sensible with this dataset, we need to know what to expect. So please go look a bit through the [Ocelot data format](https://docs.ocelot.space/data_format.html) (Go ahead, I'll wait. I have nothing but time). This will probably raise more questions than it answers, but it is at least a start.\n",
    "\n",
    "The first thing to notice is that the data format is what python calls a `dictionary`. Other languages call this a hash table. A dictionary has keys and values (like a word and its definition), and looks like this:\n",
    "\n",
    "    {\n",
    "        \"a key\": \"some value\"\n",
    "    }\n",
    "\n",
    "And we look up values using keys, the same way that we looked up indices in lists:\n",
    "\n",
    "    In[1]: our_dictionary = {\"a key\": \"some value\"}\n",
    "    In[2]: our_dictionary[\"a key\"]\n",
    "    Out[2]: \"some value\"\n",
    "\n",
    "We don't have to use strings, we can have pretty much any type of data:\n",
    "\n",
    "    {\n",
    "        True: False\n",
    "    }\n",
    "\n",
    "Speaking of types, you can use the `type()` function to get the type of an argument in Python.\n",
    "\n",
    "So let's look at the `keys` in our dataset dictionary, and get the `type` of the `values`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "end date <class 'str'>\n",
      "economic scenario <class 'str'>\n",
      "location <class 'str'>\n",
      "start date <class 'str'>\n",
      "access restricted <class 'str'>\n",
      "name <class 'str'>\n",
      "filepath <class 'str'>\n",
      "type <class 'str'>\n",
      "combined production <class 'bool'>\n",
      "parameters <class 'list'>\n",
      "technology level <class 'str'>\n",
      "exchanges <class 'list'>\n",
      "id <class 'str'>\n"
     ]
    }
   ],
   "source": [
    "for key in dataset:\n",
    "    print(key, type(dataset[key]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `str` is a string, e.g. `\"this is an example\"`. You can use `\"double\"` or `'single'` quotes.\n",
    "* `bool` is a boolean, i.e. `True` or `False`.\n",
    "* `list` is a list, e.g. [1,2,3,\"look\",\"at\",\"me\"]. Lists can have more than one type of data inside them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1: Getting data attributes by key\n",
    "\n",
    "* How can you get the name of the dataset?\n",
    "* How can you get the location of the dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Delete this comment and try your own code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2: Getting the number of exchanges\n",
    "\n",
    "The function `len()` will return the length of something, e.g. the length of a list:\n",
    "\n",
    "    In[1]: len([1,2,3])\n",
    "    Out[1]: 3\n",
    "    \n",
    "Or the length of a string:\n",
    "\n",
    "    In[1]: len(\"milk\")\n",
    "    Out[1]: 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Delete this comment and try your own code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our activity dataset has inputs and outputs. All flows for the dataset are given in the list `exchanges`. There are two types of exchanges: [technosphere exchanges](https://docs.ocelot.space/data_format.html#technosphere-exchanges-activity-exchange-schema), which are product or service flows in the supply chain, and [biosphere exchanges](https://docs.ocelot.space/data_format.html#biosphere-exchanges-elementary-exchange-schema), which are interactions with the natural world, either emissions or consumption of resources.\n",
    "\n",
    "Exchanges have a lot of data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'amount': 6.47766990291262e-10,\n",
       "  'byproduct classification': 'allocatable product',\n",
       "  'conditional exchange': False,\n",
       "  'formula': 'factor_MJ_kWh*0.0000000000556',\n",
       "  'id': '1c0c37a4-2cdb-41b1-98ba-e6862aa79543',\n",
       "  'name': 'gas power plant, 100MW electrical',\n",
       "  'properties': [],\n",
       "  'tag': 'intermediateExchange',\n",
       "  'type': 'from technosphere',\n",
       "  'uncertainty': {'mean': 6.47766990291262e-10,\n",
       "   'mu': -21.0589383934,\n",
       "   'pedigree matrix': {'completeness': 3,\n",
       "    'further technology correlation': 3,\n",
       "    'geographical correlation': 3,\n",
       "    'reliability': 3,\n",
       "    'temporal correlation': 3},\n",
       "   'type': 'lognormal',\n",
       "   'variance': 0.12,\n",
       "   'variance with pedigree uncertainty': 0.1327},\n",
       "  'unit': 'unit',\n",
       "  'variable': 'gas_power_plant'},\n",
       " {'amount': 4.1,\n",
       "  'byproduct classification': 'allocatable product',\n",
       "  'conditional exchange': False,\n",
       "  'id': '8dced74b-0677-4388-9215-bb6d30b8b084',\n",
       "  'name': 'heat, district or industrial, natural gas',\n",
       "  'production volume': {'amount': 4439289063000.0,\n",
       "   'formula': 'heat/electricity*electricity_apv',\n",
       "   'variable': 'heat_apv'},\n",
       "  'properties': [{'amount': 0.0106,\n",
       "    'id': '38f94dd1-d5aa-41b8-b182-c0c42985d9dc',\n",
       "    'name': 'price',\n",
       "    'uncertainty': {'maximum': 0.0106,\n",
       "     'minimum': 0.0106,\n",
       "     'pedigree matrix': {'completeness': 1,\n",
       "      'further technology correlation': 1,\n",
       "      'geographical correlation': 1,\n",
       "      'reliability': 1,\n",
       "      'temporal correlation': 1},\n",
       "     'standard deviation 95%': 0.0,\n",
       "     'type': 'undefined'},\n",
       "    'unit': 'EUR2005'},\n",
       "   {'amount': 0.184213553594,\n",
       "    'id': '7a3978ea-3e26-4329-bc8b-0915d58a7e6f',\n",
       "    'name': 'true value relation',\n",
       "    'uncertainty': {'maximum': 0.184213553594,\n",
       "     'minimum': 0.184213553594,\n",
       "     'pedigree matrix': {'completeness': 1,\n",
       "      'further technology correlation': 1,\n",
       "      'geographical correlation': 1,\n",
       "      'reliability': 1,\n",
       "      'temporal correlation': 1},\n",
       "     'standard deviation 95%': 0.0,\n",
       "     'type': 'undefined'},\n",
       "    'unit': 'dimensionless',\n",
       "    'variable': 'true_value_relation'}],\n",
       "  'tag': 'intermediateExchange',\n",
       "  'type': 'reference product',\n",
       "  'unit': 'MJ',\n",
       "  'variable': 'heat'},\n",
       " {'amount': 1.0,\n",
       "  'byproduct classification': 'allocatable product',\n",
       "  'conditional exchange': False,\n",
       "  'id': 'e7896def-6d42-423e-84b4-7d2c29457cbf',\n",
       "  'name': 'electricity, high voltage',\n",
       "  'production volume': {'amount': 1082753430000.0,\n",
       "   'formula': 'gross_electricity* (1-losses_gross_net)',\n",
       "   'variable': 'electricity_apv'},\n",
       "  'properties': [{'amount': 0.0977,\n",
       "    'id': '38f94dd1-d5aa-41b8-b182-c0c42985d9dc',\n",
       "    'name': 'price',\n",
       "    'uncertainty': {'maximum': 0.0977,\n",
       "     'minimum': 0.0977,\n",
       "     'pedigree matrix': {'completeness': 1,\n",
       "      'further technology correlation': 1,\n",
       "      'geographical correlation': 1,\n",
       "      'reliability': 1,\n",
       "      'temporal correlation': 1},\n",
       "     'standard deviation 95%': 0.0,\n",
       "     'type': 'undefined'},\n",
       "    'unit': 'EUR2005'},\n",
       "   {'amount': 3.6,\n",
       "    'id': '7a3978ea-3e26-4329-bc8b-0915d58a7e6f',\n",
       "    'name': 'true value relation',\n",
       "    'uncertainty': {'maximum': 3.6,\n",
       "     'minimum': 3.6,\n",
       "     'pedigree matrix': {'completeness': 1,\n",
       "      'further technology correlation': 1,\n",
       "      'geographical correlation': 1,\n",
       "      'reliability': 1,\n",
       "      'temporal correlation': 1},\n",
       "     'standard deviation 95%': 0.0,\n",
       "     'type': 'undefined'},\n",
       "    'unit': 'dimensionless'}],\n",
       "  'tag': 'intermediateExchange',\n",
       "  'type': 'byproduct',\n",
       "  'uncertainty': {'maximum': 1.0,\n",
       "   'minimum': 1.0,\n",
       "   'pedigree matrix': {'completeness': 1,\n",
       "    'further technology correlation': 1,\n",
       "    'geographical correlation': 1,\n",
       "    'reliability': 1,\n",
       "    'temporal correlation': 1},\n",
       "   'standard deviation 95%': 0.0,\n",
       "   'type': 'undefined'},\n",
       "  'unit': 'kWh',\n",
       "  'variable': 'electricity'},\n",
       " {'amount': 0.298730395817774,\n",
       "  'byproduct classification': 'allocatable product',\n",
       "  'conditional exchange': False,\n",
       "  'formula': 'factor_MJ_kWh/ natural_gas_Net_Heat_Value',\n",
       "  'id': '24251b22-d699-4ef4-af0f-c72931624ab2',\n",
       "  'name': 'natural gas, high pressure',\n",
       "  'properties': [],\n",
       "  'tag': 'intermediateExchange',\n",
       "  'type': 'from technosphere',\n",
       "  'uncertainty': {'mean': 0.298730395817774,\n",
       "   'mu': -1.10966212486,\n",
       "   'pedigree matrix': {'completeness': 1,\n",
       "    'further technology correlation': 1,\n",
       "    'geographical correlation': 1,\n",
       "    'reliability': 1,\n",
       "    'temporal correlation': 1},\n",
       "   'type': 'lognormal',\n",
       "   'variance': 0.0,\n",
       "   'variance with pedigree uncertainty': 0.0},\n",
       "  'unit': 'm3',\n",
       "  'variable': 'natural_gas'},\n",
       " {'amount': 0.622135922330097,\n",
       "  'compartment': 'air',\n",
       "  'formula': 'factor_MJ_kWh*0.0534',\n",
       "  'id': 'a53649d4-01ac-4eab-a04d-ca3815a9dd91',\n",
       "  'name': 'Carbon dioxide, fossil',\n",
       "  'properties': [],\n",
       "  'subcompartment': 'non-urban air or from high stacks',\n",
       "  'tag': 'elementaryExchange',\n",
       "  'type': 'to environment',\n",
       "  'uncertainty': {'mean': 0.622135922330097,\n",
       "   'mu': -0.376045011742,\n",
       "   'pedigree matrix': {'completeness': 3,\n",
       "    'further technology correlation': 3,\n",
       "    'geographical correlation': 3,\n",
       "    'reliability': 3,\n",
       "    'temporal correlation': 4},\n",
       "   'type': 'lognormal',\n",
       "   'variance': 0.0006,\n",
       "   'variance with pedigree uncertainty': 0.0193},\n",
       "  'unit': 'kg',\n",
       "  'variable': 'carbon_dioxide_fossil'}]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset['exchanges']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes exchanges have `properties`. For example, a property could be the dry or wet mass of a flow. In our dataset, the exchange for `heat, district or industrial, natural gas` has a property for `price`. Let's look at this exchange in more detail and we can see this a little more clearly. To do this, we will iterate over the exchanges, and pull out one based on its name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'amount': 4.1,\n",
       " 'byproduct classification': 'allocatable product',\n",
       " 'conditional exchange': False,\n",
       " 'id': '8dced74b-0677-4388-9215-bb6d30b8b084',\n",
       " 'name': 'heat, district or industrial, natural gas',\n",
       " 'production volume': {'amount': 4439289063000.0,\n",
       "  'formula': 'heat/electricity*electricity_apv',\n",
       "  'variable': 'heat_apv'},\n",
       " 'properties': [{'amount': 0.0106,\n",
       "   'id': '38f94dd1-d5aa-41b8-b182-c0c42985d9dc',\n",
       "   'name': 'price',\n",
       "   'uncertainty': {'maximum': 0.0106,\n",
       "    'minimum': 0.0106,\n",
       "    'pedigree matrix': {'completeness': 1,\n",
       "     'further technology correlation': 1,\n",
       "     'geographical correlation': 1,\n",
       "     'reliability': 1,\n",
       "     'temporal correlation': 1},\n",
       "    'standard deviation 95%': 0.0,\n",
       "    'type': 'undefined'},\n",
       "   'unit': 'EUR2005'},\n",
       "  {'amount': 0.184213553594,\n",
       "   'id': '7a3978ea-3e26-4329-bc8b-0915d58a7e6f',\n",
       "   'name': 'true value relation',\n",
       "   'uncertainty': {'maximum': 0.184213553594,\n",
       "    'minimum': 0.184213553594,\n",
       "    'pedigree matrix': {'completeness': 1,\n",
       "     'further technology correlation': 1,\n",
       "     'geographical correlation': 1,\n",
       "     'reliability': 1,\n",
       "     'temporal correlation': 1},\n",
       "    'standard deviation 95%': 0.0,\n",
       "    'type': 'undefined'},\n",
       "   'unit': 'dimensionless',\n",
       "   'variable': 'true_value_relation'}],\n",
       " 'tag': 'intermediateExchange',\n",
       " 'type': 'reference product',\n",
       " 'unit': 'MJ',\n",
       " 'variable': 'heat'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heat_process_name = 'heat, district or industrial, natural gas'\n",
    "\n",
    "for exchange in dataset['exchanges']:\n",
    "    if exchange['name'] == heat_process_name:\n",
    "        selected_exchange = exchange\n",
    "        \n",
    "selected_exchange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3: Find the numeric value (`amount`) of the property `price` for the our selected exchange.\n",
    "\n",
    "In the code above, you saw some new python syntax: `if` statements, tests for equality (`==`, can also use `<`, `>`, `!=` for not equals). You should be able to answer this question using these syntax elements!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# You know what to do here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you need a hint, start with something like this:\n",
    "\n",
    "    for property in selected_exchange['properties']:\n",
    "        if property['name'] == 'something':\n",
    "            print(property['amount'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also write functions that do what we have been already been doing. For example, here is a function that returns a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_function():\n",
    "    return [1,2,3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I can call my function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_function()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Functions can have inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_function_with_inputs(input_1, input_2):\n",
    "    print(\"Input 1 was:\", input_1)\n",
    "    print(\"Input 2 was:\", input_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input 1 was: The cow says (wait for it...)\n",
      "Input 2 was: MOOOOOOOOOOO!!!\n"
     ]
    }
   ],
   "source": [
    "my_function_with_inputs(\"The cow says (wait for it...)\", \"MOOOOOOOOOOO!!!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a function to generate some statistics on this dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def some_statistics(ds):  # `ds` is an input of a dataset\n",
    "    print(\"Number of exchanges:\", len(ds['exchanges']))\n",
    "    print(\"The sum of the exchanges:\", sum(exchange['amount'] for exchange in ds['exchanges']))\n",
    "    for index, exchange in enumerate(ds['exchanges']):\n",
    "        # We can use `.get(key, default_value)` to get a key if present, or get the default value otherwise\n",
    "        print(\"Exchange {}: {} has {} properties\".format(index, exchange['name'], len(exchange.get('properties', []))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of exchanges: 5\n",
      "The sum of the exchanges: 6.020866318795638\n",
      "Exchange 0: gas power plant, 100MW electrical has 0 properties\n",
      "Exchange 1: heat, district or industrial, natural gas has 2 properties\n",
      "Exchange 2: electricity, high voltage has 2 properties\n",
      "Exchange 3: natural gas, high pressure has 0 properties\n",
      "Exchange 4: Carbon dioxide, fossil has 0 properties\n"
     ]
    }
   ],
   "source": [
    "some_statistics(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}