06_nhst.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Null hypothesis significance testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous sections, we have measures properties of networks. We may now want to ask whether the values we observed are different from what we would expect at random. The process to reach this conclusion is as follows:\n",
    "\n",
    "- pick a network we want to test\n",
    "- estimate the probability of every interaction using a *null model*\n",
    "- generate a large number of random networks from this matrix of probabilities\n",
    "- measure the value of interest on every random network\n",
    "- test the probability that the value on the empirical network belongs to the distribution derived from the random samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we will test whether a host-parasite networks is *more* nested than expected given its degree distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m\u001b[1mActivating\u001b[22m\u001b[39m environment at `~/EcoNetworksLabs/Project.toml`\n"
     ]
    }
   ],
   "source": [
    "import Pkg; Pkg.activate(\".\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We willrequire a few packages for this example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "using EcologicalNetworks\n",
    "using Plots\n",
    "using StatsBase\n",
    "using Random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading a network\n",
    "\n",
    "To illustrate NHST, we will take one host-parasite network, remove the quantitative information, and then assign it to a variable named `N`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18×10 bipartite  ecological network (Bool, String) (L: 61)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N = convert(BipartiteNetwork, web_of_life(\"A_HP_001\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step is to get the *observed* value of the quantity of interest, for example NODF:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.639664124909223"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs = nodf(N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a second step, we need to transform this network into a probabilistic network, using one of the standard null models. In this example, we will use the type II null model, which accounts for the degree distribution of both sides of the network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18×10 bipartite probabilistic ecological network (Float64, String) (L: 60.99999999999999)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "template = null2(N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the value of $L$ might be slightly different from what it said in the deterministic network. This is because numbers are difficult to represent for computers, and this will not have too much adverse effects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this template, we want to generate *a lot* of random networks. `EcologicalNetworks` extends the usual *Julia* functions, so we can call `rand`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000-element Array{BipartiteNetwork{Bool,String},1}:\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 57)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 56)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 71)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 74)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 60)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 59)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 65)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 47)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 69)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 57)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 74)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 63)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 55)\n",
       " ⋮                                                         \n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 59)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 58)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 56)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 59)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 65)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 59)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 66)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 62)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 59)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 56)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 56)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 61)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draws = rand(template, 10_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will have generated $10^4$ networks. Each of these networks is generated in the following way: every interactions is considered to be a probability $p$, and the network is filled with the output of a Bernoulli trial with probability of success $p$. For example, if the null model says that an interaction has a probability of $0.5$, we expect that about half of the random draws will have this interaction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now have a look at the values of NODF in this random sample. The `f.(x)` notation means \"apply `f` to every element of `x`\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000-element Array{Float64,1}:\n",
       " NaN                  \n",
       " NaN                  \n",
       "   0.5264651416122004 \n",
       "   0.519256924992219  \n",
       " NaN                  \n",
       "   0.4522118477020438 \n",
       "   0.3216552546944703 \n",
       "   0.4053906006847183 \n",
       "   0.44660675381263626\n",
       " NaN                  \n",
       "   0.48483115468409593\n",
       " NaN                  \n",
       " NaN                  \n",
       "   ⋮                  \n",
       "   0.3964674758792406 \n",
       "   0.5106644880174293 \n",
       " NaN                  \n",
       "   0.37533224400871457\n",
       "   0.4710403050108931 \n",
       " NaN                  \n",
       "   0.4198879551820729 \n",
       "   0.4872642390289449 \n",
       "   0.4517403257599336 \n",
       "   0.43171802054154995\n",
       "   0.4266448801742919 \n",
       "   0.40215997510115153"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = nodf.(draws)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Weird, some of these networks have a value of `NaN`. This is because some of the random networks are *degenerate*, *i.e.* they have species with no interactions left. Let's remove them.\n",
    "\n",
    "The `filter(condition, collection)` function will select only the elements of `collection` which satisfy `condition` (which is a function). In this case, we use `!isdegenerate`, *i.e.* the opposite of \"is the network degenerate\", to return only networks with connected species. Remember to use `?` to learm more about a function -- in this case, `?filter` will give you a lot of information!\n",
    "\n",
    "Note that we use the `filter!` function - it *modifies* the collection on which it is called. By convention, *Julia* function ending with `!` will change their arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5532-element Array{BipartiteNetwork{Bool,String},1}:\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 71)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 74)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 59)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 65)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 47)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 69)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 74)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 59)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 67)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 65)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 65)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 65)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 63)\n",
       " ⋮                                                         \n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 64)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 61)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 59)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 58)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 59)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 65)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 66)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 62)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 59)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 56)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 56)\n",
       " 18×10 bipartite  ecological network (Bool, String) (L: 61)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filter!(!isdegenerate, draws)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After removing the networks with degenerate matrices, we can apply NODF to all of them. We will use the `.` syntax, where `f.(x)` means *apply f to every element of x*. In our example, we want to apply `nodf` to every element of `draws`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5532-element Array{Float64,1}:\n",
       " 0.5264651416122004 \n",
       " 0.519256924992219  \n",
       " 0.4522118477020438 \n",
       " 0.3216552546944703 \n",
       " 0.4053906006847183 \n",
       " 0.44660675381263626\n",
       " 0.48483115468409593\n",
       " 0.45798085901027075\n",
       " 0.45316187080892967\n",
       " 0.4979411764705882 \n",
       " 0.45212937026662514\n",
       " 0.43273420479302827\n",
       " 0.44676314970432607\n",
       " ⋮                  \n",
       " 0.4179084967320262 \n",
       " 0.3962254901960785 \n",
       " 0.3964674758792406 \n",
       " 0.5106644880174293 \n",
       " 0.37533224400871457\n",
       " 0.4710403050108931 \n",
       " 0.4198879551820729 \n",
       " 0.4872642390289449 \n",
       " 0.4517403257599336 \n",
       " 0.43171802054154995\n",
       " 0.4266448801742919 \n",
       " 0.40215997510115153"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nodf_random = nodf.(draws)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we want to do now, is to represent the distribution of the random draws, and compare it to the value of the network we obsered in nature. This can be done using an histogram:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip8400\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip8400)\" points=\"\n",
       "0,1600 2400,1600 2400,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip8401\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip8400)\" points=\"\n",
       "229.135,1425.62 2352.76,1425.62 2352.76,47.2441 229.135,47.2441 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip8402\">\n",
       "    <rect x=\"229\" y=\"47\" width=\"2125\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  867.257,1425.62 867.257,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1445.28,1425.62 1445.28,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2023.3,1425.62 2023.3,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,1059.06 2352.76,1059.06 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,731.518 2352.76,731.518 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,403.973 2352.76,403.973 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,76.4283 2352.76,76.4283 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,1386.61 2352.76,1386.61 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  289.238,1425.62 289.238,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  867.257,1386.61 867.257,1365.93 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1445.28,1386.61 1445.28,1365.93 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2023.3,1386.61 2023.3,1365.93 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  289.238,1059.06 321.092,1059.06 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  289.238,731.518 321.092,731.518 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  289.238,403.973 321.092,403.973 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  289.238,76.4283 321.092,76.4283 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip8400)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 867.257, 1440.61)\" x=\"867.257\" y=\"1440.61\">0.2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8400)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1445.28, 1440.61)\" x=\"1445.28\" y=\"1440.61\">0.4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8400)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2023.3, 1440.61)\" x=\"2023.3\" y=\"1440.61\">0.6</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8400)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 265.238, 1076.56)\" x=\"265.238\" y=\"1076.56\">100</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8400)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 265.238, 749.018)\" x=\"265.238\" y=\"749.018\">200</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8400)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 265.238, 421.473)\" x=\"265.238\" y=\"421.473\">300</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8400)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 265.238, 93.9283)\" x=\"265.238\" y=\"93.9283\">400</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8400)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1290.95, 1559.48)\" x=\"1290.95\" y=\"1559.48\">NODF</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8400)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 89.2861, 736.431)\" x=\"89.2861\" y=\"736.431\">Count</text>\n",
       "</g>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "982.861,1383.33 982.861,1386.61 1011.76,1386.61 1011.76,1383.33 982.861,1383.33 982.861,1383.33 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  982.861,1383.33 982.861,1386.61 1011.76,1386.61 1011.76,1383.33 982.861,1383.33 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1011.76,1380.06 1011.76,1386.61 1040.66,1386.61 1040.66,1380.06 1011.76,1380.06 1011.76,1380.06 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1011.76,1380.06 1011.76,1386.61 1040.66,1386.61 1040.66,1380.06 1011.76,1380.06 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1040.66,1376.78 1040.66,1386.61 1069.56,1386.61 1069.56,1376.78 1040.66,1376.78 1040.66,1376.78 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1040.66,1376.78 1040.66,1386.61 1069.56,1386.61 1069.56,1376.78 1040.66,1376.78 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1069.56,1366.96 1069.56,1386.61 1098.47,1386.61 1098.47,1366.96 1069.56,1366.96 1069.56,1366.96 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1069.56,1366.96 1069.56,1386.61 1098.47,1386.61 1098.47,1366.96 1069.56,1366.96 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1098.47,1334.2 1098.47,1386.61 1127.37,1386.61 1127.37,1334.2 1098.47,1334.2 1098.47,1334.2 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1098.47,1334.2 1098.47,1386.61 1127.37,1386.61 1127.37,1334.2 1098.47,1334.2 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1127.37,1347.3 1127.37,1386.61 1156.27,1386.61 1156.27,1347.3 1127.37,1347.3 1127.37,1347.3 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1127.37,1347.3 1127.37,1386.61 1156.27,1386.61 1156.27,1347.3 1127.37,1347.3 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1156.27,1311.27 1156.27,1386.61 1185.17,1386.61 1185.17,1311.27 1156.27,1311.27 1156.27,1311.27 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1156.27,1311.27 1156.27,1386.61 1185.17,1386.61 1185.17,1311.27 1156.27,1311.27 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1185.17,1304.72 1185.17,1386.61 1214.07,1386.61 1214.07,1304.72 1185.17,1304.72 1185.17,1304.72 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1185.17,1304.72 1185.17,1386.61 1214.07,1386.61 1214.07,1304.72 1185.17,1304.72 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1214.07,1239.21 1214.07,1386.61 1242.97,1386.61 1242.97,1239.21 1214.07,1239.21 1214.07,1239.21 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1214.07,1239.21 1214.07,1386.61 1242.97,1386.61 1242.97,1239.21 1214.07,1239.21 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1242.97,1196.63 1242.97,1386.61 1271.87,1386.61 1271.87,1196.63 1242.97,1196.63 1242.97,1196.63 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1242.97,1196.63 1242.97,1386.61 1271.87,1386.61 1271.87,1196.63 1242.97,1196.63 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1271.87,1157.33 1271.87,1386.61 1300.77,1386.61 1300.77,1157.33 1271.87,1157.33 1271.87,1157.33 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1271.87,1157.33 1271.87,1386.61 1300.77,1386.61 1300.77,1157.33 1271.87,1157.33 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1300.77,980.452 1300.77,1386.61 1329.67,1386.61 1329.67,980.452 1300.77,980.452 1300.77,980.452 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1300.77,980.452 1300.77,1386.61 1329.67,1386.61 1329.67,980.452 1300.77,980.452 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1329.67,885.464 1329.67,1386.61 1358.57,1386.61 1358.57,885.464 1329.67,885.464 1329.67,885.464 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1329.67,885.464 1329.67,1386.61 1358.57,1386.61 1358.57,885.464 1329.67,885.464 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1358.57,823.231 1358.57,1386.61 1387.47,1386.61 1387.47,823.231 1358.57,823.231 1358.57,823.231 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1358.57,823.231 1358.57,1386.61 1387.47,1386.61 1387.47,823.231 1358.57,823.231 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1387.47,629.979 1387.47,1386.61 1416.38,1386.61 1416.38,629.979 1387.47,629.979 1387.47,629.979 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1387.47,629.979 1387.47,1386.61 1416.38,1386.61 1416.38,629.979 1387.47,629.979 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1416.38,508.788 1416.38,1386.61 1445.28,1386.61 1445.28,508.788 1416.38,508.788 1416.38,508.788 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1416.38,508.788 1416.38,1386.61 1445.28,1386.61 1445.28,508.788 1416.38,508.788 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1445.28,420.351 1445.28,1386.61 1474.18,1386.61 1474.18,420.351 1445.28,420.351 1445.28,420.351 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1445.28,420.351 1445.28,1386.61 1474.18,1386.61 1474.18,420.351 1445.28,420.351 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1474.18,279.506 1474.18,1386.61 1503.08,1386.61 1503.08,279.506 1474.18,279.506 1474.18,279.506 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1474.18,279.506 1474.18,1386.61 1503.08,1386.61 1503.08,279.506 1474.18,279.506 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1503.08,246.752 1503.08,1386.61 1531.98,1386.61 1531.98,246.752 1503.08,246.752 1503.08,246.752 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1503.08,246.752 1503.08,1386.61 1531.98,1386.61 1531.98,246.752 1503.08,246.752 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1531.98,138.662 1531.98,1386.61 1560.88,1386.61 1560.88,138.662 1531.98,138.662 1531.98,138.662 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1531.98,138.662 1531.98,1386.61 1560.88,1386.61 1560.88,138.662 1531.98,138.662 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1560.88,171.416 1560.88,1386.61 1589.78,1386.61 1589.78,171.416 1560.88,171.416 1560.88,171.416 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1560.88,171.416 1560.88,1386.61 1589.78,1386.61 1589.78,171.416 1560.88,171.416 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1589.78,86.2547 1589.78,1386.61 1618.68,1386.61 1618.68,86.2547 1589.78,86.2547 1589.78,86.2547 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1589.78,86.2547 1589.78,1386.61 1618.68,1386.61 1618.68,86.2547 1589.78,86.2547 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1618.68,256.578 1618.68,1386.61 1647.58,1386.61 1647.58,256.578 1618.68,256.578 1618.68,256.578 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1618.68,256.578 1618.68,1386.61 1647.58,1386.61 1647.58,256.578 1618.68,256.578 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1647.58,351.566 1647.58,1386.61 1676.48,1386.61 1676.48,351.566 1647.58,351.566 1647.58,351.566 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1647.58,351.566 1647.58,1386.61 1676.48,1386.61 1676.48,351.566 1647.58,351.566 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1676.48,374.494 1676.48,1386.61 1705.39,1386.61 1705.39,374.494 1676.48,374.494 1676.48,374.494 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1676.48,374.494 1676.48,1386.61 1705.39,1386.61 1705.39,374.494 1676.48,374.494 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1705.39,564.47 1705.39,1386.61 1734.29,1386.61 1734.29,564.47 1705.39,564.47 1705.39,564.47 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1705.39,564.47 1705.39,1386.61 1734.29,1386.61 1734.29,564.47 1705.39,564.47 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1734.29,666.009 1734.29,1386.61 1763.19,1386.61 1763.19,666.009 1734.29,666.009 1734.29,666.009 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1734.29,666.009 1734.29,1386.61 1763.19,1386.61 1763.19,666.009 1734.29,666.009 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1763.19,787.201 1763.19,1386.61 1792.09,1386.61 1792.09,787.201 1763.19,787.201 1763.19,787.201 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1763.19,787.201 1763.19,1386.61 1792.09,1386.61 1792.09,787.201 1763.19,787.201 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1792.09,908.392 1792.09,1386.61 1820.99,1386.61 1820.99,908.392 1792.09,908.392 1792.09,908.392 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1792.09,908.392 1792.09,1386.61 1820.99,1386.61 1820.99,908.392 1792.09,908.392 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1820.99,987.003 1820.99,1386.61 1849.89,1386.61 1849.89,987.003 1820.99,987.003 1820.99,987.003 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1820.99,987.003 1820.99,1386.61 1849.89,1386.61 1849.89,987.003 1820.99,987.003 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1849.89,1101.64 1849.89,1386.61 1878.79,1386.61 1878.79,1101.64 1849.89,1101.64 1849.89,1101.64 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1849.89,1101.64 1849.89,1386.61 1878.79,1386.61 1878.79,1101.64 1849.89,1101.64 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1878.79,1196.63 1878.79,1386.61 1907.69,1386.61 1907.69,1196.63 1878.79,1196.63 1878.79,1196.63 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1878.79,1196.63 1878.79,1386.61 1907.69,1386.61 1907.69,1196.63 1878.79,1196.63 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1907.69,1245.76 1907.69,1386.61 1936.59,1386.61 1936.59,1245.76 1907.69,1245.76 1907.69,1245.76 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1907.69,1245.76 1907.69,1386.61 1936.59,1386.61 1936.59,1245.76 1907.69,1245.76 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1936.59,1252.31 1936.59,1386.61 1965.49,1386.61 1965.49,1252.31 1936.59,1252.31 1936.59,1252.31 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1936.59,1252.31 1936.59,1386.61 1965.49,1386.61 1965.49,1252.31 1936.59,1252.31 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1965.49,1308 1965.49,1386.61 1994.4,1386.61 1994.4,1308 1965.49,1308 1965.49,1308 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1965.49,1308 1965.49,1386.61 1994.4,1386.61 1994.4,1308 1965.49,1308 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "1994.4,1324.37 1994.4,1386.61 2023.3,1386.61 2023.3,1324.37 1994.4,1324.37 1994.4,1324.37 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1994.4,1324.37 1994.4,1386.61 2023.3,1386.61 2023.3,1324.37 1994.4,1324.37 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "2023.3,1337.48 2023.3,1386.61 2052.2,1386.61 2052.2,1337.48 2023.3,1337.48 2023.3,1337.48 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2023.3,1337.48 2023.3,1386.61 2052.2,1386.61 2052.2,1337.48 2023.3,1337.48 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "2052.2,1366.96 2052.2,1386.61 2081.1,1386.61 2081.1,1366.96 2052.2,1366.96 2052.2,1366.96 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2052.2,1366.96 2052.2,1386.61 2081.1,1386.61 2081.1,1366.96 2052.2,1366.96 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "2081.1,1376.78 2081.1,1386.61 2110,1386.61 2110,1376.78 2081.1,1376.78 2081.1,1376.78 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2081.1,1376.78 2081.1,1386.61 2110,1386.61 2110,1376.78 2081.1,1376.78 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "2110,1380.06 2110,1386.61 2138.9,1386.61 2138.9,1380.06 2110,1380.06 2110,1380.06 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2110,1380.06 2110,1386.61 2138.9,1386.61 2138.9,1380.06 2110,1380.06 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "2138.9,1383.33 2138.9,1386.61 2167.8,1386.61 2167.8,1383.33 2138.9,1383.33 2138.9,1383.33 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2138.9,1383.33 2138.9,1386.61 2167.8,1386.61 2167.8,1383.33 2138.9,1383.33 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "2167.8,1383.33 2167.8,1386.61 2196.7,1386.61 2196.7,1383.33 2167.8,1383.33 2167.8,1383.33 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2167.8,1383.33 2167.8,1386.61 2196.7,1386.61 2196.7,1383.33 2167.8,1383.33 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "2196.7,1386.61 2196.7,1386.61 2225.6,1386.61 2225.6,1386.61 2196.7,1386.61 2196.7,1386.61 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2196.7,1386.61 2196.7,1386.61 2225.6,1386.61 2196.7,1386.61 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8402)\" points=\"\n",
       "2225.6,1383.33 2225.6,1386.61 2254.5,1386.61 2254.5,1383.33 2225.6,1383.33 2225.6,1383.33 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2225.6,1383.33 2225.6,1386.61 2254.5,1386.61 2254.5,1383.33 2225.6,1383.33 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8402)\" style=\"stroke:#000000; stroke-width:8; stroke-opacity:1; fill:none\" points=\"\n",
       "  2137.93,2803.99 2137.93,-1331.13 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8400)\" points=\"\n",
       "301.135,312.204 902.282,312.204 902.282,130.764 301.135,130.764 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  301.135,312.204 902.282,312.204 902.282,130.764 301.135,130.764 301.135,312.204 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8400)\" points=\"\n",
       "325.135,215.436 469.135,215.436 469.135,167.052 325.135,167.052 325.135,215.436 \n",
       "  \" fill=\"#d3d3d3\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#d3d3d3; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  325.135,215.436 469.135,215.436 469.135,167.052 325.135,167.052 325.135,215.436 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip8400)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 493.135, 208.744)\" x=\"493.135\" y=\"208.744\">Random values</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip8400)\" style=\"stroke:#000000; stroke-width:8; stroke-opacity:1; fill:none\" points=\"\n",
       "  325.135,251.724 469.135,251.724 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip8400)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 493.135, 269.224)\" x=\"493.135\" y=\"269.224\">Observed values</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "histogram(nodf_random, lab=\"Random values\", c=:lightgrey, lc=:lightgrey, frame=:origin, legend=:topleft)\n",
    "vline!([nodf(N)], lab=\"Observed values\", c=:black, lw=2)\n",
    "xaxis!(\"NODF\")\n",
    "yaxis!(\"Count\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that the empirical network is more nested than a lot of the random networks! This would mean that the nestedness we observed in nature was *markedly higher* than what we expect if the mechanisms involved in the null model we used (the degree distribution) were acting alone.\n",
    "\n",
    "At this point, we can ask whether this difference is significant. In a permutation test, we can approximate the *p*-value as the proportion of random networks that are *more* (or *less*) nested than the empirical one, which we can measure as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0010911074740862503"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1.0-sum(nodf_random .<= nodf(N))/length(nodf_random)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we can say that the network is *more* nested than expected by chance ($p \\approx 10^{-3}$)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.2.0",
   "language": "julia",
   "name": "julia-1.2"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}