evaluation_methods.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Evaluation methods in NLP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2022\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Contents\n",
    "\n",
    "1. [Overview](#Overview)\n",
    "1. [Your projects](#Your-projects)\n",
    "1. [Set-up](#Set-up)\n",
    "1. [Data organization](#Data-organization)\n",
    "    1. [Train/dev/test](#Train/dev/test)\n",
    "    1. [No fixed splits](#No-fixed-splits)\n",
    "1. [Cross-validation](#Cross-validation)\n",
    "    1. [Random splits](#Random-splits)\n",
    "    1. [K-folds](#K-folds)\n",
    "1. [Baselines](#Baselines)\n",
    "    1. [Baselines are crucial for strong experiments](#Baselines-are-crucial-for-strong-experiments)\n",
    "    1. [Random baselines](#Random-baselines)\n",
    "    1. [Task-specific baselines](#Task-specific-baselines)\n",
    "1. [Hyperparameter optimization](#Hyperparameter-optimization)\n",
    "    1. [Rationale](#Rationale)\n",
    "    1. [The ideal hyperparameter optimization setting](#The-ideal-hyperparameter-optimization-setting)\n",
    "    1. [Practical considerations, and some compromises](#Practical-considerations,-and-some-compromises)\n",
    "    1. [Hyperparameter optimization tools](#Hyperparameter-optimization-tools)\n",
    "1. [Classifier comparison](#Classifier-comparison)\n",
    "    1. [Practical differences](#Practical-differences)\n",
    "    1. [Confidence intervals](#Confidence-intervals)\n",
    "    1. [Wilcoxon signed-rank test](#Wilcoxon-signed-rank-test)\n",
    "    1. [McNemar's test](#McNemar's-test)\n",
    "1. [Assessing models without convergence](#Assessing-models-without-convergence)\n",
    "    1. [Incremental dev set testing](#Incremental-dev-set-testing)\n",
    "    1. [Learning curves with confidence intervals](#Learning-curves-with-confidence-intervals)\n",
    "1. [The role of random parameter initialization](#The-role-of-random-parameter-initialization)\n",
    "1. [Closing remarks](#Closing-remarks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Overview\n",
    "\n",
    "This notebook is an overview of experimental methods for NLU. My primary goal is to help you with the experiments you'll be doing for your projects.  It is a companion to [the evaluation metrics notebook](evaluation_metrics.ipynb), which I suggest studying first.\n",
    "\n",
    "The teaching team will be paying special attention to how you conduct your evaluations, so this notebook should create common ground around what our values are.\n",
    "\n",
    "This notebook is far from comprehensive. I hope it covers the most common tools, techniques, and challenges in the field. Beyond that, I'm hoping the examples here suggest a perspective on experiments and evaluations that generalizes to other topics and techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Your projects\n",
    "\n",
    "1. We will never evaluate a project based on how \"good\" the results are.\n",
    "  1. Publication venues do this, because they have additional constraints on space that lead them to favor positive evidence for new developments over negative results.\n",
    "  1. In CS224u, we are not subject to this constraint, so we can do the right and good thing of valuing positive results, negative results, and everything in between.\n",
    "\n",
    "1. We __will__ evaluate your project on:   \n",
    "  1. The appropriateness of the metrics\n",
    "  1. The strength of the methods\n",
    "  1. The extent to which the paper is open and clear-sighted about the limits of its findings. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from collections import defaultdict, Counter\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "from scipy import stats\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.model_selection import train_test_split\n",
    "import sst\n",
    "from torch_rnn_classifier import TorchRNNClassifier\n",
    "from torch_shallow_neural_classifier import TorchShallowNeuralClassifier\n",
    "import utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.fix_random_seeds()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data organization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Train/dev/test\n",
    "\n",
    "Many publicly available datasets are released with a train/dev/test structure. __We're all on the honor system to do test-set runs only when development is complete.__\n",
    "\n",
    "Splits like this basically presuppose a fairly large dataset.\n",
    "\n",
    "If there is no dev set as part of the distribution, then you might create one to simulate what a test run will be like, though you have to weigh this against the reduction in train-set size.\n",
    "\n",
    "Having a fixed test set ensures that all systems are assessed against the same gold data. This is generally good, but it is problematic where the test set turns out to have unusual properties that distort progress on the task. Ideally, every task would have dozens of test sets, so that we could report average performance and related statistics. The difficulty and expense of creating so many test sets means that this ideal is rarely if ever realized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### No fixed splits\n",
    "\n",
    "Many datasets are released without predefined splits. This poses challenges for assessment, especially comparative assessment: __for robust comparisons with prior work, you really have to rerun the models using your assessment regime on your splits__. For example, if you're doing [5-fold cross-validation](#K-folds), then all the systems should be trained and assessed using exactly the same folds, to control for variation in how difficult the splits are.\n",
    "\n",
    "If the dataset is large enough, you might create a train/test or train/dev/test split right at the start of your project and use it for all your experiments. This means putting the test portion in a locked box until the very end, when you assess all the relevant systems against it. For large datasets, this will certainly simplify your experimental set-up, for reasons that will become clear when we discuss [hyperparameter optimization](#Hyperparameter-optimization) below.\n",
    "\n",
    "For small datasets, carving out dev and test sets might leave you with too little data. The most problematic symptom of this is that performance is highly variable because there isn't enough data to optimize reliably. In such situations, you might give up on having fixed splits, opting instead for some form of cross-validation, which allows you to average over multiple runs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Cross-validation\n",
    "\n",
    "In cross-validation, we take a set of examples $X$ and partition them into two or more train/test splits, and then we average over the results in some way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Random splits\n",
    "\n",
    "When creating random train/test splits, we shuffle the examples and split them, with a pre-specified percentage $t$ used for training and another pre-specified percentage (usually $1-t$) used for testing.\n",
    "\n",
    "In general, we want these splits to be __stratified__ in the sense that the train and test splits have approximately the same distribution over the classes.\n",
    "\n",
    "#### The good and the bad of random splits\n",
    "\n",
    "A nice thing about random splits is that you can create as many as you want without having this impact the ratio of training to testing examples. \n",
    "\n",
    "This can also be a liability, though, as there's no guarantee that every example will be used the same number of times for training and testing. In principle, one might even evaluate on the same split more than once (though this will be fantastically unlikely for large datasets). NLP datasets are generally large enough that this isn't a pressing concern.\n",
    "\n",
    "The function `utils.fit_classifier_with_hyperparameter_search` hard-codes a strategy of using random splits by using the [StratifiedShuffleSplit](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html#sklearn.model_selection.StratifiedShuffleSplit) utility. The benefit of decoupling the train/test ratio from the number of splits outweights the concerns about split composition.\n",
    "\n",
    "#### Random splits in scikit-learn\n",
    "\n",
    "In scikit-learn, the function [train_test_split](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) will do random splits. It is a wrapper around [ShuffleSplit](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ShuffleSplit.html#sklearn.model_selection.ShuffleSplit) or [StratifiedShuffleSplit](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html#sklearn.model_selection.StratifiedShuffleSplit), depending on how the keyword argument `stratify` is used. A potential gotcha for classification problems: `train_test_split` does not stratify its splits by default, whereas stratified splits are desired in most situations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### K-folds\n",
    "\n",
    "In K-fold cross-validation, one divides the data into $k$ folds of equal size and then conducts $k$ experiments. In each, fold $i$ is used for assessment, and all the other folds are merged together for training:\n",
    "\n",
    "$$\n",
    "\\begin{array}{c c c }\n",
    "\\textbf{Splits} & \\textbf{Experiment 1} & \\textbf{Experiment 2} & \\textbf{Experiment 3} \\\\\n",
    "\\begin{array}{|c|}\n",
    "\\hline\n",
    "\\textrm{fold } 1  \\\\\\hline\n",
    "\\textrm{fold } 2  \\\\\\hline\n",
    "\\textrm{fold } 3  \\\\\\hline\n",
    "\\end{array}\n",
    "& \n",
    "\\begin{array}{|c c|}\n",
    "\\hline\n",
    "\\textbf{Test} & \\textrm{fold } 1  \\\\\\hline\n",
    "\\textbf{Train} & \\textrm{fold } 2  \\\\\n",
    "& \\textrm{fold } 3  \\\\\\hline\n",
    "\\end{array}\n",
    "&\n",
    "\\begin{array}{|c c|}\n",
    "\\hline\n",
    "\\textbf{Test} & \\textrm{fold } 2  \\\\\\hline\n",
    "\\textbf{Train} & \\textrm{fold } 1  \\\\\n",
    "& \\textrm{fold } 3  \\\\\\hline\n",
    "\\end{array}\n",
    "&\n",
    "\\begin{array}{|c c|}\n",
    "\\hline\n",
    "\\textbf{Test} & \\textrm{fold } 3  \\\\\\hline\n",
    "\\textbf{Train} & \\textrm{fold } 1  \\\\\n",
    "& \\textrm{fold } 2  \\\\\\hline\n",
    "\\end{array}\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "#### The good and the bad of k-folds\n",
    "\n",
    "* With k-folds, every example appears in a train set exactly $k-1$ times and in a test set exactly once. We noted above that random splits do not guarantee this.\n",
    "\n",
    "* A major drawback of k-folds is that the size of $k$ determines the size of the train/test splits. With 3-fold cross validation, one trains on 67% of the data and tests on 33%. With 10-fold cross-validation, one trains on 90% and tests on 10%. These are likely to be __very__ different experimental scenarios. This is a consideration one should have in mind when [comparing models](#Classifier-comparison) using statistical tests that depend on repeated runs. This is a large enough drawback to k-folds that I generally favor random splits, [as discussed just above](#Random-splits)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### K-folds in scikit-learn\n",
    "\n",
    "* In scikit-learn, [KFold](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold) and [StratifiedKFold](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html#sklearn.model_selection.StratifiedKFold) are the primary classes for creating k-folds from a dataset. As with random splits, the stratified option is recommended for most classification problems, as one generally want to train and assess with the same label distribution.\n",
    "\n",
    "* The methods [cross_validate](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html#sklearn.model_selection.cross_validate) and [cross_val_score](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html#sklearn.model_selection.cross_val_score) are convenience methods that let you pass in a model (`estimator`), a dataset (`X` and `y`), and some cross-validation parameters, and they handle the repeated assessments. These are great. Two tips:\n",
    "  * I strongly recommend passing in a `KFold` or `StratifiedKFold` instance as the value of `cv` to ensure that you get the split behavior that you desire.\n",
    "  * Check that `scoring` has the value that you desire. For example, if you are going to report F1-scores, it's a mistake to leave `scoring=None`, as this will default to whatever your model reports with its `score` method, which is probably accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Variants  \n",
    "  \n",
    "K-folds has a number of variants and special cases. Two that frequently arise in NLU:\n",
    "\n",
    "1. [LeaveOneOut](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneOut.html#sklearn.model_selection.LeaveOneOut) is the special case where the number of folds equals the number of examples. This is especially useful for very small datasets.\n",
    "\n",
    "1. [LeavePGroupsOut](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeavePGroupsOut.html#sklearn.model_selection.LeavePGroupsOut) creates folds based on criteria that you define. This is useful in situations where the datasets have important structure that the splits need to respect – e.g., you want to assess against a graph sub-network that is never seen on training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Baselines\n",
    "\n",
    "Evaluation numbers in NLP (and throughout AI) __can never be understood properly in isolation__:\n",
    "\n",
    "* If your system gets 0.95 F1, that might seem great in absolute terms, but your readers will suspect the task is too easy and want to know what simple models achieve.\n",
    "\n",
    "* If your system gets 0.60 F1, you might despair, but it could turn out that humans achieve only 0.80, indicating that you got traction on a very challenging but basically coherent problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Baselines are crucial for strong experiments\n",
    "\n",
    "Defining baselines should not be an afterthought, but rather central to how you define your overall hypotheses. __Baselines are essential to building a persuasive case__, and they can also be used to illuminate specific aspects of the problem and specific virtues of your proposed system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Random baselines\n",
    "\n",
    "Random baselines are almost always useful to include. scikit-learn has classes [DummyClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html#sklearn.dummy.DummyClassifier) and [DummyRegressor](http://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyRegressor.html#sklearn.dummy.DummyRegressor). Each of them has a keyword argument `strategy` that allows you to specify a range of different styles of random guessing. I highly recommend using these in your work, for two central reasons:\n",
    "\n",
    "1. They will probably fit into your overall modeling pipeline.\n",
    "2. It's usual conceptually easy to describe these baselines but it can be tricky and error-prone to implement them – and the scikit-learn folks probably already did it for you flawlessly!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Task-specific baselines\n",
    "\n",
    "It is worth considering whether your problem suggests a baseline that will reveal something about the problem or the ways it is modeled. Two recent examples from NLU:\n",
    "\n",
    "1. As disussed briefly in [the NLI models notebook](nli_02_models.ipynb#Other-findings), [Leonid Keselman](https://leonidk.com/) observed [in his 2016 NLU course project](https://leonidk.com/stanford/cs224u.html) that one can do much better than chance on SNLI by processing only the hypothesis, ignoring the premise entirely. The exact interpretation of this is complex (we explore this a bit [in our NLI unit](nli_02_models.ipynb#Hypothesis-only-baselines) and [in our NLI bake-off](nli_wordentail.ipynb)), but it's certainly relevant for understanding how much a system has actually learned about reasoning from a premise to a conclusion.\n",
    " \n",
    "1. [Schwartz et al. (2017)](https://www.aclweb.org/anthology/W17-0907) develop a system for choosing between a coherent and incoherent ending for a story. Their best system achieves 75% accuracy by processing the story and the ending, but they achieve 72% using only stylistic features of the ending, ignoring the preceding story entirely. This puts the 75% – and the extent to which the system understands story completion – in a new light."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Hyperparameter optimization\n",
    "\n",
    "In machine learning, the __parameters__ of a model are those whose values are learned as part of optimizing the model itself. \n",
    "\n",
    "The __hyperparameters__ of a model are any settings that are set by a process that is outside of this optimization process. The boundary between a true setting of the model and a broader design choice will likely be blurry conceptually. For example: \n",
    "\n",
    "* The regularization term for a classifier is a clear hyperparameter – it appears in the model's objective function. \n",
    "* What about the method one uses for normalizing the feature values? This is probably not a setting of the model per se, but rather a choice point in your experimental framework.\n",
    "  \n",
    "For the purposes of this discussion, we'll construe hyperparameters very broadly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Rationale\n",
    "\n",
    "Hyperparameter optimization is one of the most important parts machine learning, and a crucial part of building a persuasive argument. For one angle on why, it's helpful to imagine that you're in an ongoing debate with a very skeptical referee:\n",
    "\n",
    "1. You ran experiments with models A, B, and C. For each, you used the default hyperparameters as given by the implementations you're using. You found that C performed the best, and so you reported that in your paper.\n",
    "1. Your reviewer doesn't have visibility into your process, and maybe doesn't fully trust you. Did you try any other values for the hyperparameters without reporting that? If not, would you have done that if C hadn't outperformed the others? There is no way for the reviewer (or perhaps anyone) to answer these questions.\n",
    "1. So, from the reviewer's perspective, all we learned from your experiments is that there is some set of hyperparameters on which C wins this competition. But, strictly speaking, this conveys no new information; we knew before you did your experiments that we could find settings that would deliver this and all other outcomes. (They might not be __sensible__ settings, but remember you're dealing with a hard-bitten, unwavering skeptic.)\n",
    "\n",
    "Our best response to this situation is to allow these models to explore a wide range of hyperparameters, choose the best ones according to performance on training or development data, and then report how they do with those settings at test time. __This gives every model its best chance to succeed.__\n",
    "\n",
    "If you do this, the strongest argument that your skeptical reviewer can muster is that you didn't pick the right space of hyperparameters to explore for one or more of the models. Alas, there is no satisfying the skeptic, but we can at least feel happy that the outcome of these experiments will have a lot more scientific value than the ones described above with fixed hyperparameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The ideal hyperparameter optimization setting\n",
    "\n",
    "When evaluating a model, the ideal regime for hyperparameter optimization is as follows:\n",
    "\n",
    "1. For each hyperparameter, identify a large set of values for it. \n",
    "2. Create a list of all the combinations of all the hyperparameter values. This will be the [cross-product](https://en.wikipedia.org/wiki/Cartesian_product) of all the values for all the features identified at step 1.\n",
    "3. For each of the settings, cross-validate it on the available training data.\n",
    "4. Choose the settings that did best in step 3, train on all the training data using those settings, and then evaluate that model on the test set.\n",
    "\n",
    "This is very demanding. First, The number of settings grows quickly with the number of hyperparameters and values. If hyperparameter $h_{1}$ has $5$ values and hyperparameter $h_{2}$ has $10$, then the number of settings is $5 \\cdot 10 = 50$. If we add a third hyperparameter $h_{3}$ with just $2$ values, then the number jumps to $100$. Second, if you're doing 5-fold cross-validation, then each model is trained 5 times. You're thus committed to training $500$ models.\n",
    "\n",
    "And it could get worse. Suppose you don't have a fixed train/test split, and you're instead reporting, say, the result of 10 random train/test splits. Strictly speaking, the optimal hyperparameters could be different for different splits. Thus, for each split, the above cross-validation should be conducted. Now you're committed to training $5,000$ systems!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Practical considerations, and some compromises\n",
    "\n",
    "The above is untenable as a set of laws for the scientific community. If we adopted it, then complex models trained on large datasets would end up disfavored, and only the very wealthy would be able to participate. Here are some pragmatic steps you can take to alleviate this problem, in descending order of attractiveness. (That is, the lower you go on this list, the more likely the skeptic is to complain!)\n",
    "\n",
    "1. [Bergstra and Bengio (2012)](http://www.jmlr.org/papers/v13/bergstra12a.html) argue that __randomly sampling__ from the space of hyperparameters delivers results like the full \"grid search\" described above with a relatively few number of samples. __Hyperparameter optimization algorithms__ like those implemented in [Hyperopt](http://hyperopt.github.io/hyperopt/) and [scikit-optimize](https://github.com/scikit-optimize/scikit-optimize) allow guided sampling from the full space. All these methods control the exponential growth in settings that comes from any serious look at one's hyperparameters. \n",
    "\n",
    "1. In large deep learning systems, __the hyperparameter search could be done on the basis of just a few iterations__. The systems likely won't have converged, but it's a solid working assumption that early performance is highly predictive of final performance. You might even be able to justify this with learning curves over these initial iterations.\n",
    "\n",
    "1. Not all hyperparameters will contribute equally to outcomes. Via heuristic exploration, it is typically possible to __identify the less informative ones and set them by hand__. As long as this is justified in the paper, it shouldn't rile the skeptic too much.\n",
    "\n",
    "1. Where repeated train/test splits are being run, one might __find optimal hyperparameters via a single split__ and use them for all the subsequent splits. This is justified if the splits are very similar.\n",
    "\n",
    "1. In the worst case, one might have to adopt hyperparameters that were optimal for other experiments that have been published. The skeptic will complain that these findings don't translate to your new data sets. That's true, but it could be the only option. For example, how would one compare against [Rajkomar et al. (2018)](https://arxiv.org/abs/1801.07860) who report that \"the performance of all above neural networks were [sic] tuned automatically using Google Vizier [35] with a total of >201,000 GPU hours\"?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Hyperparameter optimization tools\n",
    "\n",
    "* scikit-learn's [model_selection](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection) package has classes `GridSearchCV` and `RandomizedSearchCV`. These are very easy to use. (We used `GridSearchCV` in our course code, in `utils.fit_classifier_with_hyperparameter_search`.)\n",
    "\n",
    "* [scikit-optimize](https://github.com/scikit-optimize/scikit-optimize) offers a variety of methods for guided search through the grid of hyperparameters. [This post](https://roamanalytics.com/2016/09/15/optimizing-the-hyperparameter-of-which-hyperparameter-optimizer-to-use/) assesses these methods against grid search and fully randomized search, and it also provides [starter code](https://github.com/roamanalytics/roamresearch/tree/master/BlogPosts/Hyperparameter_tuning_comparison) for using these implementations with sklearn-style classifiers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Classifier comparison\n",
    "\n",
    "Suppose you've assessed two classifier models. Their performance is probably different to some degree. What can be done to establish whether these models are different in any meaningful sense?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Practical differences\n",
    "\n",
    "One very simple step one can take is to simply count up how many examples the models actually differ on. \n",
    "\n",
    "* If the test set has 1,000 examples, then a difference of 1% in accuracy or F1 will correspond to roughly 10 examples. We'll likely have intuitions about whether that difference has any practical import. \n",
    "\n",
    "* If the test set has 1M examples, then 1% will correspond to 10,000 examples, which seems sure to matter. Unless other considerations (e.g., cost, understandability) favor the less accurate model, the choice seems clear.\n",
    "\n",
    "Even where the numbers suggest a practical difference, we might still wonder whether the difference is stable across different runs, and thus we might still want to gather more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Confidence intervals\n",
    "\n",
    "If you can afford to run the model multiple times, then reporting confidence intervals based on the resulting scores could suffice to build an argument about whether the models are meaningfully different.\n",
    "\n",
    "The following will calculate a simple 95% confidence interval for a vector of scores `vals`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_ci(vals):\n",
    "    if len(set(vals)) == 1:\n",
    "        return (vals[0], vals[0])\n",
    "    loc = np.mean(vals)\n",
    "    scale = np.std(vals) / np.sqrt(len(vals))\n",
    "    return stats.t.interval(0.95, len(vals)-1, loc=loc, scale=scale)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's very likely that these confidence intervals will look very large relative to the variation that you actually observe. You probably can afford to do no more than 10–20 runs. Even if your model is performing very predictably over these runs (which it will, assuming your method for creating the splits is sound), the above intervals will be large in this situation. This might justify bootstrapping the confidence intervals. I recommend [scikits-bootstrap](https://github.com/cgevans/scikits-bootstrap) for this.\n",
    "\n",
    "__Important__: when evaluating multiple systems via repeated train/test splits or cross-validation, all the systems have to be run on the same splits. This is the only way to ensure that all the systems face the same challenges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Wilcoxon signed-rank test\n",
    "\n",
    "NLPers always choose tables over plots for some reason, and confidence intervals are hard to display in tables. This might mean that you want to calculate a p-value. \n",
    "\n",
    "Where you can afford to run the models at least 10 times with different splits (and preferably more like 20), [Demšar (2006)](http://www.jmlr.org/papers/v7/demsar06a.html) recommends the [Wilcoxon signed-rank test](https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test). This is implemented in scipy as [scipy.stats.wilcoxon](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html). This test relies only on the absolute differences between scores for each split and makes no assumptions about how the scores are distributed.\n",
    "\n",
    "Take care not to confuse this with [scipy.stats.ranksums](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ranksums.html), which does the Wilcoxon rank-sums test. This is also known as the [Mann–Whitney U test](https://en.wikipedia.org/wiki/Mann–Whitney_U_test), though SciPy distinguishes this as a separate test ([scipy.stats.mannwhitneyu](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html#scipy.stats.mannwhitneyu)). In any case, the heart of this is that the signed-rank variant is more appropriate for classifier assessments, where we are always comparing systems trained and assessed on the same underlying pool of data.\n",
    "\n",
    "Like all tests of this form, we should be aware of what they can tell us and what they can't: \n",
    "\n",
    "* The test says __nothing__ about the practical importance of any differences observed. \n",
    "\n",
    "* __Small p-values do not reliably indicate large effect sizes__. (A small p-value will more strongly reflect the number of samples you have.)\n",
    "\n",
    "* Large p-values simply mean that the available evidence doesn't support a conclusion that the systems are different, not that there is no difference in fact. And even that limited conclusion is only relative to this particular, quite conservative test. \n",
    "\n",
    "All this is to say that these values should not be asked to stand on their own, but rather presented as part of a larger, evidence-driven argument."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### McNemar's test\n",
    "\n",
    "[McNemar's test](https://en.wikipedia.org/wiki/McNemar%27s_test) operates directly on the vectors of predictions for the two models being compared. As such, it doesn't require repeated runs, which is good where optimization is expensive.\n",
    "\n",
    "The basis for the test is a contingency table with the following form, for two models A and B:\n",
    "\n",
    "$$\\begin{array}{|c | c |}\n",
    "\\hline\n",
    "\\textrm{number of examples}        & \\textrm{number of examples} \\\\\n",
    "\\textrm{where A and B are correct} & \\textrm{where A is correct, B incorrect} \n",
    "\\\\\\hline\n",
    "\\textrm{number of examples}        & \\textrm{number of examples} \\\\\n",
    "\\textrm{where A is correct, B incorrect} & \\textrm{where both A and B are incorrect} \\\\\\hline\n",
    "\\end{array}$$\n",
    "\n",
    "Following [Dietterich (1998)](http://sci2s.ugr.es/keel/pdf/algorithm/articulo/dietterich1998.pdf), let the above be abbreviated to\n",
    "\n",
    "$$\\begin{array}{|c | c |}\n",
    "\\hline\n",
    "n_{11} & n_{10}\n",
    "\\\\\\hline\n",
    "n_{01} & n_{00} \\\\\n",
    "\\hline\n",
    "\\end{array}$$\n",
    "\n",
    "The null hypothesis tested is that the two models have the same error rate, i.e., that $n_{01} = n_{10}$. The test statistic is\n",
    "\n",
    "$$\n",
    "\\frac{\n",
    "    \\left(|(n_{01} - n_{10}| - 1\\right)^{2}\n",
    "}{\n",
    "    n_{01} + n_{10}\n",
    "}$$\n",
    "\n",
    "which has an approximately chi-squared distribution with 1 degree of freedom. \n",
    "\n",
    "An implementation is available in this repository: `utils.mcnemar`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Assessing models without convergence\n",
    "\n",
    "When working with linear models, convergence issues rarely arise. Typically, the implementation has a fixed number of iterations it performs, or a threshold on the error, and the model stops when it reaches one of these points. We mostly don't reflect on this because of the speed and stability of these models.\n",
    "\n",
    "With neural networks, convergence takes center stage. The models rarely converge, or they converge at different rates between runs, and their performance on the test data is often heavily dependent on these differences. Sometimes a model with a low final error turns out to be great, and sometimes it turns out to be worse than one that finished with a higher error. Who knows?!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Incremental dev set testing\n",
    "\n",
    "The key to addressing this uncertainty is to __regularly collect information about dev set performance as part of training__. For example, after every epoch, one could make predictions on the dev set and store that vector of predictions, or just whatever assessment metric one is using. These assessments can provide direct information about how the model is doing on the actual task we care about, which will be a better indicator than the errors.\n",
    "\n",
    "All the PyTorch models for this course accept a keyword argument `early_stopping`. The behavior should closely resemble that of `sklearn.neural_network` models. If `early_stopping=True`, then part of the dataset given to the `fit` method is reserved for incremental testing. The amount can be controlled with `validation_fraction` (default: `0.10`). After every epoch, this data will be used to evaluate the model using its `score` method. The parameters of the best model are stored. If an improvement of more than `tol` (default: `1e-5`) isn't seen within `n_iter_no_change` steps (default: `10`), then optimization stops, and the parameters of the numerically best model seen are used as the final model.\n",
    "\n",
    "It's important to see just how different this dev set performance can be from the training loss. In particular, the training loss can continue to go down even as the model grows worse and worse in evaluations on held-out data. This is a common form of __over-fitting__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Dataset to illustrate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "SST_HOME = os.path.join('data', 'sentiment')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unigrams_phi(text):\n",
    "    return Counter(text.lower().split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = sst.build_dataset(\n",
    "    sst.train_reader(SST_HOME),\n",
    "    phi=unigrams_phi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev = sst.build_dataset(\n",
    "    sst.dev_reader(SST_HOME),\n",
    "    phi=unigrams_phi,\n",
    "    vectorizer=train['vectorizer'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model without early stopping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "mod_no_stopping = TorchShallowNeuralClassifier(\n",
    "    early_stopping=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Stopping after epoch 757. Training loss did not improve more than tol=1e-05. Final error is 0.00045882913400419056."
     ]
    }
   ],
   "source": [
    "_ = mod_no_stopping.fit(train['X'], train['y'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.624     0.598     0.611       428\n",
      "     neutral      0.264     0.231     0.247       229\n",
      "    positive      0.624     0.689     0.655       444\n",
      "\n",
      "    accuracy                          0.559      1101\n",
      "   macro avg      0.504     0.506     0.504      1101\n",
      "weighted avg      0.549     0.559     0.553      1101\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(dev['y'], mod_no_stopping.predict(dev['X']), digits=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Model with early stopping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod_stopping = TorchShallowNeuralClassifier(\n",
    "    early_stopping=True,\n",
    "    n_iter_no_change=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Stopping after epoch 82. Validation score did not improve by tol=1e-05 for more than 50 epochs. Final error is 0.1478035654872656"
     ]
    }
   ],
   "source": [
    "_ = mod_stopping.fit(train['X'], train['y'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.641     0.673     0.657       428\n",
      "     neutral      0.285     0.179     0.220       229\n",
      "    positive      0.644     0.736     0.687       444\n",
      "\n",
      "    accuracy                          0.596      1101\n",
      "   macro avg      0.523     0.529     0.521      1101\n",
      "weighted avg      0.568     0.596     0.578      1101\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(dev['y'], mod_stopping.predict(dev['X']), digits=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Errors vs. incremental performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtcAAAEKCAYAAADZxnkxAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABLbklEQVR4nO3deXxU5dn/8c81k31fCTsECDsCGnZwQVxrtS6tS7VWq1br+tQ+rT6/bvbp032xVlv3pdVqrXutO66A7KDsEPawhCQEQhKyzdy/P2bAEAIEyORk+b5fr3kxc+bMnG9COFy55zr3bc45RERERETk+Pm8DiAiIiIi0lGouBYRERERaSEqrkVEREREWoiKaxERERGRFqLiWkRERESkhai4FhERERFpISquRUSk2czscTPbYWZLG2zLMLN3zWxN+M90LzOKiHhJxbWIiByNJ4GzG227C5junMsDpocfi4h0SqZFZERE5GiYWV/gdefc8PDjVcCpzrltZtYN+NA5N8jLjCIiXonyOkBLysrKcn379vU6hojIUVuwYEGJcy7b6xzHKMc5tw0gXGB3OdSOZnYDcANAYmLiSYMHD26liCIiLedw5+wOVVz37duX+fPnex1DROSomdlGrzO0Bufcw8DDAPn5+U7nbBFpjw53zlbPtYiIHK+icDsI4T93eJxHRMQzKq5FROR4vQZcHb5/NfCqh1lERDyl4lpERJrNzJ4FPgUGmVmhmX0L+BVwhpmtAc4IPxYR6ZQ6VM+1iIhElnPu8kM8dXqrBhERaaM0ci0iIiIi0kJUXIuIiIiItBAV1yIiIiIiLUTFtbRJH6zawbKtu72OISJtyM7KWu59bzUrtpV7HUVE5JB0QaO0OdV1AW5+ZiHpCTG8991TiI/xex1JRNoAA+59bw1+M4Z0S/E6johIkzRyLcdtzrpSJv5yOmuLK1rk/T5cVUxVbYAtu/by1w8LWuQ9RaT9S0+MYXDXZD5dV+p1FBGRQ1JxLcft7WVFbN1dzfdf+JxA0B33+725dBvpCdF86YRuPPjxOjaWVrZAShHpCCb0z2TBxjJq6gNeRxERaZKKazluczeUkhwXxYKNZfzt0w3H9V7VdQGmr9jBWcO68uPzhhLtM3727+UtE1RE2r3x/TKpqQ/y2WZdkyEibZOKazku5dV1LN9azjWTcjl1UDa/eWsVm3dWHfP7zVhTQkVNPeeM6EZOShy3T8tj+sodTF9R1IKpRaS9GpebgRl8ulatISLSNqm4luOyYGMZQQfjczP4xYUj8PuMu176HOeOrT3kjSXbSI2PZmL/TAC+OTGX/tmJ3PPv5VTV1rdkdBFph9ISYhjSNYXZ6rsWkTZKxbUcl7nrdxLlM0b3Tqd7Wjx3nzuYmQWlPDNn01G/V019gHdXFHHG0Byi/aEfzZgoH/934Qg2l1Xxo1eWtXR8EWmHJvTPZMGmMqrr1HctIm2PiutOaOmW3cfVutHQ3PU7GdEzdf90eZeP6c2UvCz+9/XlRz0X7ayCUvZU13PuiK4HbB/fL5Nbp+bx4sJCXlhQ2CK5RaT9Gt8vk9r6IIs37/I6iojIQVRcdzLVdQGueGQ2//XPxS3yXp8X7mJsbsb+bT6f8cdLR5EaH83NzyykouaLVg7nHMu3llMXCDb5fm8s2UZybBSTBmQd9Nztp+cxvl8GP3plKWuK9hx3dhFpv8aq71pE2jAV153Mm0u3UV5dz/yNZQcVqdV1Aa56bA6/enPlQR+3VtcF+HDVjgMK40WbdlEXcIxrUFwDZCXFct/lo9lQWsn/vLQE5xxriyv4xuNzOfe+T7j68bmUV9cd8Jq6QJB3lhcxbWgOsVEHLxrj9xl/umw0CTF+bv7HQvbWHvvHwc45bvz7Ai57+FPu+fcy/jV/MxtKNN2fSHuRGh/NsO7quxaRtkkrNHYyz83dTPfUOIoranhu3mZ+dN7Q/c+9sKCQT9aU8MmaEt5etp1fXjSCkT3T+MfcTTz00Vp27Knh1qkDuPPMQUCoJcQMTuqTcdBxxvfL5M4zB/Hbt1dRWVPPx2uKiYvyc9X4Pjw7dxNfe/BTnrhmDN1S4ynYUcGv3lzB7r11nDO860HvtU9OShx/vHQUVz8xl9+9s+qA7Edj4aYy3lq2ndysRBZv3kR1XRC/z/ivaXncdOoA/D4DoD4Q5OVFW1iwsYzy6jr2VNcT5TPuOX84vTMTjunYItIyJvTL5KlZG6muCxAXrVVcRaTtUHHdiawrrmDO+p18/+xBLN2ym5cWFvL9swcRG+WnLhDkwY/WMrp3Gt87cxB3v7SEyx6eTWp8NLv31jGhXyZDuqXw4EdruWBUdwZ0SWbuhlKGdE0hNT66yePddEp/5q7fyfSVO7jkpJ784OzBZCfHctawrtz49AIu+ssspuRl8eLCLcRH+/n+2YM4Y2jOYb+Gkwdmc9mY3jw5awOXjunFwJzko/4+7Dvev2+dTHy0n/Ulldw3fQ2/e2c1H68u4fdfG8nizbv447urWVdSSWZiDOmJMSTHRVFQVMG3nprHS9+ZSHJc01+3iETe+H6ZPPLJehZuKmNi/4NbyUREvKLiuhN5fn4hfp9xyYk9Gd49lTeWbOftZUWcP7I7ry3eSmHZXu45fxiTBmTx1h1TuG96AZt2VvLNibmMzc2gpKKG03//Ef/v5aX8/VvjWLCxjMvG9D7k8Xw+46GrTmLb7mpysxL3b5+cl8U/vz2ea56Yx0sLt3DV+D7cOnUAmUmxzfo6/vusQbyxZBs/fW0Zz1w3DjNr9vegui7A659t5ezhXUmKDf34D+iSxJ8uG8Wpg7L58avLOOW3HxB0MCgnmYevOokzhubsP8asghK+8fhcbnt2EY9ePWb/KLeItK4xuRn4DGavLVVxLSJtiorrTqIuEOSFBYWcNqgLXVLiyEqKpWd6PM/N3cR5I7rxlw8LGNw1mamDuwCQEBPFXecMPuA9spJiueucwdz90hLu+fcyquuCB1zM2JS4aP8BhfU+w7qn8tYdJ1NTH6BbavxRfS0ZiTHceeZAfvzqMt5cup1zR3Rr9mvfX7mD8up6LjqxxwHbzYyLTuxJfp8M7p2+mpPzsvnyyO4HFc8TB2Tx0/OH8cNXlvKrN1fw/750bK0pInJ8UuKiGd4jlVlrS/mu12FERBrQBY2dxPsrd1BSUcNlY3oBoVHly8b0YtbaUh76eB1riyu5+bQBRxwFvjS/Fyf1Sd8/j/WYvocvrg8nIzHmqAvrfa4Y25sh3VL4+euHXlxmV1UtlTUHPvfSwkJyUmIPOdLVOzOBP3xtFF8Z3eOQo9JXju/D1RP68Mgn63l18ZZjyi8ix++Ugdks3FRGWWWt11FERPZTcd2O1QWClFTUNGvff87bTJfkWE4dlL1/21fze+H3Gb9+ayW5WYnNGgH2+Yz/u3A4UT6jX3Yi2cnNa+VoaVF+H/ecP4ytu6v539eX83nhLipr6gkEHR+u2sFNTy8g/+fvcd6fZ7BjTzUAJRU1fLiq+LCFc3P96LyhDMxJOqbFckSkZUwbkkPQhQYPRETaCrWFtGN/fHc1T83awPvfO5WclLhD7rextDJUcJ7anyj/F79P5aTEcdqgLry3ooibTunf7IJzcNcUfnPJCSTEePvjMzY3g0vze/Hs3M08O3czAEmxUVTU1JORGMOlY3rx0sItfOOxuTx3w3heW7yV+qDjotE9j/vYUX4fU/KyeXr2RmrqA01OHygikTWiRyo5KbG8t6KIi086/n/XIiItQcV1OxUIOl5YUEhlbYB731vDLy8a0eR+O8qrufrxuSTGRHH52IMvPrxjWh4ZidF8ZXSPJl59aBed2Db+I/vVxSO4/uRcCnZUsKaogq279zIlL5tpQ3KIifJxzvBuXPvkPL75xDyq6wIM75HCoK5HP8NIU8b0zeCxGetZumV3k9MRikhk+XzGtCE5vLxoi6bkE5E2Q20h7dSstSXs2FNDXpcknp+/mYIdFQftU1pRw9cfncOOPTU8ee0YeqYfPDfz8B6p/OaSkcREtc8fBTNjQJdkzh7ejVtPz+OXF53AuSO67f96Judl8ecrRrNky25Wbt/TIqPW++T3TQdg7vqyFntPETk604bmUFUb4FMtKCMibUREKyozO9vMVplZgZnd1cTzp5rZbjNbHL79uNHzfjNbZGavRzJne/Tyoi0kx0Xx1LVjiY/289u3Vx7w/O6qOq56bC6bdlbx6NX5nXpk9axhXfnD10YyunfaUY/QH05WUiz9shOZt2Fni72niBydif0zSYzx897yIq+jiIgAESyuzcwPPACcAwwFLjezpuYt+8Q5Nyp8+1mj524HVkQqY3tVVVvP20u386UR3eieFs8NJ/fj7WVFLNgYGkGdu34nF/5lJmt27OGhq07SHLDABaN68PJ3JpGRGNOi7zu2bwbzN+wkGHQt+r4i0jyxUX5OHpjNeyuK9O9QRNqESI5cjwUKnHPrnHO1wHPABc19sZn1BL4EPBqhfJ76x5xNXPHIbHZVHXoKqR17QjNhnH3vxwe0fby7vIjK2sD+UdjrpuSSlRTLL95YwY9eWcrXHvqU2kCQp64dy6mDukT8a+nMxvTNoLy6nlVFe7yOItJpnTE0h6LyGpZu3e11FBGRiF7Q2APY3OBxITCuif0mmNlnwFbge865ZeHt9wLfB1rm6rM25uPVxcxaW8qVj83hmW+NJzXhi6W0i/fU8NBHa3l6zkZq64PER/u54W/zefnmSaTGR/Pyoi30SItnbHiO6YSYKO6YlscPX1nKwk1lXDsplzvPHEhirK5XjbR9i+jM37CTId1SPE7T8qrrAvz7s60kxERx9vCuB8wos7c2wJtLt9E3K5ETe6cf8Lqi8mr+NX8zlbUBonyGz4zx/TKZ0D+ztb8E6QROG9QFv894d3kRJ/RM8zqOiHRykay+mprXrfFndguBPs65CjM7F3gFyDOz84AdzrkFZnbqYQ9idgNwA0Dv3odeirut2VZeTbfUOFZvr+Abj8/h79eNIxBwPPTxOp6atYGa+gAXju7JLVMHULynhq8/Opvbnl3Eby45gU/WlPDtk/vha1DoXDqmFzsra5k0IIuT+qQf5sjSknqmx5OTEsvcDWVcNaHv/u2BoCPoHFE+O6rl2duK8uo6np69kcdnbNg/l/qALkncfnoepw3uwj/mbOThj9dRUhH65GXygCxuOz2PvlkJPPjhOp6Zs5HaQJBon4/6YJCggydmrueTH0wlNT76gGM556gLOGrqA1TXBUmI8esXQzkq6Ykx5PdJ593lRdx55iCv44hIJxfJ/8EKgV4NHvckNDq9n3OuvMH9N8zsL2aWBUwCzg8X3HFAipk97Zy7svFBnHMPAw8D5Ofnt5uGu+3hKePOHtaVm55ZwIUPzKSovIbK2nouGNmd207Po192EgC5WYncc/5w/uflJVzxyGwCQceFjS7Mi/b7uO30PC++lE7NzBjTN4N563finMPMKCyr4qsPfsq23aHFa6L9xgk903jkG/kt3vMdCSu3l3PpQ7PZvbeOKXlZ3HTKKHbtrePe91Zz67OLiPIZ9UHHlLwsbjylP8u3lvPQx+v42kOfEuUzHHDR6B7cOjWP3pmhGWqWbtnNeX+ewRMz13PHtIH7j1VdF+CKR2azcNOu/dvio/385esnctpgtTRJ850xNIef/2cFm3dW0Svj4JmRRERaSySL63mERqFzgS3AZcAVDXcws65AkXPOmdlYQj3gpc65u4G7w/ucSqhd5KDCur2qDwQp3lNDt9Q4pg3N4YErTuS25xYxdXAX7pg2kIE5B3fCXDGuN8u37ebp2ZsY1j2FvCb2EW+Mzc3g9c+3UVi2l26pcdzx3GL2VNfzX9MGEggG2VsX4G+fbuTKR+fwj+vHkZbQdIFdFwiysbSKAV2SWvkr+EIg6PjBC5/j9xn/vmUyI3qm7n/u7GFd+c+SbcxeV8pFJ/bc/wnJpAFZXDWhD8/O3cTmnXv5xoQ+9M1KPOB9h/dI5cyhOTw2Yz3XTMrdP3p9//sFLNy0i2+f3I/s5Fhio/08N3cT1/9tPr/76sgWnd1FOrZTB3Xh5/9ZwYyCkibn9BcRaS0RK66dc/VmdgvwNuAHHnfOLTOzG8PPPwhcAtxkZvXAXuAy51y7GX0+VsUVNQQddE0Nrap45rCuLLvn7COukPiTLw/Db8bUITmtEVOaaUy4933u+p1s2lnF/I1l/OmyUVww6ovCcHJeNtc/NZ+rHpvL09eNO6g1oqq2nm//fQGfrCnh2evHt1hvcjDoDmgfOpInZq7ns8Ld/OmyUQcU1hBasOPLI7vz5ZHdD3pdXLSfayblHva975g2kHfu+4THZqznu2cMZPnWch78aC0Xn9iTu88dsn+/r4zqzvV/m88d/1xMWVXtEd830hZtKuOxGes5fUgXLmzBedKlZfXPTiQnJZaZKq5FxGMRnefaOfeGc26gc66/c+7/wtseDBfWOOfud84Nc86NdM6Nd87NauI9PnTOnRfJnK1tX7tAt9QvlixvztLj0X4f91wwnFMGZkcsmxy9QTnJJMdF8bdPN/Dn99dw0Yk9DiisAU4ZmM2DV53Iyu3lfOPxuQfM/rKrqpYrH53DzIIS0hKi+eWbK1pkSjHnHBc/OIvbnl3UrPfbvLOK37+zmqmDu3B+EwX08RraPYWzh3XliRnrKa2o4Qcvfk5aQjQ/Om/IAfslx0Xz5DVjOXNoDvf8eznfeWYBqxvMxrK7qo4/vruas/74Mb95ayWl4Z7wljZvw06uemwOF/5lFq9/vo0fvryUovLqiBxLjp+ZMbF/Fp+uLaUTjNGISBvWPpfla+e2h4vrrinxHieRluDzGfl90vmscDc90xP42QXDm9xv6uBQC9CKbeVM+8NHfO2hT3l+/mYufWg2S7eU85evn8QPvzSUzwt3858l244716LNu1i0aRevfbaVe99bfcBzW3bt5eZnFvLjV5fun6f7f15egt9n/PwrwyN2Eebt0/LYU1PP1x76lCVbdnPP+cObbJOJC/dd3356Hh+tKuasez/mtmcX8eu3VjLp1+/zp+lriIny8deP1jL51x/w89eXt1jh65zj9++s4qsPfsryreXcdc5g3rhtCnVBxy/f0LT7bdnE/pmUVtayuujgFWtFRFqLLsn3wP7iusHItbRvU/KymVFQwn2XjybpMDNdnDmsK7PumsoLCwp5du4mvv/C5yTE+HnimjFMGpBFIOh49JN1/PbtVZw1rOtxLUv/yqItxEb5OGtYV+57v4BBXVP40gnd+GzzLq7723wqa+oJBB1/+3QjWUkxlFTU8r8XDKN7WuR+6RvSLYVzR3TljSXbOWNoDueO6HrIfaP8Pv7rjIFcPbEvj3wSmkVnb12Ac0d045bTBjCkWwoFO/bwwAdreXzmep6ctYFzR3Tj6ol9ObF32jH9ghAMOn72+nKenLWBS/N78dPzhxEf4wfgxpP7cd/7BVw+tjfj+mlKwaaY2X8B1xGaGWoJcI1zrtWG+ycOCC2YNbOghEFddV2KiHjDOtLHZ/n5+W7+/PlexziiX7yxgidnbWDV/57dLqdpk4PVB4LsrKqlS3Lzf2EKBh1z1u8kKynmgAtUP1y1g28+MY+ffHnoMfcb19YHGfeL95icl83vvnoCVzwyh2Vbd3P76QP50/TVZCXF8sQ3x9A1NY53lxeF5rKOjeLPl40+qh7tY7GxtJI/vruau88dQk5K879fu6pqqaoNNFn8byyt5KlZG/nX/M3sqannhJ6pfGtyLueO6Ea0P/QLyopt5TzwQQFriiqYNCCL04d0YUzfjP2/wNQHgvzgxSW8uLCQ66fk8j/nDjng3+fe2gDT/vARyXFRvH7rZKL8LfvBn5ktcM7lt+ibtiIz6wHMAIY65/aa2fPAG865Jw/1mkics0/97QcM6JLMo1e322+liLQDhztna+TaA9t2h+a4VmHdcUT5fUdVWEOonaSpCxdPGZjNpAGZ3Dd9DRef1JOUuAMvfnx18RZ+/85qfv6V4Zx8iP77j1cXU1ZVx4WjuxMb5eevV57IBffP5NdvreTE3mk8/I18spJiAbjoxJ5cdGLrXajXJzORey8bfdSvS0uIIe0QM6z1yUzkx18eyp1nDuSlhYU8MXMDtz+3mF+/uZKvj+/D4s27eHd5EUmxUZzQM5Wn52zk8ZnriY/2kxwXOg3WBYKUVdXx3TMGcuvUAQf9+4yP8fOj84Zw49MLeXr2Rr7p8YWWbVQUEG9mdUACjaZfbQ0T+mfx+mdbqQ8EW/wXIBGR5lBx7YHtu/fS9ShG7KRzMTPuPmcI5/15Bpc9NJtvTc7lSyd0o6YuyA9fXcq/P9uKGfzm7ZVMyctq8pe0lxdvISMxhil5oeK7S3IcT107lneWbee6Kf2Ii/a39pfVKhJjo7hqQl++Pq4P76/cwSPhFpuUuNAqptdMzCU1IZqq2npmFpQys6CEmvrA/tePzc047IwgZw3rypS8LH755koKiiu4ekJfTYsZ5pzbYma/AzYRmv3pHefcO62dY9KATJ6du4mlW8sZ1SuttQ8vIqLi2gvby6sPWi5apKHhPVL546Ujuf/9Au7812f84o0VRPmN0opavnfmQDISY/mfl5fw0epiTh104GIr5dV1vLe8iMvG9NrfEgEwMCe5yTnUOyKfz5g2NIdpQ3PYWFpJZlLsAb3wCTFRnDE0hzOGHt20lmbG7786kt++vYrn5xfy9OxNTB6QxQ0n9zvkLzqdhZmlAxcAucAu4F9mdqVz7ulG+0V0Vd3x4X74mQUlKq5FxBP6zKyVBYOOot01uphRjujC0T1577un8Mx14zipTzq9MxJ46TsTuWVqHpec1JPuqXE88EHBQa97a8l2auqDWoAlrE9m4mEvMj1aXVLi+O1XR/LpXVP577MGsWbHHr7x+Fy+8pdZvLe8qDNPAzcNWO+cK3bO1QEvARMb7+Sce9g5l++cy8/ObvlpRbOSYhncNZlP15a2+HuLiDSHiutWtrOqltpAkG5qC5FmMDMmDcji4W/k868bJ3JCzzQAYqJ83HByP+ZtKGPOugOLiJcXbaFvZoJG7SIsMymWm08bwMffP41fXDiC0ooarvvbfL7ywEzqA0Gv43lhEzDezBIsNIR/OuDJ3IUT+2cxb8NOqusCR95ZRKSFqbhuZV9Mw6c5ruX4XDa2N1lJMdzfYPR64aYyZq8v5Suje3TqFoXWFBvl54pxvfnge6fyu6+OZOrgnE55IZ1zbg7wArCQ0DR8PuBhL7JM7J9JTX2QRZt2eXF4Eenk1HPdyjTHtbSUuGg/107O5TdvreLFBYW8s3w7by8rIj0hmktO0jLdrS3a7+v033fn3E+An3idY1y/DPw+Y9bakiZn5BERiaTON7zisW3lBy99LnKsrhrfh5S4KO7812fMWlvKHdPy+Oj7p9Ez/RBz1ol0Aslx0Qzvkcrsdeq7FpHWp5HrVrZ99178Pts/x7DI8UiOi+Y3l4xkbXEFV47rQ2pC9JFfJNIJjM/N4ImZG6iuC3TYqSdFpG3SyHUr27a7mpzkWPwRXgVPOo+zh3fl5tMGqLAWaWBsbga1AfVdi0jrU3HdyorKq8lRS4iISETl983ADOasV2uIiLQuFdetbN/S5yIiEjmp8dEM7ZbC3PU7vY4iIp2MiutW5Jxj++5quqZoGj4RkUgbm5vBwk1l1NZ3ynnHRcQjKq5b0Z6aeqpqAxq5FhFpBeNyM6muC/J54S6vo4hIJ6LiuhVpjmsRkdYzNjcDgDlqDRGRVqTiuhVtU3EtItJqMhJjGJiTpOJaRFqViutWtH33XgC6pqi4FhFpDWNzM1iwYSf1AfVdi0jrUHHdirbvrgEgR8W1iEirGJebSWVtgGVby72OIiKdhIrrVrS9fC9ZSbHEROnbLiLSGsbt77vWfNci0jpU5bWibbur6ZqqZc9FRFpLl5Q4crMSNd+1iLQaFdetSHNci4i0vnG5Gcxdv5NA0HkdRUQ6ARXXrWh7uVZnFBFpbeP6ZVBeXc+Kbeq7FpHIU3HdSor31LCrqo5eGRq5FhFpTRP6ZQHw6Vr1XYtI5EW0uDazs81slZkVmNldTTx/qpntNrPF4duPw9t7mdkHZrbCzJaZ2e2RzNkaZq0tAb44yYuISOvomhpHv+zE/edhEZFIiorUG5uZH3gAOAMoBOaZ2WvOueWNdv3EOXdeo231wJ3OuYVmlgwsMLN3m3htu/HJmhLSEqIZ2j3F6ygiIp3OhH6ZvLJoC3WBINF+fWgrIpETyTPMWKDAObfOOVcLPAdc0JwXOue2OecWhu/vAVYAPSKWNMKcc8xYU8Kk/ln4feZ1HBGRTmdi/ywqawMs2bLb6ygi0sFFsrjuAWxu8LiQpgvkCWb2mZm9aWbDGj9pZn2B0cCcpg5iZjeY2Xwzm19cXNwCsVve2uIKtpdXMzlPLSEiIl4Y3y8037X6rkUk0iJZXDc1RNt4HqSFQB/n3Ejgz8ArB7yBWRLwInCHc67Jy7ydcw875/Kdc/nZ2dnHnzoCZqwJ9flNHqDiWkTEC5lJsQzumqy+axGJuEgW14VArwaPewJbG+7gnCt3zlWE778BRJtZFoCZRRMqrJ9xzr0UwZwRN6OghD6ZCfTKSPA6iohIpzWhfybzN5RRUx/wOoqIdGCRLK7nAXlmlmtmMcBlwGsNdzCzrmZm4ftjw3lKw9seA1Y45/4QwYwRVxcIMnvdTo1ai4h4bGL/LGrqgyzatMvrKCLSgUWsuHbO1QO3AG8TuiDxeefcMjO70cxuDO92CbDUzD4D7gMuc845YBJwFTC1wTR950YqayR9tnkXFTX1Kq5FRDw2NjcDn8Es9V2LSARFbCo+2N/q8UajbQ82uH8/cH8Tr5tB0z3b7c4na0rwWWjEREREvJMaH83wHqnMXlsamiRWRCQCjjhybWYJZvYjM3sk/DjPzBrPSy2HMKOghBE900hNiPY6iohIpzehfyaLNpdRVVvvdRQR6aCa0xbyBFADTAg/LgR+HrFEHcie6joWb97F5AGZXkcRERFCnyLWBRzzN5R5HUVEOqjmFNf9nXO/AeoAnHN76SAtG5E2e91OAkHH5AFtc4pAEZHOZkzfdKJ8xkxNySciEdKc4rrWzOIJz1FtZv0JjWTLEazZsQeAkb1SPU4iIiIACTFRnNg7nVkFuqhRRCKjOcX1T4C3gF5m9gwwHfh+RFN1EKUVtSTG+EmIieh1oyIichQmDchi6dbdlFXWeh1FRDqgwxbXZuYD0oGLgG8CzwL5zrkPI56sAyitqCEzKdbrGCIi0sDkvEycg0/XafRaRFreYYtr51wQuMU5V+qc+49z7nXnnBrVmqm0spbMpBivY4iISAMje6aRFBvFJ2v035mItLzmtIW8a2bfM7NeZpax7xbxZB1ASUUtmYkqrkVE2pIov4/x/TKZWaDiWkRaXnOK62uBm4GPgQXh2/xIhuooSitqyExUW4iISFszeUAmm3ZWsam0yusoItLBHPFKO+dcbmsE6Wicc+xUW4iIdCJmlgY8CgwnNMPUtc65Tz0NdQiT80Kr5s5cW0LvzN4epxGRjqQ5KzRGm9ltZvZC+HaLmWm5wSMo31tPfdDpgkYR6Uz+BLzlnBsMjARWeJznkPpnJ5GTEssMtYaISAtrzhxxfwWigb+EH18V3nZdpEJ1BCWVoanAszRyLSKdgJmlACcTmlkK51wt0GbnujMzJg3I4oOVOwgGHT6f1kYTkZbRnJ7rMc65q51z74dv1wBjIh2svSutCP2fop5rEekk+gHFwBNmtsjMHjWzxMY7mdkNZjbfzOYXFxe3fsoGJg/IoqyqjuXbyj3NISIdS3OK60B4VUYAzKwfEIhcpI6htCI0cq2eaxHpJKKAE4G/OudGA5XAXY13cs497JzLd87lZ2dnt3bGA0weEOq7VmuIiLSk5hTX/w18YGYfmtlHwPvAnZGN1f6VVO4buVZxLSKdQiFQ6JybE378AqFiu83qkhLHwJwkTcknIi2qObOFTDezPGAQYMBK51xNxJO1c/tGrtNVXItIJ+Cc225mm81skHNuFXA6sNzrXEcyaUAW/5izieq6AHHRfq/jiEgH0JzZQm4G4p1znzvnPgMSzOw7kY/WvpVW1JKWEE20vzkfDoiIdAi3As+Y2efAKOAX3sY5spPzsqmpDzJ3/U6vo4hIB9Gcyu9659yufQ+cc2XA9RFL1EGUVtaoJUREOhXn3OJwP/UJzrmvhP+/aNPG9csgxu/j49XeXlwpIh1Hc4prn5ntn6PIzPyAqsYjKK2o1RzXIiJtXEJMFGNy0/lkjfquRaRlNKe4fht43sxON7OpwLPAW5GN1f6VVtZqjmsRkXbg5LxsVhXtYfvuaq+jiEgH0Jzi+gfAdOAm4Obw/e9HMlRHUFpRozmuRUTagSl5oSkBP16j1hAROX5HLK6dc0Hn3IPOuUsI9Vp/6pzTPNeHUR8IUlZVR4Z6rkVE2rwh3ZLJTo5Va4iItIjmzBbyoZmlmFkGsJjQ6lt/iHiydmxnVWiOa7WFiIi0fWbGlLwsZqwpJhB0XscRkXauOW0hqc65cuAi4Ann3EnAtMjGat/2L32uCxpFRNqFUwZmU1ZVx9Itu72OIiLtXHOK6ygz6wZ8DXg9wnk6hP3FtdpCRETahUnhpdA/Ud+1iByn5hTXPyM0Y0iBc26emfUD1kQ2VvtWWhlanVEj1yIi7UNWUizDe6Tw8Wr1XYvI8WnOBY3/Ci8I8J3w43XOuYub8+ZmdraZrTKzAjO7q4nnTzWz3Wa2OHz7cXNf25btG7lWz7WItFVm5jOziV7naEtOzstm4aYy9lTXeR1FRNqxiK3NHV5s5gHgHGAocLmZDW1i10+cc6PCt58d5WvbpNLKGqJ8RkpctNdRRESa5JwLAr/3OkdbcvLAbOqDjpkFGr0WkWMXseIaGEuolWSdc64WeA64oBVe67nSilrSE2Pw+ezIO4uIeOcdM7u44Sq8ndlJfdJJjY/mnWVFXkcRkXYsksV1D2Bzg8eF4W2NTTCzz8zsTTMbdpSvxcxuMLP5Zja/uLhtXIhSUlGrixlFpD34LvAvoNbMys1sj5mVex3KK9F+H6cP6cJ7K4qoCwS9jiMi7dQxFddmdk1zdmtiW+MJRBcCfZxzI4E/A68cxWtDG5172DmX75zLz87ObkasyCutrCFLFzOKSBvnnEt2zvmcc9HOuZTw4xSvc3nprGFdKa+uZ866nV5HEZF26lhHru9pxj6FQK8Gj3sCWxvu4Jwrd85VhO+/AUSbWVZzXtuWlVbUkqmLGUWkHTCz883sd+HbeV7n8drJednERft4e9l2r6OISDt1yOLazD4/xG0JkNOM954H5JlZrpnFAJcBrzU6Rtd9vX5mNjacp7Q5r23LSitqyEzUyLWItG1m9ivgdmB5+HZ7eFunFR/j55SB2byzfDtBrdYoIscg6jDP5QBnAWWNthsw60hv7JyrN7NbCM2R7Qced84tM7Mbw88/CFwC3GRm9cBe4DLnnAOafO3RfWneqK4LUFkb0Mi1iLQH5wKjwjOHYGZPAYuAdjX9aUs7a1hX3l5WxGeFuxjdO93rOCLSzhyuuH4dSHLOLW78hJl92Jw3D7d6vNFo24MN7t8P3N/c17YHpZWa41pE2pU0YF+DcaqHOdqM0wfnEOUz3l5WpOJaRI7aIdtCnHPfcs7NOMRzV0QuUvtWWhFanTFDbSEi0vb9AlhkZk+GR60XhLd1aqkJ0Yzvl8k7y7YT+jBVRKT5DtdzfVGD+/rVvZn2rc6othARacvMzAcEgfHAS+HbBOfcc54GayPOGpbDupJKCnZUeB1FRNqZw80W8sMG96dHOkhHURIeuc7SyLWItGHhPutbnHPbnHOvOededc5pioywM4Z2BdCsISJy1A5XXNsh7sth7Ou51si1iLQD75rZ98ysl5ll7Lt5Haot6Joax+jeafxniYprETk6h7ugMd7MRhMqwOPC9/cX2c65hZEO1x6VVtQQF+0jIcbvdRQRkSO5NvznzQ22OaCfB1nanPNHdueefy9nTdEe8nKSvY4jIu3E4YrrbcAfwve3N7gPoZPv1EiFas9KK2rJTIwlPH23iEibFO65vss590+vs7RV553Qnf99fTmvLN7Cf5812Os4ItJOHLK4ds6d1ppBOoqSylpNwycibZ5zLmhmNwMqrg8hOzmWSQOyeHXxVr535iANmohIsxzr8udyCDsra8hM0sWMItIuqOf6CL4yqgeFZXtZuKnxemoiIk1Tcd3CtpTtJSclzusYIiLNcS2hfuuPCc1xvQCY72miNuas4V2JjfLx6uKtXkcRkXZCxXULKquspayqjv7ZiV5HERE5IudcbhM3XczYQFJsFNOG5vD659uoCwS9jiMi7UCzimszO9/Mfhe+fTnSodqrdSWVAORmqbgWkbbLzL7f4P5XGz3X6VdobOwro3qws7KWGQUlXkcRkXbgiMW1mf0SuB1YHr7dFt4mjaxXcS0i7cNlDe7f3ei5s1szSHtwysBsUuOjeXXRFq+jiEg7cLip+Pb5EjAqvJoXZvYUsIiDT8id3vqSCqJ8Rq+MBK+jiIgczuEWCdOUGI3ERPk4d0Q3Xl28haraehJimvNfp4h0Vs3tuU5rcD81Ajk6hPUllfTOSCDar1Z2EWnT3CHuN/VYgItO7EFVbYA3tGKjiBxBc379/gWwyMw+IDSicTIatW7SuuJKtYSISHsw0szKCZ3T48P3CT/WdEdNyO+TTr+sRJ6ft5lLTurpdRwRacMOO8QaXsErCIwHXgrfJjjnnmuFbO1KMOhYX6LiWkTaPuec3zmX4pxLds5Fhe/vexztdb62yMz4an4v5m7YybriCq/jiEgbdtjiOtxnfYtzbptz7jXn3KvOOX0m1oRt5dXU1AfJ1TR8ItKJmZnfzBaZ2eteZ2lpF5/YA7/PeH5+oddRRKQNa05zsFbwaob1xaGZQvplJXmcRETEU7cDK7wOEQldUuI4bVA2Ly4spF5zXovIITSnuNYKXs2wviT0MWE/jVyLSCdlZj0JzTD1qNdZIuVr+b0o3lPDh6uKvY4iIm3UES9odM7ltkaQ9m5tcSUJMX66JMd6HUVExCv3At8Hkg+1g5ndANwA0Lt379ZJ1YJOG9yFrKRY/jl/M9OG5ngdR0TaoOYsInOzmaU1eJxuZt+JaKp2aN/FjGaaIlZEOh8zOw/Y4ZxbcLj9nHMPO+fynXP52dnZrZSu5UT7fVx8Ug/eX7mDHXuqvY4jIm1Qc9pCrnfO7dr3wDlXBlwfsUTt1PqSSvplq99aRDqtScD5ZrYBeA6YamZPexspMr6W34tA0PHCAl3YKCIHa05x7bMGw7Fm5gdiIhep/ampD1BYVqVp+ESk03LO3e2c6+mc60toefX3nXNXehwrIvpnJzFpQCZPzdpATX3A6zgi0sY0p7h+G3jezE43s6nAs8BbkY3VvmzeWUXQQT8V1yIincJNpwygqLyGlxdu8TqKiLQxzSmufwC8D9xEaNaQ6YQuWJGwteFp+DRyLSICzrkPnXPneZ0jkiYNyGREj1Qe+ngdgaBWjBeRLxyxuHbOBZ1zf3XOXeKcu9g595Bzrlmfg5nZ2Wa2yswKzOyuw+w3xswCZnZJg23/ZWbLzGypmT1rZm12Sd71JeHiWtPwiYh0CmbGTaf2Z31JJW8v09pqIvKF5swWkmdmL5jZcjNbt+/WjNf5gQeAc4ChwOVmNvQQ+/2aUPvJvm09gNuAfOfccMBPqIevTVpfXElWUiwpcVo1WESkszhrWFdysxL564drcU6j1yIS0py2kCeAvwL1wGnA34C/N+N1Y4EC59w651wtoavHL2hiv1uBF4EdjbZHAfFmFgUkAFubcUxPrC+pVL+1iEgn4/cZ3z65H0u27GZmQanXcUSkjWhOcR3vnJsOmHNuo3Pup8DUZryuB7C5wePC8Lb9wiPUFwIPNtzunNsC/A7YBGwDdjvn3mnGMT2xrqRC/dYiIp3QhSf2oEtyLH/5sMDrKCLSRjSnuK42Mx+wxsxuMbMLgS7NeF1Tq6k0/tzsXuAHjXu4zSyd0Ch3LtAdSDSzJqd0MrMbzGy+mc0vLm795Wh3762jpKJWy56LiHRCsVF+rp/Sj1lrS1m4qczrOCLSBjSnuL6DUFvGbcBJwFXA1c14XSHQq8Hjnhzc2pEPPBdedOAS4C9m9hVgGrDeOVfsnKsDXgImNnUQr1f72lCimUJERDqzK8b1Jj0hmj9PX+N1FBFpA6KOtINzbl74bgVwzVG89zwgz8xygS2ELki8otF75+67b2ZPAq87514xs3HAeDNLAPYCpwPzj+LYrWbNjgoA+nfR6owiIp1RYmwU103px2/fXsXSLbsZ3iPV60gi4qFDFtdm9trhXuicO/8Iz9eb2S2EZgHxA48755aZ2Y3h5x88zGvnmNkLwEJCF1IuAh4+3PG8srpoDzFRPvpmauRaRKSzumpCHx76aC1/fn8ND12V73UcEfHQ4UauJxC6IPFZYA5N91AflnPuDeCNRtuaLKqdc99s9PgnwE+O9pitbdX2PQzITsLvO+pvj4iIdBApcdF8c1Iu901fw8rt5QzumuJ1JBHxyOF6rrsC/wMMB/4EnAGUOOc+cs591Brh2oPVRXsY1DXZ6xgiIuKxayf1JTHGz/3va+YQkc7skMW1cy7gnHvLOXc1MB4oAD40s1tbLV0bV15dx7bd1QzMUXEtItLZpSXE8I2JffnPkm0UhK/HEZHO57CzhZhZrJldBDwN3AzcR2jmDgHWFO0BYFBXXcwoIiLwrcm5JET7+eUbK7yOIiIeOWRxbWZPAbOAE4F7nHNjnHP/G17gRYBV20MjExq5FhERgKykWG6flsf0lTt4f2WR13FExAOHG7m+ChgI3A7MMrPy8G2PmZW3Try2bXXRHhJj/PRIi/c6ioiItBHfnJhLv+xEfvbv5dTUB478AhHpUA7Xc+1zziWHbykNbsnOOV0GTWimkLycZMw0U4iIiITERPn46ZeHsaG0ikc/We91HBFpZc1ZoVEOYc2OPQxSS4iIiDRy8sBszhqWw/3vF7B1116v44hIK1JxfYxKKmooqahloKbhExGRJvzwS0MJOsfP/7Pc6ygi0opUXB+j1ftmCtHItYiINKFXRgK3nZ7HG0u28+pizQUg0lmouD5Gq7eHiuuBmoZPREQO4dsn9+OkPun88JWlag8R6SRUXB+jVUUVpCVEk50U63UUERFpo6L8Pv7wtZEEg447n/+MYNB5HUlEIkzF9TFaXbSHgZopREREjqBPZiI//vJQPl1XyuMzNXuISEen4voYOOdYXaSZQkREpHm+lt+LaUNy+M1bq1i5XUtFiHRkKq6PwfbyavZU12umEBERaRYz41cXjyAlPoo7nlusxWVEOjAV18dg1XbNFCIiIkcnKymW31xyAiu37+H376z2Oo6IRIiK62Owbxq+gTmaKURERJpv6uAcvj6uN498so5P15Z6HUdEIkDF9TFYvrWcLsmxpCXEeB1FRETamf/3pSH0zUzkzucXs3tvnddxRKSFqbg+Sruqanl7WRGnDMz2OoqIiLRDCTFR/PHSURTtqeGuFz/X9HwiHYyK66P03LzN7K0LcM2kXK+jiIhIOzWqVxp3nT2YN5du577313gdR0RaUJTXAdqTukCQp2ZtYGL/TIZ2T/E6joiItGPXTcllxfZy7n1vDYNykjlnRDevI4lIC9DI9VF4a+l2tu2u5lqNWouIyHEyM35x4QhG907ju89/xrKtu72OJCItQMX1UXh85nr6ZiYwdXAXr6OIiEgHEBft56GrTiItIZrrn5rP+pJKryOJyHFScd1MCzeVsWjTLq6ZlIvPpyXPRUSkZXRJjuOxq8dQXR/k4r/O4rPNu7yOJCLHQcV1Mz0+Yz3JcVFcclJPr6OIiEgHM7R7Ci/eNJHEWD+XPzKbj1YXex1JRI6RiutmKKmo4c2l27l8bG8SY3UNqIhIY2bWy8w+MLMVZrbMzG73OlN7k5uVyIs3TaRvZiLfenIe//l8m9eRROQYqLhuhpkFJQSCjvNO0JXcIiKHUA/c6ZwbAowHbjazoR5nane6JMfxz2+PZ3TvNO745yI+XLXD60gicpQiWlyb2dlmtsrMCszsrsPsN8bMAmZ2SYNtaWb2gpmtDI+ETIhk1sP5eHUJaQnRDOue6lUEEZE2zTm3zTm3MHx/D7AC6OFtqvYpOS6ax745hoE5ydz49ALmbdjpdSQROQoRK67NzA88AJwDDAUub2oUI7zfr4G3Gz31J+At59xgYCShE3Wrc84xo6CYSf2z8OtCRhGRIzKzvsBoYE4Tz91gZvPNbH5xsfqKDyUlLpqnrh1L99R4rn1iHku3aJo+kfYikiPXY4EC59w651wt8BxwQRP73Qq8COz/7MvMUoCTgccAnHO1zrldEcx6SAU7Kigqr2FyXpYXhxcRaVfMLInQOf0O51x54+edcw875/Kdc/nZ2dmtH7AdyUqK5e/XjSM5LoorH5vD3PUawRZpDyJZXPcANjd4XEijjwjNrAdwIfBgo9f2A4qBJ8xskZk9amaJTR0k0qMgn6wpAWDyABXXIiKHY2bRhArrZ5xzL3mdpyPokRbPczdMICMhhisfncNrn231OpKIHEEki+umeihco8f3Aj9wzgUabY8CTgT+6pwbDVQCTfZsR3oUZEZBCblZifTKSGjx9xYR6SjMzAh92rjCOfcHr/N0JL0zE3jxpomM6pXGbc8u4oEPCnCu8X+nItJWRLK4LgR6NXjcE2j8K3c+8JyZbQAuAf5iZl8Jv7bQObevX+8FQsV2q6qtDzJ7XalGrUVEjmwScBUw1cwWh2/neh2qo0hPjOHv143l/JHd+e3bq/jG43PZVFrldSwRaUIkJ22eB+SZWS6wBbgMuKLhDs653H33zexJ4HXn3Cvhx5vNbJBzbhVwOrA8glmbtHBTGVW1AfVbi4gcgXNuBk1/YiktJDbKz72XjuKkPun89u1VnHnvR9wxbSDfmpxLtF8z64q0FRH71+icqwduITQLyArgeefcMjO70cxubMZb3Ao8Y2afA6OAX0Qq66HMWFOC32dM6J/Z2ocWERE5iM9nXD2xL+9+92Sm5GXzqzdXcvFfZ7GxtNLraCISFtHlBp1zbwBvNNrW+OLFfdu/2ejxYkJtI575pKCEUb3SSImL9jKGiIjIAbqlxvPIN/J5Y8k27nrxc7503wx+edEIvjyyu9fRRDo9fY50CLuqallSuEv91iIi0madO6Ibb9w+hbycJG59dhF3v/Q5lTX1XscS6dRUXB/CrLWlBB1MUb+1iIi0YT3TE3j+2xO48ZT+PDdvM2fd+zEzwtPIikjrU3F9CIs37yImysfIXmleRxERETmsaL+Pu84ZzPPfnkCM38eVj83h7pc+Z3dVndfRRDodFdeHsKVsLz3S4nUFtoiItBtj+mbwxu1T+PbJ/fjnvM1M/NV0/u8/y9m2e6/X0UQ6DVWOh7BlV6i4FhERaU/iov3cfe4Q/nPbFKYNzeHxmRuY8usPuOvFz9lVVet1PJEOT8X1IWzZtZfuaXFexxARETkmQ7ql8KfLRvPh907lyvF9eGFBIWf88WPeW17kdTSRDk3FdRNq6gMU76mhR5qWPBcRkfatV0YCPz1/GK/eMonMxBiu+9t8vvv8YrbsUquISCREdJ7r9mrbrmoAjVyLiEiHMax7Kq/dMpn731/DAx+u5eVFW5iSl83X8ntyxtAcYqP8XkcU6RBUXDdha/i3+R7p6rkWEZGOIybKx3fPHMRX83vxrwWFvDB/M7f8YxHpCdF8ZXQPLh3Ti8FdU7yOKdKuqbhuQuG+4loXNIqISAfUKyOB754xkNtPz2NmQQn/nL+ZZ2Zv4omZGxjZK40bpvTj7OFd8fvM66gi7Y6K6yZs3bUXs9DysiIiIh2V32ecPDCbkwdms7OyllcWbeHp2Ru5+R8L6Z+dyHdOHcD5o7prWlqRo6DiuglbyvbSJTmWmCidTEREpHPISIzh2sm5XD2xL28u3cb97xdw578+4yevLWN8v0xOHpjFKQOz6ZOZ6HVUkTZNxXUTtu7eS3e1hIiISCfk9xnnndCdL43oxoeri3lnWREzCop5b0VoCr9RvdK4+KSenH9Cd1IToj1OK9L2qLhuwpayvQzvkep1DBEREc+YGacN6sJpg7oAsLG0kreXbefFBVv40StL+d9/Lye/bzrjcjMZ1y+DUb3SiIvWjCMiKq4bCQYdW3dXc9awrl5HERERaTP6ZCZyw8n9uX5KP5ZtLeeVRVuYubaUe6evxr0H8dF+JudlcfrgLkwd0oUuyZrOVjonFdeNlFTWUFsf1DR8IiIiTTAzhvdI3f8J7+6qOuZu2MnHq4uZvqKId8MrQA7umszkAVlMGpDF2NwMEmNVckjnoJ/0RrbuW0BGM4WIiIgcUWpCNGcMzeGMoTn87IJhrNy+h/dX7mDW2hL+Nnsjj85YT7TfGN07nckDspicl8XInmma5k86LBXXjWwp0wIyIiIix8LMGNIthSHdUrj5tAFU1wWYv6GMGQUlzCgo5o/vreYP764mIzGG0wZ14fQhXRjRI5WuqXGa7k86DBXXjWzZVQWg2UJERESOU1y4D3tyXhYwmJ2VtXyyppgPVu7gvRVFvLiwEACfQU5KHLlZiUzol8nEAZmc0DNNBbe0SyquG9m6q5rk2ChS4zW9kIiISEvKSIzhglE9uGBUD+oDQRZv3sXa4gq2lO1ly65qVmwr5w/vreb370JCjJ9RvdIY3TuN0b3SGdI9hZzkWKJUcEsbp+K6kcIyzXEtIiISaVF+H/l9M8jvm3HA9rLKWmavK+XTdaUs3FTGgx+tIxB0wBcj3D3T4zmhZxr5fdI5qU86XVI0M4m0HSquG9m6a6/6rUVERDySnhjDOSO6cc6IbgDsrQ2wZMtu1hZXsG1XaIR7Y2klT8/eyGMz1gOQlRRLv6xE+mYl0DUljur6IJU19VTXBRnSLZnTBnehX1YiZrqIUiJPxXUjW3bt5cQ+aV7HEBERESA+xs/Y3AzG5h44wl1bH2TZ1t0s2FjGmqIK1pdU8v7KYkoqaoiP9pMQ4yfKb7y4sJCf/2cFvTMSGN8vg9ysJHLDhXh2UixpCTGauURalIrrBipq6tm9t44eaQleRxEREZHDiInyMbp3OqN7px+w3Tl3wAj15p1VfLg6dBHl9BU7KK0sPGB/n0F6QgxdU+Pom5lI78wEeqTFkxQbRXxMqEjvkhxqRdFc3dIcnf6npOE/wq27NA2fiIhIe9a49aNXRgJXje/DVeP7AFBeXceGkko2llZRWlFDaWUtJRW1bNu9l+Xbynl72Xbqwz3ejaUnRNM9LZ4uybFkh29dkuPISYmlS0oc2UmxZCbFkBDT6curTq1T/+3PKijhZ68v55nrxpGZFMuWfcV1mi6MEBER6YhS4qI5oWcaJ/RMa/L5+kCQ4ooaqmoD7K0NUFUboKi8msKyvWwuq2Lrrr0UV9SwfFs5JRW1+y+2bCgu2kdmYizpidGkJ8TsHxnvmR5Pz/R4clLiSIyJIiHGT3yMn9goP9F+U094BxHR4trMzgb+BPiBR51zvzrEfmOA2cClzrkXGmz3A/OBLc6581o6X0ZSDOuKK/mfl5fw4JUnfbGAjNpCREREOqUov49uzVylORh07Kyqpai8mh3lNRRX1LCzspadlbWUVtRSVhW6bdpZxdvLqqmpDx72/WKifCTHRtElJY6uKbHkpMTRIy2eHunx9ExPICX+i7LNb0ZyXDQp8VHER/tVmLchESuuw4XxA8AZQCEwz8xec84tb2K/XwNvN/E2twMrgJRIZBzcNYU7zxzIL99cyYsLt7B1116ifEZ2cmwkDiciIiIdiM9nZCXFkpUUy7Duh9/XOUdxRQ2FZXvZUV5NVXhUfG9tgJr6ALX1QWrqg5RX17GjvIYde2pYsqWckoqaI+aI9hvpCTFkJsWSlRQaKU+NDxXeqfHRJMZGkRS+Nb6fGOsnLsqPTxd1tphIjlyPBQqcc+sAzOw54AJgeaP9bgVeBMY03GhmPYEvAf8HfDdSIa+b0o/pK3fw09eWMbRbCt3S4nTVsIiIiLQoM6NLchxdko+u9bS6LsDWXXvZXLaXqpr6/dvrg4491aGJGHbvraOsspbSyhpKKkIj5eV76yivrm+ybaUp8dF+EmNDbSr7WlZS4qNJjosmOS6KuCg/0VFGrN9HbLSf5LgvCvSE8IWfcdF+4qP9xEb7iYvyER3lw7nQCH/QORJjo4iL9h/V198eRbK47gFsbvC4EBjXcAcz6wFcCEylUXEN3At8H0g+3EHM7AbgBoDevXsfdUi/z/j9V0dyzp8+Ye6GnYxrNNWPiIiIiFfiov30y06iX3bSUb/WOUdlbYDKmnr2VNdTUVNPZU3oz4rqeipr6xuMoH9xv7Im9NzOylo2lFSyp7qemvogtYEgtUdobTmS2Cgf6QkxJMVFEe33EeM3ov2+/UV9YmwUMVFGMAgOh8+M1PhoUhOiSYuPISHGT7TfR7Tf8PuMQNARdKGvNTbaR0LMgaPyybHRxEX7MDOccwSCDr8vsv3tkSyum0rd+Nene4EfOOcCDb9IMzsP2OGcW2Bmpx7uIM65h4GHAfLz85v361kjvTIS+MmXh/LfL3yumUJERESkQzCz/S0gOS3UYOuco6Y+uL9Ar6ipZ2/dFwV6dV2QmvoA1XVB6gJBzAyfgc+Mytp6dlfVsauqjoqaemoDoX1q64Psqa6nqLyayppA+HVgGAHn2L237riK+n0lpgtXiT5j//clKS6Kf1w/nqyklmsJjmRxXQj0avC4J7C10T75wHPhwjoLONfM6gmNcJ9vZucCcUCKmT3tnLsyUmEvOaknO/bUaORaROQYNfcidhFpv8yMuOhQC0hLFqRHsrc2QFlVLdV1AeqDjtr64P5RaJ8ZZlBTH6QqPDJfWVtPRU2Aiup6qmrrsXB2v8+o3ffLQfgXhJZuVYlkcT0PyDOzXGALcBlwRcMdnHO5++6b2ZPA6865V4BXgLvD208FvhfJwjp8HG4+bUAkDyEi0mE19yJ2EZFjER/jJz6mfXQX+CL1xs65euAWQrOArACed84tM7MbzezGSB1XREQ8sf8idudcLbDvInYRkU4lovNcO+feAN5otO3BQ+z7zUNs/xD4sIWjiYhIyzriRexw/Behi4i0dREbuRYRkU6lORex45x72DmX75zLz87OboVYIiKtS8W1iIi0hOZcxC4i0uGpuBYRkZaw/yJ2M4shdBH7ax5nEhFpdRHtuRYRkc7BOVdvZvsuYvcDjzvnlnkcS0Sk1am4FhGRFtHURewiIp2N2kJERERERFqIOXdMK4a3SWZWDGw8ypdlASURiHMs2koW5TiQchyoreSAtpOlJXL0cc51qukzjvGcDR3r770lKMeBlONgbSVLR8pxyHN2hyquj4WZzXfO5XudA9pOFuVQjvaQA9pOlraSo7NoK99v5VCO9pAD2k6WzpJDbSEiIiIiIi1ExbWIiIiISAtRcQ0Pex2ggbaSRTkOpBwHais5oO1kaSs5Oou28v1WjgMpx4HaSg5oO1k6RY5O33MtIiIiItJSNHItIiIiItJCVFyLiIiIiLSQTl1cm9nZZrbKzArM7K5WPO7jZrbDzJY22JZhZu+a2Zrwn+mtkKOXmX1gZivMbJmZ3e5FFjOLM7O5ZvZZOMc9XuRokMdvZovM7HWPc2wwsyVmttjM5nuVxczSzOwFM1sZ/lmZ4MHPyKDw92HfrdzM7vDo+/Ff4Z/TpWb2bPjn15Ofkc7Gq3N2+Nien7d1zj5kHp2zD8yhc/aBWVr9nN1pi2sz8wMPAOcAQ4HLzWxoKx3+SeDsRtvuAqY75/KA6eHHkVYP3OmcGwKMB24Ofw9aO0sNMNU5NxIYBZxtZuM9yLHP7cCKBo+9ygFwmnNuVIP5OL3I8ifgLefcYGAkoe9Nq+Zwzq0Kfx9GAScBVcDLrZ3DzHoAtwH5zrnhgB+4rLVzdEYen7OhbZy3dc5ums7ZB9I5O8yzc7ZzrlPegAnA2w0e3w3c3YrH7wssbfB4FdAtfL8bsMqD78mrwBleZgESgIXAOC9yAD3D/9CmAq97+XcDbACyGm1r1SxACrCe8MXPXuVodOwzgZkefT96AJuBDCAKeD2cx/N/vx395vU5O3zMNnXe1jlb5+wmMuicfeCxPTlnd9qRa774hu9TGN7mlRzn3DaA8J9dWvPgZtYXGA3M8SJL+GO9xcAO4F3nnCc5gHuB7wPBBtu8+rtxwDtmtsDMbvAoSz+gGHgi/LHro2aW6EGOhi4Dng3fb9UczrktwO+ATcA2YLdz7p3WztFJtbVzNnj4965z9n73onN2QzpnN+DVObszF9fWxLZOOS+hmSUBLwJ3OOfKvcjgnAu40MdHPYGxZja8tTOY2XnADufcgtY+9iFMcs6dSOhj8JvN7GQPMkQBJwJ/dc6NBirxsOXBzGKA84F/eXT8dOACIBfoDiSa2ZVeZOmEdM4O0zk7ROfsJumcfeDxPTlnd+biuhDo1eBxT2CrR1kAisysG0D4zx2tcVAziyZ0kn7GOfeSl1kAnHO7gA8J9Ta2do5JwPlmtgF4DphqZk97kAMA59zW8J87CPWqjfUgSyFQGB6VAniB0Inbq5+Rc4CFzrmi8OPWzjENWO+cK3bO1QEvARM9yNEZtbVzNnjw965z9gF0zj6YztkH8uSc3ZmL63lAnpnlhn+zugx4zcM8rwFXh+9fTaiXLqLMzIDHgBXOuT94lcXMss0sLXw/ntA/hpWtncM5d7dzrqdzri+hn4f3nXNXtnYOADNLNLPkffcJ9Ygtbe0szrntwGYzGxTedDqwvLVzNHA5X3y8iAc5NgHjzSwh/O/ndEIXC3n1/ehM2to5G1r/XKlzdgM6Zx9M5+yDeHPOjmQjeVu/AecCq4G1wP9rxeM+S6j3p47Qb5nfAjIJXZSxJvxnRivkmEzoY9XPgcXh27mtnQU4AVgUzrEU+HF4e6t/TxpkOpUvLo7x4u+mH/BZ+LZs38+nR1lGAfPDfz+vAOke5UgASoHUBtu8yHEPoUJiKfB3INbLn9XOdPPqnB0+tufnbZ2zD5tJ5+wvsuicfWCOVj9na/lzEREREZEW0pnbQkREREREWpSKaxERERGRFqLiWkRERESkhai4FhERERFpISquRURERERaiIpr6RTMLGBmixvcWmzFKjPra2ZLW+r9REQ6O52zpT2L8jqASCvZ60JL9YqISNunc7a0Wxq5lk7NzDaY2a/NbG74NiC8vY+ZTTezz8N/9g5vzzGzl83ss/BtYvit/Gb2iJktM7N3wquWYWa3mdny8Ps859GXKSLSIeicLe2BimvpLOIbfcR4aYPnyp1zY4H7gXvD2+4H/uacOwF4BrgvvP0+4CPn3EjgREIrcQHkAQ8454YBu4CLw9vvAkaH3+fGyHxpIiIdjs7Z0m5phUbpFMyswjmX1MT2DcBU59w6M4sGtjvnMs2sBOjmnKsLb9/mnMsys2Kgp3OupsF79AXedc7lhR//AIh2zv3czN4CKggtQfuKc64iwl+qiEi7p3O2tGcauRYBd4j7h9qnKTUN7gf44nqGLwEPACcBC8xM1zmIiBwfnbOlTVNxLQKXNvjz0/D9WcBl4ftfB2aE708HbgIwM7+ZpRzqTc3MB/Ryzn0AfB9IAw4aiRERkaOic7a0afqNTDqLeDNb3ODxW865fVM7xZrZHEK/bF4e3nYb8LiZ/TdQDFwT3n478LCZfYvQaMdNwLZDHNMPPG1mqYABf3TO7Wqhr0dEpCPTOVvaLfVcS6cW7t/Ld86VeJ1FREQOT+dsaQ/UFiIiIiIi0kI0ci0iIiIi0kI0ci0iIiIi0kJUXIuIiIiItBAV1yIiIiIiLUTFtYiIiIhIC1FxLSIiIiLSQv4/RYPAm/28HLIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "scores = mod_stopping.validation_scores\n",
    "errors = mod_no_stopping.errors[: len(scores)]\n",
    "\n",
    "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))\n",
    "\n",
    "ax1 = pd.Series(scores).plot(ax=ax1)\n",
    "ax1.set_xlabel(\"Epochs\")\n",
    "ax1.set_ylabel(\"Macro F1 score\")\n",
    "\n",
    "ax2 = pd.Series(errors).plot(ax=ax2)\n",
    "ax2.set_xlabel(\"Epochs\")\n",
    "_ = ax2.set_ylabel(\"Error\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Learning curves with confidence intervals\n",
    "\n",
    "I frankly think the best response to all this is to accept that incremental performance plots like the above are how we should be assessing our models. This exposes all of the variation that we actually observe. \n",
    "\n",
    "In addition, in deep learning, we're often dealing with classes of models that are in principle capable of learning anything. The real question is implicitly how efficiently they can learn given the available data and other resources. Learning curves bring this our very clearly.\n",
    "\n",
    "We can improve the curves by adding confidence intervals to them derived from repeated runs. Here's a plot from a paper I wrote with Nick Dingwall ([Dingwall and Potts 2018](https://arxiv.org/abs/1803.09901)):\n",
    "\n",
    "<img src=\"fig/diagnosis-curve.png\" />\n",
    "\n",
    "I think this shows very clearly that, once all is said and done, the Mittens model (red) learns faster than the others, but is indistinguishable from the Clinical text GloVe model (blue) after enough training time. Furthermore, it's clear that the other two models are never going to catch up in the current experimental setting. A lot of this information would be lost if, for example, we decided to stop training when dev set performance reached its peak and report only a single F1 score per class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The role of random parameter initialization\n",
    "\n",
    "Most deep learning models have their parameters initialized randomly, perhaps according to some heuristics related to the number of parameters ([Glorot and Bengio 2010](http://proceedings.mlr.press/v9/glorot10a.html)) or their internal structure ([Saxe et al. 2014](https://arxiv.org/abs/1312.6120)). This is meaningful largely because of the non-convex optimization problems that these models define, but it can impact simpler models that have multiple optimal solutions that still differ at test time. \n",
    "\n",
    "There is growing awareness that these random choices have serious consequences. For instance, [Reimers and Gurevych (2017)](https://www.aclweb.org/anthology/D17-1035) report that different initializations for neural sequence models can lead to statistically significant results, and they show that a number of recent systems are indistinguishable in terms of raw performance once this source of variation is taken into account.\n",
    "\n",
    "This shouldn't surprise practitioners, who have long struggled with the question of what to do when a system experiences a catastrophic failure as a result of unlucky initialization. (I think the answer is to report this failure rate.)\n",
    "\n",
    "The code snippet below lets you experience this phenomenon for yourself. The XOR logic operator, which is true just in case its two arguments have the same value, is famously not learnable by a linear classifier but within reach of a neural network with a single hidden layer and a non-linear activation function ([Rumelhart et al. 1986](https://www.nature.com/articles/323533a0)). But how consistently do such models actually learn XOR? No matter what settings you choose, you rarely if ever see perfect performance across multiple runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Finished epoch 500 of 500; error is 0.012734206393361092"
     ]
    },
    {
     "data": {
      "text/plain": [
       "defaultdict(int, {'correct': 7, 'incorrect': 3})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def xor_eval(n_trials=10):\n",
    "    xor = [\n",
    "        ([1.,1.], 1),\n",
    "        ([1.,0.], 0),\n",
    "        ([0.,1.], 0),\n",
    "        ([0.,0.], 1)]\n",
    "    X, y = zip(*xor)\n",
    "    results = defaultdict(int)\n",
    "    for trial in range(n_trials):\n",
    "        model = TorchShallowNeuralClassifier(\n",
    "            hidden_dim=2,\n",
    "            max_iter=500,\n",
    "            eta=0.01)\n",
    "        model.fit(X, y)\n",
    "        preds = tuple(model.predict(X))\n",
    "        result = 'correct' if preds == y else 'incorrect'\n",
    "        results[result] += 1\n",
    "    return results\n",
    "\n",
    "xor_eval(n_trials=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For better or worse, the only response we have to this situation is to __report scores for multiple complete runs of a model with different randomly chosen initializations__. [Confidence intervals](#Confidence-intervals) and [statistical tests](#Wilcoxon-signed-rank-test) can be used to summarize the variation observed. If the evaluation regime already involves comparing the results of multiple train/test splits, then ensuring a new random initializing for each of those would seem sufficient.\n",
    "\n",
    "Arguably, these observations are incompatible with evaluation regimes involving only a single train/test split, as in [McNemar's test](#McNemar's-test). However, [as discussed above](#Practical-considerations,-and-some-compromises), we have to be realistic. If multiple run aren't feasible, then a more heuristic argument will be needed to try to convince skeptics that the differences observed are larger than we would expect from just different random initializations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Closing remarks\n",
    "\n",
    "We can summarize most of the above with a few key ideas:\n",
    "    \n",
    "1. Your evaluation should be based around a few systems that are related in ways that illuminate your hypotheses and help to convey what the best models are learning.\n",
    "\n",
    "1. Every model you assess should be given its best chance to shine (but we need to be realistic about how many experiments this entails!).\n",
    "\n",
    "1. The test set should play no role whatsoever in optimization or model selection. The best way to ensure this is to have the test set locked away until the final batch of experiments that will be reported in the paper, but this separation is simulated adequately by careful cross-validation set-ups.\n",
    "\n",
    "1. Strive to base your model comparisons in multiple runs on the same splits. This is especially important for deep learning, where a single model can perform in very different ways on the same data, depending on the vagaries of optimization."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}