4.4-overfitting-and-underfitting.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.2.4'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 과대적합과 과소적합\n",
    "\n",
    "이 노트북은 [케라스 창시자에게 배우는 딥러닝](https://tensorflow.blog/케라스-창시자에게-배우는-딥러닝/) 책의 4장 4절의 코드 예제입니다. 책에는 더 많은 내용과 그림이 있습니다. 이 노트북에는 소스 코드에 관련된 설명만 포함합니다. 이 노트북의 설명은 케라스 버전 2.2.2에 맞추어져 있습니다. 케라스 최신 버전이 릴리스되면 노트북을 다시 테스트하기 때문에 설명과 코드의 결과가 조금 다를 수 있습니다.\n",
    "\n",
    "----\n",
    "\n",
    "이전 장에 있던 세 가지 예제인 영화 리뷰 예측, 토픽 분류, 주택 가격 회귀 모두 홀드아웃 데이터에서 모델의 성능이 몇 번의 에포크 후에 최고치에 다다랐다가 감소되기 시작했습니다. 즉, 모델이 금방 훈련 데이터에 과대적합되기 시작합니다. 과대적합은 모든 머신 러닝 문제에서 발생합니다. 머신 러닝을 마스터하려면 과대적합을 다루는 방법을 꼭 배워야 합니다.\n",
    "\n",
    "머신 러닝의 근본적인 이슈는 최적화와 일반화 사이의 줄다리기입니다. 최적화는 가능한 훈련 데이터에서 최고의 성능을 얻으려고 모델을 조정하는 과정입니다(머신 러닝의 학습). 반면 일반화는 훈련된 모델이 이전에 본 적 없는 데이터에서 얼마나 잘 수행되는지를 의미합니다. 물론 모델을 만드는 목적은 좋은 일반화 성능을 얻는 것입니다. 하지만 일반화 성능을 제어할 방법이 없습니다. 단지 훈련 데이터를 기반으로 모델을 조정할 수만 있습니다.\n",
    "\n",
    "훈련 초기에 최적화와 일반화는 상호 연관되어 있습니다. 훈련 데이터의 손실이 낮을수록 테스트 데이터의 손실이 낮습니다. 이런 상황이 발생할 때 모델이 과소적합되었다고 말합니다. 모델의 성능이 계속 발전될 여지가 있습니다. 즉, 네트워크가 훈련 데이터에 있는 관련 특성을 모두 학습하지 못했습니다. 하지만 훈련 데이터에 여러 번 반복 학습하고 나면 어느 시점부터 일반화 성능이 더이상 높아지지 않습니다. 검증 세트의 성능이 멈추고 감소되기 시작합니다. 즉, 모델이 과대적합되기 시작합니다. 이는 훈련 데이터에 특화된 패턴을 학습하기 시작했다는 의미입니다. 이 패턴은 새로운 데이터와 관련성이 적어 잘못된 판단을 하게 만듭니다.\n",
    "\n",
    "모델이 관련성이 없고 좋지 못한 패턴을 훈련 데이터에서 학습하지 못하도록 하려면 가장 좋은 방법은 더 많은 훈련 데이터를 모으는 것입니다. 더 많은 데이터에서 훈련된 모델은 자연히 일반화 성능이 더 뛰어 납니다. 데이터를 더 모으는 것이 불가능할 때 차선책은 모델이 수용할 수 있는 정보의 양을 조절하거나 저장할 수 있는 정보에 제약을 가하는 것입니다. 네트워크가 적은 수의 패턴만을 기억할 수 있다면 최적화 과정에서 가장 중요한 패턴에 집중하게 될 것입니다. 이런 패턴은 더 나은 일반화 성능을 제공할 수 있습니다.\n",
    "\n",
    "이런 식으로 과대적합을 피하는 처리 과정을 규제라고 합니다. 가장 널리 사용되는 규제 기법을 알아보고 이전 장에서 본 영화 리뷰 분류 모델에 실제로 적용해 성능을 향상시켜 보겠습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "노트: 이 노트북에서는 편의상 IMDB 테스트 세트를 검증 세트로 사용합니다.\n",
    "\n",
    "3장 5절에 있는 코드를 사용해 데이터를 준비합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "import numpy as np\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    # 크기가 (len(sequences), dimension))이고 모든 원소가 0인 행렬을 만듭니다\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.  # results[i]에서 특정 인덱스의 위치를 1로 만듭니다\n",
    "    return results\n",
    "\n",
    "# 훈련 데이터를 벡터로 변환합니다\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# 테스트 데이터를 벡터로 변환합니다\n",
    "x_test = vectorize_sequences(test_data)\n",
    "# 레이블을 벡터로 변환합니다\n",
    "y_train = np.asarray(train_labels).astype('float32')\n",
    "y_test = np.asarray(test_labels).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 과대적합\n",
    "\n",
    "## 네트워크 크기 축소\n",
    "\n",
    "과대적합을 막는 가장 단순한 방법은 모델의 크기, 즉 모델에 있는 학습 파라미터의 수를 줄이는 것입니다. 파라미터의 수는 층의 수와 각 층의 유닛 수에 의해 결정됩니다. 딥러닝에서 모델에 있는 학습 파라미터의 수를 종종 모델의 용량이라고 말합니다. 당연하게 파라미터가 많은 모델이 기억 용량이 더 많습니다. 훈련 샘플과 타깃 사이를 딕셔너리와 같은 1:1 매핑으로 완벽하게 학습할 수도 있습니다. 이런 매핑은 일반화 능력이 없습니다. 예를 들어 500,000개의 이진 파라미터가 있는 모델은 MNIST 훈련 세트의 숫자 이미지의 클래스를 쉽게 모두 학습할 수 있습니다. 50,000개의 숫자 이미지 하나마다 10개의 이진 파라미터만 있으면 됩니다. 하지만 이런 모델은 새로운 숫자 샘플을 분류하는 용도로는 쓸모가 없습니다. 항상 유념해야 할 것은 딥러닝 모델은 훈련 데이터에 잘 맞으려는 경향을 가진다는 점입니다. 하지만 진짜 문제는 최적화가 아니고 일반화입니다.\n",
    "\n",
    "다른 한편으로 네트워크가 기억 용량에 제한이 있다면 이런 매핑을 쉽게 학습하지 못할 것입니다. 따라서 손실을 최소화하기 위해 타깃에 대한 예측 성능을 가진 압축된 표현을 학습해야 합니다. 정확히 이런 표현이 우리의 관심 대상입니다. 동시에 기억해야할 것은 과소적합되지 않도록 충분한 파라미터를 가진 모델을 사용해야 한다는 점입니다. 모델의 기억 용량이 부족해서는 안 됩니다. 너무 많은 용량과 충분하지 않은 용량 사이의 절충점을 찾아야 합니다.\n",
    "\n",
    "안타깝지만 알맞은 층의 수나 각 층의 유닛 수를 결정할 수 있는 마법 같은 공식은 없습니다. 데이터에 알맞는 모델 크기를 찾으려면 각기 다른 구조를 (당연히 테스트 세트가 아니고 검증 세트에서) 평가해 보아야 합니다. 적절한 모델 크기를 찾는 일반적인 작업 흐름은 비교적 적은 수의 층과 파라미터로 시작합니다. 그다음 검증 손실이 감소되기 시작할 때까지 층이나 유닛의 수를 늘리는 것입니다.\n",
    "\n",
    "영화 리뷰 분류 모델에 적용해 보죠. 원래 네트워크는 다음과 같습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "original_model = models.Sequential()\n",
    "original_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "original_model.add(layers.Dense(16, activation='relu'))\n",
    "original_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "original_model.compile(optimizer='rmsprop',\n",
    "                       loss='binary_crossentropy',\n",
    "                       metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "더 작은 네트워크로 바꾸어 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "smaller_model = models.Sequential()\n",
    "smaller_model.add(layers.Dense(6, activation='relu', input_shape=(10000,)))\n",
    "smaller_model.add(layers.Dense(6, activation='relu'))\n",
    "smaller_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "smaller_model.compile(optimizer='rmsprop',\n",
    "                      loss='binary_crossentropy',\n",
    "                      metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "원본 네트워크와 축소된 네트워크의 검증 손실을 비교해 보죠. 점으로 표현된 것이 작은 네트워크이고 덧셈 기호가 원래 네트워크 입니다(검증 손실이 작은 것이 좋은 모델입니다)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 3s 117us/step - loss: 0.4440 - acc: 0.8251 - val_loss: 0.3286 - val_acc: 0.8835\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.2573 - acc: 0.9078 - val_loss: 0.2864 - val_acc: 0.8882\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.1991 - acc: 0.9292 - val_loss: 0.2821 - val_acc: 0.8891\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.1666 - acc: 0.9412 - val_loss: 0.2939 - val_acc: 0.8844\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 2s 84us/step - loss: 0.1435 - acc: 0.9501 - val_loss: 0.3116 - val_acc: 0.8804\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.1257 - acc: 0.9558 - val_loss: 0.3483 - val_acc: 0.8721\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1104 - acc: 0.9615 - val_loss: 0.3598 - val_acc: 0.8722\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0977 - acc: 0.9669 - val_loss: 0.3975 - val_acc: 0.8662\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 2s 84us/step - loss: 0.0841 - acc: 0.9721 - val_loss: 0.4339 - val_acc: 0.8611\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0755 - acc: 0.9757 - val_loss: 0.4997 - val_acc: 0.8524\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0684 - acc: 0.9778 - val_loss: 0.4872 - val_acc: 0.8587\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0563 - acc: 0.9831 - val_loss: 0.5220 - val_acc: 0.8557\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 2s 80us/step - loss: 0.0513 - acc: 0.9843 - val_loss: 0.5351 - val_acc: 0.8579\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0423 - acc: 0.9881 - val_loss: 0.6093 - val_acc: 0.8501\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0363 - acc: 0.9894 - val_loss: 0.6115 - val_acc: 0.8558\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.0310 - acc: 0.9909 - val_loss: 0.6426 - val_acc: 0.8536\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0265 - acc: 0.9926 - val_loss: 0.7694 - val_acc: 0.8417\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0208 - acc: 0.9946 - val_loss: 0.7221 - val_acc: 0.8516\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.0198 - acc: 0.9949 - val_loss: 0.7739 - val_acc: 0.8476\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.0151 - acc: 0.9965 - val_loss: 0.9725 - val_acc: 0.8298\n"
     ]
    }
   ],
   "source": [
    "original_hist = original_model.fit(x_train, y_train,\n",
    "                                   epochs=20,\n",
    "                                   batch_size=512,\n",
    "                                   validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 2s 87us/step - loss: 0.5670 - acc: 0.7410 - val_loss: 0.4849 - val_acc: 0.8256\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.3924 - acc: 0.8895 - val_loss: 0.3609 - val_acc: 0.8780\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.2791 - acc: 0.9174 - val_loss: 0.3012 - val_acc: 0.8879\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.2213 - acc: 0.9278 - val_loss: 0.2823 - val_acc: 0.8900\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1877 - acc: 0.9375 - val_loss: 0.2816 - val_acc: 0.8863\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1649 - acc: 0.9448 - val_loss: 0.2858 - val_acc: 0.8856\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1467 - acc: 0.9518 - val_loss: 0.2954 - val_acc: 0.8817\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1330 - acc: 0.9558 - val_loss: 0.3067 - val_acc: 0.8796\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1196 - acc: 0.9620 - val_loss: 0.3223 - val_acc: 0.8773\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.1095 - acc: 0.9648 - val_loss: 0.3358 - val_acc: 0.8752\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.0988 - acc: 0.9690 - val_loss: 0.3549 - val_acc: 0.8712\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.0898 - acc: 0.9728 - val_loss: 0.3790 - val_acc: 0.8692\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.0816 - acc: 0.9759 - val_loss: 0.3900 - val_acc: 0.8676\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0743 - acc: 0.9772 - val_loss: 0.4130 - val_acc: 0.8654\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.0675 - acc: 0.9806 - val_loss: 0.4395 - val_acc: 0.8631\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0605 - acc: 0.9829 - val_loss: 0.4665 - val_acc: 0.8603\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0554 - acc: 0.9840 - val_loss: 0.4742 - val_acc: 0.8618\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.0500 - acc: 0.9868 - val_loss: 0.5050 - val_acc: 0.8590\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0456 - acc: 0.9883 - val_loss: 0.5209 - val_acc: 0.8591\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 2s 83us/step - loss: 0.0408 - acc: 0.9896 - val_loss: 0.5726 - val_acc: 0.8534\n"
     ]
    }
   ],
   "source": [
    "smaller_model_hist = smaller_model.fit(x_train, y_train,\n",
    "                                       epochs=20,\n",
    "                                       batch_size=512,\n",
    "                                       validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = range(1, 21)\n",
    "original_val_loss = original_hist.history['val_loss']\n",
    "smaller_model_val_loss = smaller_model_hist.history['val_loss']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XucFPWZ7/HPw80JiLcwSVAugwQj4AwII8iaBFhE0RPA1URFXucEEsPxjp5sjq5mZdC4iTkxnniNuDGYDdH1sl5ODhuNChJdNQwuoMAyIAEdcXXEABJALj77R9U0TdMz3TPd1dU9/X2/XvXqrupfVT/TNPV0/W5l7o6IiAhAp7gDEBGR4qGkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCZElBTN7wMw+MLM3W3jdzOwOM1tvZivNbERUsYiISHa6RHjs+cBdwK9aeP0sYFC4jAbuDR9b1atXL6+qqspPhCIiZWLZsmUfuntlpnKRJQV3X2JmVa0UmQr8yoN5Nl41s6PMrLe7v9facauqqqivr89jpCIiHZ+ZbcqmXJxtCscB7yStN4bbDmFms8ys3szqm5qaChKciEg5ijMpWJptaWfnc/d57l7r7rWVlRmvfkREpJ3iTAqNQN+k9T7A5phiERERom1ozuRp4Aoze5iggXlbpvaEluzdu5fGxkZ2796d1wAlOhUVFfTp04euXbvGHYqIJIksKZjZQ8A4oJeZNQJzgK4A7v5zYCFwNrAe2AnMbO97NTY20rNnT6qqqjBLVyslxcTd2bJlC42NjQwYMCDucEQkSZS9j6ZleN2By/PxXrt371ZCKCFmxmc/+1nUaUCkberqgiVKHWZEsxJCadG/l0jbzZ0b/Xt0mKQgIiK5U1LIk8bGRqZOncqgQYMYOHAgs2fPZs+ePWnLbt68ma9//esZj3n22WezdevWdsVTV1fHT37yk3btm6358+dzxRVX5FxGRFpWVwdmwQIHnkdVjVTWSSFfH6q7c+6553LOOeewbt06Ghoa2LFjBzfccMMhZfft28exxx7LY489lvG4Cxcu5KijjspPkCJSkurqwD1Y4MBzJYUI5Kt+7oUXXqCiooKZM4MOVJ07d+b222/ngQceYOfOncyfP59vfOMbTJ48mTPOOIONGzdy0kknAbBz507OP/98ampquOCCCxg9enRiGo+qqio+/PBDNm7cyODBg/nOd77D0KFDOeOMM9i1axcA999/P6eccgrDhg3jvPPOY+fOna3GOmPGDC699FLGjx/P8ccfz4svvsi3vvUtBg8ezIwZMxLlHnroIaqrqznppJO49tprE9t/+ctfcsIJJzB27FhefvnlxPampibOO+88TjnlFE455ZSDXhOR0lHWSSFfVq1axciRIw/adsQRR9CvXz/Wr18PwCuvvMKDDz7ICy+8cFC5e+65h6OPPpqVK1fy93//9yxbtizte6xbt47LL7+cVatWcdRRR/H4448DcO6557J06VJWrFjB4MGD+cUvfpEx3j//+c+88MIL3H777UyePJlrrrmGVatW8cYbb7B8+XI2b97MtddeywsvvMDy5ctZunQpTz75JO+99x5z5szh5Zdf5ve//z2rV69OHHP27Nlcc801LF26lMcff5yLL764TZ+hiGQ2Z0707xHn4LVY1NUdfIXQXE83Z077L8fcPW1vmuTtEydO5JhjjjmkzEsvvcTs2bMBOOmkk6ipqUn7HgMGDGD48OEAjBw5ko0bNwLw5ptv8v3vf5+tW7eyY8cOzjzzzIzxTp48GTOjurqaz3/+81RXVwMwdOhQNm7cyKZNmxg3bhzNU4pMnz6dJUuWABy0/YILLqChoQGA55577qAksX37dj7++OOMsYhI9qLujgplmhSaP1izA/V0uRg6dGjil3uz7du388477zBw4ECWLVtGjx490u7rWQZw2GGHJZ537tw5UX00Y8YMnnzySYYNG8b8+fNZvHhx1sfq1KnTQcft1KkT+/bto0uXlr8WLXUl/fTTT3nllVf4zGc+k82fIyJFStVHeTBhwgR27tzJr34V3Dpi//79fPe732XGjBl079691X2//OUv88gjjwCwevVq3njjjTa998cff0zv3r3Zu3cvCxYsaN8fkGL06NG8+OKLfPjhh+zfv5+HHnqIsWPHMnr0aBYvXsyWLVvYu3cvjz76aGKfM844g7vuuiuxvnz58rzEIiKFVdZJIV/1c2bGE088waOPPsqgQYM44YQTqKio4B/+4R8y7nvZZZfR1NRETU0Nt956KzU1NRx55JFZv/fNN9/M6NGjmThxIieeeGIuf0ZC7969+eEPf8j48eMZNmwYI0aMYOrUqfTu3Zu6ujrGjBnD6aefzogRB26Wd8cdd1BfX09NTQ1Dhgzh5z//eV5iEZHCsmyrL4pFbW2tp95kZ82aNQwePDimiHKzf/9+9u7dS0VFBW+99RYTJkygoaGBbt26xR1a5Er5302k1JjZMnevzVSu7NoUis3OnTsZP348e/fuxd259957yyIhiEhxUlKIWc+ePXV7UREpGmXdpiAiIgdTUhARkQQlBRERSVBSEBGRBCWFPLnlllsYOnQoNTU1DB8+nNdeey0vxz388MMBDppErxiMGzcuYwN5NmVEpLiUZVJYsACqqqBTp+Ax14HAr7zyCr/97W95/fXXWblyJc899xx9+/bNR6jttn///ljfX0RKU6RJwcwmmdlaM1tvZteleb2/mT1vZivNbLGZ9YkyHggSwKxZsGlTMO/Rpk3Bei6J4b333qNXr16JeYR69erFscceCwTTX19//fWMGTOG2tpaXn/9dc4880wGDhyYGPW7Y8cOJkyYwIgRI6iuruapp55q9f3279/P9773PU455RRqamq47777AFi8eDHjx4/noosuSkxyl+zwww/n2muvZeTIkZx++un88Y9/ZNy4cRx//PE8/fTTQHC/65kzZ1JdXc3JJ5/MokWLANi1axcXXnhhYorv5rmXAJ599lnGjBnDiBEj+MY3vsGOHTva/2GKSLzcPZIF6Ay8BRwPdANWAENSyjwKfDN8/tfAP2U67siRIz3V6tWrD9nWkv79m29RcfDSv3/WhzjExx9/7MOGDfNBgwb5pZde6osXL056v/5+zz33uLv71Vdf7dXV1b59+3b/4IMPvLKy0t3d9+7d69u2bXN396amJh84cKB/+umn7u7eo0cPd3f/05/+5EOHDnV39/vuu89vvvlmd3ffvXu3jxw50jds2OCLFi3y7t27+4YNG9LGCfjChQvd3f2cc87xiRMn+p49e3z58uU+bNgwd3f/yU9+4jNmzHB39zVr1njfvn19165dftttt/nMmTPd3X3FihXeuXNnX7p0qTc1NflXvvIV37Fjh7u7/+hHP/K5c+e6u/vYsWN96dKlLX5ubfl3E5HcAPWexbk7ysFro4D17r4BwMweBqYCq5PKDAGuCZ8vAp6MMB4A3n67bduzcfjhh7Ns2TL+8Ic/sGjRIi644AJ+9KMfJW5aM2XKFACqq6vZsWMHPXv2pGfPnlRUVLB161Z69OjB9ddfz5IlS+jUqRPvvvsu77//Pl/4whfSvt+zzz7LypUrE3dv27ZtG+vWraNbt26MGjWKAQMGpN2vW7duTJo0KRHLYYcdRteuXamurk5Mxf3SSy9x5ZVXAnDiiSfSv39/GhoaWLJkCVdddRUANTU1iSm+X331VVavXs1pp50GwJ49exgzZkz7P0wRiVWUSeE44J2k9UZgdEqZFcB5wM+AvwF6mtln3X1LVEH16xdUGaXbnovOnTszbtw4xo0bR3V1NQ8++GAiKWSaqnrBggU0NTWxbNkyunbtSlVVFbt3727xvdydO++885B7JyxevLjFKboBunbtmpj6OjmW5jiaj92Slu4ZMXHiRB566KEW9xOR0hFlm0K6ifdTzzh/C4w1s38HxgLvAvsOOZDZLDOrN7P6pqamnIK65RZInc26e/dge3utXbuWdevWJdaXL19O//79s95/27ZtfO5zn6Nr164sWrSITemyVpIzzzyTe++9l7179wLQ0NDAX/7yl/YFn+KrX/1qYgruhoYG3n77bb70pS8dtP3NN99k5cqVAJx66qm8/PLLiTvM7dy5M3HjHREpPVFeKTQCyV1w+gCbkwu4+2bgXAAzOxw4z923pR7I3ecB8yCYJTWXoKZPDx5vuCGoMurXL0gIzdvbY8eOHVx55ZVs3bqVLl268MUvfpF58+a1IabpTJ48mdraWoYPH55xCuyLL76YjRs3MmLECNydyspKnnwyPzVvl112GZdccgnV1dV06dKF+fPnc9hhh3HppZcyc+bMRJfbUaNGAVBZWcn8+fOZNm0an3zyCQA/+MEPOOGEE/ISj4gUVmRTZ5tZF6ABmEBwBbAUuMjdVyWV6QV85O6fmtktwH53v7G143a0qbPLmf7dRAon26mzI6s+cvd9wBXAM8Aa4BF3X2VmN5nZlLDYOGCtmTUAnwdyqMQREZFcRTp1trsvBBambLsx6fljwGNRxiAiItnrMCOao6oGk2jo30ukOHWIpFBRUcGWLVt0oikR7s6WLVuoqKiIOxQRSdEh7rzWp08fGhsbybW7qhRORUUFffpEPquJiLRRh0gKXbt2bXEUr4iIZK9DVB+JiEh+KCmIiEiCkoKIiCQoKYiISIKSgoiIJCgpiIhIgpKCiIgkKCmIiEiCkoKIiCQoKYiISIKSgoiIJCgpiIhIgpKCiIgkKCmIiEiCkoKIiCQoKYiISEKkScHMJpnZWjNbb2bXpXm9n5ktMrN/N7OVZnZ2lPGIiEjrIksKZtYZuBs4CxgCTDOzISnFvg884u4nAxcC90QVj4iIZBbllcIoYL27b3D3PcDDwNSUMg4cET4/EtgcYTwiIpJBlPdoPg54J2m9ERidUqYOeNbMrgR6AKdHGI+IiGQQ5ZWCpdnmKevTgPnu3gc4G/gnMzskJjObZWb1Zlbf1NQUQagiIgLRJoVGoG/Seh8OrR76NvAIgLu/AlQAvVIP5O7z3L3W3WsrKysjCldERKJMCkuBQWY2wMy6ETQkP51S5m1gAoCZDSZICroUEBGJSWRJwd33AVcAzwBrCHoZrTKzm8xsSljsu8B3zGwF8BAww91Tq5hERKRAomxoxt0XAgtTtt2Y9Hw1cFqUMYiIFIu6umApZhrRLCJlI+4T8ty58b5/NpQURKRslMJJOW5KCiIiEaqrA7NggQPP475qaYmSgoh0aHGflOvqwD1Y4MDzYk0KVmqdfWpra72+vj7uMESkBJkdODmX2/ub2TJ3r81UTlcKIiIFMmdO3BFkpqQgImUj7pNysVYZJVNSEJGyUQon5bgpKYiISELGpGBmPZpnLjWzE8xsipl1jT40EREptGyuFJYAFWZ2HPA8MBOYH2VQIiISj2ySgrn7TuBc4E53/xuC22uKiEgHk1VSMLMxwHTg/4fbIp1IT0RE4pFNUrga+DvgiXDq6+OBRdGGJSIicciYFNz9RXef4u63hg3OH7r7VQWITUSkqJRDl9Zseh/9xsyOMLMewGpgrZl9L/rQRESKSznMsppN9dEQd98OnENww5x+wH+PNCoREYlFNkmhazgu4RzgKXffC5TWLHoiIu0U9yyrhZZNUrgP2Aj0AJaYWX9ge5RBiYgUi1Kb+jpXGbuWuvsdwB1JmzaZ2fjoQhIRkbhk09B8pJn91Mzqw+U2gquGjMxskpmtNbP1ZnZdmtdvN7Pl4dJgZlvb8TeIiBRE3LOsFkI21UcPAB8D54fLduCXmXYys87A3cBZBCOgp5nZQSOh3f0adx/u7sOBO4F/aVv4IiKF01GrjJJlMzJ5oLufl7Q+18yWZ7HfKGC9u28AMLOHgakE3VrTmQaUQR4WESle2Vwp7DKzLzevmNlpwK4s9jsOeCdpvTHcdoiw8XoA8EILr89qrr5qamrK4q1FRKQ9srlSuBR40MyOBAz4CJiRxX6WZltLXVkvBB5z9/3pXnT3ecA8CO7RnMV7i4hIO2TT+2g5MMzMjgjXs+2O2gj0TVrvA2xuoeyFwOVZHldERCLSYlIws//VwnYA3P2nGY69FBhkZgOAdwlO/BelOd6XgKOBV7ILWUREotLalULPXA7s7vvM7ArgGaAz8EA4y+pNQL27Px0WnQY87O6qFhIRiZmV2rm4trbW6+vr4w5DRKSkmNkyd6/NVC6b3kciIlImlBRERCRBSUFERBIydkk1s8OA84Cq5PLuflN0YYmISByyGbz2FLANWAZ8Em04IiISp2ySQh93nxR5JCIiErts2hT+zcyqI49ERERil82VwpeBGWb2J4LqIwPc3WsijUxERAoum6RwVuRRiIhkoa6uPO5pEKeM1Ufuvgk4CpgcLkeF20RECmru3Lgj6PiyuR3nbGAB8Llw+bWZXRl1YCIiUnjZNDR/Gxjt7je6+43AqcB3og1LRCRQVwdmwQIHnqsaKRrZtCkYkHzzm/2kv4GOiEjeJbcjmEGJzeFZcrJJCr8EXjOzJ8L1c4BfRBeSiIjEJZs7r/3UzBYTdE01YKa7/3vUgYmIpJozJ+4IOr7W7rx2hLtvN7NjgI3h0vzaMe7+UfThiYgcoHaE6LV2pfAb4GsEcx4l1+JZuH58hHGJiEgMWux95O5fCx8HuPvxScsAd1dCEClDuf5S1y/94pfNOIXns9kmIh1froPHNPis+LWYFMysImxP6GVmR5vZMeFSBRybzcHNbJKZrTWz9WZ2XQtlzjez1Wa2ysx+054/QkRE8qO1K4X/SdCecGL42Lw8Bdyd6cBm1jksdxYwBJhmZkNSygwC/g44zd2HAle3428QkQjlOnhMg89Ki3mGkSBmdqW739nmA5uNAerc/cxw/e8A3P2HSWV+DDS4+z9me9za2lqvr69vazgikge5Dh7T4LP4mNkyd6/NVC6bcQp3mtlJBL/2K5K2/yrDrscB7yStNwKjU8qcEAb7MtCZIIn8LlNMIiISjWzu0TwHGEeQFBYSVAe9BGRKCummwkj9jdAFGBQevw/wBzM7yd23psQwC5gF0K9fv0whi0hEch08psFnxS+bCfG+DkwA/tPdZwLDgMOy2K8R6Ju03gfYnKbMU+6+193/BKwlSBIHcfd57l7r7rWVlZVZvLWIREFdUju+bJLCLnf/FNhnZkcAH5DdwLWlwCAzG2Bm3YALgadTyjwJjAcws14E1Ukbsg1eRETyK5sJ8erN7CjgfoLeRzuAP2bayd33mdkVwDME7QUPuPsqM7sJqHf3p8PXzjCz1QSzr37P3be0828REZEcZex9dFDhYIzCEe6+MqqAMlHvIxGRtsu291Frg9dGpC7AMUCX8LmIiBTAggVQVQWdOgWPCxZE916tVR/dFj5WALXACoIeRTXAawRTaYuISIQWLIBZs2DnzmB906ZgHWD69Py/X2sT4o139/HAJmBE2PtnJHAysD7/oYiISKobbjiQEJrt3Blsj0I2vY9OdPc3mlfc/U1geDThiIhIsrffbtv2XGWTFNaY2T+a2TgzG2tm9wNroglHRESStTReN6pxvNkkhZnAKmA2wYR1q8NtIiISsVtuge7dD97WvXuwPQrZzH20G7g9XEREpICaG5NvuCGoMurXL0gIUTQyQ+v3aH7E3c83szc4dM4i3L0mmpBERCTZ9OnRJYFUrV0pzA4fv1aIQEREJH4tJgV3fy983FS4cEREJE6tjWj+2My2p1k+NrPthQxSRPJDs5RKJq0NXuvp7kekWXq6+xGFDFJE8mPu3LgjkGKXzSypAJjZ5zj4zmsRDZ0QEZG4ZBynYGZTzGwd8CfgRWAj8K8RxyUieVJXF9wb2cJ7ITY/V1WSpJPN4LWbgVOBBncfQHAXtpcjjUpE0mrPibyuDtyDBQ48V1IonELOcpqrbJLC3vDGN53MrJO7L0JzH4nEQm0Cpad5ltNNm4Jk3DzLabEmhmySwlYzOxxYAiwws58B+6INS0SiMGdO3BGUn0LPcpqrbJLCVGAXcA3wO+AtYHKUQYnIAflsE1CVUeEVepbTXLV4O04zuwv4jbv/W2FDap1uxynlzOxA24CUhqqqoMooVf/+sHFj4eLI+XacwDrgNjPbaGa3mpnaEUSkLOXSUFzoWU5z1drgtZ+5+xhgLPAR8EszW2NmN5rZCdkc3MwmmdlaM1tvZteleX2GmTWZ2fJwubjdf4lIGVCbQOHl2lA8fTrMmxdcGZgFj/PmFW6Cu7ZqsfoobWGzk4EHgBp375yhbGegAZgINAJLgWnuvjqpzAyg1t2vyDYGVR9JXOrqVCdfjoql+idX+ag+aj5QVzObbGYLCAatNQDnZRHDKGC9u29w9z3AwwSN1iIlSd1By1OpNRTnqrUJ8Saa2QMEv/JnAQuBge5+gbs/mcWxjwPeSVpvDLelOs/MVprZY2bWt4VYZplZvZnVNzU1ZfHWIiL5UejbYcattSuF64FXgMHuPtndF7j7X9pwbEuzLbWu6v8BVeENe54DHkx3IHef5+617l5bWVnZhhBEcqPuoFJqDcW5aq2heby73+/uH7Xz2I1A8i//PsDmlPfY4u6fhKv3AyPb+V6tKqUh5lJc8jlFhKqfSlOpNRTnKutZUtthKTDIzAYA7wIXAhclFzCz3s038wGmAGvyHURzz4HmEYXNPQeg4/6jikh+FfJ2mHHLZkRzu7j7PuAK4BmCk/0j7r7KzG4ysylhsavMbJWZrQCuAmbkO45SG2Iuxas93UE1Q6mUmjZ1SS0Gbe2S2qlT+hGgZvDpp3kMTCQDjUaOz4IFwQ/Bt98OGohvuaV8fvk3y1uX1FJXbj0HRORgpTZLadw6fFIot54DUrw0Grn9cuksoirktunwSaHceg5I8VI7Qvvk+ku/3Aaf5arDtymISGnLdZqJjjJNRa7UpiAiHUKuv/RVhdw2SgoiUtRy7SyiKuS2UVIQkaKWj1/606cHVUWffho8KiG0TElBRIqafukXVpTTXIiI5EU5TTMRN10piEjkNCll6VBSkLKhcQLx0Iji0qJxClI2NPdQPDROoDhonIKIFAWNKC4tSgrSoWnq6vhpUsrSoqQgHVo+75wm7aMRxaVFSUFEIqVxBqVFSUHKhqaubr9cu5RqRHHp0OA1KRuqMmof3ee8vOhKQURapZvUlJdIk4KZTTKztWa23syua6Xc183MzSxjH1oRKSx1KS0vkSUFM+sM3A2cBQwBppnZkDTlegJXAa9FFYuItJ+6lJaXKK8URgHr3X2Du+8BHgampil3M/BjYHeEsUgHoDaBeKhLaXmJMikcB7yTtN4Ybksws5OBvu7+2wjjkA5i7ty4IyhP6lJaXqJMCpZmW2LmGTPrBNwOfDfjgcxmmVm9mdU3NTXlMUSR8qAupZKtKJNCI9A3ab0PsDlpvSdwErDYzDYCpwJPp2tsdvd57l7r7rWVlZURhizFRtNU5E6zlEpbRDZLqpl1ARqACcC7wFLgIndf1UL5xcDfunurU6BqltTypVlO20ezlAoUwSyp7r4PuAJ4BlgDPOLuq8zsJjObEtX7inREuVT/qEuptEWkI5rdfSGwMGXbjS2UHRdlLFL6ynWailxHFPfrl/5KQV1KJR2NaJaSUa7tCLmOKFaXUmkLJQWRIpdr9Y+6lEpbaEI8kSKXj+qf6dOVBCQ7ulLIQq59vEVyoeofKSQlhQzUxzt/yrVNIFeq/pFCimycQlQKPU5Bfbzzp1zHGSxYEDQKv/12UOVzyy06oUvhZTtOQW0KGaiPt+RCN6iRUqPqoww0bfAB7an+KfdpKnSDGik1qj7KIPWXHgSNfOVYp5tr9U85Vh916pT+bzYLJpcTKZTYp7noKNTIJ7nQlaaUGiWFLJTztMH5rP4p1WkqcumSrO6kUmpUfSRZK8fqn3xUH6r3kRSDbKuPyiop1NWVTwNnFMoxKahLsnQUalNIQ7dzzE2pVv/kQl2SpdyUVVKQ3JTqVVYubQJqKJZy0+GTQrn3ky93uU5TooZiKTdl1aZQjnXi5S4fbQJqKJaOQG0KcohyvDrKR5tAOXdJlvJTVkmhHBtKk5VjQ7vaBETapqySQjn+Uu4INHhMpHAiTQpmNsnM1prZejO7Ls3rl5jZG2a23MxeMrMhUcYTlzhv0lPqDe25NhRrmhKRtomsodnMOgMNwESgEVgKTHP31UlljnD37eHzKcBl7j6pteOW2ojmYppQrxQb2jV4TCQ/iqGheRSw3t03uPse4GFganKB5oQQ6gGU2Ckrs3xOnVwqv+5T5XKlpMFjIoUVZVI4Dngnab0x3HYQM7vczN4CfgxcFWE8scjnSS3XhuI4Gtpzrf5RQ7FIYUWZFCzNtkOuBNz9bncfCFwLfD/tgcxmmVm9mdU3NTXlOcxoFdNJLY4rjVyvlNRQLFJYUSaFRqBv0nofYHMr5R8Gzkn3grvPc/dad6+trKzMY4ht056Taq4ntWJoKI6z+kcNxSIF5u6RLAT3f94ADAC6ASuAoSllBiU9nwzUZzruyJEjPS7Qvv1+/Wv3/v2D/fv3D9bj2N+s7fv/+tfu3bsH7928dO+e/TGa405d+vdv298gIrnJ5vzq4WkuysRwNkEPpLeAG8JtNwFTwuc/A1YBy4FFqUkj3VKKSSGX/XM9Kcd9Us/1/UUkP4oiKUSxFDopzJmT/qQ4Z07bj9WepJDrSTnX/c3S72+W/d+Qy5WKiORHtkmhrCbEy1V7+vnX1aXvNTRnTnbtArne+D3X/TVOQKRjKIZxCkJw4m/+fQ0HnmfbUJxr76Vc91fvH5HyoqTQBnH088/1pJzr/ur9I1JelBTaINduoO1JKrmelPNxUtfU0SLlQ20KIiJlQG0KIiLSZkoKIiKSoKQgIiIJSgoiIpKgpCAiIgkl1/vIzJqANGNsi0Iv4MO4g2iF4stNsccHxR+j4stNLvH1d/eM00yXXFIoZmZWn02Xr7govtwUe3xQ/DEqvtwUIj5VH4mISIKSgoiIJCgp5Ne8uAPIQPHlptjjg+KPUfHlJvL41KYgIiIJulIQEZEEJYU2MrO+ZrbIzNaY2Sozm52mzDgz22Zmy8PlxgLHuNHM3gjf+5DZAy1wh5mtN7PCB7RbAAAF+UlEQVSVZjaigLF9KelzWW5m283s6pQyBf/8zOwBM/vAzN5M2naMmf3ezNaFj0e3sO83wzLrzOybBYrt/5jZf4T/fk+Y2VEt7NvqdyHiGOvM7N2kf8ezW9h3kpmtDb+P1xUwvn9Oim2jmS1vYd9IP8OWzimxff+yuT2blqT7l0JvYET4vCfBPaiHpJQZB/w2xhg3Ar1aef1s4F8BA04FXospzs7AfxL0n4718wO+CowA3kza9mPguvD5dcCtafY7BtgQPh4dPj+6ALGdAXQJn9+aLrZsvgsRx1gH/G0W34G3gOOBbsCK1P9PUcWX8vptwI1xfIYtnVPi+v7pSqGN3P09d389fP4xsAY4Lt6o2mwq8CsPvAocZWa9Y4hjAvCWu8c+GNHdlwAfpWyeCjwYPn8QOCfNrmcCv3f3j9z9z8DvgUlRx+buz7r7vnD1VaBPPt+zrVr4/LIxCljv7hvcfQ/wMMHnnletxWdmBpwPPJTv981GK+eUWL5/Sgo5MLMq4GTgtTQvjzGzFWb2r2Y2tKCBgQPPmtkyM5uV5vXjgHeS1huJJ7FdSMv/EeP8/Jp93t3fg+A/LvC5NGWK4bP8FsGVXzqZvgtRuyKs4nqgheqPYvj8vgK87+7rWni9YJ9hyjkllu+fkkI7mdnhwOPA1e6+PeXl1wmqRIYBdwJPFji809x9BHAWcLmZfTXldUuzT0G7oZlZN2AK8Gial+P+/Noi1s/SzG4A9gELWiiS6bsQpXuBgcBw4D2CKppUsX8XgWm0fpVQkM8wwzmlxd3SbMvp81NSaAcz60rwj7fA3f8l9XV33+7uO8LnC4GuZtarUPG5++bw8QPgCYJL9GSNQN+k9T7A5sJEl3AW8Lq7v5/6QtyfX5L3m6vVwscP0pSJ7bMMGxW/Bkz3sII5VRbfhci4+/vuvt/dPwXub+G9Y/0umlkX4Fzgn1sqU4jPsIVzSizfPyWFNgrrH38BrHH3n7ZQ5gthOcxsFMHnvKVA8fUws57NzwkaJN9MKfY08D/CXkinAtuaL1MLqMVfZ3F+fimeBpp7c3wTeCpNmWeAM8zs6LB65IxwW6TMbBJwLTDF3Xe2UCab70KUMSa3U/1NC++9FBhkZgPCq8cLCT73Qjkd+A93b0z3YiE+w1bOKfF8/6JqUe+oC/BlgsuzlcDycDkbuAS4JCxzBbCKoCfFq8BfFTC+48P3XRHGcEO4PTk+A+4m6PXxBlBb4M+wO8FJ/sikbbF+fgQJ6j1gL8Gvr28DnwWeB9aFj8eEZWuBf0za91vA+nCZWaDY1hPUJTd/B38elj0WWNjad6GAn98/hd+vlQQnuN6pMYbrZxP0uHkrqhjTxRdun9/8vUsqW9DPsJVzSizfP41oFhGRBFUfiYhIgpKCiIgkKCmIiEiCkoKIiCQoKYiISIKSgkjIzPbbwTO45m3GTjOrSp6hU6RYdYk7AJEissvdh8cdhEicdKUgkkE4n/6tZvbHcPliuL2/mT0fTvj2vJn1C7d/3oJ7HKwIl78KD9XZzO4P58x/1sw+E5a/ysxWh8d5OKY/UwRQUhBJ9pmU6qMLkl7b7u6jgLuA/xtuu4tgCvIaggnp7gi33wG86MGEfiMIRsICDALudvehwFbgvHD7dcDJ4XEuieqPE8mGRjSLhMxsh7sfnmb7RuCv3X1DOHHZf7r7Z83sQ4KpG/aG299z915m1gT0cfdPko5RRTDv/aBw/Vqgq7v/wMx+B+wgmA32SQ8nAxSJg64URLLjLTxvqUw6nyQ938+BNr3/RjAX1UhgWThzp0gslBREsnNB0uMr4fN/I5jVE2A68FL4/HngUgAz62xmR7R0UDPrBPR190XA/waOAg65WhEpFP0iETngM3bwzdt/5+7N3VIPM7PXCH5ITQu3XQU8YGbfA5qAmeH22cA8M/s2wRXBpQQzdKbTGfi1mR1JMHvt7e6+NW9/kUgbqU1BJIOwTaHW3T+MOxaRqKn6SEREEnSlICIiCbpSEBGRBCUFERFJUFIQEZEEJQUREUlQUhARkQQlBRERSfgvnvXAzLy2+CMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ‘b+’는 파란색 덧셈 기호을 의미합니다\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "# ‘bo’는 파란색 점을 의미합니다\n",
    "plt.plot(epochs, smaller_model_val_loss, 'bo', label='Smaller model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "여기서 볼 수 있듯이 작은 네트워크가 기본 네트워크보다 더 나중에 과대적합되기 시작했습니다(네 번째 에포크가 아니라 여섯 번째 에포크에서). 과대적합이 시작되었을 때 성능이 더 천천히 감소되었습니다.\n",
    "\n",
    "이번에는 문제에 필요한 것보다 훨씬 더 많은 용량을 가진 네트워크를 비교해 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigger_model = models.Sequential()\n",
    "bigger_model.add(layers.Dense(1024, activation='relu', input_shape=(10000,)))\n",
    "bigger_model.add(layers.Dense(1024, activation='relu'))\n",
    "bigger_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "bigger_model.compile(optimizer='rmsprop',\n",
    "                     loss='binary_crossentropy',\n",
    "                     metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 3s 133us/step - loss: 0.5454 - acc: 0.7882 - val_loss: 0.2760 - val_acc: 0.8891\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 3s 111us/step - loss: 0.2188 - acc: 0.9147 - val_loss: 0.3125 - val_acc: 0.8754\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 2s 100us/step - loss: 0.1151 - acc: 0.9607 - val_loss: 0.3470 - val_acc: 0.8831\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 3s 105us/step - loss: 0.0658 - acc: 0.9850 - val_loss: 0.4694 - val_acc: 0.8829\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 2s 96us/step - loss: 0.0979 - acc: 0.9854 - val_loss: 0.4798 - val_acc: 0.8785\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 3s 105us/step - loss: 0.0027 - acc: 0.9996 - val_loss: 0.6154 - val_acc: 0.8806\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 2s 98us/step - loss: 8.7515e-04 - acc: 0.9998 - val_loss: 0.7370 - val_acc: 0.8802\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 3s 108us/step - loss: 0.1479 - acc: 0.9849 - val_loss: 0.8596 - val_acc: 0.8716\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 3s 104us/step - loss: 9.8797e-04 - acc: 0.9998 - val_loss: 0.7918 - val_acc: 0.8764\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 3s 102us/step - loss: 3.0235e-05 - acc: 1.0000 - val_loss: 0.8231 - val_acc: 0.8813\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 2s 97us/step - loss: 4.1083e-06 - acc: 1.0000 - val_loss: 0.9013 - val_acc: 0.8801\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 3s 104us/step - loss: 0.1174 - acc: 0.9915 - val_loss: 1.0090 - val_acc: 0.8724\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 2s 95us/step - loss: 6.8146e-04 - acc: 1.0000 - val_loss: 0.9307 - val_acc: 0.8774\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 2s 95us/step - loss: 0.1304 - acc: 0.9871 - val_loss: 1.0209 - val_acc: 0.8652\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 3s 105us/step - loss: 0.0512 - acc: 0.9931 - val_loss: 2.1249 - val_acc: 0.7613\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 2s 99us/step - loss: 0.0201 - acc: 0.9975 - val_loss: 0.8756 - val_acc: 0.8756\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 2s 98us/step - loss: 1.0459e-05 - acc: 1.0000 - val_loss: 0.9152 - val_acc: 0.8774\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 3s 101us/step - loss: 1.5831e-06 - acc: 1.0000 - val_loss: 0.9598 - val_acc: 0.8780\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 2s 97us/step - loss: 3.9690e-07 - acc: 1.0000 - val_loss: 1.0252 - val_acc: 0.8777\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 3s 103us/step - loss: 1.6289e-07 - acc: 1.0000 - val_loss: 1.0794 - val_acc: 0.8782\n"
     ]
    }
   ],
   "source": [
    "bigger_model_hist = bigger_model.fit(x_train, y_train,\n",
    "                                     epochs=20,\n",
    "                                     batch_size=512,\n",
    "                                     validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "다음은 더 큰 네트워크가 기본 네트워크에 비해 얼마나 차이나는지를 보여줍니다. 점이 용량이 큰 네트워크의 검증 손실이고 덧셈 기호는 원본 네트워크의 검증 손실입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3X2cFNWd7/HPDySOKAgqcVGEQa9ERAaEASR6BZY4PkTAh0TF2SiaSOLDhri5e9WQyGjWZL0mMRKTuLgieDPBjTEa773GRxTjU8LgRREwgAo64oYBNEhGIwO//aOqsRm6Z2qmu7q6p7/v16tf3XWqqutHTdO/rnNOnWPujoiISHu6JR2AiIiUBiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMEREJJJ9kg4gnw455BCvrKxMOgwRkZKxbNmyze7eL8q2XSphVFZW0tDQkHQYIiIlw8w2RN1WVVIiIhKJEoaIiESihCEiIpF0qTaMTHbs2EFjYyMfffRR0qFIBBUVFQwYMIAePXokHYqItNLlE0ZjYyO9evWisrISM0s6HGmDu7NlyxYaGxsZPHhw0uGISCtdvkrqo48+4uCDD1ayKAFmxsEHH6yrwTJSXw+VldCtW/BcX590RNKWLn+FAShZlBD9rcpHfT3MnAnNzcHyhg3BMkBtbXJxSXZd/gpDRIrT7NmfJIuU5uagXIqTEkYBNDY2Mm3aNI4++miOOuooZs2axccff5xx240bN/KFL3yh3fc844wzeP/99zsVT11dHT/4wQ86tW9UCxYs4Kqrrsp5G+m63nqrY+WSPCWMLOrq8vM+7s4555zDWWedxdq1a1mzZg3bt29ndoafUS0tLRx22GH8+te/bvd9H374Yfr06ZOfIEUSMHBgx8oleUoYWdxwQ37eZ/HixVRUVHDJJZcA0L17d2699Vbmz59Pc3MzCxYs4Itf/CJTpkyhpqaG9evXc9xxxwHQ3NzMeeedR1VVFeeffz7jxo3bPfRJZWUlmzdvZv369QwdOpTLLruMYcOGUVNTw4cffgjAnXfeyZgxYxgxYgTnnnsuza2v/1uZMWMGl19+OZMmTeLII49kyZIlXHrppQwdOpQZM2bs3m7RokUMHz6c4447jmuuuWZ3+d13382QIUOYMGECzz333O7ypqYmzj33XMaMGcOYMWP2WCfl66aboGfPPct69gzKpTgpYcRs5cqVjB49eo+y3r17M3DgQNatWwfACy+8wMKFC1m8ePEe2/3sZz+jb9++vPLKK3znO99h2bJlGY+xdu1arrzySlauXEmfPn24//77ATjnnHNYunQpL7/8MkOHDuWuu+5qN9733nuPxYsXc+uttzJlyhSuvvpqVq5cyYoVK1i+fDkbN27kmmuuYfHixSxfvpylS5fy4IMP8u677zJnzhyee+45Hn/8cVatWrX7PWfNmsXVV1/N0qVLuf/++/nKV77SoXMoXVNtLcybB4MGgVnwPG+eGryLWVn0koqqrm7PK4tUh505czpfReXuGXv+pJefcsopHHTQQXtt8+yzzzJr1iwAjjvuOKqqqjIeY/DgwYwcORKA0aNHs379egBeffVVvv3tb/P++++zfft2Tj311HbjnTJlCmbG8OHDOfTQQxk+fDgAw4YNY/369WzYsIGJEyfSr18wuGVtbS3PPPMMwB7l559/PmvWrAHgiSee2COBbNu2jQ8++KDdWKTrq61VgiglShhp6uo+SQxm4J77ew4bNmz3L/6Ubdu28fbbb3PUUUexbNky9t9//4z7esQA9t13392vu3fvvrtKasaMGTz44IOMGDGCBQsW8PTTT0d+r27duu3xvt26daOlpYV99sn+kcnWJXbXrl288MIL7LffflH+OSJSpFQlFbPJkyfT3NzMPffcA8DOnTv55je/yYwZM+jZugK3lZNOOolf/epXAKxatYoVK1Z06NgffPAB/fv3Z8eOHdTn6Y6ocePGsWTJEjZv3szOnTtZtGgREyZMYNy4cTz99NNs2bKFHTt2cN999+3ep6amhttvv3338vLly/MSi4gUlhJGFnPm5Od9zIwHHniA++67j6OPPpohQ4ZQUVHB9773vXb3veKKK2hqaqKqqoqbb76ZqqoqDjzwwMjH/u53v8u4ceM45ZRTOOaYY3L5Z+zWv39/vv/97zNp0iRGjBjBqFGjmDZtGv3796euro7x48fzuc99jlGjRu3eZ+7cuTQ0NFBVVcWxxx7LHXfckZdYRKSwLGq1Rymorq721hMorV69mqFDhyYUUW527tzJjh07qKio4PXXX2fy5MmsWbOGT33qU0mHFqtS/puJlBozW+bu1VG2VRtGEWtubmbSpEns2LEDd+fnP/95l08WIlK8YksYZnYEcA/wd8AuYJ6739ZqGwNuA84AmoEZ7v5SuO5i4Nvhpv/i7gvjirVY9erVS1POikjRiPMKowX4pru/ZGa9gGVm9ri7r0rb5nTg6PAxDvg5MM7MDgLmANWAh/s+5O7vxRiviIi0IbZGb3d/N3W14O4fAKuBw1ttNg24xwMvAn3MrD9wKvC4u28Nk8TjwGlxxSoiIu0rSC8pM6sEjgf+0GrV4cDbacuNYVm2chERSUjsCcPMDgDuB77h7ttar86wi7dRnun9Z5pZg5k1NDU15RasiIhkFWvCMLMeBMmi3t1/k2GTRuCItOUBwMY2yvfi7vPcvdrdq1PDUhSb7t27M3LkyN33LTz//PNA9KHMi9kBBxyQl21EpPjFljDCHlB3Aavd/UdZNnsIuMgCJwB/cfd3gUeBGjPra2Z9gZqwLHZxTBm53377sXz5cl5++WW+//3vc9111wFEHso8Fy0tLbG+v4iUjzivME4EvgT8vZktDx9nmNnXzOxr4TYPA28A64A7gSsA3H0r8F1gafi4MSyLVWrKyA0bgnGkUlNG5nOe4W3bttG3b1+AyEOZ33XXXQwZMoSJEydy2WWX7Z50KNuw4XV1dcycOZOamhouuuiiPY7/9NNPM2HCBM477zyGDBnCtddeS319PWPHjmX48OG8/vrrAGzYsIHJkydTVVXF5MmTeSuc1ebNN99k/PjxjBkzhu985zt7vPctt9zCmDFjqKqqYk6+bpUXkeLh7l3mMXr0aG9t1apVe5VlM2iQe5Aq9nwMGhT5LTLq1q2bjxgxwj/zmc947969vaGhwd3d33zzTR82bJi7u99yyy0+c+ZMd3dfsWKFd+/e3ZcuXervvPOODxo0yLds2eIff/yxn3TSSX7llVe6u/v06dP997//vbu7b9iwwY855hh3d58zZ46PGjXKm5ub94rlqaee8gMPPNA3btzoH330kR922GF+/fXXu7v7j3/8Y581a5a7u5955pm+YMECd3e/6667fNq0ae7uPmXKFF+4cKG7u99+++2+//77u7v7o48+6pdddpnv2rXLd+7c6Z///Od9yZIl7u67t4mqI38zEckN0OARv2M1llSauKaMTFVJvfbaazzyyCNcdNFFe41E++yzz3LBBRcAew5l/sc//pEJEyZw0EEH0aNHD774xS/u3ueJJ57gqquuYuTIkUydOnWPYcOnTp2adXTYMWPG0L9/f/bdd1+OOuooampqABg+fPjuodFfeOEFLrzwQgC+9KUv8eyzzwLw3HPPMX369N3lKY899hiPPfYYxx9/PKNGjeK1115j7dq1OZ03ESkuGhokzcCBQTVUpvJ8GT9+PJs3b6Z1j67WCaS9cmh72PBsQ6YDew1bnj6kebY2j/Shy7PN73Hdddfx1a9+NetxRaS06QojTSGmjHzttdfYuXMnBx988B7l2YYyHzt2LEuWLOG9996jpaVlj7k14hw2/LOf/Sz33nsvAPX19Zx00kkAnHjiiXuUp5x66qnMnz+f7du3A/DOO++wadOmvMUjIsnTFUaa1Mxfs2cH1VADBwbJItcZwT788MPdM+K5OwsXLqR79+57bHPFFVdw8cUXU1VVxfHHH797KPPDDz+cb33rW4wbN47DDjuMY489dvcQ53PnzuXKK6+kqqqKlpYWTj755LwNHT537lwuvfRSbrnlFvr168fdd98NwG233caFF17Ibbfdxrnnnrt7+5qaGlavXs348eOBoCvtL37xCz796U/nJR4RSZ6GNy8SbQ1lvn37dg444ABaWlo4++yzufTSSzn77LOTDjk2pfI3E+kKNLx5CWprKPO6ujqeeOIJPvroI2pqajjrrLMSjlZEypESRpFoayjzH/zgBwWORkRkb2XR6N2Vqt26Ov2tRIpXl08YFRUVbNmyRV9EJcDd2bJlCxUVFUmHIiIZdPkqqQEDBtDY2LjXfQ9SnCoqKhgwYEDSYYhIBl0+YfTo0YPBgwcnHYaISMnr8lVSIiKSH0oYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiEQS230YZjYfOBPY5O7HZVj/z0Bq4PB9gKFAP3ffambrgQ+AnUBL1JEURUQkPnFeYSwATsu20t1vcfeR7j4SuA5Y4u5b0zaZFK5XshARKQKxJQx3fwbY2u6GgenAorhiERGR3CXehmFmPQmuRO5PK3bgMTNbZmYz29l/ppk1mFmDxosSEYlP4gkDmAI816o66kR3HwWcDlxpZidn29nd57l7tbtX9+vXL+5YRUTKVjEkjAtoVR3l7hvD503AA8DYBOISEZE0iSYMMzsQmAD8Nq1sfzPrlXoN1ACvJhOhiIikxNmtdhEwETjEzBqBOUAPAHe/I9zsbOAxd/9r2q6HAg+YWSq+X7r7I3HFKSIi0cSWMNx9eoRtFhB0v00vewMYEU9UIiLSWcXQhiEiIiVACUNERCJRwhARkUiUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMEREJBIlDBERiSS2hGFm881sk5llnI/bzCaa2V/MbHn4uD5t3Wlm9iczW2dm18YVo4iIRNduwjCz/c2sW/h6iJlNNbMeEd57AXBaO9v83t1Hho8bw2N0B34KnA4cC0w3s2MjHE9ERGIU5QrjGaDCzA4HngQuodU83Jm4+zPA1k7ENBZY5+5vuPvHwL3AtE68j4iI5FGUhGHu3gycA/zE3c8m+OWfD+PN7GUz+52ZDQvLDgfeTtumMSwTEZEERUoYZjYeqAX+X1i2Tx6O/RIwyN1HAD8BHkwdL8O23kZwM82swcwampqa8hCWiIhkEiVhfAO4DnjA3Vea2ZHAU7ke2N23ufv28PXDQA8zO4TgiuKItE0HABvbeJ957l7t7tX9+vXLNSwREcmi3SsFd18CLAEIG783u/vXcz2wmf0d8Gd3dzMbS5C8tgDvA0eb2WDgHeAC4MJcjyciIrlpN2GY2S+BrwE7gWXAgWb2I3e/pZ39FgETgUPMrBGYA/QAcPc7gC8Al5tZC/AhcIG7O9BiZlcBjwLdgfnuvrKT/z4REckTC76j29jAbLm7jzSzWmA0cA2wzN2rChFgR1RXV3tDQ0PSYYiIlAwzW+bu1VG2jdKG0SO87+Is4LfuvoM2GqFFRKRripIw/g1YD+wPPGNmg4BtcQYlIiLFJ0qj91xgblrRBjObFF9IIiJSjKIMDXKgmf0oda+Dmf2Q4GpDRETKSJQqqfnAB8B54WMbcHecQYmISPGJcsf2Ue5+btryDWa2PK6ARESkOEW5wvjQzE5KLZjZiQT3TYiISBmJcoVxObDQzA4kGOdpKzAjzqBERKT4ROkltRwYYWa9w2V1qRURKUNZE4aZ/VOWcgDc/UcxxSQiIkWorSuMXgWLQkREil7WhOHuNxQyEBERKW5RekmJiIgoYYiISDRKGCIiEkmUCZT2Bc4FKtO3d/cb4wtLRESKTZQb934L/IVgtr2/xRuOiIgUqygJY4C7nxZ7JCIiUtSitGE8b2bDO/rGZjbfzDaZ2atZ1tea2Svh43kzG5G2br2ZrTCz5WamOVdFRIpAlCuMk4AZZvYmQZWUAR5hTu8FwO3APVnWvwlMcPf3zOx0YB4wLm39JHffHCE+EREpgCgJ4/TOvLG7P2NmlW2sfz5t8UVgQGeOIyIihdFulZS7bwD6AFPCR5+wLJ++DPwu/bDAY2a2zMxm5vlYIiLSCVGmaJ0F1AOfDh+/MLN/zFcA4fzgXwauSSs+0d1HEVzdXGlmJ7ex/8zU9LFNTU35CktERFqJ0uj9ZWCcu1/v7tcDJwCX5ePgZlYF/Dswzd23pMrdfWP4vAl4ABib7T3cfZ67V7t7db9+/fIRloiIZBAlYRiwM215Z1iWEzMbCPwG+JK7r0kr39/MeqVeAzVAxp5WIiJSOFEave8G/mBmD4TLZwF3tbeTmS0CJgKHmFkjMAfoAeDudwDXAwcDPwvn2Ghx92rgUOCBsGwf4Jfu/kgH/k0iIhIDc/f2NzIbRdC91oBn3P3/xx1YZ1RXV3tDg27bEBGJysyWhT/W29XWjHu93X2bmR0ErA8fqXUHufvWXAMVEZHS0VaV1C+BMwnGkEq/DLFw+cgY4xIRkSLT1ox7Z4bPgwsXjoiIFKso92E8GaVMRES6trbaMCqAngS9nPrySVfa3sBhBYhNRESKSFttGF8FvkGQHJbxScLYBvw05rhERKTItNWGcRtwm5n9o7v/pIAxiYhIEWr3xj13/4mZHQccC1SklWcbtlxERLqgKHN6zyG4Y/tY4GGCAQGfJfs8FyIi0gVFGUvqC8Bk4D/d/RJgBLBvrFGJiEjRiZIwPnT3XUCLmfUGNqGb9kREEldfD5WV0K1b8FxfH+/xogw+2GBmfYA7CXpLbQf+GGtUIiLSpvp6mDkTmpuD5Q0bgmWA2tp4jhlp8MHdGwdTrvZ291fiCSc3GnxQRMpFZWWQJFobNAjWr4/+PvkafHBUW+vc/aXoIYmISD699VbHyvOhrSqpH4bPFUA18DLBzXtVwB8IhjsXEZEEDByY+Qpj4MD4jpm10dvdJ7n7JGADMCqcBnU0cDywLr6QRESkPTfdBD177lnWs2dQHpcovaSOcfcVqQV3fxUYGV9IIlIqCt1Lp6vJ5fzV1sK8eUGbhVnwPG9efA3eEKHRO5xq9a/ALwjmwfgH4AB3nx5fWJ2jRm+RwmndSweCX7hxf2l1FcVy/jrS6B3lCuMSYCUwi2AwwlVhWZRA5pvZJjN7Nct6M7O5ZrbOzF5Jb2g3s4vNbG34uDjK8UTKTZK/8GfP3vPLDoLl2bMLF0MpK8Xz16FutR1+c7OTCe7buMfdj8uw/gzgH4EzgHHAbe4+LpwWtoGgsd0J7v8Y7e7vtXU8XWFIOUn6F2q3bpDp68MMdu2K//ilrljOX16uMMzsV+HzivDX/x6PKG/u7s8Abc39PY0gmbi7vwj0MbP+wKnA4+6+NUwSjwOnRTmmSLlI+hdqtt44cfbS6UpK8fy1VSU1K3w+E5iS4ZEPhwNvpy03hmXZykUklEQ//HRJ9NLpSvJ5/urq8hJSu9rqVvtu+Lwh0yNPx7cMZd5G+d5vYDbTzBrMrKGpqSlPYYkUv6R/oSbRS6cryef5u+GG/MeXSVtVUh+Y2bYMjw/MbFuejt8IHJG2PADY2Eb5Xtx9XniPSHW/fv3yFJZI8SuGX/i1tcEwFLt2Bc/llixy7XRQauevrSuMXu7eO8Ojl7v3ztPxHwIuCntLnQD8JbyyeRSoMbO+4XziNWGZiIT0Cz9ZqU4HGzYEjdepwf8K1VOtri74u1tYH5N6HWf1VOReUmb2afacca/dmtLwHo6JwCHAn4E5QI9w/zvMzIDbCRq0m4FL3L0h3PdS4FvhW93k7ne3dzz1khKRQsnX4H/5YJa5x1W0faP3kopy495UgnGlDiOYC2MQsNrdh3UuvPgoYYh0TH190KvqrbeCto+bbtIVSlTF0i02dcxCJIwoN+59FzgBWOPugwlm33uuc6GJSLFIukql1CXd6SDdnDmFOU6UhLHD3bcA3cysm7s/hcaSEil5Sd/HUQxyabQuhk4HKYl3q03zvpkdADwD1JvZbUBLvGGJlIckh/ZI+j6OpOV6hVWOnQ6itGHsD3xEcG9ELXAgUB9edRQVtWFIKUl6aI9iarRNQrn/+1PyNTTI7Wb2WXf/q7vvdPcWd1/o7nOLMVmIlJqkq4SKqUolCeV+hdUZbVVJrQV+aGbrzexmM1O7hUgeJf2F1RWqVHKp0iumRutS0daNe7e5+3hgAsEAgneb2Wozu97MhhQsQpEuqhi+sErtTuN0ubZBlPsVVme02+gdjh11s7sfD1wInA2sjj0ykQhKecY3fWHlJtcqva5whVVoURq9exDciX0BwT0YS4BF7v5g/OF1jBq9y0vSjcb5oBvnOq+YbpwrZXm509vMTgGmA58H/gjcCzzo7n/NV6D5poRRXtTLpbzp758f+brT+1vAC8BQd5/i7vXFnCyk/CTdaAylXSVW6lSlV3j7ZFvh7pMKGYhIRw0cmPkXZqEajVtXiaUaXUHVSoWQOseq0iucWOf0LjRVSZWXpNswVCUiXUG+Bx8UKUpJ93IphioxkUJSwpCSlut9BLrxSyQ6JQwpW7rxS6RjlDCkbOnGL0kp1PDgxXr8qNToLWVLN35JSi4z1pX68Yum0dvMTjOzP5nZOjO7NsP6W81sefhYY2bvp63bmbbuoTjjlPKkNgiRjoktYZhZd+CnwOnAscB0Mzs2fRt3v9rdR7r7SOAnwG/SVn+YWufuU+OKU8qX2iDKW11d8MveLFhOvS5U9VDSx++MOK8wxgLr3P0Nd/+YYGiRaW1sPx1YFGM8UoSSvFNabRDlra4uqAZKVQWlXhcyYSR5/M6IM2EcDrydttwYlu3FzAYBg4HFacUVZtZgZi+a2VnZDmJmM8PtGpqamvIRtxRIrr2U8qGUh/eW4lHMX/L5FGfCsAxl2Zp1LgB+7e4708oGhg0xFwI/NrOjMu3o7vPcvdrdq/v165dbxFJQSc84J5IyZ05u+99wQ7LHL5Q4E0YjcETa8gBgY5ZtL6BVdZS7bwyf3wCeBo7Pf4iSJN0pLcUi6SuEpI8fVZwJYylwtJkNNrNPESSFvXo7mdlngL4EI+Omyvqa2b7h60OAE4FVMcYqCVAvJSllpdhonavYEoa7twBXAY8SzND3K3dfaWY3mll6r6fpwL2+5w0hQ4EGM3sZeAr4V3dXwuhi1EtJSlkpNlrnSjfuSaI045x0BUnf+JeLjty4l3U+DJFCqK1VgpDSVyqN1rnSWFIiIjnqytVQ6ZQwREQkEiUMESl55fILP2lKGCJS8nK9cU6iUcIQEZFIlDAkJ0kOHijFI9cqoc7sX443ziVN92FIp6UGD0wfD6pnT434Wo5yvQ8h6f3LWdFMoCRdmwYPFCkvShjSaRo8sLzlWiWUzyqlcrlxLmmqkpJOq6wM5rBobdCgYG4JKR+qUipdqpKSgtDggSLlRQlDOk1TnEpKrlVCqlIqDaqSEhEpY6qSEpEO0b0LEoUShohoaA2JRAlDpAvQFYIUQqwJw8xOM7M/mdk6M7s2w/oZZtZkZsvDx1fS1l1sZmvDx8VxxilS6jpzhaChNaSjYmv0NrPuwBrgFKARWApMT5+b28xmANXuflWrfQ8CGoBqwIFlwGh3f6+tY6rRW5JSV5fsF63ug5DOKpZG77HAOnd/w90/Bu4FpkXc91TgcXffGiaJx4HTYopTJGe5tgFo8D0pBXEmjMOBt9OWG8Oy1s41s1fM7NdmdkQH95UcabTZ4tDZKiX3T64MUq81tIbEJc6EYRnKWl/0/h+g0t2rgCeAhR3YN9jQbKaZNZhZQ1NTU6eDLVW5fOGnRpvdsCH4otmwIVhW0oimK/3CL8WYpfDiTBiNwBFpywOAjekbuPsWd/9buHgnMDrqvmnvMc/dq929ul+/fnkJvFTk+oWv0WZzk+svfA2+J6UmzkbvfQgavScD7xA0el/o7ivTtunv7u+Gr88GrnH3E8JG72XAqHDTlwgavbe2dcxya/TOdfC/bt0yN3Sawa5duUZXXtToLKWqI43e+8QVhLu3mNlVwKNAd2C+u680sxuBBnd/CPi6mU0FWoCtwIxw361m9l2CJANwY3vJohzlOrz4wIGZE87AgZ2PqVzpF76UA40lVcJyvcLQjHmfSLpbbNLHl/JVLN1qJWa5Di+u0WY/kfTQGEoWUgpiq5KS+KW+2GfPDqqhBg4MkkVHvvBra8szQYhIx+kKo8TV1gbVT7t2Bc/68o+uK3WLFSkEJQzpEjp7p3S+bnwTKQdKGNIlJN0GIVIOlDBEULdYkSiUMKRk5bMNQtVQIu1TwpCioDYIkeKnhJGwrjJabK5f0mqDECl+ShgJ6kqjxSb9ha82CJH4KWGEkqjGKPfRYtUGIVJalDBCSfxCznXwwKTl+oWvNgiR0qKEkaNc2iCyjQqbxGixanQWkfaUdcLI9Rdyrm0QuQ4emE9qgxCR9mh481BnJrDJdXhxCJJLLoMHpuQ6PHauE/hoeG6R0qThzQskH20QqcEDr78+t8EDO3OFoEZnEemIsk8YqTYISLYNIokqIbVBiEhHlHXCSG+DgNJrg9Dw3CJSSLEmDDM7zcz+ZGbrzOzaDOv/ycxWmdkrZvakmQ1KW7fTzJaHj4fiiC/X+yDWrs28/9q10fYvpm6panQWkfbE1uhtZt2BNcApQCOwFJju7qvStpkE/MHdm83scmCiu58frtvu7gd05JgdbfTu1i1zQ69ZMCFRR+TaaJz0/iJSnoql0XsssM7d33D3j4F7gWnpG7j7U+6e+o3+IjAgxnj2Ukz3QeRKVwgiErc4E8bhwNtpy41hWTZfBn6XtlxhZg1m9qKZnRVHgPlsg8j1CzvX/dVuISJx2yfG97YMZRkrTczsH4BqYEJa8UB332hmRwKLzWyFu7+eYd+ZwEyAgR28NEh1Yc3XfRC50Be+iBS7OBNGI3BE2vIAYGPrjczsc8BsYIK7/y1V7u4bw+c3zOxp4Hhgr4Th7vOAeRC0YXQ0yNrazt/7ICJSTuKskloKHG1mg83sU8AFwB69nczseODfgKnuvimtvK+Z7Ru+PgQ4EViFiIgkJrYrDHdvMbPK05XlAAAGqElEQVSrgEeB7sB8d19pZjcCDe7+EHALcABwnwV9S99y96nAUODfzGwXQVL71/TeVSIiUngaS0pEpIwVS7daERHpQpQwREQkki5VJWVmTUCGAceLwiHA5qSDaIPiy43iy43iy00u8Q1y935RNuxSCaOYmVlD1HrCJCi+3Ci+3Ci+3BQqPlVJiYhIJEoYIiISiRJG4cxLOoB2KL7cKL7cKL7cFCQ+tWGIiEgkusIQEZFIlDDyyMyOMLOnzGy1ma00s1kZtploZn9Jm03w+gLHuN7MVoTH3uu2eAvMDWdJfMXMRhUwts+knZflZrbNzL7RapuCnj8zm29mm8zs1bSyg8zscTNbGz73zbLvxeE2a83s4gLGd4uZvRb+/R4wsz5Z9m3zsxBjfHVm9k7a3/CMLPu2OWNnjPH9R1ps681seZZ9C3H+Mn6nJPYZdHc98vQA+gOjwte9CGYcPLbVNhOB/5tgjOuBQ9pYfwbBvCQGnEAwI2IScXYH/pOgj3hi5w84GRgFvJpW9r+Aa8PX1wI3Z9jvIOCN8Llv+LpvgeKrAfYJX9+cKb4on4UY46sD/keEv//rwJHAp4CXW/9fiiu+Vut/CFyf4PnL+J2S1GdQVxh55O7vuvtL4esPgNW0PWlUMZoG3OOBF4E+ZtY/gTgmA6+7e6I3Yrr7M8DWVsXTgIXh64VApgm+TgUed/et7v4e8DhwWiHic/fH3L0lXCz4TJatYsl0/qJod8bOfGgrPgtGRD0PWJTv40bVxndKIp9BJYyYmFklwRwef8iweryZvWxmvzOzYQUNLJjE6jEzWxZOPtVaR2dKjMsFZP+PmuT5AzjU3d+F4D808OkM2xTLebyUPWeyTNfeZyFOV4VVZvOzVKcUw/n778Cf3X1tlvUFPX+tvlMS+QwqYcTAzA4A7ge+4e7bWq1+iaCaZQTwE+DBAod3oruPAk4HrjSzk1utjzxTYlwsmD9lKnBfhtVJn7+oiuE8zgZagPosm7T3WYjLz4GjgJHAuwTVPq0lfv6A6bR9dVGw89fOd0rW3TKU5XQOlTDyzMx6EPxh6939N63Xu/s2d98evn4Y6GHBJFEF4Z/MZLgJeIDg0j9dpJkSY3Y68JK7/7n1iqTPX+jPqWq68HlThm0SPY9hA+eZQK2HFdqtRfgsxMLd/+zuO919F3BnluMmff72Ac4B/iPbNoU6f1m+UxL5DCph5FFY53kXsNrdf5Rlm78Lt8PMxhL8DbYUKL79zaxX6jVB4+irrTZ7CLgo7C11AvCX1KVvAWX9ZZfk+UvzEJDqcXIx8NsM2zwK1Fgwe2RfgnP9aCGCM7PTgGsIZrJszrJNlM9CXPGlt4mdneW47c7YGbPPAa+5e2OmlYU6f218pyTzGYyzhb/cHsBJBJd8rwDLw8cZwNeAr4XbXAWsJOj18SLw2QLGd2R43JfDGGaH5enxGfBTgh4qK4DqAp/DngQJ4MC0ssTOH0HiehfYQfCL7cvAwcCTwNrw+aBw22rg39P2vRRYFz4uKWB86wjqrlOfwTvCbQ8DHm7rs1Cg+P53+Nl6heCLr3/r+MLlMwh6Bb1eyPjC8gWpz1zatkmcv2zfKYl8BnWnt4iIRKIqKRERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDpB1mttP2HEU3byOnmlll+kipIsVsn6QDECkBH7r7yKSDEEmarjBEOimcD+FmM/tj+PhvYfkgM3syHFzvSTMbGJYfasH8FC+Hj8+Gb9XdzO4M5zt4zMz2C7f/upmtCt/n3oT+mSK7KWGItG+/VlVS56et2+buY4HbgR+HZbcTDBFfRTDw39ywfC6wxIOBE0cR3CEMcDTwU3cfBrwPnBuWXwscH77P1+L6x4lEpTu9RdphZtvd/YAM5euBv3f3N8IB4v7T3Q82s80Ew13sCMvfdfdDzKwJGODuf0t7j0qCOQuODpevAXq4+7+Y2SPAdoIReR/0cNBFkaToCkMkN57ldbZtMvlb2uudfNK2+HmCcb1GA8vCEVRFEqOEIZKb89OeXwhfP08wuipALfBs+PpJ4HIAM+tuZr2zvamZdQOOcPengP8J9AH2usoRKST9YhFp335mtjxt+RF3T3Wt3dfM/kDw42t6WPZ1YL6Z/TPQBFwSls8C5pnZlwmuJC4nGCk1k+7AL8zsQIIRhG919/fz9i8S6QS1YYh0UtiGUe3um5OORaQQVCUlIiKR6ApDREQi0RWGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpH8F0byR/8g3dS9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bigger_model_val_loss = bigger_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, bigger_model_val_loss, 'bo', label='Bigger model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "용량이 큰 네트워크는 첫 번째 에포크 이후 거의 바로 과대적합이 시작되어 갈수록 더 심해집니다. 검증 손실도 매우 불안정합니다.\n",
    "\n",
    "다음은 두 네트워크의 훈련 손실입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xt8VNW99/HPj4giiopI+6BIghyogASEAKW1oocabwW8K/JUEWtahVc5ts85Qmkl2mN9rJ5aOdpaWhH6kGq9nHo8fal4QWm1XggWRS4KKMGIrYAX5BWpBH7PH3syTuJMMsnMnj2T+b5fr3nN7DVr7/3LzmR+2WuvvZa5OyIiIgBdog5ARETyh5KCiIjEKSmIiEickoKIiMQpKYiISJySgoiIxCkpiIhInJKCiIjEKSmIiEjcflEH0F5HHHGEl5WVRR2GiEhBWbly5XZ3791WvYJLCmVlZdTW1kYdhohIQTGzunTqqflIRETilBRERCROSUFEROIK7pqCiOTOnj17qK+vZ/fu3VGHImnq1q0bffv2pWvXrh1aX0lBRFKqr6+nR48elJWVYWZRhyNtcHd27NhBfX09/fv379A2iqL5qKYGysqgS5fguaYm6ohECsPu3bvp1auXEkKBMDN69eqV0Zldpz9TqKmBqipoaAiW6+qCZYCpU6OLS6RQKCEUlkx/X53+TGHu3M8SQpOGhqBcRESa6/RJYcuW9pWLSH6pr69n8uTJDBw4kAEDBjBr1iw+/fTTpHW3bt3Keeed1+Y2zzjjDD788MMOxVNdXc0tt9zSoXXTtWjRImbOnJlxnY7o9EmhX7/2lYtI5qqrs7Mdd+ecc87hrLPOYsOGDbzxxhvs2rWLuUlO9RsbGznyyCN54IEH2tzuI488wmGHHZadIDuZTp8UbrgBundvXta9e1AuIuG47rrsbGfZsmV069aNyy67DICSkhJuvfVWFi5cSENDA4sWLeL8889n4sSJVFZWsnnzZo477jgAGhoauOCCCygvL+fCCy9k7Nix8SFyysrK2L59O5s3b2bw4MFcccUVDB06lMrKSj755BMAfv3rXzN69GiGDx/OueeeS0PLdugWpk2bxpVXXsnJJ5/MMcccw/Lly5k+fTqDBw9m2rRp8Xr33HMPw4YN47jjjuOaa66Jl999990MGjSI8ePH89xzz8XLt23bxrnnnsvo0aMZPXp0s/fC0OmTwtSpsGABlJaCWfC8YIEuMosUgjVr1jBq1KhmZYcccgj9+vVj48aNADz//PMsXryYZcuWNav3i1/8gp49e/Lqq6/yox/9iJUrVybdx4YNG5gxYwZr1qzhsMMO48EHHwTgnHPOYcWKFbzyyisMHjyYu+66q814P/jgA5YtW8att97KxIkTufrqq1mzZg2rV69m1apVbN26lWuuuYZly5axatUqVqxYwUMPPcS7777LvHnzeO6553jiiSdYu3ZtfJuzZs3i6quvZsWKFTz44IN861vfatcxbK9O3/sIggSgJCASrurq5mcITZ1g5s3reHOSuyftTZNYfsopp3D44Yd/rs6zzz7LrFmzADjuuOMoLy9Puo/+/fszYsQIAEaNGsXmzZsBeO211/jhD3/Ihx9+yK5duzj11FPbjHfixImYGcOGDeOLX/wiw4YNA2Do0KFs3ryZuro6TjrpJHr3DgYrnTp1Kn/6058AmpVfeOGFvPHGGwA8+eSTzZLEzp07+fjjj9uMpaOKIimISPiqqz/78jcD98y3OXTo0Ph/7k127tzJ22+/zYABA1i5ciUHHXRQ0nU9zQAOOOCA+OuSkpJ489G0adN46KGHGD58OIsWLeKZZ55Je1tdunRptt0uXbrQ2NjIfvul/spN1ZV03759PP/88xx44IHp/DgZ6/TNRyJSuCZMmEBDQwO//e1vAdi7dy/f//73mTZtGt1bXixs4YQTTuC+++4DYO3ataxevbpd+/7444/p06cPe/bsoSZLd7yOHTuW5cuXs337dvbu3cs999zD+PHjGTt2LM888ww7duxgz5493H///fF1Kisruf322+PLq1atykosqSgpiEjWzZuXne2YGX/4wx+4//77GThwIIMGDaJbt2785Cc/aXPdq666im3btlFeXs5NN91EeXk5hx56aNr7/vGPf8zYsWM55ZRTOPbYYzP5MeL69OnDjTfeyMknn8zw4cMZOXIkkydPpk+fPlRXVzNu3Di+/vWvM3LkyPg68+fPp7a2lvLycoYMGcKdd96ZlVhSsXRPsfJFRUWFa5IdkdxYt24dgwcPjjqMDtm7dy979uyhW7dubNq0iQkTJvDGG2+w//77Rx1a6JL93sxspbtXtLWurimISKfU0NDAySefzJ49e3B3fvnLXxZFQsiUkoKIdEo9evTQ1L0doGsKIiISp6QgIiJxoSYFMzvNzF43s41mNjvJ+9PMbJuZrYo9wr1VT0REWhXaNQUzKwHuAE4B6oEVZvawu69tUfX37p79of5ERKTdwjxTGANsdPc33f1T4F5gcoj7E5FOqKSkhBEjRsT79f/lL38B0h8mO58dfPDBWamTTWEmhaOAtxOW62NlLZ1rZq+a2QNmdnSI8YhIyMKY+vbAAw9k1apVvPLKK9x4443MmTMHIO1hsjPR2NgY6vbzUZhJIdlAHi3vlPsfoMzdy4EngcVJN2RWZWa1Zla7bdu2LIcpItnQNPVtXV0w7lHT1LfZnBN9586d9OzZEyDtYbLvuusuBg0axEknncQVV1wRn5gm1ZDU1dXVVFVVUVlZySWXXNJs/8888wzjx4/nggsuYNCgQcyePZuamhrGjBnDsGHD2LRpEwB1dXVMmDCB8vJyJkyYwJbYrF5vvfUW48aNY/To0fzoRz9qtu2bb76Z0aNHU15ezrxs3RLeEe4eygMYByxNWJ4DzGmlfgnwUVvbHTVqlItIbqxduzbtuqWl7kE6aP4oLc0shi5duvjw4cP9S1/6kh9yyCFeW1vr7u5vvfWWDx061N3db775Zq+qqnJ399WrV3tJSYmvWLHC33nnHS8tLfUdO3b4p59+6ieccILPmDHD3d2nTJnif/7zn93dva6uzo899lh3d583b56PHDnSGxoaPhfL008/7Yceeqhv3brVd+/e7UceeaRfe+217u7+85//3GfNmuXu7t/4xjd80aJF7u5+1113+eTJk93dfeLEib548WJ3d7/99tv9oIMOcnf3pUuX+hVXXOH79u3zvXv3+plnnunLly93d4/XaY9kvzeg1tP47g7zTGEFMNDM+pvZ/sBFwMOJFcysT8LiJGBdiPGISIjCmvq2qflo/fr1PPbYY1xyySWfGwH12Wef5aKLLgKaD5P90ksvMX78eA4//HC6du3K+eefH1/nySefZObMmYwYMYJJkyY1G5J60qRJKUclHT16NH369OGAAw5gwIABVFZWAjBs2LD4sNvPP/88F198MQDf/OY3efbZZwF47rnnmDJlSry8yeOPP87jjz/O8ccfz8iRI1m/fj0bNmzI6Lh1VGi9j9y90cxmAksJzgIWuvsaM7ueIGM9DHzXzCYBjcD7wLSw4hGRcPXrFzQZJSvPlnHjxrF9+3ZaNiO3TBJtlUPrQ1KnGo4b+NyQ2InDZae6BpE4LHaq+SHmzJnDt7/97ZT7zZVQ71Nw90fcfZC7D3D3G2Jl18YSAu4+x92Huvtwdz/Z3deHGY+IhCcXU9+uX7+evXv30qtXr2blqYbJHjNmDMuXL+eDDz6gsbGx2dwMYQ5J/ZWvfIV7770XgJqaGk444QQAvvrVrzYrb3LqqaeycOFCdu3aBcA777zDe++9l7V42kNjH4lIVjTNbjh3btBk1K9fkBAynfXwk08+ic+M5u4sXryYkpKSZnWuuuoqLr30UsrLyzn++OPjw2QfddRR/OAHP2Ds2LEceeSRDBkyJD589vz585kxYwbl5eU0NjZy4oknZm1Y6vnz5zN9+nRuvvlmevfuzd133w3AbbfdxsUXX8xtt93GueeeG69fWVnJunXrGDduHBB0Q12yZAlf+MIXshJPe2jobBFJqVCGzm5tmOxdu3Zx8MEH09jYyNlnn8306dM5++yzow45VBo6W0SKWmvDZFdXV/Pkk0+ye/duKisrOeussyKONr8pKYhIwWttmOxbbrklx9EUNo2SKiKtKrQm5mKX6e9LSUFEUurWrRs7duxQYigQ7s6OHTvo1q1bh7eh5iMRSalv377U19d/7r4AyV/dunWjb9++HV5fSUFEUuratSv9+/ePOgzJITUfiYhInJKCiIjEKSmIiEickoKIiMQVVVKoro46AhGR/FZUSeG666KOQEQkvxVVUhARkdZ1+qRQXQ1mwQM+e62mJBGRzyuqobPNglljRUSKTbpDZ3f6MwUREUlfUSWFefOijkBEJL8VVVLQdQQRkdYVVVIQEZHWKSmIiEickoKIiMQpKYiISJySgoiIxCkpiIhInJKCiIjEhZoUzOw0M3vdzDaa2exW6p1nZm5mbd6CLSIi4QktKZhZCXAHcDowBJhiZkOS1OsBfBd4MaxYREQkPWGeKYwBNrr7m+7+KXAvMDlJvR8DPwV2hxiLiIikIcykcBTwdsJyfawszsyOB4529z+GGIeIiKQpzKRgScriA1ebWRfgVuD7bW7IrMrMas2sdtu2bVkMUUREEoWZFOqBoxOW+wJbE5Z7AMcBz5jZZuDLwMPJLja7+wJ3r3D3it69e4cYsohIcQszKawABppZfzPbH7gIeLjpTXf/yN2PcPcydy8DXgAmuXvHZtAREZGMhZYU3L0RmAksBdYB97n7GjO73swmhbVfERHpuP3C3Li7PwI80qLs2hR1TwozFhERaZvuaBYRkTglBRERiVNSEBGROCUFERGJU1IQEZE4JQUREYlTUhARkTglBRERiVNSEBGROCUFERGJa1dSsMBBYQUjIiLRajMpmNlvzewQM+sOrAHeMrPvhR+aiIjkWjpnCsPcfSdwFvA4wbwI08IMSkREopFOUtjfzPYjmF/5odh8y/vCDUtERKKQTlL4DbAF6AksN7N+wK5QoxIRkUi0mRTc/VZ3P9LdK93dgbeBfw4/NBERybV0LjTPNLNDYq9/BbwIfC3swEREJPfSaT6qcvedZlYJHAVcCfw03LBERCQK6SQFjz2fDtzt7ivTXE9ERApMOl/ur5jZI8BE4FEzO5jPEoWIiHQi+6VR5zJgFLDR3RvM7Ajg8nDDEhGRKLSZFNx9bywRnGNmAMvd/dHQIxMRkZxLp/fRDcC/AW/GHv9qZv8edmAiIpJ76TQfTQRGunsjgJktBF4GfhhmYCIiknvp9iLqkeK1iIh0IumcKfwUeNnMngIMOAm4NsygREQkGulcaF5iZk8DYwmSwrXu/k7okYmISM6lTApmVt6iaGPsuZeZ9XL3V8MLS0REotDamcIdrbznwIltbdzMTgNuA0qA37j7/23x/neAGcBegpFXq9x9bVvbFRGRcKRMCu6e0aB3ZlZCkFhOAeqBFWb2cIsv/d+5+52x+pOAnwGnZbJfERHpuDDHMBpDcBf0m7GJee4lmKgnLjajW5OD0PAZIiKRSqf3UUcdRTD3QpN6govVzZjZDOB7wP6kmKfBzKqAKoB+/fplPVAREQmEeaZgSco+dybg7ne4+wDgGlLcEOfuC9y9wt0revfuneUwRUSkSZtnCkl6IQF8BLzt7q3N1VwPHJ2w3BfY2kr9e4FfthWPiIiEJ53mo7uAEcAagv/+BwOvAYeaWZW7P5VivRXAQDPrD7wDXARcnFjBzAa6+4bY4pnABkREJDLpNB9tAEa5+wh3H04wjPYq4FTgP1KtFBsraSawFFgH3Ofua8zs+lhPI4CZZrbGzFYRXFe4NIOfRUREMpTOmcLgxBvV3H21mY10942xobRTcvdHgEdalF2b8HpWO+MVEZEQpZMUNpnZfxK0+QNcCGw0swOAxtAiExGRnEun+egSgovGs4E5BBeLLyVICBPCC01ERHItnQHxGoCbYo+WPsp6RCIiEpl0uqR+GZgHlCbWd/dBIcYlIiIRSOeawt0E03GuJBi4TkREOql0ksJOd/+f0CMREZHIpZMUlpnZjcB/Af9oKtR8CiIinU86SeGEFs+Q5nwKnU11dfAQEemszL2wRquuqKjw2traSPZtBgV2uEREADCzle5e0Va91qbjnOLu95jZd5O97+7zMwlQRETyT2s3r/WMPfdO8SgK1dXBGULTiB5Nr9WMJCKdkZqP2kHNRyJSqDJuPkrY0BHAdKCM5jevVWUSoIiI5J90eh/9N/AC8CxFfvPavHlRRyAiEq50ksJB7v790CMpALqOICKdXTqjpD5qZpWhRyIiIpFLJyl8B3jMzHaZ2ftm9oGZvR92YCIiknvpNB8dEXoUIiKSF1q7eW2gu28AhqaoorGPREQ6mdbOFGYDlwN3JHmvKMc+EhHp7FImBXe/PPb8tdyFIyIiUUrnQjNmdqyZnWNmFzc9wg4sn9TUQFkZdOkSPNfURB2RiEg40rmj+YdAJXAssBQ4leBGtt+FG1p+qKmBqipoaAiW6+qCZYCpU6OLS0QkDOmcKVwInAy86+7fBIaTXq+lTmHu3M8SQpOGhqBcRKSzSScpfOLue4FGM+sB/A04Jtyw8seWLe0rFxEpZOkkhb+a2WHAQqAWeAl4OdSo8ki/fu0rFxEpZK0mBTMzoNrdP3T3O4AzgW+7+yU5iS4P3HADdO/evKx796BcRKSzaTUpeDDZwh8Tlje6e9pnCWZ2mpm9bmYbzWx2kve/Z2ZrzexVM3vKzErbFX0OTJ0KCxZAaWkwn0JpabCsi8wi0hml03z0kpmNbO+GzayE4Ma304EhwBQzG9Ki2l+BCncvBx4Aftre/eTC1KmweTPs2xc8dzQhaJRVEcl3KZOCmTX1MDqBIDG8bmYvm9lfzSyds4UxwEZ3f9PdPwXuBSYnVnD3p929qW/PC0Df9v8IheO666KOQESkda11LX0JGAmc1cFtHwW8nbBcD4xtpf7lwKPJ3jCzKqAKoJ+u8IqIhKa15iMDcPdNyR5pbNuSlCWd4djM/jdQAdyc7H13X+DuFe5e0bt37zR2nT+qq4NrERY7Gk2v1ZQkIvmotTOF3mb2vVRvuvvP2th2PXB0wnJfYGvLSmb2dWAuMN7d/9HGNgtOdfVnCcAMPGlaFBHJD60lhRLgYJL/x5+OFcBAM+sPvANcBDQbM8nMjgd+BZzm7u91cD8iIpIlrSWFd939+o5u2N0bzWwmwXhJJcBCd19jZtcDte7+MEFz0cHA/cEtEWxx90kd3We+mzcv6ghERFpnnqI9w8z+6u7H5zieNlVUVHhtbW3UYYiIFBQzW+nuFW3Va+1C84QsxiMiIgUgZVJw9/dzGYiIiEQvrUl2RESkOCgpiIhInJJCAdENbyISNiWFAqKxk0QkbEoKIiISp6SQ5zR2kojkUsqb1/JVMd+8prGTRKSjsnHzmoiIFBklhQJSrGMn1dRAWRl06RI819REHZFI59XagHiSZ4rxOkJNDVRVQUNsfr66umAZNE+2SBh0plBECjGpzJ37WUJo0tAQlItI9ikpFJFCvM9hy5b2lYtIZpQUJK+lmpJbU3WLhENJoZMr9PscbrgBundvXta9e1AuItmn+xSKSKHe51BTE1xD2LIlOEO44QZdZBZpL92nIJ3G1KmweTPs2xc8KyHknroFFw8lhSKS6X0OhdLkJNnV1C24ri4402zqFqzE0Dmp+UjSVqjNT5KZsrIgEbRUWhqcuUlhUPORiGSFugUXFyUFaVWh917qLKJs01e34OKipJADhXyRrro6aDJqajZqeq2kkDtRt+mrW3BxUVIIWdR/0FL4oh7qY+pUWLAguIZgFjwvWKBeYJ2VkkLIov6Dzib1XopGPrTpq1tw8VDvo5B16ZK8x45Z8AdWTNR7qWPU+0eyQb2P8oQu0kmm1KYvuRRqUjCz08zsdTPbaGazk7x/opm9bGaNZnZemLFEpdj/oNV7KXNq05dcCi0pmFkJcAdwOjAEmGJmQ1pU2wJMA34XVhxRy4c/6Ch7P2Wz91IxJxK16UuuhHZNwczGAdXufmpseQ6Au9+YpO4i4I/u/kBb2y20awpRazlzGQRnKlH8p5npNQVdkxDpuHy4pnAU8HbCcn2sTHIon3o/Fesc0yKFJMykYEnKOvR/nplVmVmtmdVu27Ytw7CKSz50Z2zS0SYjXZMQyZ0wk0I9cHTCcl9ga0c25O4L3L3C3St69+6dleCKRaH3ftId1SK5FWZSWAEMNLP+ZrY/cBHwcIj7kySKvfdTIiUSkbaFlhTcvRGYCSwF1gH3ufsaM7vezCYBmNloM6sHzgd+ZWZrwoqnWOVD76dsyfSaxHXXZScOkc5MdzRL0VDvJSlm+dD7SCRyulAt0j5KCtKp6eY5kfZRUhBJk65JSDFQUpCiEfXNczrTkEKgpFAACnnmtnwS9c1zOtOQQqDeR3kun8YuKnYau0kKmXofdRL5NHaRtJ96P0mhUVLIc/k0dlGx68g1iWwP06FkImFTUshzhT52UWeSD1/Iui4hYVNSyHMau6jziLr3E0SX2NRZonAoKeS5zjR2UbHLpMmokHtANXWWqKsLms7q6oJlJYb8pN5HIgWkEHtAlZUFiaCl0tJgalHJDfU+EhEg+h5Q6ixRWJQURApI1D2gOrKOOksUFiUFkQISdQ+ojlyTUGeJwqKkIFJEougBldhZAtRZIt8pKYgUkajGf5o69bOLyps3KyHkMyUFEWlVtu/KzjQWCZeSgoiEKp/us1BSaZvuUxCRtFVXZ/bFGvV9FsU8Uq3uUxCRrIuqyShfRpothjMNJQURyZmO3mexZEnz3ktLlqT/Ba3mq/ZR85GI5LVsTjQVdfNTputn0nyn5iMR6RSinmgqn5qvcjGgoZKCiOS1bI6dFMUwIfmUVNKhpCAieS2bYydFdXG6kJKKkoKI5LV8GjspimFCcn3zYKhJwcxOM7PXzWyjmc1O8v4BZvb72PsvmllZmPFIx2jWLIlSPk00lekXcT7MvteW0JKCmZUAdwCnA0OAKWY2pEW1y4EP3P2fgFuBm8KKRzomG7NmRZ1UMt1/1OtnQz7EkImmsZP27evY2ElR/w6b1r/++szWhxz8/tw9lAcwDliasDwHmNOizlJgXOz1fsB2Yt1kUz1GjRrlkjulpU0nq80fpaXprb9kiXv37s3X7d49KM+FTPcf9frZkA8xRCnq32HU6zcBaj2d7+50KnXkAZwH/CZh+ZvA7S3qvAb0TVjeBBzR2naVFHLLLHlSMEtv/UyTSqYy3X/U62dDPsQQpah/h1Gv3yTdpBDmNQVLUtbyto106mBmVWZWa2a127Zty0pwkp5Me35EPRVjpvuPev1syIcYohT17zDq9dsrzKRQDxydsNwX2JqqjpntBxwKvN9yQ+6+wN0r3L2id+/eIYUryWTa8yPqqRgz3X/U62dDPsQQpah/h1Gv327pnE505EFwjeBNoD+wP/AKMLRFnRnAnbHXFwH3tbVdNR/l3pIlwamqWfDcnrbMqNuzo27Pjfrnz5cYohT17zDq9ZsQ9TWFIAbOAN4guFYwN1Z2PTAp9robcD+wEXgJOKatbSopFJ5Mkko+7D/q9bMhH2KIUtS/w6jXd08/KWhAPBGRIqAB8UREpN2UFEREJE5JQURE4pQUREQkTklBRETiCq73kZltA+qijiOFIwjGb8pXii8z+R4f5H+Mii8zmcRX6u5t3v1bcEkhn5lZbTpdvqKi+DKT7/FB/seo+DKTi/jUfCQiInFKCiIiEqekkF0Log6gDYovM/keH+R/jIovM6HHp2sKIiISpzMFERGJU1JoJzM72syeNrN1ZrbGzGYlqXOSmX1kZqtij2tzHONmM1sd2/fnRg+0wHwz22hmr5rZyBzG9qWE47LKzHaa2b+0qJPz42dmC83sPTN7LaHscDN7wsw2xJ57plj30lidDWZ2aY5iu9nM1sd+f38ws8NSrNvqZyHkGKvN7J2E3+MZKdY9zcxej30eZ+cwvt8nxLbZzFalWDfUY5jqOyWyz186Q6nq0Ww48D7AyNjrHgRDgw9pUeck4I8RxriZVqY1JRjS/FGCme++DLwYUZwlwN8I+k9HevyAE4GRwGsJZT8FZsdezwZuSrLe4QTzhhwO9Iy97pmD2CqB/WKvb0oWWzqfhZBjrAb+TxqfgU3AMXw278qQXMTX4v3/AK6N4him+k6J6vOnM4V2cvd33f3l2OuPgXXAUdFG1W6Tgd964AXgMDPrE0EcE4BN7h75zYju/ic+P+vfZGBx7PVi4Kwkq54KPOHu77v7B8ATwGlhx+buj7t7Y2zxBYKZDSOT4vilYwyw0d3fdPdPgXsJjntWtRafmRlwAXBPtvebjla+UyL5/CkpZMDMyoDjgReTvD3OzF4xs0fNbGhOAwvmuX7czFaaWVWS948C3k5YrieaxHYRqf8Qozx+Tb7o7u9C8IcLfCFJnXw4ltMJzvySaeuzELaZsSauhSmaP/Lh+H0N+Lu7b0jxfs6OYYvvlEg+f0oKHWRmBwMPAv/i7jtbvP0yQZPIcOA/gYdyHN5X3X0kcDoww8xObPG+JVknp93QzGx/YBLBzHstRX382iPSY2lmc4FGoCZFlbY+C2H6JTAAGAG8S9BE01Lkn0VgCq2fJeTkGLbxnZJytSRlGR0/JYUOMLOuBL+8Gnf/r5bvu/tOd98Ve/0I0NXMjshVfO6+Nfb8HvAHglP0RPXA0QnLfYGtuYku7nTgZXf/e8s3oj5+Cf7e1KwWe34vSZ3IjmXsouI3gKkea2BuKY3PQmjc/e/uvtfd9wG/TrHvSD+LZrYfcA7w+1R1cnEMU3ynRPL5U1Jop1j7413AOnf/WYo6/ytWDzMbQ3Ccd+QovoPMrEfTa4ILkq+1qPYwcEmsF9KXgY+aTlNzKOV/Z1EevxYeBpp6c1wK/HeSOkuBSjPrGWseqYyVhcrMTgOuIZjvvCFFnXQ+C2HGmHid6uwU+14BDDSz/rGzx4sIjnuufB1Y7+71yd7MxTFs5Tslms9fWFfUO+sDOIHg9OxVYFXscQbwHeA7sTozgTUEPSleAL47wpn9AAACgElEQVSSw/iOie33lVgMc2PlifEZcAdBr4/VQEWOj2F3gi/5QxPKIj1+BAnqXWAPwX9flwO9gKeADbHnw2N1K4DfJKw7HdgYe1yWo9g2ErQlN30G74zVPRJ4pLXPQg6P3/+Lfb5eJfiC69MyxtjyGQQ9bjaFFWOy+GLli5o+dwl1c3oMW/lOieTzpzuaRUQkTs1HIiISp6QgIiJxSgoiIhKnpCAiInFKCiIiEqekIBJjZnut+QiuWRux08zKEkfoFMlX+0UdgEge+cTdR0QdhEiUdKYg0obYePo3mdlLscc/xcpLzeyp2IBvT5lZv1j5Fy2Y4+CV2OMrsU2VmNmvY2PmP25mB8bqf9fM1sa2c29EP6YIoKQgkujAFs1HFya8t9PdxwC3Az+Pld1OMAR5OcGAdPNj5fOB5R4M6DeS4E5YgIHAHe4+FPgQODdWPhs4Prad74T1w4mkQ3c0i8SY2S53PzhJ+Wbgn939zdjAZX9z915mtp1g6IY9sfJ33f0IM9sG9HX3fyRso4xg3PuBseVrgK7u/u9m9hiwi2A02Ic8NhigSBR0piCSHk/xOlWdZP6R8Hovn13TO5NgLKpRwMrYyJ0ikVBSEEnPhQnPz8de/4VgVE+AqcCzsddPAVcCmFmJmR2SaqNm1gU42t2fBv4NOAz43NmKSK7oPxKRzxxozSdvf8zdm7qlHmBmLxL8IzUlVvZdYKGZ/SuwDbgsVj4LWGBmlxOcEVxJMEJnMiXAEjM7lGD02lvd/cOs/UQi7aRrCiJtiF1TqHD37VHHIhI2NR+JiEiczhRERCROZwoiIhKnpCAiInFKCiIiEqekICIicUoKIiISp6QgIiJx/x8d/MiLtNf2yAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "original_train_loss = original_hist.history['loss']\n",
    "bigger_model_train_loss = bigger_model_hist.history['loss']\n",
    "\n",
    "plt.plot(epochs, original_train_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, bigger_model_train_loss, 'bo', label='Bigger model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Training loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "여기서 볼 수 있듯이 용량이 큰 네트워크는 훈련 손실이 매우 빠르게 0에 가까워집니다. 용량이 많은 네트워크일수록 더 빠르게 훈련 데이터를 모델링할 수 있습니다(결국 훈련 손실이 낮아집니다). 하지만 더욱 과대적합에 민감해집니다(결국 훈련과 검증 손실 사이에 큰 차이가 발생합니다)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 가중치 규제 추가\n",
    "\n",
    "오캄의 면도날 이론을 알고 있을지 모르겠습니다. 어떤 것에 대한 두 가지의 설명이 있다면 더 적은 가정을 필요하는 간단한 설명이 옳을 것이라는 이론입니다. 이 개념은 신경망으로 학습되는 모델에도 적용됩니다. 어떤 훈련 데이터와 네트워크 구조가 주어졌을 때 데이터를 설명할 수 있는 가중치 값의 집합은 여러 개(여러 개의 모델)입니다. 간단한 모델이 복잡한 모델보다 덜 과대적합될 가능성이 높습니다.\n",
    "\n",
    "여기에서 간단한 모델은 파라미터 값 분포의 엔트로피가 작은 모델입니다(또는 앞 절에서 본 것처럼 적은 수의 파라미터를 가진 모델입니다). 그러므로 과대적합을 완화시키기 위한 일반적인 방법은 네트워크의 복잡도에 제한을 두어 가중치가 작은 값을 가지도록 강제하는 것입니다. 가중치 값의 분포가 더 균일하게 됩니다. 이를 가중치 규제라고 하고 네트워크의 손실 함수에 큰 가중치에 연관된 비용을 추가합니다. 두 가지 형태의 비용이 있습니다.\n",
    "\n",
    "* L1 규제 : 가중치의 절대값에 비례하는 비용이 추가됩니다(가중치의 L1 노름).\n",
    "* L2 규제 : 가중치의 제곱에 비례하는 비용이 추가됩니다(가중치의 L2 노름). L2 규제는 신경망에서 가중치 감쇠라고도 부릅니다. 다른 이름 때문에 혼돈하지 마세요. 가중치 감쇠는 수학적으로 L2 규제와 동일합니다.\n",
    "\n",
    "케라스에서 가중치 규제 인스턴스를 층의 키워드 매개변수로 전달하여 가중치 규제를 추가할 수 있습니다. 영화 리뷰 분류 네트워크에 L2 가중치 규제를 추가해 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import regularizers\n",
    "\n",
    "l2_model = models.Sequential()\n",
    "l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),\n",
    "                          activation='relu', input_shape=(10000,)))\n",
    "l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),\n",
    "                          activation='relu'))\n",
    "l2_model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "l2_model.compile(optimizer='rmsprop',\n",
    "                 loss='binary_crossentropy',\n",
    "                 metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`l2(0.001)`는 가중치 행렬의 모든 원소를 제곱하고 `0.001`을 곱하여 네트워크의 전체 손실에 더해진다는 의미입니다. 이 페널티 항은 훈련할 때만 추가됩니다. 이 네트워크의 손실은 테스트보다 훈련할 때 더 높을 것입니다.\n",
    "\n",
    "L2 규제 페널티의 효과를 확인해 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 2s 68us/step - loss: 0.4879 - acc: 0.8152 - val_loss: 0.3895 - val_acc: 0.8656\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 2s 68us/step - loss: 0.3100 - acc: 0.9061 - val_loss: 0.3306 - val_acc: 0.8891\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 2s 61us/step - loss: 0.2657 - acc: 0.9202 - val_loss: 0.3301 - val_acc: 0.8872\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.2459 - acc: 0.9286 - val_loss: 0.3413 - val_acc: 0.8821\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 2s 68us/step - loss: 0.2323 - acc: 0.9352 - val_loss: 0.3818 - val_acc: 0.8685\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 2s 64us/step - loss: 0.2237 - acc: 0.9390 - val_loss: 0.3676 - val_acc: 0.8752\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 2s 61us/step - loss: 0.2173 - acc: 0.9409 - val_loss: 0.3748 - val_acc: 0.8735\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 1s 60us/step - loss: 0.2106 - acc: 0.9445 - val_loss: 0.3726 - val_acc: 0.8752\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 2s 69us/step - loss: 0.2045 - acc: 0.9466 - val_loss: 0.3762 - val_acc: 0.8753\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.2009 - acc: 0.9500 - val_loss: 0.3822 - val_acc: 0.8748\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 1s 60us/step - loss: 0.1950 - acc: 0.9495 - val_loss: 0.3838 - val_acc: 0.8755\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 2s 69us/step - loss: 0.1922 - acc: 0.9518 - val_loss: 0.3918 - val_acc: 0.8745\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 2s 66us/step - loss: 0.1876 - acc: 0.9548 - val_loss: 0.3972 - val_acc: 0.8734\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 2s 68us/step - loss: 0.1820 - acc: 0.9567 - val_loss: 0.4331 - val_acc: 0.8642\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 2s 66us/step - loss: 0.1816 - acc: 0.9574 - val_loss: 0.4568 - val_acc: 0.8566\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.1757 - acc: 0.9596 - val_loss: 0.4463 - val_acc: 0.8643\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 2s 68us/step - loss: 0.1717 - acc: 0.9613 - val_loss: 0.4222 - val_acc: 0.8696\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.1665 - acc: 0.9637 - val_loss: 0.4374 - val_acc: 0.8672\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 2s 62us/step - loss: 0.1648 - acc: 0.9642 - val_loss: 0.4623 - val_acc: 0.8594\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 2s 68us/step - loss: 0.1616 - acc: 0.9667 - val_loss: 0.4412 - val_acc: 0.8665\n"
     ]
    }
   ],
   "source": [
    "l2_model_hist = l2_model.fit(x_train, y_train,\n",
    "                             epochs=20,\n",
    "                             batch_size=512,\n",
    "                             validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XucFPWZ7/HPw4gZUbxFYlAugxw03AaQATQXEG+oUfCuLGdPwETiBUXNMRoxMprjGj2J7nqJiReCrkRUjEoSNjGKwtElWQblqstFF3QWVgcMIo6sAz7nj6ppmqF7ume6q6tn+vt+veo1XdW/qnq6KOqp+tWvfmXujoiICECHuAMQEZHioaQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJkSUFM5thZh+a2co035uZ3Wtm68xsuZkdG1UsIiKSnX0iXPZM4H7g8TTfnw70CYcRwIPh32YddthhXlFRkZ8IRURKxJIlSza7e5dM5SJLCu6+0MwqmikyDnjcg342/mJmB5tZV3ff1NxyKyoqqKmpyWOkIiLtn5ltyKZcnPcUjgTeTxqvDaftxcwmm1mNmdXU1dUVJDgRkVIUZ1KwFNNS9s7n7g+5e5W7V3XpkvHqR0REWinOpFALdE8a7wZsjCkWEREh2hvNmcwFppjZbIIbzB9nup+QTkNDA7W1tezYsSOvAUr7VV5eTrdu3ejYsWPcoYgUlciSgpk9CZwAHGZmtcB0oCOAu/8SmAecAawD6oFJrV1XbW0tnTt3pqKiArNUtVIiu7k7W7Zsoba2ll69esUdjkhRibL10fgM3ztwZT7WtWPHDiUEyZqZ8eUvfxk1WpC2pro6GKLUbp5oVkKQltD+Im3RrbdGv452kxRERCR3Sgp5Ultby7hx4+jTpw+9e/dm6tSpfP755ynLbty4kfPPPz/jMs844wy2bt3aqniqq6v52c9+1qp5szVz5kymTJmScxkRSa+6GsyCAXZ/jqoaqaSTQr42qrtz7rnncvbZZ7N27VrWrFnD9u3bmTZt2l5ld+7cyRFHHMGcOXMyLnfevHkcfPDB+QlSRNqk6mpwDwbY/VlJIQL5qp+bP38+5eXlTJoUNKAqKyvjnnvuYcaMGdTX1zNz5kwuuOACzjrrLE499VTWr1/PgAEDAKivr+fCCy+ksrKSiy66iBEjRiS68aioqGDz5s2sX7+evn37cumll9K/f39OPfVUPvvsMwAefvhhhg0bxqBBgzjvvPOor69vNtaJEydy+eWXM3r0aI466igWLFjAJZdcQt++fZk4cWKi3JNPPsnAgQMZMGAAN9xwQ2L6r3/9a44++mhGjRrF66+/npheV1fHeeedx7Bhwxg2bNge34lI21HSSSFfVq1axdChQ/eYduCBB9KjRw/WrVsHwKJFi3jssceYP3/+HuV+8YtfcMghh7B8+XJ+/OMfs2TJkpTrWLt2LVdeeSWrVq3i4IMP5tlnnwXg3HPPZfHixSxbtoy+ffvy6KOPZoz3b3/7G/Pnz+eee+7hrLPO4tprr2XVqlWsWLGCpUuXsnHjRm644Qbmz5/P0qVLWbx4Mc8//zybNm1i+vTpvP766/z5z3/mrbfeSixz6tSpXHvttSxevJhnn32W733vey3ahiKS2fTp0a8jzofXYlFdvecVQmM93fTprb8cc/eUrVmSp59yyikceuihe5V57bXXmDp1KgADBgygsrIy5Tp69erF4MGDARg6dCjr168HYOXKldx8881s3bqV7du3M2bMmIzxnnXWWZgZAwcO5PDDD2fgwIEA9O/fn/Xr17NhwwZOOOEEGrsUmTBhAgsXLgTYY/pFF13EmjVrAHjppZf2SBLbtm3jk08+yRiLiGQv6uaoUKJJoXHDmu2up8tF//79E2fujbZt28b7779P7969WbJkCfvvv3/KeT3LAL70pS8lPpeVlSWqjyZOnMjzzz/PoEGDmDlzJq+++mrWy+rQocMey+3QoQM7d+5kn33S7xbpmnJ+8cUXLFq0iP322y+bnyMiRUrVR3lw0kknUV9fz+OPB6+O2LVrFz/4wQ+YOHEinTp1anbeb37zmzz99NMAvPXWW6xYsaJF6/7kk0/o2rUrDQ0NzJo1q3U/oIkRI0awYMECNm/ezK5du3jyyScZNWoUI0aM4NVXX2XLli00NDTwzDPPJOY59dRTuf/++xPjS5cuzUssIlJYJZ0U8lU/Z2Y899xzPPPMM/Tp04ejjz6a8vJy/uEf/iHjvFdccQV1dXVUVlZy5513UllZyUEHHZT1un/yk58wYsQITjnlFL72ta/l8jMSunbtyh133MHo0aMZNGgQxx57LOPGjaNr165UV1dz/PHHc/LJJ3PssbtflnfvvfdSU1NDZWUl/fr145e//GVeYhGRwrJsqy+KRVVVlTd9yc7bb79N3759Y4ooN7t27aKhoYHy8nLeeecdTjrpJNasWcO+++4bd2jtXlveb0RaysyWuHtVpnIld0+h2NTX1zN69GgaGhpwdx588EElBBGJjZJCzDp37qzXi4pI0SjpewoiIrInJQUREUlQUhARkQQlBRERSVBSyJMDDjhgr2l33303/fr1o7KykpNOOokNGzYUPK7WdKE9d+5cfvrTn+a87hNOOCHym+gTJ07M2ONsNmVEJFCSSWHWLKiogA4dgr95ehB4L0OGDKGmpobly5dz/vnn88Mf/jDjPDt37owmmCzt3LmTsWPHcuONN8Yah4jEI9KkYGanmdlqM1tnZnsdZcysp5m9bGbLzexVM+sWZTwQJIDJk2HDhqDfow0bgvEoEsPo0aMT3Vwcd9xx1NbWpiw3ceJErrvuOkaPHs0NN9zAp59+yiWXXMKwYcMYMmQIL7zwAtB8N9vJVypz5szZoxvsRum62W66/uQX4wwePDgx7LfffixYsCBtfJ999hkXX3xxIr7G/pmaqqio4KabbuL444+nqqqKN954gzFjxtC7d+/Ek9DuzvXXX8+AAQMYOHAgTz31VGL6lClT6NevH9/+9rf58MMPE8tdsmQJo0aNYujQoYwZM4ZNmzZl9w8lIgmRPadgZmXAA8ApQC2w2MzmuvtbScV+Bjzu7o+Z2YnAHcDfRxUTwLRp0PSVA/X1wfQJE6Jb76OPPsrpp5+e9vs1a9bw0ksvUVZWxk033cSJJ57IjBkz2Lp1K8OHD+fkk0/mwQcfTHSzvXLlykSvqdk699xzufTSSwG4+eabefTRR7nqqqv2Wv/MmTMT8zT2YfS73/2Ou+66i69//etMnz49ZXy/+tWv6NSpE8uXL2f58uV7dIPRVPfu3Vm0aBHXXnstEydO5PXXX2fHjh3079+fyy67jN/+9rcsXbqUZcuWsXnzZoYNG8bIkSNZtGgRq1evZsWKFXzwwQf069ePSy65hIaGBq666ipeeOEFunTpwlNPPcW0adOYMWNGi7aRSKmL8uG14cA6d38XwMxmA+OA5KTQD7g2/PwK8HyE8QDw3nstm54PTzzxBDU1NSxYsCBtmQsuuICysjIAXnzxRebOnZu4F7Bjxw7ee++9rLvZTqe5braT19/U2rVruf7665k/fz4dO3ZMG9/ChQu5+uqrAaisrGw2vrFjxwIwcOBAtm/fTufOnencuTPl5eVs3bqV1157jfHjx1NWVsbhhx/OqFGjWLx4MQsXLkxMP+KIIzjxxBMBWL16NStXruSUU04Bgu5Dunbt2qLtIyLRJoUjgfeTxmuBEU3KLAPOA/4JOAfobGZfdvctUQXVo0dQZZRqehReeuklbr/9dhYsWJDopnratGn84Q9/AHafiSd3re3uPPvssxxzzDF7LKu5fqqSu7TesWNHyjLNdbOdrmvvTz/9lAsvvJCHH36YI444otn4msbRnEzdd2f7Wxu5O/3792fRokVZrV9EUovynkKqo0PT/+n/GxhlZm8Co4D/BPa602pmk82sxsxq6urqcgrq9tuhaW/WnToF0/PtzTff5Pvf/z5z587lK1/5SlIMt7N06dK03UuPGTOG++67L3FgfPPNN4Hmu9k+/PDDefvtt/niiy947rnnUi63Nd1sT5o0iUmTJvGtb30rY3wjR45MLHflypUsX748q3WkMnLkSJ566il27dpFXV0dCxcuZPjw4YwcOZLZs2eza9cuNm3axCuvvALAMcccQ11dXSIpNDQ0sGrVqlavX6RURXmlUAt0TxrvBmxMLuDuG4FzAczsAOA8d/+46YLc/SHgIQh6Sc0lqMb7BtOmBVVGPXoECSHX+wn19fV067b7Pvl1113HvHnz2L59OxdccAEAPXr0YO7cuRmX9eMf/5hrrrmGyspK3J2Kigp+//vfc8UVV/Cd73yHyspKhgwZskc32z/96U8588wz6d69OwMGDGD79u17Lbexm+2ePXsycODAjG9G27BhA3PmzGHNmjWJuvlHHnkkbXyXX345kyZNorKyksGDBzN8+PCst19T55xzDosWLWLQoEGYGXfddRdf/epXOeecc5g/fz4DBw5MvCsaYN9992XOnDlcffXVfPzxx+zcuZNrrrmG/v37tzoGkVIUWdfZZrYPsAY4ieAKYDHwd+6+KqnMYcBH7v6Fmd0O7HL3W5pbbnvrOrsl1M12fpXKfiMCRdB1trvvNLMpwJ+AMmCGu68ys9uAGnefC5wA3GFmDiwErowqnvZA3WyLSNQi7Trb3ecB85pMuyXp8xxAj5pmSd1si0jU2s0TzW3tDXISL+0vIqm1i6RQXl7Oli1b9B9dsuLubNmyhfLy8rhDESk67eLNa926daO2tpZcm6tK6SgvL9+jtZiIBNpFUujYsSO9evWKOwwRkTavXVQfiYhIfigpiIhIgpKCiIgkKCmIiEiCkoKIiCQoKYiISIKSgoiIJCgpiIhIgpKCiIgkKCmIiEiCkoKIiCQoKYiISIKSgoiIJCgpiIhIgpKCiIgkKCmIiEhCpEnBzE4zs9Vmts7MbkzxfQ8ze8XM3jSz5WZ2RpTxiIhI8yJLCmZWBjwAnA70A8abWb8mxW4Gnnb3IcDFwC+iikdERDKL8kphOLDO3d9198+B2cC4JmUcODD8fBCwMcJ4REQkgyjf0Xwk8H7SeC0wokmZauBFM7sK2B84OcJ4REQkgyivFCzFNG8yPh6Y6e7dgDOAfzazvWIys8lmVmNmNXV1dRGEKiIiEG1SqAW6J413Y+/qoe8CTwO4+yKgHDis6YLc/SF3r3L3qi5dukQUroiIRJkUFgN9zKyXme1LcCN5bpMy7wEnAZhZX4KkoEsBEZGYRJYU3H0nMAX4E/A2QSujVWZ2m5mNDYv9ALjUzJYBTwIT3b1pFZOIiBRIlDeacfd5wLwm025J+vwW8I0oYxARKRbV1cFQzPREs4iUjLgPyLfeGu/6s6GkICIloy0clOOmpCAiEqHqajALBtj9Oe6rlnSUFESkXYv7oFxdDe7BALs/F2tSsLbW2KeqqspramriDkNE2iCz3QfnUlu/mS1x96pM5XSlICJSINOnxx1BZkoKIlIy4j4oF2uVUTIlBREpGW3hoBw3JQUREUnImBTMbP/GnkvN7GgzG2tmHaMPTURECi2bK4WFQLmZHQm8DEwCZkYZlIiIxCObpGDuXg+cC9zn7ucQvF5TRETamaySgpkdD0wA/hBOi7QjPRERiUc2SeEa4EfAc2HX10cBr0QbloiIxCFjUnD3Be4+1t3vDG84b3b3qwsQm4hIUSmFJq3ZtD76jZkdaGb7A28Bq83s+uhDExEpLqXQy2o21Uf93H0bcDbBC3N6AH8faVQiIhKLbJJCx/C5hLOBF9y9AWhbveiJiLRS3L2sFlo2SeFXwHpgf2ChmfUEtkUZlIhIsWhrXV/nKmPTUne/F7g3adIGMxsdXUgiIhKXbG40H2Rmd5tZTTj8nOCqISMzO83MVpvZOjO7McX395jZ0nBYY2ZbW/EbREQKIu5eVgshm+qjGcAnwIXhsA34daaZzKwMeAA4neAJ6PFmtseT0O5+rbsPdvfBwH3Ab1sWvohI4bTXKqNk2TyZ3Nvdz0sav9XMlmYx33Bgnbu/C2Bms4FxBM1aUxkPlEAeFhEpXtlcKXxmZt9sHDGzbwCfZTHfkcD7SeO14bS9hDevewHz03w/ubH6qq6uLotVi4hIa2RzpXA58JiZHQQY8BEwMYv5LMW0dE1ZLwbmuPuuVF+6+0PAQxC8ozmLdYuISCtk0/poKTDIzA4Mx7NtjloLdE8a7wZsTFP2YuDKLJcrIiIRSZsUzOy6NNMBcPe7Myx7MdDHzHoB/0lw4P+7FMs7BjgEWJRdyCIiEpXmrhQ657Jgd99pZlOAPwFlwIywl9XbgBp3nxsWHQ/MdndVC4mIxMza2rG4qqrKa2pq4g5DRKRNMbMl7l6VqVw2rY9ERKREKCmIiEiCkoKIiCRkbJJqZl8CzgMqksu7+23RhSUiInHI5uG1F4CPgSXAf0cbjoiIxCmbpNDN3U+LPBIREYldNvcU/tXMBkYeiYiIxC6bK4VvAhPN7D8Iqo8McHevjDQyEREpuGySwumRRyEikoXq6tJ4p0GcMlYfufsG4GDgrHA4OJwmIlJQt94adwTtXzav45wKzAK+Eg5PmNlVUQcmIiKFl82N5u8CI9z9Fne/BTgOuDTasEREAtXVYBYMsPuzqpGikc09BQOSX36zi9Qv0BERybvk+whm0Mb68GxzskkKvwb+ambPheNnA49GF5KIiMQlmzev3W1mrxI0TTVgkru/GXVgIiJNTZ8edwTtX3NvXjvQ3beZ2aHA+nBo/O5Qd/8o+vBERHbTfYToNXel8BvgTII+j5Jr8SwcPyrCuEREJAZpWx+5+5nh317uflTS0MvdlRBESlCuZ+o60y9+2Tyn8HI200Sk/cv14TE9fFb80iYFMysP7yccZmaHmNmh4VABHJHNws3sNDNbbWbrzOzGNGUuNLO3zGyVmf2mNT9CRETyo7krhe8T3E/4Wvi3cXgBeCDTgs2sLCx3OtAPGG9m/ZqU6QP8CPiGu/cHrmnFbxCRCOX68JgePmtbzDM8CWJmV7n7fS1esNnxQLW7jwnHfwTg7ncklbkLWOPuj2S73KqqKq+pqWlpOCKSB7k+PKaHz+JjZkvcvSpTuWyeU7jPzAYQnO2XJ01/PMOsRwLvJ43XAiOalDk6DPZ1oIwgifwxU0wiIhKNbN7RPB04gSApzCOoDnoNyJQUUnWF0fQcYR+gT7j8bsD/M7MB7r61SQyTgckAPXr0yBSyiEQk14fH9PBZ8cumQ7zzgZOA/3L3ScAg4EtZzFcLdE8a7wZsTFHmBXdvcPf/AFYTJIk9uPtD7l7l7lVdunTJYtUiEgU1SW3/skkKn7n7F8BOMzsQ+JDsHlxbDPQxs15mti9wMTC3SZnngdEAZnYYQXXSu9kGLyIi+ZVNh3g1ZnYw8DBB66PtwL9lmsndd5rZFOBPBPcLZrj7KjO7Dahx97nhd6ea2VsEva9e7+5bWvlbREQkRxlbH+1ROHhG4UB3Xx5VQJmo9ZGISMvl3PrIzI5t7jt3f6O1wYmISHFqrvro5+HfcqAKWEbQoqgS+CtBV9oiItKONNch3mh3Hw1sAI4NW/8MBYYA6woVoIiIFE42rY++5u4rGkfcfSUwOLqQREQkLtm0PnrbzB4BniB4+Ox/Am9HGpWIiMQim6QwCbgcmBqOLwQejCwiERGJTTZ9H+0A7gkHERFpx5prkvq0u19oZivYu88i3L0y0shERKTgmrtSaKwuOrMQgYiISPzSJgV33xT+3VC4cEREJE7NvY7zEzPblmL4xMy2FTJIEckP9VIqmTT38Fpndz8wxdDZ3Q8sZJAikh+33hp3BFLssmmSCoCZfYU937z2XiQRiYhIbDI+0WxmY81sLfAfwAJgPfAvEcclInlSXR28G9nCdyE2flZVkqSSTTcXPwGOA9a4ey+Ct7C9HmlUIpJSaw7k1dXgHgyw+7OSgqSSTVJoCF9808HMOrj7K6jvI5FY6J6ARC2bewpbzewAgu4tZpnZh8DOaMMSkShMnx53BFLssrlSGAd8BlwL/BF4BzgryqBEZLd83hNQlZFkkvZ1nGZ2P/Abd//XwobUPL2OU0qZ2e57AyItke3rOJu7UlgL/NzM1pvZnWam+wgiIu1ccw+v/ZO7Hw+MAj4Cfm1mb5vZLWZ2dDYLN7PTzGy1ma0zsxtTfD/RzOrMbGk4fK/Vv0SkBOiegEQt4z0Fd9/g7ne6+xDg74BzyOIlO2ZWBjwAnA70A8abWb8URZ9y98Hh8EjLwhcpnGKojy+GGKR9y+bhtY5mdpaZzSJ4aG0NcF4Wyx4OrHP3d939c2A2wU1rkTZJzUGlFDTXId4pZjYDqAUmA/OA3u5+kbs/n8WyjwTeTxqvDac1dZ6ZLTezOWbWvQWxi4hInjV3pXATsAjo6+5nufssd/+0Bcu2FNOatpv4HVARvrDnJeCxlAsym2xmNWZWU1dX14IQRHKj5qBSatI2Sc15wWbHA9XuPiYc/xGAu9+RpnwZ8JG7H9TcctUkVeKSa3NQNSeVOOWjSWquFgN9zKyXme0LXAzMTS5gZl2TRseSxQ1sERGJTmRJwd13AlOAPxEc7J9291VmdpuZjQ2LXW1mq8xsGXA1MDGqeERy1ZrmoOqhVNqayKqPoqLqI2mrVH0kcSqG6iMREWljlBRECkRPI0tboKQgUiC6j1C6Zs2Cigro0CH4O2tW3BGll/U7mkVEpOVmzYLJk6G+PhjfsCEYB5gwIb640tGVgohEri2dKefbtGm7E0Kj+vpgejHSlYKIRKqtnSnn23vvtWx63HSlICKRamtnyvnWo0fLpsdNSUFEIpWPM+W2XP10++3QqdOe0zp1CqYXIyUFEYlUrmfKjdVPGzYED/81Vj+1lcQwYQI89BD07Bk8wNizZzDekqqzQiZFJQURiVSuZ8rFUP2U60F5wgRYvx6++CL429KEUMikqKQgJUPPCcQj1zPluG/Uxn2lUuikqL6PpGSo76HWmzUrOAi9915Q7XP77YVrOVRRERyIm+rZMzjrbu/r79Ah9X5rFlx5ZEt9H4lIXsR9phz3jdq4r1QK3XpJSUHaNXVdnbu46/TzcaM2F3E3KS14UnT3NjUMHTrURVoD4o6gbTILtl3TwSzuyLL3xBPuPXsGMffsGYy3ZN5Onfb87Z06tWwZucol/kZAjWdxjNUTzSLSrB49UtepF+vDV03l+kR1Y5m47qk0xlCo9an6SEqGuq5unbjr9HOVj+qvXJqUtjVKClIydB+hdeKu089V3DeK2xpVH4lIRoWsvsi3tl79VWiRXimY2WlmttrM1pnZjc2UO9/M3MwytqEVEWmJtl79VWiRJQUzKwMeAE4H+gHjzaxfinKdgauBv0YVi4iUrrZe/VVoUV4pDAfWufu77v45MBsYl6LcT4C7gB0RxiLtgO4JtF5b7mU0H0rpRnGuokwKRwLvJ43XhtMSzGwI0N3dfx9hHNJO3Hpr3BG0TXE/kSxtS5RJwVJMS/TgYWYdgHuAH2RckNlkM6sxs5q6uroWB1LqZ0nS9uWyD8f9RLK0LVEmhVqge9J4N2Bj0nhnYADwqpmtB44D5qa62ezuD7l7lbtXdenSpUVB6CypbVM3Fbnvw2qSKS2SzWPPrRkImru+C/QC9gWWAf2bKf8qUJVpuS3t5qJnz9SP6Pfs2aLFtGn5eES+GMTVTUWu2y/X+XPdh/V/QNyz7+Yi0n6KgDOANcA7wLRw2m3A2BRlI0kK7aHfllwUQ78t+RJHUsh1++Vj++e6D7enfUBaryiSQhSDrhRaphh+f76uVKZPL/z6i+EsPR/LaC9Xi9J6SgqhUj9LivtKKR/bP84eLnPdfvnY/qW+D0t+KCkkKeWzpLjPMnNdf64HxLjP9PN1pVbK+7Dkh5KCuHv8deK5ninnelCNuz5eZ/lSLJQUJKEt16nHnVTc4299JJIP2SYFC8q2HVVVVV5TUxN3GCUj15eGN33BCQSdkWXb90yuL03Pdf0i7YWZLXH3jJ2O6n0K0qxc30+b3BkZtLwzslx7uFRnaCItoysFaVY+z7TNUl91ZBNDnK9CFGkPdKUgeZHrmX4+uqlQD5cihaMrBclaa8/08zW/iLSerhTaEfXyKiKFoqRQ5OLu5TWfvZROn57PyEQkCqo+KnK5NsnMJ1X/iLRdqj7Ko1yrb3KZX33hi0ghlVRSaE2VR67VN7nOn+tzAvmk6h+R9q+kqo9aU/2Ra/WNnsgVkWKg6qM8ybX6Jtf59USuiBRSu08KubaeybX6Jh/VP3p4S0QKpSSSQmPfmLD7c7ZJIde+d3KdP59K6WX3ItI67T4p5CrX6ptiqv659dbCr1NE2paSutFcXV3aZ8t6zkCkdBXFjWYzO83MVpvZOjO7McX3l5nZCjNbamavmVm/KOMpxYSQzyeSRaT9i+xKwczKgDXAKUAtsBgY7+5vJZU50N23hZ/HAle4+2nNLbfUnmjOJ10piJSuYrhSGA6sc/d33f1zYDYwLrlAY0II7Q/okCUiEqMok8KRwPtJ47XhtD2Y2ZVm9g5wF3B1hPG0eblW+eiJZBHJJMqkYCmm7XUl4O4PuHtv4Abg5pQLMptsZjVmVlNXV5fnMNuOXFsP6T6CiGQSZVKoBbonjXcDNjZTfjZwdqov3P0hd69y96ouXbrkMcSW0UFVRNq7KJPCYqCPmfUys32Bi4G5yQXMrE/S6LeBtRHGk7M4ztTVekhECinS5xTM7AzgH4EyYIa7325mtwE17j7XzP4JOBloAP4GTHH3Vc0tM87WR3G/jlKth0SktYqh9RHuPs/dj3b33u5+ezjtFnefG36e6u793X2wu4/OlBDioDN1ESkl6uYig1z7TtLrLEWkLSmpbi5ypeofEWmriqL6qL3RmbqItHdKCi2gh8dEpL1TUigg3ZwWkWKnpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJbe7hNTOrAzbEHUcahwGb4w6iGYovN8UeHxR/jIovN7nE19PdM3Yz3eaSQjEzs5psnhiMi+LLTbHHB8Ufo+LLTSHiU/WRiIgkKCmIiEiCkkJ+PRR3ABkovtwUe3xQ/DEqvtxEHp+fsJ12AAAGKElEQVTuKYiISIKuFEREJEFJoYXMrLuZvWJmb5vZKjObmqLMCWb2sZktDYdbChzjejNbEa57r5dPWOBeM1tnZsvN7NgCxnZM0nZZambbzOyaJmUKvv3MbIaZfWhmK5OmHWpmfzazteHfQ9LM+52wzFoz+06BYvu/Zvbv4b/fc2Z2cJp5m90XIo6x2sz+M+nf8Yw0855mZqvD/fHGAsb3VFJs681saZp5I92G6Y4pse1/7q6hBQPQFTg2/NwZWAP0a1LmBOD3Mca4Hjisme/PAP4FMOA44K8xxVkG/BdB++lYtx8wEjgWWJk07S7gxvDzjcCdKeY7FHg3/HtI+PmQAsR2KrBP+PnOVLFlsy9EHGM18L+z2AfeAY4C9gWWNf3/FFV8Tb7/OXBLHNsw3TElrv1PVwot5O6b3P2N8PMnwNvAkfFG1WLjgMc98BfgYDPrGkMcJwHvuHvsDyO6+0LgoyaTxwGPhZ8fA85OMesY4M/u/pG7/w34M3Ba1LG5+4vuvjMc/QvQLZ/rbKk02y8bw4F17v6uu38OzCbY7nnVXHxmZsCFwJP5Xm82mjmmxLL/KSnkwMwqgCHAX1N8fbyZLTOzfzGz/gUNDBx40cyWmNnkFN8fCbyfNF5LPIntYtL/R4xz+zU63N03QfAfF/hKijLFsC0vIbjySyXTvhC1KWEV14w01R/FsP2+BXzg7mvTfF+wbdjkmBLL/qek0EpmdgDwLHCNu29r8vUbBFUig4D7gOcLHN433P1Y4HTgSjMb2eR7SzFPQZuhmdm+wFjgmRRfx739WiLWbWlm04CdwKw0RTLtC1F6EOgNDAY2EVTRNBX7vgiMp/mrhIJswwzHlLSzpZiW0/ZTUmgFM+tI8I83y91/2/R7d9/m7tvDz/OAjmZ2WKHic/eN4d8PgecILtGT1QLdk8a7ARsLE13C6cAb7v5B0y/i3n5JPmisVgv/fpiiTGzbMrypeCYwwcMK5qay2Bci4+4fuPsud/8CeDjNumPdF81sH+Bc4Kl0ZQqxDdMcU2LZ/5QUWiisf3wUeNvd705T5qthOcxsOMF23lKg+PY3s86NnwluSK5sUmwu8L/CVkjHAR83XqYWUNqzszi3XxNzgcbWHN8BXkhR5k/AqWZ2SFg9cmo4LVJmdhpwAzDW3evTlMlmX4gyxuT7VOekWfdioI+Z9QqvHi8m2O6FcjLw7+5em+rLQmzDZo4p8ex/Ud1Rb68D8E2Cy7PlwNJwOAO4DLgsLDMFWEXQkuIvwNcLGN9R4XqXhTFMC6cnx2fAAwStPlYAVQXehp0IDvIHJU2LdfsRJKhNQAPB2dd3gS8DLwNrw7+HhmWrgEeS5r0EWBcOkwoU2zqCuuTGffCXYdkjgHnN7QsF3H7/HO5fywkOcF2bxhiOn0HQ4uadqGJMFV84fWbjfpdUtqDbsJljSiz7n55oFhGRBFUfiYhIgpKCiIgkKCmIiEiCkoKIiCQoKYiISIKSgkjIzHbZnj245q3HTjOrSO6hU6RY7RN3ACJF5DN3Hxx3ECJx0pWCSAZhf/p3mtm/hcP/CKf3NLOXww7fXjazHuH0wy14x8GycPh6uKgyM3s47DP/RTPbLyx/tZm9FS5ndkw/UwRQUhBJtl+T6qOLkr7b5u7DgfuBfwyn3U/QBXklQYd094bT7wUWeNCh37EET8IC9AEecPf+wFbgvHD6jcCQcDmXRfXjRLKhJ5pFQma23d0PSDF9PXCiu78bdlz2X+7+ZTPbTNB1Q0M4fZO7H2ZmdUA3d//vpGVUEPR73yccvwHo6O7/x8z+CGwn6A32eQ87AxSJg64URLLjaT6nK5PKfyd93sXue3rfJuiLaiiwJOy5UyQWSgoi2bko6e+i8PO/EvTqCTABeC38/DJwOYCZlZnZgekWamYdgO7u/grwQ+BgYK+rFZFC0RmJyG772Z4vb/+juzc2S/2Smf2V4ERqfDjtamCGmV0P1AGTwulTgYfM7LsEVwSXE/TQmUoZ8ISZHUTQe+097r41b79IpIV0T0Ekg/CeQpW7b447FpGoqfpIREQSdKUgIiIJulIQEZEEJQUREUlQUhARkQQlBRERSVBSEBGRBCUFERFJ+P8XLmH4PBz8+QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "l2_model_val_loss = l2_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, l2_model_val_loss, 'bo', label='L2-regularized model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "여기서 볼 수 있듯이 두 모델이 동일한 파라미터 수를 가지고 있더라도 L2 규제를 사용한 모델(점)이 기본 모델(덧셈 기호)보다 훨씬 더 과대적합에 잘 견디고 있습니다.\n",
    "\n",
    "케라스에서 L2 규제 대신에 다음과 같은 가중치 규제 중 하나를 사용할 수 있습니다:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "from keras import regularizers\n",
    "\n",
    "# L1 규제\n",
    "regularizers.l1(0.001)\n",
    "\n",
    "# L1과 L2 규제 병행\n",
    "regularizers.l1_l2(l1=0.001, l2=0.001)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### l1(0.0001) 모델과 비교"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "l1_model = models.Sequential()\n",
    "l1_model.add(layers.Dense(16, kernel_regularizer=regularizers.l1(0.0001),\n",
    "                          activation='relu', input_shape=(10000,)))\n",
    "l1_model.add(layers.Dense(16, kernel_regularizer=regularizers.l1(0.0001),\n",
    "                          activation='relu'))\n",
    "l1_model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "l1_model.compile(optimizer='rmsprop',\n",
    "                 loss='binary_crossentropy',\n",
    "                 metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 2s 72us/step - loss: 0.5502 - acc: 0.8246 - val_loss: 0.4253 - val_acc: 0.8731\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.3636 - acc: 0.8954 - val_loss: 0.3726 - val_acc: 0.8845\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 2s 62us/step - loss: 0.3272 - acc: 0.9053 - val_loss: 0.3768 - val_acc: 0.8760\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.3062 - acc: 0.9136 - val_loss: 0.3538 - val_acc: 0.8876\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 2s 64us/step - loss: 0.2960 - acc: 0.9168 - val_loss: 0.3572 - val_acc: 0.8858\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.2885 - acc: 0.9204 - val_loss: 0.3910 - val_acc: 0.8732\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.2795 - acc: 0.9244 - val_loss: 0.3763 - val_acc: 0.8805\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.2768 - acc: 0.9260 - val_loss: 0.3679 - val_acc: 0.8845\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 2s 62us/step - loss: 0.2742 - acc: 0.9264 - val_loss: 0.3745 - val_acc: 0.8839\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 2s 62us/step - loss: 0.2659 - acc: 0.9307 - val_loss: 0.3783 - val_acc: 0.8809\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.2603 - acc: 0.9330 - val_loss: 0.3844 - val_acc: 0.8816\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.2636 - acc: 0.9314 - val_loss: 0.3998 - val_acc: 0.8741\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.2557 - acc: 0.9351 - val_loss: 0.4169 - val_acc: 0.8708\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 2s 62us/step - loss: 0.2509 - acc: 0.9382 - val_loss: 0.4548 - val_acc: 0.8604\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.2483 - acc: 0.9384 - val_loss: 0.4024 - val_acc: 0.8763\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.2402 - acc: 0.9427 - val_loss: 0.4454 - val_acc: 0.8611\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 2s 63us/step - loss: 0.2397 - acc: 0.9434 - val_loss: 0.4198 - val_acc: 0.8718\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 2s 61us/step - loss: 0.2328 - acc: 0.9454 - val_loss: 0.4076 - val_acc: 0.8759\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 2s 64us/step - loss: 0.2272 - acc: 0.9495 - val_loss: 0.4145 - val_acc: 0.8746\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 2s 62us/step - loss: 0.2222 - acc: 0.9493 - val_loss: 0.4149 - val_acc: 0.8752\n"
     ]
    }
   ],
   "source": [
    "l1_model_hist = l1_model.fit(x_train, y_train,\n",
    "                             epochs=20,\n",
    "                             batch_size=512,\n",
    "                             validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XucFOWd7/HPjxEzYvAWWYMOMMhBw21AGCDmAoIX1Ch4SVTC2bNgItGIEpP1YERlNIc1ehLdeIkJRoKuxBtGJScmGkXlaEiWwXCV5aILOgurA4YgjqwD/vaPqmmaoae7Z7qrq2f6+3696tVd1U9X/bqmp35dz/PUU+buiIiIAHSKOwARESkeSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpIQWVIws7lm9p6ZrW7hdTOzu8xso5mtNLOhUcUiIiLZOSjCdc8D7gEeauH1s4C+4TQSuC98TOvoo4/2ysrK/EQoIlIili1bts3du2UqF1lScPfFZlaZpsgE4CEPxtn4k5kdYWbd3X1ruvVWVlZSW1ubx0hFRDo+M9ucTbk42xSOA95Jmq8Llx3AzKaaWa2Z1dbX1xckOBGRUhRnUrAUy1KOzufuc9y92t2ru3XLePYjIiJtFGdSqAN6JM1XAFtiikVERIi2oTmThcA0M3uUoIH5b5naE1rS2NhIXV0du3fvzmuA0nGVl5dTUVFB586d4w5FpKhElhTM7BHgFOBoM6sDZgGdAdz9Z8CzwNnARqABmNLWbdXV1dG1a1cqKysxS1UrJbKPu7N9+3bq6uro3bt33OGIFJUoex9NzPC6A1fmY1u7d+9WQpCsmRmf+cxnUKcFaW9qaoIpSh3mimYlBGkNfV+kPbr55ui30WGSgoiI5E5JIU/q6uqYMGECffv2pU+fPkyfPp2PP/44ZdktW7bw1a9+NeM6zz77bHbs2NGmeGpqavjRj37Upvdma968eUybNi3nMiLSspoaMAsm2Pc8qmqkkk4K+dqp7s4FF1zAeeedx4YNG1i/fj27du1i5syZB5Tds2cPxx57LAsWLMi43meffZYjjjgiP0GKSLtUUwPuwQT7nispRCBf9XOLFi2ivLycKVOCDlRlZWXceeedzJ07l4aGBubNm8fXvvY1zj33XM444ww2bdrEwIEDAWhoaOCiiy6iqqqKiy++mJEjRyaG8aisrGTbtm1s2rSJfv36cdlllzFgwADOOOMMPvroIwDuv/9+hg8fzuDBg7nwwgtpaGhIG+vkyZO54oorGDNmDMcffzyvvPIKl156Kf369WPy5MmJco888giDBg1i4MCBzJgxI7H8l7/8JSeccAKjR4/mtddeSyyvr6/nwgsvZPjw4QwfPny/10Sk/SjppJAva9asYdiwYfstO+yww+jZsycbN24EYMmSJTz44IMsWrRov3I//elPOfLII1m5ciU33ngjy5YtS7mNDRs2cOWVV7JmzRqOOOIInnzySQAuuOACli5dyooVK+jXrx8PPPBAxnj/+te/smjRIu68807OPfdcrrnmGtasWcOqVatYvnw5W7ZsYcaMGSxatIjly5ezdOlSnn76abZu3cqsWbN47bXX+MMf/sAbb7yRWOf06dO55pprWLp0KU8++STf/OY3W7UPRSSzWbOi30acF6/FoqZm/zOEpnq6WbPafjrm7il7syQvP/300znqqKMOKPPqq68yffp0AAYOHEhVVVXKbfTu3ZshQ4YAMGzYMDZt2gTA6tWrueGGG9ixYwe7du1i3LhxGeM999xzMTMGDRrEMcccw6BBgwAYMGAAmzZtYvPmzZxyyik0DSkyadIkFi9eDLDf8osvvpj169cD8MILL+yXJHbu3MkHH3yQMRYRyV7U3VGhRJNC044121dPl4sBAwYkfrk32blzJ++88w59+vRh2bJlHHrooSnf61kG8KlPfSrxvKysLFF9NHnyZJ5++mkGDx7MvHnzePnll7NeV6dOnfZbb6dOndizZw8HHdTy16KlrpyffPIJS5Ys4ZBDDsnm44hIkVL1UR6ceuqpNDQ08NBDwa0j9u7dy/e+9z0mT55Mly5d0r73S1/6Eo8//jgAb7zxBqtWrWrVtj/44AO6d+9OY2Mj8+fPb9sHaGbkyJG88sorbNu2jb179/LII48wevRoRo4cycsvv8z27dtpbGzkiSeeSLznjDPO4J577knML1++PC+xiEhhlXRSyFf9nJnx1FNP8cQTT9C3b19OOOEEysvL+ad/+qeM7/32t79NfX09VVVV3HbbbVRVVXH44Ydnve0f/OAHjBw5ktNPP53Pfe5zuXyMhO7du3PrrbcyZswYBg8ezNChQ5kwYQLdu3enpqaGk08+mdNOO42hQ/fdLO+uu+6itraWqqoq+vfvz89+9rO8xCIihWXZVl8Ui+rqam9+k521a9fSr1+/mCLKzd69e2lsbKS8vJw333yTU089lfXr13PwwQfHHVqH156/NyKtZWbL3L06U7mSa1MoNg0NDYwZM4bGxkbcnfvuu08JQURio6QQs65du+r2oiJSNEq6TUFERPanpCAiIglKCiIikqCkICIiCUoKefLpT3/6gGWLFy9m6NChHHTQQVmNihqFtgyhvXDhQn74wx/mvO1TTjkl8kb0yZMnZ9y32ZQRkUBJJoX586GyEjp1Ch7zdCHwAXr27Mm8efP4+te/nvV79uzZE00wrdj++PHjue6662KNQ0TiEWlSMLMzzWydmW00swOOMmbWy8xeNLOVZvaymVVEGQ8ECWDqVNi8ORj3aPPmYD6KxFBZWUlVVRWdOqXfzZMnT+a73/0uY8aMYcaMGXz44YdceumlDB8+nJNOOolnnnkGSD/MdvKZyoIFC/YbBrtJS8NsN99+8o1xhgwZkpgOOeQQXnnllRbj++ijj7jkkksS8TWNz5Rqv1x//fWcfPLJVFdX8/rrrzNu3Dj69OmTuBLa3bn22msZOHAggwYN4rHHHkssnzZtGv379+crX/kK7733XmK9y5YtY/To0QwbNoxx48axdevWjH8jEdlfZNcpmFkZcC9wOlAHLDWzhe7+RlKxHwEPufuDZjYWuBX4+6hiApg5E5rfcqChIVg+aVKUW05v/fr1vPDCC5SVlXH99dczduxY5s6dy44dOxgxYgSnnXYa9913X2KY7dWrVydGTc3WBRdcwGWXXQbADTfcwAMPPMBVV111wPbnzZuXeE/TGEa/+c1vuP322/nCF77ArFmzUsb385//nC5durBy5UpWrly53zAYzfXo0YMlS5ZwzTXXMHnyZF577TV2797NgAEDuPzyy/n1r3/N8uXLWbFiBdu2bWP48OGMGjWKJUuWsG7dOlatWsW7775L//79ufTSS2lsbOSqq67imWeeoVu3bjz22GPMnDmTuXPntvIvIVLaorx4bQSw0d3fAjCzR4EJQHJS6A9cEz5/CXg6wngAePvt1i0vlK997WuUlZUB8Pzzz7Nw4cJEW8Du3bt5++23sx5muyXphtlO3n5zGzZs4Nprr2XRokV07ty5xfgWL17M1VdfDUBVVVXa+MaPHw/AoEGD2LVrF127dqVr166Ul5ezY8cOXn31VSZOnEhZWRnHHHMMo0ePZunSpSxevDix/Nhjj2Xs2LEArFu3jtWrV3P66acDwfAh3bt3b9X+EZFok8JxwDtJ83XAyGZlVgAXAj8Bzge6mtln3H17VEH17BlUGaVaXigzZ87kt7/9LbDvl3jy0NruzpNPPsmJJ5643/vSjVOVPKT17t27U5ZJN8x2S0N7f/jhh1x00UXcf//9HHvssWnjax5HOpmG7872szZxdwYMGMCSJUuy2r6IpBZlm0Kqo0Pz//R/BEab2V+A0cB/AAe0tJrZVDOrNbPa+vr6nIKaPRuaj2bdpUuwvFBmz57N8uXLWxxeety4cdx9992JA+Nf/vIXIP0w28cccwxr167lk08+4amnnkq53rYMsz1lyhSmTJnCl7/85YzxjRo1KrHe1atXs3Llyqy2kcqoUaN47LHH2Lt3L/X19SxevJgRI0YwatQoHn30Ufbu3cvWrVt56aWXADjxxBOpr69PJIXGxkbWrFnT5u2LlKook0Id0CNpvgLYklzA3be4+wXufhIwM1z2t+Yrcvc57l7t7tVNd/1qq0mTYM4c6NUruMlOr17BfK7tCQ0NDVRUVCSmO+64g6VLl1JRUcETTzzBt771LQYMGJDVum688UYaGxupqqpi4MCB3HjjjUD6YbZ/+MMfcs455zB27NgWq01aO8z25s2bWbBgAXPnzk00NtfW1rYY3xVXXMGuXbuoqqri9ttvZ8SIEVl93lTOP/98qqqqGDx4MGPHjuX222/ns5/9LOeffz59+/Zl0KBBXHHFFYwePRqAgw8+mAULFjBjxgwGDx7MkCFD+OMf/9jm7YuUqsiGzjazg4D1wKkEZwBLga+7+5qkMkcD77v7J2Y2G9jr7jelW29HGzq7NTTMdn6VyvdGBIpg6Gx332Nm04DngDJgrruvMbNbgFp3XwicAtxqZg4sBq6MKp6OQMNsi0jUIh06292fBZ5ttuympOcLAF1qmiUNsy0iUeswVzS3tzvISbz0fRFJrUMkhfLycrZv365/dMmKu7N9+3bKy8vjDkWk6HSIO69VVFRQV1dHrt1VpXSUl5dTURH5qCoi7U6HSAqdO3emd+/ecYchItLudYjqIxERyQ8lBRERSVBSEBGRBCUFERFJUFIQEZEEJQUREUlQUhARkQQlBRERSVBSEBGRBCUFERFJUFIQEZEEJQUREUlQUhARkQQlBRERSVBSEBGRBCUFERFJiDQpmNmZZrbOzDaa2XUpXu9pZi+Z2V/MbKWZnR1lPCIikl5kScHMyoB7gbOA/sBEM+vfrNgNwOPufhJwCfDTqOIREZHMojxTGAFsdPe33P1j4FFgQrMyDhwWPj8c2BJhPCIikkGU92g+Dngnab4OGNmsTA3wvJldBRwKnBZhPCIikkGUZwqWYpk3m58IzHP3CuBs4F/M7ICYzGyqmdWaWW19fX0EoYqICESbFOqAHknzFRxYPfQN4HEAd18ClANHN1+Ru89x92p3r+7WrVtE4YqISJRJYSnQ18x6m9nBBA3JC5uVeRs4FcDM+hEkBZ0KiIjEJLKk4O57gGnAc8Bagl5Ga8zsFjMbHxb7HnCZma0AHgEmu3vzKiYRESmQKBuacfdngWebLbsp6fkbwBejjEFEpFjU1ARTMdMVzSJSMuI+IN98c7zbz4aSgoiUjPZwUI6bkoKISIRqasAsmGDf87jPWlqipCAiHVrcB+WaGnAPJtj3vFiTgrW3zj7V1dVeW1sbdxgi0g6Z7Ts4l9r2zWyZu1dnKqczBRGRApk1K+4IMlNSEJGSEfdBuVirjJIpKYhIyWgPB+W4KSmIiEhCxqRgZoc2jVxqZieY2Xgz6xx9aCIiUmjZnCksBsrN7DjgRWAKMC/KoEREJB7ZJAVz9wbgAuBudz+f4PaaIiLSwWSVFMzsZGAS8NtwWaQD6YmISDyySQrfAb4PPBUOfX088FK0YYmISBwyJgV3f8Xdx7v7bWGD8zZ3v7oAsYmIFJVS6NKaTe+jX5nZYWZ2KPAGsM7Mro0+NBGR4lIKo6xmU33U3913AucR3DCnJ/D3kUYlIiKxyCYpdA6vSzgPeMbdG4H2NYqeiEgbxT3KaqFlkxR+DmwCDgUWm1kvYGeUQYmIFIv2NvR1rjJ2LXX3u4C7khZtNrMx0YUkIiJxyaah+XAzu8PMasPpxwRnDRmZ2Zlmts7MNprZdSlev9PMlofTejPb0YbPICJSEHGPsloI2VQfzQU+AC4Kp53ALzO9yczKgHuBswiugJ5oZvtdCe3u17j7EHcfAtwN/Lp14YuIFE5HrTJKls2VyX3c/cKk+ZvNbHkW7xsBbHT3twDM7FFgAkG31lQmAiWQh0VEilc2ZwofmdmXmmbM7IvAR1m87zjgnaT5unDZAcLG697AohZen9pUfVVfX5/FpkVEpC2yOVO4AnjQzA4HDHgfmJzF+yzFspa6sl4CLHD3valedPc5wBwI7tGcxbZFRKQNsul9tBwYbGaHhfPZdketA3okzVcAW1ooewlwZZbrFRGRiLSYFMzsuy0sB8Dd78iw7qVAXzPrDfwHwYH/6ynWdyJwJLAku5BFRCQq6c4UuuayYnffY2bTgOeAMmBuOMrqLUCtuy8Mi04EHnV3VQuJiMTM2tuxuLq62mtra+MOQ0SkXTGzZe5enalcNr2PRESkRCgpiIhIgpKCiIgkZOySamafAi4EKpPLu/st0YUlIiJxyObitWeAvwHLgP+KNhwREYlTNkmhwt3PjDwSERGJXTZtCn80s0GRRyIiIrHL5kzhS8BkM/t3guojA9zdqyKNTERECi6bpHBW5FGIiGShpqY07mkQp4zVR+6+GTgCODecjgiXiYgU1M03xx1Bx5fN7TinA/OBvwunh83sqqgDExGRwsumofkbwEh3v8ndbwI+D1wWbVgiIoGaGjALJtj3XNVI0cimTcGA5Jvf7CX1DXRERPIuuR3BDNrZGJ7tTjZJ4ZfAn83sqXD+POCB6EISEZG4ZHPntTvM7GWCrqkGTHH3v0QdmIhIc7NmxR1Bx5fuzmuHuftOMzsK2BROTa8d5e7vRx+eiMg+akeIXrozhV8B5xCMeZRci2fh/PERxiUiIjFosfeRu58TPvZ29+OTpt7uroQgUoJy/aWuX/rFL5vrFF7MZpmIdHy5Xjymi8+KX4tJwczKw/aEo83sSDM7KpwqgWOzWbmZnWlm68xso5ld10KZi8zsDTNbY2a/asuHEBGR/Eh3pvAtgvaEz4WPTdMzwL2ZVmxmZWG5s4D+wEQz69+sTF/g+8AX3X0A8J02fAYRiVCuF4/p4rP2xTzDlSBmdpW7393qFZudDNS4+7hw/vsA7n5rUpnbgfXu/ots11tdXe21tbWtDUdE8iDXi8d08Vl8zGyZu1dnKpfNdQp3m9lAgl/75UnLH8rw1uOAd5Lm64CRzcqcEAb7GlBGkER+nykmERGJRjb3aJ4FnEKQFJ4lqA56FciUFFINhdH8N8JBQN9w/RXA/zezge6+o1kMU4GpAD179swUsohEJNeLx3TxWfHLZkC8rwKnAv/p7lOAwcCnsnhfHdAjab4C2JKizDPu3uju/w6sI0gS+3H3Oe5e7e7V3bp1y2LTIhIFdUnt+LJJCh+5+yfAHjM7DHiP7C5cWwr0NbPeZnYwcAmwsFmZp4ExAGZ2NEF10lvZBi8iIvmVzYB4tWZ2BHA/Qe+jXcC/ZnqTu+8xs2nAcwTtBXPdfY2Z3QLUuvvC8LUzzOwNgtFXr3X37W38LCIikqOMvY/2Kxxco3CYu6+MKqBM1PtIRKT1cu59ZGZD073m7q+3NTgRESlO6aqPfhw+lgPVwAqCHkVVwJ8JhtIWEZEOJN2AeGPcfQywGRga9v4ZBpwEbCxUgCIiUjjZ9D76nLuvappx99XAkOhCEhGRuGTT+2itmf0CeJjg4rP/CayNNCoREYlFNklhCnAFMD2cXwzcF1lEIiISm2zGPtoN3BlOIiLSgaXrkvq4u19kZqs4cMwi3L0q0shERKTg0p0pNFUXnVOIQEREJH4tJgV33xo+bi5cOCIiEqd0t+P8wMx2ppg+MLOdhQxSRPJDo5RKJukuXuvq7oelmLq6+2GFDFJE8uPmm+OOQIpdNl1SATCzv2P/O6+9HUlEIiISm4xXNJvZeDPbAPw78AqwCfhdxHGJSJ7U1AT3RrbwXohNz1WVJKlkM8zFD4DPA+vdvTfBXdheizQqEUmpLQfymhpwDybY91xJQVLJJik0hje+6WRmndz9JTT2kUgs1CYgUcumTWGHmX2aYHiL+Wb2HrAn2rBEJAqzZsUdgRS7bM4UJgAfAdcAvwfeBM6NMigR2SefbQKqMpJMWrwdp5ndA/zK3f9Y2JDS0+04pZSZ7WsbEGmNbG/Hme5MYQPwYzPbZGa3mZnaEUREOrh0F6/9xN1PBkYD7wO/NLO1ZnaTmZ2QzcrN7EwzW2dmG83suhSvTzazejNbHk7fbPMnESkBahOQqGVsU3D3ze5+m7ufBHwdOJ8sbrJjZmXAvcBZQH9gopn1T1H0MXcfEk6/aF34IoVTDPXxxRCDdGzZXLzW2czONbP5BBetrQcuzGLdI4CN7v6Wu38MPErQaC3SLqk7qJSCdAPinW5mc4E6YCrwLNDH3S9296ezWPdxwDtJ83XhsuYuNLOVZrbAzHq0InYREcmzdGcK1wNLgH7ufq67z3f3D1uxbkuxrHm/id8AleENe14AHky5IrOpZlZrZrX19fWtCEEkN+oOKqWmxS6pOa/Y7GSgxt3HhfPfB3D3W1soXwa87+6Hp1uvuqRKXHLtDqrupBKnfHRJzdVSoK+Z9Tazg4FLgIXJBcyse9LseLJowBYRkehElhTcfQ8wDXiO4GD/uLuvMbNbzGx8WOxqM1tjZiuAq4HJUcUjkqu2dAfVCKXS3kRWfRQVVR9Je6XqI4lTMVQfiYhIO6OkIFIguhpZ2gMlBZECUTuCtAdKCiIikqCkICIZzZ8PlZXQqVPwOH9+3BFJVLK585qIlLD582HqVGhoCOY3bw7mASZNii8uiYbOFEQkrZkz9yWEJg0NwXLpeJQURCStt99u3fIoqPqqcJQURCStnj1btzzfmqqvNm8OLv5rqr5SYoiGkoKIpDV7NnTpsv+yLl2C5YWg6qvCUlKQkqHrBNpm0iSYMwd69QqG6ujVK5gvVCNzMVRflRKNfSQlQ2MPtU+VlUGVUXO9esGmTYWOpv3S2Eci0iHEXX1VakoiKajnQunS0NXtX9zVV6Wmw1cfNb/wBoJfGfpSlR5VH5Wu+fODhum33w56Tc2eXXr//6o+Cqnngkhpny2rS2vrdPikoJ4L0qRUh64u9YNiMfwwzDUpFzKpd/jqI/VckFJX6v8DnTqlrjY0g08+iX77uVZh56sKXNVHIfVckFJX6mfL+bgiO5df6rmeqRT6TCfSpGBmZ5rZOjPbaGbXpSn3VTNzM8uYxVpLPRek1MU9TAXE26aR6w/DXKvfck3KBU/q7h7JBJQBbwLHAwcDK4D+Kcp1BRYDfwKqM6132LBhLlJqHn7YvVcvd7Pg8eGHW/feLl3cg0NaMHXp0rp15CLu7TfF0Nb916vX/rE3Tb16tY/3NwFqPZtjdzaF2jIBJwPPJc1/H/h+inL/DJwDvKykIOnMmhV3BPHIx0E1l4NirvJ1UIuLWer4zbJ7f65/v3wl1WJICl8FfpE0//fAPc3KnAQ8GT5XUpC0IO4I4lHqB9W45WP/55qU85HUs00KUbYpWIpliT4AZtYJuBP4XsYVmU01s1ozq62vr89jiJKNUu7jXgzae0NxMbRp5CIfnVUmTQp6en3ySfDY2jbNXN/fGlEmhTqgR9J8BbAlab4rMBB42cw2AZ8HFqZqbHb3Oe5e7e7V3bp1izBkaS7uPu4apkIH1biVXGeVbE4n2jIR3P/5LaA3+xqaB6Qp/zKqPio6xVR10V6rj/JRdRB3Q22u4mzTkABxVx+5+x5gGvAcsBZ43N3XmNktZjY+qu1KfrX3qot8yKX6LB9nWh3hl2ohqz8kR9lkjmKa2uOZQnv+lVQMjWxN2tr7KM7unMV0piWljbh7H0U1xZEU2nMf8VwVS3e6tor7oN7ee95Ix6GkkCdxH1SKQZwX/sS9/VwP6h3h7y8dQ7ZJocMPiJerXAcTi3swrrjl+vlzHQws1+3n+vfX/TykWGhAvDzJtaG1vXcnzFWunz/XwcBy3X6u3Sk7QiOxlBYlhQziPqjkQ3sejCzXpFwMB3X1vJF2JZs6pmKa2lubQtM64rrEPe6G3qYYmurW21ubhEhHgRqa8yfOg0pHauhuy8VnxZDURDqCbJOCqo+yEOfpf6516vm8+KwtQ0vkOkyF6uRFCku9j4pc3L1nmm8zl69Lru8XkbZT76MOoiM0dItI+6GkUOTi7j2Tz1FKZ81q/XtEpLBUfdQOzJ8ftCG8/XZwhjB7djx16qr+EWm/sq0+OqgQwUhuJk1Sw6qIFEZJVR+V0o1ZoqDqH5GOr6Sqj1T9ISKlSr2PRESk1Tp8UtA9fkVEslcSSaFpgATY97wUk0IpfmYRaZ0OnxRkn5tvjjsCESl2JZUU1HtGRCS9SJOCmZ1pZuvMbKOZXZfi9cvNbJWZLTezV82sf5TxlGL1idpURKQ1IuuSamZlwHrgdKAOWApMdPc3ksoc5u47w+fjgW+7+5np1luKVzTni7rkipSuYuiSOgLY6O5vufvHwKPAhOQCTQkhdCigQ5aISIyiTArHAe8kzdeFy/ZjZlea2ZvA7cDVEcbT7uVa5aM2FRHJJMqkYCmWHXAm4O73unsfYAZwQ8oVmU01s1ozq62vr89zmO1Hrr2H1I4gIplEmRTqgB5J8xXAljTlHwXOS/WCu89x92p3r+7WrVseQ2wdHVRFpKOLMiksBfqaWW8zOxi4BFiYXMDM+ibNfgXYEGE8OYvjl7p6D4lIIUU6IJ6ZnQ38M1AGzHX32WZ2C8ENpBea2U+A04BG4K/ANHdfk26dcfY+ivt2lOo9JCJtVQy9j3D3Z939BHfv4+6zw2U3ufvC8Pl0dx/g7kPcfUymhBAH/VIXkVJSUlc0t0WuYyfpdpYi0p6U1P0UcqXqHxFpr4qi+qij0S91EenolBRaQRePiUhHp6RQQGqcFpFip6QgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCe3u4jUzqwc2xx1HC44GtsUdRBqKLzfFHh8Uf4yKLze5xNfL3TMOM93ukkIxM7PabK4YjIviy02xxwfFH6Piy00h4lP1kYiIJCgpiIhIgpJCfs2JO4AMFF9uij0+KP4YFV9uIo9PbQoiIpKgMwUREUlQUmglM+thZi+Z2VozW2Nm01OUOcXM/mZmy8PppgLHuMnMVoXbPuDmExa4y8w2mtlKMxtawNhOTNovy81sp5l9p1mZgu8/M5trZu+Z2eqkZUeZ2R/MbEP4eGQL7/2HsMwGM/uHAsX2f83s38K/31NmdkQL7037XYg4xhoz+4+kv+PZLbx7YLTDAAAFnUlEQVT3TDNbF34frytgfI8lxbbJzJa38N5I92FLx5TYvn/urqkVE9AdGBo+7wqsB/o3K3MK8P9ijHETcHSa188GfgcY8HngzzHFWQb8J0H/6Vj3HzAKGAqsTlp2O3Bd+Pw64LYU7zsKeCt8PDJ8fmQBYjsDOCh8fluq2LL5LkQcYw3wj1l8B94EjgcOBlY0/3+KKr5mr/8YuCmOfdjSMSWu75/OFFrJ3be6++vh8w+AtcBx8UbVahOAhzzwJ+AIM+seQxynAm+6e+wXI7r7YuD9ZosnAA+Gzx8Ezkvx1nHAH9z9fXf/K/AH4MyoY3P35919Tzj7J6Ain9tsrRb2XzZGABvd/S13/xh4lGC/51W6+MzMgIuAR/K93WykOabE8v1TUsiBmVUCJwF/TvHyyWa2wsx+Z2YDChoYOPC8mS0zs6kpXj8OeCdpvo54EtsltPyPGOf+a3KMu2+F4B8X+LsUZYphX15KcOaXSqbvQtSmhVVcc1uo/iiG/fdl4F1339DC6wXbh82OKbF8/5QU2sjMPg08CXzH3Xc2e/l1giqRwcDdwNMFDu+L7j4UOAu40sxGNXvdUrynoN3QzOxgYDzwRIqX495/rRHrvjSzmcAeYH4LRTJ9F6J0H9AHGAJsJaiiaS727yIwkfRnCQXZhxmOKS2+LcWynPafkkIbmFlngj/efHf/dfPX3X2nu+8Knz8LdDazowsVn7tvCR/fA54iOEVPVgf0SJqvALYUJrqEs4DX3f3d5i/Evf+SvNtUrRY+vpeiTGz7MmxUPAeY5GEFc3NZfBci4+7vuvted/8EuL+Fbcf6XTSzg4ALgMdaKlOIfdjCMSWW75+SQiuF9Y8PAGvd/Y4Wynw2LIeZjSDYz9sLFN+hZta16TlBg+TqZsUWAv8r7IX0eeBvTaepBdTir7M4918zC4Gm3hz/ADyTosxzwBlmdmRYPXJGuCxSZnYmMAMY7+4NLZTJ5rsQZYzJ7VTnt7DtpUBfM+sdnj1eQrDfC+U04N/cvS7Vi4XYh2mOKfF8/6JqUe+oE/AlgtOzlcDycDobuBy4PCwzDVhD0JPiT8AXChjf8eF2V4QxzAyXJ8dnwL0EvT5WAdUF3oddCA7yhycti3X/ESSorUAjwa+vbwCfAV4ENoSPR4Vlq4FfJL33UmBjOE0pUGwbCeqSm76DPwvLHgs8m+67UMD99y/h92slwQGue/MYw/mzCXrcvBlVjKniC5fPa/reJZUt6D5Mc0yJ5funK5pFRCRB1UciIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgEjKzvbb/CK55G7HTzCqTR+gUKVYHxR2ASBH5yN2HxB2ESJx0piCSQTie/m1m9q/h9D/C5b3M7MVwwLcXzaxnuPwYC+5xsCKcvhCuqszM7g/HzH/ezA4Jy19tZm+E63k0po8pAigpiCQ7pFn10cVJr+109xHAPcA/h8vuIRiCvIpgQLq7wuV3Aa94MKDfUIIrYQH6Ave6+wBgB3BhuPw64KRwPZdH9eFEsqErmkVCZrbL3T+dYvkmYKy7vxUOXPaf7v4ZM9tGMHRDY7h8q7sfbWb1QIW7/1fSOioJxr3vG87PADq7+/8xs98DuwhGg33aw8EAReKgMwWR7HgLz1sqk8p/JT3fy742va8QjEU1DFgWjtwpEgslBZHsXJz0uCR8/keCUT0BJgGvhs9fBK4AMLMyMzuspZWaWSegh7u/BPxv4AjggLMVkULRLxKRfQ6x/W/e/nt3b+qW+ikz+zPBD6mJ4bKrgblmdi1QD0wJl08H5pjZNwjOCK4gGKEzlTLgYTM7nGD02jvdfUfePpFIK6lNQSSDsE2h2t23xR2LSNRUfSQiIgk6UxARkQSdKYiISIKSgoiIJCgpiIhIgpKCiIgkKCmIiEiCkoKIiCT8Nxwl3JaGZZSNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "l1_model_val_loss = l1_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, l1_model_val_loss, 'bo', label='L1-regularized model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### l1_l2(l1=0.0001, l2=0.0001)모델과 비교"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "l1l2_model = models.Sequential()\n",
    "l1l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l1_l2(l1=0.0001, l2=0.0001),\n",
    "                            activation='relu', input_shape=(10000,)))\n",
    "l1l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l1_l2(l1=0.0001, l2=0.0001),\n",
    "                            activation='relu'))\n",
    "l1l2_model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "l1l2_model.compile(optimizer='rmsprop',\n",
    "                   loss='binary_crossentropy',\n",
    "                   metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 2s 92us/step - loss: 0.5541 - acc: 0.8168 - val_loss: 0.4444 - val_acc: 0.8563\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.3722 - acc: 0.8947 - val_loss: 0.3823 - val_acc: 0.8836\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.3349 - acc: 0.9060 - val_loss: 0.3800 - val_acc: 0.8812\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.3230 - acc: 0.9090 - val_loss: 0.3645 - val_acc: 0.8873\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.3103 - acc: 0.9145 - val_loss: 0.3826 - val_acc: 0.8787\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.3044 - acc: 0.9175 - val_loss: 0.3736 - val_acc: 0.8849\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2967 - acc: 0.9188 - val_loss: 0.3726 - val_acc: 0.8846\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2963 - acc: 0.9190 - val_loss: 0.3773 - val_acc: 0.8839\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2882 - acc: 0.9245 - val_loss: 0.3889 - val_acc: 0.8806\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2818 - acc: 0.9268 - val_loss: 0.3815 - val_acc: 0.8831\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2789 - acc: 0.9266 - val_loss: 0.3825 - val_acc: 0.8822\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2749 - acc: 0.9294 - val_loss: 0.3872 - val_acc: 0.8812\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2696 - acc: 0.9312 - val_loss: 0.3976 - val_acc: 0.8744\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2610 - acc: 0.9350 - val_loss: 0.3933 - val_acc: 0.8780\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2579 - acc: 0.9371 - val_loss: 0.4073 - val_acc: 0.8743\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2505 - acc: 0.9389 - val_loss: 0.4054 - val_acc: 0.8758\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2478 - acc: 0.9396 - val_loss: 0.4036 - val_acc: 0.8767\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2420 - acc: 0.9426 - val_loss: 0.4476 - val_acc: 0.8634\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2408 - acc: 0.9409 - val_loss: 0.4184 - val_acc: 0.8746\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2369 - acc: 0.9448 - val_loss: 0.4186 - val_acc: 0.8748\n"
     ]
    }
   ],
   "source": [
    "l1l2_model_hist = l1l2_model.fit(x_train, y_train,\n",
    "                                 epochs=20,\n",
    "                                 batch_size=512,\n",
    "                                 validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XucFOWd7/HPjwEzoniLxKBcBjlouA23ATSJIN5Qo+BdCbsnYCLxgqJmjUaMM5rDGj2J7nqJCSpBV+INo6Jho1EUji7JMihXXS66oLOwOqCIOLIO+Dt/VE3TDD3TPdNdXT0z3/frVa/uqn6q+tc1PfXreup5njJ3R0REBKBd3AGIiEjhUFIQEZEEJQUREUlQUhARkQQlBRERSVBSEBGRBCUFERFJUFIQEZGEyJKCmc00s4/MbGUDr5uZ3W1m68xsuZkNiSoWERHJTPsItz0LuBd4pIHXTwN6h9MI4P7wsVGHHnqol5SU5CZCEZE2YsmSJZvdvXO6cpElBXdfaGYljRQZBzziwTgbfzWzg8ysi7tvamy7JSUlVFZW5jBSEZHWz8w2ZFIuzmsKRwAfJM1Xhcv2YmaTzazSzCqrq6vzEpyISFsUZ1KwFMtSjs7n7jPcvczdyzp3Tnv2IyIizRRnUqgCuiXNdwU2xhSLiIgQ7YXmdOYCU8zscYILzJ+mu57QkNraWqqqqtixY0dOAxRJpbi4mK5du9KhQ4e4QxHJuciSgpk9BhwPHGpmVUA50AHA3X8LzANOB9YBNcCk5r5XVVUVnTp1oqSkBLNUtVIiueHubNmyhaqqKnr27Bl3OCI5F2Xro/FpXnfgily8144dO5QQJC/MjK9//euowYPEoaIimKLUano0KyFIvui7JnG55Zbo36PVJAUREcmekkKOVFVVMW7cOHr37k2vXr2YOnUqX375ZcqyGzdu5Lzzzku7zdNPP52tW7c2K56Kigp+9atfNWvdTM2aNYspU6ZkXUZEGlZRAWbBBLufR1WN1KaTQq52qrtzzjnncNZZZ7F27VrWrFnD9u3bmTZt2l5ld+7cyeGHH86cOXPSbnfevHkcdNBBuQlSRFqkigpwDybY/VxJIQK5qp+bP38+xcXFTJoUNKAqKirirrvuYubMmdTU1DBr1izOP/98zjzzTE455RTWr19P//79AaipqeGCCy6gtLSUCy+8kBEjRiSG8SgpKWHz5s2sX7+ePn36cMkll9CvXz9OOeUUvvjiCwAeeOABhg0bxsCBAzn33HOpqalpNNaJEydy2WWXMXr0aI488kgWLFjAxRdfTJ8+fZg4cWKi3GOPPcaAAQPo378/119/fWL573//e4466ihGjRrFG2+8kVheXV3Nueeey7Bhwxg2bNger4lIy9Gmk0KurFq1iqFDh+6x7IADDqB79+6sW7cOgEWLFvHwww8zf/78Pcr95je/4eCDD2b58uX8/Oc/Z8mSJSnfY+3atVxxxRWsWrWKgw46iKeffhqAc845h8WLF7Ns2TL69OnDQw89lDbeTz75hPnz53PXXXdx5plncs0117Bq1SpWrFjB0qVL2bhxI9dffz3z589n6dKlLF68mGeffZZNmzZRXl7OG2+8wV/+8hfefvvtxDanTp3KNddcw+LFi3n66af50Y9+1KR9KCLplZdH/x5xdl6LRUXFnmcIdfV05eXNPx1z95QtUpKXn3zyyRxyyCF7lXn99deZOnUqAP3796e0tDTle/Ts2ZNBgwYBMHToUNavXw/AypUruemmm9i6dSvbt29nzJgxaeM988wzMTMGDBjAYYcdxoABAwDo168f69evZ8OGDRx//PHUDSkyYcIEFi5cCLDH8gsvvJA1a9YA8PLLL++RJLZt28Znn32WNhYRyVzUzVGhjSaFuh1rtrueLhv9+vVL/HKvs23bNj744AN69erFkiVL2G+//VKu6xkG8LWvfS3xvKioKFF9NHHiRJ599lkGDhzIrFmzeO211zLeVrt27fbYbrt27di5cyft2zf8tWioOeZXX33FokWL2HfffTP5OCJSoFR9lAMnnngiNTU1PPJIcOuIXbt28ZOf/ISJEyfSsWPHRtf97ne/y5NPPgnA22+/zYoVK5r03p999hldunShtraW2bNnN+8D1DNixAgWLFjA5s2b2bVrF4899hijRo1ixIgRvPbaa2zZsoXa2lqeeuqpxDqnnHIK9957b2J+6dKlOYlFRPKrTSeFXNXPmRnPPPMMTz31FL179+aoo46iuLiYf/zHf0y77uWXX051dTWlpaXcfvvtlJaWcuCBB2b83r/4xS8YMWIEJ598Mt/61rey+RgJXbp04bbbbmP06NEMHDiQIUOGMG7cOLp06UJFRQXHHnssJ510EkOG7L5Z3t13301lZSWlpaX07duX3/72tzmJRUTyyzKtvigUZWVlXv8mO++88w59+vSJKaLs7Nq1i9raWoqLi3n33Xc58cQTWbNmDfvss0/coUkjWvJ3TtomM1vi7mXpyrW5awqFpqamhtGjR1NbW4u7c//99yshiEhslBRi1qlTJ91eVEQKRpu+piAiIntSUhARkQQlBRERSVBSEBGRBCWFHNl///33WrZw4UKGDBlC+/btGx0VNdW6d955J3379qW0tJQTTzyRDRs25DTeTDRn+O25c+fyy1/+Muv3Pv744yO/AD9x4sS0o9VmUkakNWmTSWH2bCgpgXbtgsccdQTeS/fu3Zk1axbf//73m7zu4MGDqaysZPny5Zx33nn89Kc/TbvOzp07mxNmzuzcuZOxY8dyww03xBqHiDRfpEnBzE41s9Vmts7M9jpSmFkPM3vFzJab2Wtm1jXKeCBIAJMnw4YNwbhHGzYE81EkhpKSEkpLS2nXrum7efTo0YkhMo455hiqqqpSlps4cSLXXnsto0eP5vrrr+fzzz/n4osvZtiwYQwePJjnnnsOaHyI7uQzlTlz5uwxhHadhoborv/+yTfVGTRoUGLad999WbBgQYPxffHFF1x00UWJ+OrGdkq1T2+88UaOPfZYysrKePPNNxkzZgy9evVK9KJ2d6677jr69+/PgAEDeOKJJxLLp0yZQt++ffne977HRx99lNjukiVLGDVqFEOHDmXMmDFs2rQpsz+USCsTWT8FMysC7gNOBqqAxWY2193fTir2K+ARd3/YzE4AbgP+PqqYAKZNg/q3HKipCZZPmBDlOzffQw89xGmnndbg62vWrOHll1+mqKiIG2+8kRNOOIGZM2eydetWhg8fzkknncT999+fGKJ75cqViRFXM3XOOedwySWXAHDTTTfx0EMPceWVV+71/rNmzUqsUzf+0fPPP88dd9zBt7/9bcrLy1PG97vf/Y6OHTuyfPlyli9fvscQGvV169aNRYsWcc011zBx4kTeeOMNduzYQb9+/bj00kv54x//yNKlS1m2bBmbN29m2LBhjBw5kkWLFrF69WpWrFjBhx9+SN++fbn44oupra3lyiuv5LnnnqNz58488cQTTJs2jZkzZzZpH4m0BlF2XhsOrHP39wDM7HFgHJCcFPoC14TPXwWejTAeAN5/v2nL4/boo49SWVnJggULGixz/vnnU1RUBMBLL73E3LlzE9cCduzYwfvvv5/xEN0NaWyI7uT3r2/t2rVcd911zJ8/nw4dOjQY38KFC7nqqqsAKC0tbTS+sWPHAjBgwAC2b99Op06d6NSpE8XFxWzdupXXX3+d8ePHU1RUxGGHHcaoUaNYvHgxCxcuTCw//PDDOeGEEwBYvXo1K1eu5OSTTwaCoUe6dOnSpP0j0lpEmRSOAD5Imq8CRtQrsww4F/hn4Gygk5l93d23RBVU9+5BlVGq5YXm5ZdfZvr06SxYsCAxxPW0adP405/+BOz+JZ48LLe78/TTT3P00Ufvsa3GxrhKHg57x44dKcs0NkR3Q8OCf/7551xwwQU88MADHH744Y3GVz+OxqQb+jvTz1rH3enXrx+LFi3K6P1FWrMorymk+g+v/9/6D8AoM3sLGAX8F7DX1VIzm2xmlWZWWV1dnVVQ06dD/dGsO3YMludTuhFN33rrLX784x8zd+5cvvGNbySWT58+naVLlzY4NPWYMWO45557EgfGt956C2h8iO7DDjuMd955h6+++opnnnkm5XabM0T3pEmTmDRpEscdd1za+EaOHJnY7sqVK1m+fHlG75HKyJEjeeKJJ9i1axfV1dUsXLiQ4cOHM3LkSB5//HF27drFpk2bePXVVwE4+uijqa6uTiSF2tpaVq1a1ez3F2nJojxTqAK6Jc13BTYmF3D3jcA5AGa2P3Cuu39af0PuPgOYAcEoqdkEVXfdYNq0oMqoe/cgIWR7PaGmpoauXXdfJ7/22ms57rjjOPvss/nkk094/vnnKS8vZ9WqVWzevHmPX7Op1p03bx7bt2/n/PPPB4KWTHPnzk0bx89//nOuvvpqSktLcXdKSkp44YUXuPzyy/nBD35AaWkpgwcP3mOI7l/+8pecccYZdOvWjf79+7N9+/a9tls3RHePHj0YMGBA2ruqbdiwgTlz5rBmzZpE3fyDDz7YYHyXXXYZkyZNorS0lEGDBjF8+PC0n7UhZ599NosWLWLgwIGYGXfccQff/OY3Ofvss5k/fz4DBgxI3GcaYJ999mHOnDlcddVVfPrpp+zcuZOrr76afv36NTsGkZYqsqGzzaw9sAY4keAMYDHwfXdflVTmUOBjd//KzKYDu9z95sa22xqGzn7hhRd47733EnXo+aAhunOrpX3nRGIfOtvdd5rZFOBFoAiY6e6rzOxWoNLd5wLHA7eZmQMLgSuiiqeQnHHGGXl/Tw3RLSKZiHTobHefB8yrt+zmpOdzAHUXzQMN0S0imWg1PZpb2h3kpOXSd01as1aRFIqLi9myZYv+WSVy7s6WLVsoLi6OOxSRSLSKO6917dqVqqoqsm2uKpKJ4uLiPVqLibQmrSIpdOjQgZ49e8YdhohIi9cqqo9ERCQ3lBRERCRBSUFERBKUFEREJEFJQUREEpQUREQkQUlBREQSlBRERCRBSUFERBKUFEREJEFJQUREEpQUREQkQUlBREQSlBRERCRBSUFERBKUFEREJCHSpGBmp5rZajNbZ2Y3pHi9u5m9amZvmdlyMzs9ynhERKRxkSUFMysC7gNOA/oC482sb71iNwFPuvtg4CLgN1HFIyIi6UV5pjAcWOfu77n7l8DjwLh6ZRw4IHx+ILAxwnhERCSNKO/RfATwQdJ8FTCiXpkK4CUzuxLYDzgpwnhERCSNKM8ULMUyrzc/Hpjl7l2B04F/MbO9YjKzyWZWaWaV1dXVEYQqIiIQbVKoArolzXdl7+qhHwJPArj7IqAYOLT+htx9hruXuXtZ586dIwpXRESiTAqLgd5m1tPM9iG4kDy3Xpn3gRMBzKwPQVLQqYCISEwiSwruvhOYArwIvEPQymiVmd1qZmPDYj8BLjGzZcBjwER3r1/FJCIieRLlhWbcfR4wr96ym5Oevw18J8oYREQKRUVFMBUy9WgWkTYj7gPyLbfE+/6ZUFIQkTajJRyU46akICISoYoKMAsm2P087rOWhigpiEirFvdBuaIC3IMJdj8v1KRgLa2xT1lZmVdWVsYdhoi0QGa7D85t7f3NbIm7l6UrpzMFEZE8KS+PO4L0lBREpM2I+6BcqFVGyZQURKTNaAkH5bgpKYiISELapGBm+9WNXGpmR5nZWDPrEH1oIiKSb5mcKSwEis3sCOAVYBIwK8qgREQkHpkkBXP3GuAc4B53P5vg9poiItLKZJQUzOxYYALwp3BZpAPpiYhIPDJJClcDPwOeCYe+PhJ4NdqwREQkDmmTgrsvcPex7n57eMF5s7tflYfYREQKSlto0ppJ66M/mNkBZrYf8Daw2syuiz40EZHC0hZGWc2k+qivu28DziK4YU534O8jjUpERGKRSVLoEPZLOAt4zt1rgZY1ip6ISDPFPcpqvmWSFH4HrAf2AxaaWQ9gW5RBiYgUipY29HW20jYtdfe7gbuTFm0ws9HRhSQiInHJ5ELzgWZ2p5lVhtOvCc4a0jKzU81stZmtM7MbUrx+l5ktDac1Zra1GZ9BRCQv4h5lNR8yqT6aCXwGXBBO24Dfp1vJzIqA+4DTCHpAjzezPXpCu/s17j7I3QcB9wB/bFr4IiL501qrjJJl0jO5l7ufmzR/i5ktzWC94cA6d38PwMweB8YRNGtNZTzQBvKwiEjhyuRM4Qsz+27djJl9B/gig/WOAD5Imq8Kl+0lvHjdE5jfwOuT66qvqqurM3hrERFpjkzOFC4DHjazAwEDPgYmZrCepVjWUFPWi4A57r4r1YvuPgOYAcE9mjN4bxERaYZMWh8tBQaa2QHhfKbNUauAbknzXYGNDZS9CLgiw+2KiEhEGkwKZnZtA8sBcPc702x7MdDbzHoC/0Vw4P9+iu0dDRwMLMosZBERiUpjZwqdstmwu+80synAi0ARMDMcZfVWoNLd54ZFxwOPu7uqhUREYmYt7VhcVlbmlZWVcYchItKimNkSdy9LVy6T1kciItJGKCmIiEiCkoKIiCSkbZJqZl8DzgVKksu7+63RhSUiInHIpPPac8CnwBLgf6INR0RE4pRJUujq7qdGHomIiMQuk2sK/2ZmAyKPREREYpfJmcJ3gYlm9p8E1UcGuLuXRhqZiIjkXSZJ4bTIoxARyUBFRdu4p0Gc0lYfufsG4CDgzHA6KFwmIpJXt9wSdwStXya345wKzAa+EU6PmtmVUQcmIiL5l8mF5h8CI9z9Zne/GTgGuCTasEREAhUVYBZMsPu5qpGikck1BQOSb36zi9Q30BERybnk6whm0MLG8GxxMkkKvwf+ZmbPhPNnAQ9FF5KIiMQlkzuv3WlmrxE0TTVgkru/FXVgIiL1lZfHHUHr19id1w5w921mdgiwPpzqXjvE3T+OPjwRkd10HSF6jZ0p/AE4g2DMo+RaPAvnj4wwLhERiUGDrY/c/Yzwsae7H5k09XR3JQSRNijbX+r6pV/4Mumn8Eomy0Sk9cu285g6nxW+BpOCmRWH1xMONbODzeyQcCoBDs9k42Z2qpmtNrN1ZnZDA2UuMLO3zWyVmf2hOR9CRERyo7EzhR8TXE/4VvhYNz0H3Jduw2ZWFJY7DegLjDezvvXK9AZ+BnzH3fsBVzfjM4hIhLLtPKbOZy2LeZqeIGZ2pbvf0+QNmx0LVLj7mHD+ZwDufltSmTuANe7+YKbbLSsr88rKyqaGIyI5kG3nMXU+i4+ZLXH3snTlMumncI+Z9Sf4tV+ctPyRNKseAXyQNF8FjKhX5qgw2DeAIoIk8ud0MYmISDQyuUdzOXA8QVKYR1Ad9DqQLimkGgqj/m+E9kDvcPtdgf9nZv3dfWu9GCYDkwG6d++eLmQRiUi2ncfU+azwZTIg3nnAicB/u/skYCDwtQzWqwK6Jc13BTamKPOcu9e6+38CqwmSxB7cfYa7l7l7WefOnTN4axGJgpqktn6ZJIUv3P0rYKeZHQB8RGYd1xYDvc2sp5ntA1wEzK1X5llgNICZHUpQnfRepsGLiEhuZTIgXqWZHQQ8QND6aDvw7+lWcvedZjYFeJHgesFMd19lZrcCle4+N3ztFDN7m2D01evcfUszP4uIiGQpbeujPQoHfRQOcPflUQWUjlofiYg0Xdatj8xsSGOvufubzQ1OREQKU2PVR78OH4uBMmAZQYuiUuBvBENpi4hIK9LYgHij3X00sAEYErb+GQoMBtblK0AREcmfTFoffcvdV9TNuPtKYFB0IYmISFwyaX30jpk9CDxK0Pns74B3Io1KRERikUlSmARcBkwN5xcC90cWkYiIxCaTsY92AHeFk4iItGKNNUl90t0vMLMV7D1mEe5eGmlkIiKSd42dKdRVF52Rj0BERCR+DSYFd98UPm7IXzgiIhKnxm7H+ZmZbUsxfWZm2/IZpIjkhkYplXQa67zWyd0PSDF1cvcD8hmkiOTGLbfEHYEUukyapAJgZt9gzzuvvR9JRCIiEpu0PZrNbKyZrQX+E1gArAf+NeK4RCRHKiqCeyNbeC/EuueqSpJUMhnm4hfAMcAad+9JcBe2NyKNSkRSas6BvKIC3IMJdj9XUpBUMkkKteGNb9qZWTt3fxWNfSQSC10TkKhlck1hq5ntTzC8xWwz+wjYGW1YIhKF8vK4I5BCl8mZwjjgC+Aa4M/Au8CZUQYlIrvl8pqAqowknQZvx2lm9wJ/cPd/y29IjdPtOKUtM9t9bUCkKTK9HWdjZwprgV+b2Xozu93MdB1BRKSVa6zz2j+7+7HAKOBj4Pdm9o6Z3WxmR2WycTM71cxWm9k6M7shxesTzazazJaG04+a/UlE2gBdE5Copb2m4O4b3P12dx8MfB84mwxusmNmRcB9wGlAX2C8mfVNUfQJdx8UTg82LXyR/CmE+vhCiEFat0w6r3UwszPNbDZBp7U1wLkZbHs4sM7d33P3L4HHCS5ai7RIag4qbUFjA+KdbGYzgSpgMjAP6OXuF7r7sxls+wjgg6T5qnBZfeea2XIzm2Nm3ZoQu4iI5FhjZwo3AouAPu5+prvPdvfPm7BtS7GsfruJ54GS8IY9LwMPp9yQ2WQzqzSzyurq6iaEIJIdNQeVtqbBJqlZb9jsWKDC3ceE8z8DcPfbGihfBHzs7gc2tl01SZW4ZNscVM1JJU65aJKarcVAbzPraWb7ABcBc5MLmFmXpNmxZHABW0REohNZUnD3ncAU4EWCg/2T7r7KzG41s7FhsavMbJWZLQOuAiZGFY9ItprTHFQjlEpLE1n1UVRUfSQtlaqPJE6FUH0kIiItjJKCSJ6oN7K0BEoKInmi6wjSEigpiIhIgpKCiIgkKCmIiEiCkoKIiCS0iaQwezaUlEC7dsHj7NlxRyQiLUlbOoa0jzuAqM2eDZMnQ01NML9hQzAPMGFCfHGJSMvQ1o4hrb5Hc0lJ8Eesr0cPWL8+Z2GJSCvVWo4h6tEcev/9pi2X1kv9BKQ52toxpNUnhe7dm7ZcWi/dOU2ao60dQ1p9Upg+HTp23HNZx47BchGRdNraMaTVJ4UJE2DGjKD+zyx4nDGjdV4gkr1p6GrJVls7hrT6C80idTR0tbRlutAsIlIgsu3nkM9+Eq2+n4JIHQ1dLXHItp9DvvtJqPpIRCRC2fZzyFU/CVUfiUjBaEvDRNSXbT+HfPeTiDQpmNmpZrbazNaZ2Q2NlDvPzNzM0mYxEWlZ6qo/NmwILvTXVX80JTG0pDr5+rLt55D3fhLuHskEFAHvAkcC+wDLgL4pynUCFgJ/BcrSbXfo0KEuIi1Hjx7uQTrYc+rRI7P1H33UvWPHPdft2DFYno/1s1Uo8QOVnsmxO5NCzZmAY4EXk+Z/BvwsRbl/As4AXlNSkMaUl8cdQcv16KPBQdgseMzXAdE9eM9UScEss/WzTSrZrp8L2e7/XPz9Mk0KUVYfHQF8kDRfFS5LMLPBQDd3fyHCOKSV0DAVzRN39U221R8trU4+lQkTgovCX30VPDa11VC26zdFlEnBUixLNHUys3bAXcBP0m7IbLKZVZpZZXV1dQ5DlLagLV/kBJg2bXdzxjo1NcHyTGSbVLIdJqLF1cm3dJmcTjRnIk31EXAgsBlYH047gI2kqUJS9VHbUl6e+tQ/06qkuOuTcyWb6oO4q2+yjb9Q6uRbOgrgmkJ74D2gJ7svNPdrpPxr6RKCx5QU4qyPzYWWHn8daPo6cR/QcrV+Nge1bPdBtkklF+L+G7QGsSeFIAZOB9YQtEKaFi67FRibomxBJoVc/MqI8wvZmn4lNScpZHtAK4RfqXG33imEC7WSvYJIClFM+U4Kcf9DZqs1/EPXJdW6uPN5QI17fffc/FKPs/pGCoOSQo4UQn1sNuI+oGS7fty/1LPdf7nY/3F/h9xV/dIaKCnkSEuvj437TKcQqi6yOaAVwpmCfqlLLigp5EhLP6jFHX9LT6pxJ8Xk7eiXumRDSSGHWnL1R7bxx1190hqqTnRAl0KgpFBA4qy+yFbcZwqqOhHJjUyTgobOzoNsuqjH3UU/296o2a7f1u6PKxI3JYUCF3cX/WwPysnrQ/MO6vkc90WkrVNSKHDZ/tLOhbqD8s03N/2gXFEBf/d3u+8ctWFDMF9Rkfs4RSR7uh1nCzB7djB42fvvB2cI06fH82vZLKjVj2t9EWm+TG/H2T4fwUh2JkxQlYmI5Ieqj6RRFRXBL3wLB0Kve96c6p/y8lxGJiJRUPWRZEzVPyItV6bVRzpTEBGRhDaVFNTiJTuq/hFp/dpU9ZGqP0SkrVL1kYiINFmrTwq5bD0jItLatYmkUDeUGux+3haTQlv8zCLSNK0+Kchut9wSdwQiUujaVFJQ6xkRkcZFmhTM7FQzW21m68zshhSvX2pmK8xsqZm9bmZ9o4ynLVaf6JqKiDRFZE1SzawIWAOcDFQBi4Hx7v52UpkD3H1b+HwscLm7n9rYdtWjufnUJFek7SqEJqnDgXXu/p67fwk8DoxLLlCXEEL7ATpkiYjEKMqkcATwQdJ8VbhsD2Z2hZm9C9wBXBVhPC1etlU+uqYiIulEmRQsxbK9zgTc/T537wVcD9yUckNmk82s0swqq6urcxxmy5Ft6yFdRxCRdKJMClVAt6T5rsDGRso/DpyV6gV3n+HuZe5e1rlz5xyG2DQ6qIpIaxdlUlgM9Daznma2D3ARMDe5gJn1Tpr9HrA2wniyFscvdbUeEpF8inRAPDM7HfgnoAiY6e7TzexWoNLd55rZPwMnAbXAJ8AUd1/V2DbjbH0U9+0o1XpIRJqrEFof4e7z3P0od+/l7tPDZTe7+9zw+VR37+fug9x9dLqEEAf9UheRtqRN9WhujmzHTtLtLEWkJWlT91PIlqp/RKSlKojqo9ZGv9RFpLVTUmgCdR4TkdZOSSGPdHFaRAqdkoKIiCQoKYiISIKSgoiIJCgpiIhIgpKCiIgktLjOa2ZWDWyIO44GHApsjjuIRii+7BR6fFD4MSq+7GQTXw93TzvMdItLCoXMzCoz6TEYF8WXnUKPDwo/RsWXnXzEp+ojERFJUFIQEZEEJYXcmhF3AGkovuwUenxQ+DEqvuxEHp+uKYiISILOFEREJEFJoYnMrJuuMJvwAAAGGklEQVSZvWpm75jZKjObmqLM8Wb2qZktDaeb8xzjejNbEb73XjefsMDdZrbOzJab2ZA8xnZ00n5ZambbzOzqemXyvv/MbKaZfWRmK5OWHWJmfzGzteHjwQ2s+4OwzFoz+0GeYvu/ZvYf4d/vGTM7qIF1G/0uRBxjhZn9V9Lf8fQG1j3VzFaH38cb8hjfE0mxrTezpQ2sG+k+bOiYEtv3z901NWECugBDwuedgDVA33pljgdeiDHG9cChjbx+OvCvgAHHAH+LKc4i4L8J2k/Huv+AkcAQYGXSsjuAG8LnNwC3p1jvEOC98PHg8PnBeYjtFKB9+Pz2VLFl8l2IOMYK4B8y+A68CxwJ7AMsq///FFV89V7/NXBzHPuwoWNKXN8/nSk0kbtvcvc3w+efAe8AR8QbVZONAx7xwF+Bg8ysSwxxnAi86+6xd0Z094XAx/UWjwMeDp8/DJyVYtUxwF/c/WN3/wT4C3Bq1LG5+0vuvjOc/SvQNZfv2VQN7L9MDAfWuft77v4l8DjBfs+pxuIzMwMuAB7L9ftmopFjSizfPyWFLJhZCTAY+FuKl481s2Vm9q9m1i+vgYEDL5nZEjObnOL1I4APkuariCexXUTD/4hx7r86h7n7Jgj+cYFvpChTCPvyYoIzv1TSfReiNiWs4prZQPVHIey/44AP3X1tA6/nbR/WO6bE8v1TUmgmM9sfeBq42t231Xv5TYIqkYHAPcCzeQ7vO+4+BDgNuMLMRtZ73VKsk9dmaGa2DzAWeCrFy3Hvv6aIdV+a2TRgJzC7gSLpvgtRuh/oBQwCNhFU0dQX+3cRGE/jZwl52YdpjikNrpZiWVb7T0mhGcysA8Efb7a7/7H+6+6+zd23h8/nAR3M7NB8xefuG8PHj4BnCE7Rk1UB3ZLmuwIb8xNdwmnAm+7+Yf0X4t5/ST6sq1YLHz9KUSa2fRleVDwDmOBhBXN9GXwXIuPuH7r7Lnf/CniggfeO9btoZu2Bc4AnGiqTj33YwDEllu+fkkIThfWPDwHvuPudDZT5ZlgOMxtOsJ+35Cm+/cysU91zgguSK+sVmwv877AV0jHAp3WnqXnU4K+zOPdfPXOButYcPwCeS1HmReAUMzs4rB45JVwWKTM7FbgeGOvuNQ2UyeS7EGWMydepzm7gvRcDvc2sZ3j2eBHBfs+Xk4D/cPeqVC/mYx82ckyJ5/sX1RX11joB3yU4PVsOLA2n04FLgUvDMlOAVQQtKf4KfDuP8R0Zvu+yMIZp4fLk+Ay4j6DVxwqgLM/7sCPBQf7ApGWx7j+CBLUJqCX49fVD4OvAK8Da8PGQsGwZ8GDSuhcD68JpUp5iW0dQl1z3HfxtWPZwYF5j34U87r9/Cb9fywkOcF3qxxjOn07Q4ubdqGJMFV+4fFbd9y6pbF73YSPHlFi+f+rRLCIiCao+EhGRBCUFERFJUFIQEZEEJQUREUlQUhARkQQlBZGQme2yPUdwzdmInWZWkjxCp0ihah93ACIF5At3HxR3ECJx0pmCSBrhePq3m9m/h9P/Cpf3MLNXwgHfXjGz7uHywyy4x8GycPp2uKkiM3sgHDP/JTPbNyx/lZm9HW7n8Zg+pgigpCCSbN961UcXJr22zd2HA/cC/xQuu5dgCPJSggHp7g6X3w0s8GBAvyEEPWEBegP3uXs/YCtwbrj8BmBwuJ1Lo/pwIplQj2aRkJltd/f9UyxfD5zg7u+FA5f9t7t/3cw2EwzdUBsu3+Tuh5pZNdDV3f8naRslBOPe9w7nrwc6uPv/MbM/A9sJRoN91sPBAEXioDMFkcx4A88bKpPK/yQ938Xua3rfIxiLaiiwJBy5UyQWSgoimbkw6XFR+PzfCEb1BJgAvB4+fwW4DMDMiszsgIY2ambtgG7u/irwU+AgYK+zFZF80S8Skd32tT1v3v5nd69rlvo1M/sbwQ+p8eGyq4CZZnYdUA1MCpdPBWaY2Q8JzgguIxihM5Ui4FEzO5Bg9Nq73H1rzj6RSBPpmoJIGuE1hTJ33xx3LCJRU/WRiIgk6ExBREQSdKYgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKS8P8Bvscn5jA+wfgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "l1l2_model_val_loss = l1l2_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, l1l2_model_val_loss, 'bo', label='L1,L2-regularized model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 드롭아웃 추가\n",
    "\n",
    "드롭아웃은 토론토 대학의 제프리 힌튼과 그의 학생들이 개발했습니다. 신경망을 위해 사용되는 규제 기법 중에서 가장 효과적이고 널리 사용되는 방법 중에 하나입니다. 네트워크의 층에 드롭아웃을 적용하면 훈련하는 동안 무작위로 층의 일부 출력 특성을 제외시킵니다(0으로 만듭니다). 한 층이 정상적으로 훈련하는 동안에는 어떤 입력 샘플에 대해 `[0.2, 0.5, 1.3, 0.8, 1.1]` 벡터를 출력한다고 가정합시다. 드롭아웃을 적용하면 이 벡터의 일부가 무작위로 0으로 바뀝니다. 예를 들면 `[0, 0.5, 1.3, 0, 1.1]`이 됩니다. 드롭아웃 비율은 0이 될 특성의 비율입니다. 보통 0.2에서 0.5 사이로 지정됩니다. 테스트 단계에서는 어떤 유닛도 드롭아웃되지 않습니다. 대신에 층의 출력을 드롭아웃 비율에 비례하여 줄여 줍니다. 훈련할 때보다 더 많은 유닛이 활성화되기 때문입니다.\n",
    "\n",
    "크기가 `(batch_size, features)`인 한 층의 출력을 담고 있는 넘파이 행렬을 생각해 보겠습니다. 훈련할 때는 이 행렬의 값의 일부가 랜덤하게 0이 됩니다:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "# 훈련할 때 유닛의 출력 중 50%를 버립니다\n",
    "layer_output *= np.random.randint(0, high=2, size=layer_output.shape)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "테스트할 때는 드롭아웃 비율로 출력을 낮추어 주어야 합니다. 여기에서는 0.5배만큼 스케일을 조정했습니다(앞에서 절반의 유닛을 드롭아웃했으므로):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "# 테스트 단계\n",
    "layer_output *= 0.5\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "훈련 단계에 이 두 연산을 포함시켜 테스트 단계에는 출력을 그대로 두도록 구현할 수 있습니다. 실제로 종종 이런 방식으로 구현합니다:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "# 훈련 단계\n",
    "layer_output *= np.randint(0, high=2, size=layer_output.shape)\n",
    "# 여기에서 스케일을 낮추는 대신 높입니다![image.png](attachment:image.png)\n",
    "layer_output /= 0.5\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 기법이 이상하고 무계획적으로 보일 수 있습니다. 왜 드롭아웃이 과대적합을 줄이는 데 도움이 될까요? 힌튼은 은행에서 사용하는 부정 방지 메커니즘에서 착안했다고 합니다. 그의 말을 빌리면 “은행에 갔을 때 행원들이 계속 바뀌길래 왜 그런지를 물었습니다. 자신들도 이유는 모르지만 자주 업무가 바뀐다고 했습니다. 나는 은행에서 부정 행위를 하려면 직원들 사이의 유대가 필요하기 때문이라고 판단했습니다. 각 샘플에 대해 뉴런의 일부를 무작위하게 제거하면 뉴런의 부정한 협업을 방지하고 결국 과대적합을 감소시킨다는 것을 깨달았습니다.\"\n",
    "\n",
    "핵심 아이디어는 층의 출력 값에 노이즈를 추가하여 중요하지 않은 우연한 패턴(힌튼이 이야기한 부정한 협업)을 깨뜨리는 것입니다. 노이즈가 없다면 네트워크가 이 패턴을 기억하기 시작할 것입니다.\n",
    "\n",
    "케라스에서는 층의 출력 바로 뒤에 `Dropout` 층을 추가하여 네트워크에 드롭아웃을 적용할 수 있습니다:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "model.add(layers.Dropout(0.5))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMDB 네트워크에 두 개의 `Dropout` 층을 추가하고 과대적합을 얼마나 줄여주는지 확인해 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "dpt_model = models.Sequential()\n",
    "dpt_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "dpt_model.add(layers.Dropout(0.5))\n",
    "dpt_model.add(layers.Dense(16, activation='relu'))\n",
    "dpt_model.add(layers.Dropout(0.5))\n",
    "dpt_model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "dpt_model.compile(optimizer='rmsprop',\n",
    "                  loss='binary_crossentropy',\n",
    "                  metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 2s 94us/step - loss: 0.5963 - acc: 0.6720 - val_loss: 0.4509 - val_acc: 0.8563\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.4414 - acc: 0.8197 - val_loss: 0.3414 - val_acc: 0.8836\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.3557 - acc: 0.8696 - val_loss: 0.2940 - val_acc: 0.8886\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2968 - acc: 0.9008 - val_loss: 0.2823 - val_acc: 0.8896\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2575 - acc: 0.9134 - val_loss: 0.2827 - val_acc: 0.8896\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2298 - acc: 0.9246 - val_loss: 0.2863 - val_acc: 0.8879\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.2073 - acc: 0.9342 - val_loss: 0.3196 - val_acc: 0.8852\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1864 - acc: 0.9401 - val_loss: 0.3235 - val_acc: 0.8854\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1702 - acc: 0.9457 - val_loss: 0.3446 - val_acc: 0.8832\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1566 - acc: 0.9507 - val_loss: 0.3812 - val_acc: 0.8823\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1478 - acc: 0.9541 - val_loss: 0.3830 - val_acc: 0.8811\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1396 - acc: 0.9563 - val_loss: 0.4123 - val_acc: 0.8812\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1346 - acc: 0.9596 - val_loss: 0.4472 - val_acc: 0.8770\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1218 - acc: 0.9621 - val_loss: 0.4721 - val_acc: 0.8803\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1242 - acc: 0.9618 - val_loss: 0.4887 - val_acc: 0.8773\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1169 - acc: 0.9643 - val_loss: 0.4918 - val_acc: 0.8766\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1149 - acc: 0.9656 - val_loss: 0.4986 - val_acc: 0.8719\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1098 - acc: 0.9677 - val_loss: 0.5318 - val_acc: 0.8737\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1038 - acc: 0.9685 - val_loss: 0.5627 - val_acc: 0.8746\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 2s 82us/step - loss: 0.1042 - acc: 0.9691 - val_loss: 0.5747 - val_acc: 0.8742\n"
     ]
    }
   ],
   "source": [
    "dpt_model_hist = dpt_model.fit(x_train, y_train,\n",
    "                               epochs=20,\n",
    "                               batch_size=512,\n",
    "                               validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "결과 그래프를 그려 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xt8VPW57/HPQ0AR5aIFrZVLAids5Rox3E6pSL3hBVGpFTa7R7SV0sqW2tajrW2JWnerdevZije0grbUC1qR0+OuWhUtFluCjRdAuQaJeImoIEVrgOf8sVaGIUwyk8ysmUzm+3695jWz1vxmzZNhWM+s3++3nmXujoiICEC7XAcgIiKth5KCiIjEKCmIiEiMkoKIiMQoKYiISIySgoiIxCgpiIhIjJKCiIjERJYUzOxeM3vfzF5v5Hkzs1vMbJ2ZvWpmw6KKRUREUtM+wm3PB+YA9zfy/GlAaXgbCdwR3jepe/fuXlxcnJkIRUQKxIoVKz5w9x7J2kWWFNz9BTMrbqLJROB+D+psvGRm3czsSHd/p6ntFhcXU1lZmcFIRUTaPjPblEq7XI4pHAVsjluuCdftx8ymm1mlmVXW1tZmJTgRkUKUy6RgCdYlrM7n7nPdvdzdy3v0SHr0IyIiLZTLpFAD9Ipb7glsyVEsIiJCtAPNySwGZprZgwQDzNuSjSc0pq6ujpqaGj777LOMBiiSSR07dqRnz5506NAh16GINCqypGBmDwAnAN3NrAaYDXQAcPc7gSeA04F1wE7gwpa+V01NDZ07d6a4uBizRL1SIrnl7mzdupWamhpKSkpyHY5Io6KcfTQlyfMOXJKJ9/rss8+UEKRVMzO+8IUvoIkSko6KiuAWpTZzRrMSgrR2+o5Kuq6+Ovr3aDNJQURE0qekkCE1NTVMnDiR0tJS+vXrx6xZs/j8888Ttt2yZQtf+9rXkm7z9NNP5+OPP25RPBUVFdx4440tem2q5s+fz8yZM9NuIyKNq6gAs+AGex9H1Y1U0EkhUx+qu3Puuedy9tlns3btWtasWcOOHTu46qqr9mu7a9cuvvSlL/HII48k3e4TTzxBt27dMhOkiOSligpwD26w97GSQgQy1T/37LPP0rFjRy68MJhAVVRUxM0338y9997Lzp07mT9/Pueddx4TJkzglFNOobq6mkGDBgGwc+dOvv71rzNkyBDOP/98Ro4cGSvjUVxczAcffEB1dTXHHHMMF198MQMHDuSUU07h008/BeDuu+9m+PDhDB06lEmTJrFz584mY502bRrf+c53GDduHH379uX555/noosu4phjjmHatGmxdg888ACDBw9m0KBBXHHFFbH18+bNo3///owdO5YXX3wxtr62tpZJkyYxfPhwhg8fvs9zIpI/CjopZMrKlSs57rjj9lnXpUsXevfuzbp16wBYtmwZ9913H88+++w+7W6//XYOPfRQXn31VX7605+yYsWKhO+xdu1aLrnkElauXEm3bt149NFHATj33HNZvnw5r7zyCscccwy//vWvk8b70Ucf8eyzz3LzzTczYcIELrvsMlauXMlrr71GVVUVW7Zs4YorruDZZ5+lqqqK5cuXs2jRIt555x1mz57Niy++yNNPP82qVati25w1axaXXXYZy5cv59FHH+Vb3/pWsz5DEUlu9uzo3yOXJ6/lREXFvkcI9f10s2e3/HDM3RPOLIlff/LJJ3PYYYft12bp0qXMmjULgEGDBjFkyJCE71FSUkJZWRkAxx13HNXV1QC8/vrr/OQnP+Hjjz9mx44dnHrqqUnjnTBhAmbG4MGDOeKIIxg8eDAAAwcOpLq6mk2bNnHCCSdQX1Jk6tSpvPDCCwD7rD///PNZs2YNAH/605/2SRLbt2/nk08+SRqLiKQu6umoUKBJof6DNdvbT5eOgQMHxn6519u+fTubN2+mX79+rFixgoMPPjjhaz3FAA488MDY46Kiolj30bRp01i0aBFDhw5l/vz5LFmyJOVttWvXbp/ttmvXjl27dtG+feNfi8amVe7Zs4dly5Zx0EEHpfLniEgrpe6jDDjxxBPZuXMn998fXDpi9+7d/OAHP2DatGl06tSpydeOGTOGhx9+GIBVq1bx2muvNeu9P/nkE4488kjq6upYsGBBy/6ABkaOHMnzzz/PBx98wO7du3nggQcYO3YsI0eOZMmSJWzdupW6ujoWLlwYe80pp5zCnDlzYstVVVUZiUVEsqugk0Km+ufMjMcee4yFCxdSWlpK//796dixI//xH/+R9LXf/e53qa2tZciQIVx//fUMGTKErl27pvze1157LSNHjuTkk0/m6KOPTufPiDnyyCP5xS9+wbhx4xg6dCjDhg1j4sSJHHnkkVRUVDB69GhOOukkhg3be7G8W265hcrKSoYMGcKAAQO48847MxKLiGSXpdp90VqUl5d7w4vsrF69mmOOOSZHEaVn9+7d1NXV0bFjR9avX8+JJ57ImjVrOOCAA3IdmkQgn7+rkt/MbIW7lydrV3BjCq3Nzp07GTduHHV1dbg7d9xxhxKCiOSMkkKOde7cWZcXFZFWo6DHFEREZF9KCiIiEqOkICIiMUoKIiISo6SQIUVFRZSVlTFw4ECGDh3KTTfdxJ49e3IWz6JFi/YpO5FNhxxySLNfk06Z8HpLlizhzDPPTGsbycQXM0ynjUhrVZBJYcECKC6Gdu2C+0ycCHzQQQdRVVXFypUrefrpp3niiSe4OkEZ1l27dqX/ZilINSm4e06TV/37q0y4SOsQaVIws/Fm9qaZrTOzKxM838fMnjGzV81siZn1jDIeCBLA9OmwaVNQ92jTpmA5QxUiADj88MOZO3cuc+bMwd33K53t7lx++eUMGjSIwYMH89BDDwHBL93jjz+ec845hwEDBjBjxozYDruxUtbxv8ofeeQRpk2bxl/+8hcWL17M5ZdfTllZGevXr98nvvpS3N/97ncZNmwYmzdv5qmnnmL06NEMGzaM8847jx07dgDBNR2OPvpoxowZw6WXXhr7Jd7wIj6DBg2KFemrt2PHDk488USGDRvG4MGDefzxxxt9//oy4XfeeSdlZWWUlZVRUlLCuHHjABqN749//GMsvt///vcJ/z3mz5/P2WefzYQJEygpKWHOnDncdNNNHHvssYwaNYoPP/wQCEpzjBo1iiFDhnDOOefw0UcfAbBixQqGDh3K6NGjue2222Lb3b17N5dffjnDhw9nyJAh3HXXXUm/GyKtnrtHcgOKgPVAX+AA4BVgQIM2C4ELwsdfBX6TbLvHHXecN7Rq1ar91jWmT5/6S1Tse+vTJ+VNJHTwwQfvt65bt27+7rvv+rx58/yoo47yrVu3urv7I4884ieddJLv2rXL3333Xe/Vq5dv2bLFn3vuOT/wwAN9/fr1vmvXLj/ppJN84cKF/vbbb3uvXr38/fff97q6Oh83bpw/9thj+73vwoUL/YILLnB39wsuuMAXLlyYMNaNGze6mfmyZcvc3b22tta/8pWv+I4dO9zd/Ze//KVfffXV/umnn3rPnj19w4YN7u4+efJkP+OMM9zdffbs2f6rX/0qts2BAwf6xo0b94mprq7Ot23bFnuPfv36+Z49e/Z7f3f3Pn36eG1tbWz5888/9zFjxvjixYuTxrdmzRrfs2ePn3feebH44s2bN8/79evn27dv9/fff9+7dOnid9xxh7u7f+973/Obb77Z3d0HDx7sS5YscXf3n/70pz5r1qz91v/whz/0gQMHurv7XXfd5ddee627u3/22Wd+3HHH+YYNG3zjxo2xNg0157sqkklApaew747ySGEEsM7dN7j758CDwMQGbQYAz4SPn0vwfMa99Vbz1qfD40qIxJfOXrp0KVOmTKGoqIgjjjiCsWPHsnz5cgBGjBhB3759KSoqYsqUKSxdupTly5fHSla3b99+n1LWLdWnTx9GjRoFwEsvvcSqVav48pe/TFlZGffddx+bNm3ijTfeoG/fvpSUlAAwZcqUZv/9P/7xjxkyZAgnnXQSb7/9Nu+9995+75/IrFmz+OpXv8qECROajK+kpITS0lLMjH/7t39rdHvjxo2jc+fO9OjRg65duzJhwgQABg8eTHV1Ndu2bePjjz9m7NixAFxwwQW88MIL+63/xje+EdvmU089xf33309ZWRkjR45k69atrF27tlmfkUhrE+UZzUcBm+OWa4CRDdq8AkwC/gs4B+hsZl9w961RBdW7d9BllGh9Jm3YsIGioiIOP/xwgH1KZ8cni4YalqY2s5Tbf/bZZwnbbN68ObYTnDFjBuPHj98vnpNPPpkHHnhgn9f9/e9/b/R927dvv89YRKL3XrBgAbW1taxYsYIOHTpQXFwca9dYKXEIuns2bdoUq7raWHxVVVWNlvJuqGGJ8Pjy4U2N83gj18qof+7WW2/d7xoWDbvRRPJJlEcKif4nNdy7/RAYa2Z/B8YCbwP7/Q81s+lmVmlmlbW1tWkFdd110LCadadOwfpMqa2tZcaMGcycOTPhDuX444/noYceYvfu3dTW1vLCCy8wYsQIAP72t7+xceNG9uzZw0MPPcSYMWMaLWUNcMQRR7B69Wr27NnDY489FnuPzp07xy5y06tXL6qqqqiqqmLGjBn7xTNq1ChefPHF2FXidu7cyZo1azj66KPZsGFDbCdXP/YBwaVCX375ZQBefvllNm7cuN92t23bxuGHH06HDh147rnn2JQoGzewYsUKbrzxRn7729/Srl27pPFt3LgxNmbSMGk0R9euXTn00EP585//DMBvfvMbxo4dS7du3ejatStLly4F2Kc8+amnnsodd9xBXV0dAGvWrOEf//hHi2MQaQ2iPFKoAXrFLfcEtsQ3cPctwLkAZnYIMMndtzXckLvPBeZCUCU1naCmTg3ur7oq6DLq3TtICPXrW+rTTz+lrKyMuro62rdvzze+8Q2+//3vJ2x7zjnnsGzZMoYOHYqZccMNN/DFL36RN954g9GjR3PllVfy2muvxQad27VrFytl7e6cfvrpTJwY9LT98pe/5Mwzz6RXr14MGjQoNgA7efJkLr74Ym655RYeeeQR+vXr12jsPXr0YP78+UyZMoV//vOfAPz85z+nf//+3H777YwfP57u3bvHEhfApEmTYl0nw4cPp3///vttd+rUqUyYMIHy8nLKyspSKu09Z84cPvzww9gAc3l5Offcc0+j8c2dO5czzjiD7t27M2bMGF5//fWk79GY++67jxkzZrBz50769u3LvHnzgOC61BdddBGdOnXa56jgW9/6FtXV1QwbNgx3p0ePHixatKjF7y/SGkRWOtvM2gNrgBMJjgCWA//q7ivj2nQHPnT3PWZ2HbDb3X/W1HbbWunseEuWLOHGG2/kD3/4Q65DidmxYweHHHII7s4ll1xCaWkpl112Wa7Dyltt5bsq+SfV0tmRdR+5+y5gJvAksBp42N1Xmtk1ZnZW2OwE4E0zWwMcAWSwE0cy4e67746dlLdt2za+/e1v5zokEYmQLrIjkkX6rkqu5PxIIdvyLblJ4dF3VPJBm0gKHTt2ZOvWrfpPJ62Wu7N161Y6duyY61BEmtQmrrzWs2dPampqSHe6qkiUOnbsSM+ekVdyEUlLm0gKHTp0iJ11KyIiLdcmuo9ERCQzlBRERCRGSUFERGKUFEREJEZJQUREYpQUREQkRklBRERilBRERCRGSUFERGKUFEREJEZJQUREYpQUREQkRklBRERilBRERCRGSUFERGKUFEREJCbSpGBm483sTTNbZ2ZXJni+t5k9Z2Z/N7NXzez0KOMREZGmRZYUzKwIuA04DRgATDGzAQ2a/QR42N2PBSYDt0cVj4iIJBflkcIIYJ27b3D3z4EHgYkN2jjQJXzcFdgSYTwiIpJElNdoPgrYHLdcA4xs0KYCeMrM/h04GDgpwnhERCSJKI8ULME6b7A8BZjv7j2B04HfmNl+MZnZdDOrNLPK2traCEIVERGINinUAL3ilnuyf/fQN4GHAdx9GdAR6N5wQ+4+193L3b28R48eEYUrIiJRJoXlQKmZlZjZAQQDyYsbtHkLOBHAzI4hSAo6FBARyZHIkoK77wJmAk8CqwlmGa00s2vM7Kyw2Q+Ai83sFeABYJq7N+xiEhGRLIlyoBl3fwJ4osG6n8U9XgV8OcoYRERai4qK4Naa6YxmESkYud4hX311bt8/FUoKIlIw8mGnnGtKCiIiEaqoALPgBnsf5/qopTFKCiLSpuV6p1xRAe7BDfY+bq1JwfJtsk95eblXVlbmOgwRyUNme3fOhfb+ZrbC3cuTtdORgohIlsyenesIklNSEJGCkeudcmvtMoqnpCAiBSMfdsq5pqQgIiIxSZOCmR1cX7nUzPqb2Vlm1iH60EREJNtSOVJ4AehoZkcBzwAXAvOjDEpERHIjlaRg7r4TOBe41d3PIbi8poiItDEpJQUzGw1MBf5fuC7SQnoiIpIbqSSF7wE/Ah4LS1/3BZ6LNiwREcmFpEnB3Z9397Pc/fpwwPkDd780C7GJiLQqhTClNZXZR78zsy5mdjCwCnjTzC6PPjQRkdalEKqsptJ9NMDdtwNnE1wwpzfwjUijEhGRnEglKXQIz0s4G3jc3euA/KqiJyLSQrmuspptqSSFu4Bq4GDgBTPrA2yPMigRkdYi30pfpyvp1FJ3vwW4JW7VJjMbF11IIiKSK6kMNHc1s5vMrDK8/SfBUUNSZjbezN40s3VmdmWC5282s6rwtsbMPm7B3yAikhW5rrKaDal0H90LfAJ8PbxtB+Yle5GZFQG3AacRnAE9xcz2ORPa3S9z9zJ3LwNuBX7fvPBFRLKnrXYZxUvlzOR+7j4pbvlqM6tK4XUjgHXuvgHAzB4EJhJMa01kClAAeVhEpPVK5UjhUzMbU79gZl8GPk3hdUcBm+OWa8J1+wkHr0uAZxt5fnp991VtbW0Kby0iIi2RypHCd4D7zKwrYMCHwLQUXmcJ1jU2lXUy8Ii77070pLvPBeZCcI3mFN5bRERaIJXZR1XAUDPrEi6nOh21BugVt9wT2NJI28nAJSluV0REItJoUjCz7zeyHgB3vynJtpcDpWZWArxNsOP/1wTb+xfgUGBZaiGLiEhUmjpS6JzOht19l5nNBJ4EioB7wyqr1wCV7r44bDoFeNDd1S0kIpJjlm/74vLycq+srMx1GCIiecXMVrh7ebJ2qcw+EhGRAqGkICIiMUoKIiISk3RKqpkdCEwCiuPbu/s10YUlIiK5kMrJa48D24AVwD+jDUdERHIplaTQ093HRx6JiIjkXCpjCn8xs8GRRyIiIjmXypHCGGCamW0k6D4ywN19SKSRiYhI1qWSFE6LPAoRkRRUVBTGNQ1yKWn3kbtvAroBE8Jbt3CdiEhWXX11riNo+1K5HOcsYAFweHj7rZn9e9SBiYhI9qUy0PxNYKS7/8zdfwaMAi6ONiwRkUBFBZgFN9j7WN1I0UhlTMGA+Ivf7CbxBXRERDIufhzBDPKshmfeSSUpzAP+amaPhctnA7+OLiQREcmVVK68dpOZLSGYmmrAhe7+96gDExFpaPbsXEfQ9jV15bUu7r7dzA4DqsNb/XOHufuH0YcnIrKXxhGi19SRwu+AMwlqHsX34lm43DfCuEREJAcanX3k7meG9yXu3jfuVuLuSggiBSjdX+r6pd/6pXKewjOprBORti/dk8d08lnr12hSMLOO4XhCdzM71MwOC2/FwJdS2biZjTezN81snZld2Uibr5vZKjNbaWa/a8kfISIimdHUkcK3CcYTjg7v62+PA7cl27CZFYXtTgMGAFPMbECDNqXAj4Avu/tA4Hst+BtEJELpnjymk8/yi3mSM0HM7N/d/dZmb9hsNFDh7qeGyz8CcPdfxLW5AVjj7vekut3y8nKvrKxsbjgikgHpnjymk89yx8xWuHt5snapnKdwq5kNIvi13zFu/f1JXnoUsDluuQYY2aBN/zDYF4EigiTyx2QxiYhINFK5RvNs4ASCpPAEQXfQUiBZUkhUCqPhb4T2QGm4/Z7An81skLt/3CCG6cB0gN69eycLWUQiku7JYzr5rPVLpSDe14ATgXfd/UJgKHBgCq+rAXrFLfcEtiRo87i717n7RuBNgiSxD3ef6+7l7l7eo0ePFN5aRKKgKaltXypJ4VN33wPsMrMuwPukduLacqDUzErM7ABgMrC4QZtFwDgAM+tO0J20IdXgRUQks1IpiFdpZt2AuwlmH+0A/pbsRe6+y8xmAk8SjBfc6+4rzewaoNLdF4fPnWJmqwiqr17u7ltb+LeIiEiaks4+2qdxcI5CF3d/NaqAktHsIxEpNAsWwFVXwVtvQe/ecN11MHVq87aR9uwjMxvW1HPu/nLzQhIRkeZasACmT4edO4PlTZuCZWh+YkhFo0cKZvZc+LAjUA68QjCjaAjwV3cfk/lwktORgogUkuLiIBE01KcPVFenvp1UjxSaKog3zt3HAZuAYeHsn+OAY4F1qYciIiIt9dZbzVufrlRmHx3t7q/VL7j760BZNOGIiLQ9CxYEv/jbtQvuFyxI/bWNnZoV1SlbqSSF1WZ2j5mdYGZjzexuYHU04YiItC31YwKbNgUlPurHBFJNDNddB5067buuU6dgfRRSSQoXAiuBWQQF61aF60REJImrrto7SFxv585gfSqmToW5c4MxBLPgfu7caAaZoZlTUlsDDTSLSD5p1y5xEUAz2LMne3FkYkrqw+7+dTN7jf1rFuHuQ9KMUUSkzevdO/HsodZaxq2pM5pnhfdnZiMQEZG26Lrr9j3PAKIdE0hXo0nB3d8J7xPkOBERSUV933+6ZyRnS1OX4/zEzLYnuH1iZtuzGaSIZIaqlObG1KnBiWZ79gT3rTUhQNMnr3V29y4Jbp3dvUs2gxSRzLj66lxHkJ/SOc8g36RSJRUAMzucfa+8FtH5dCIirUe2aw/lWtLzFMzsLDNbC2wEngeqgf+OOC4RyZCKimD6o4XXQqx/rK6k1KR7nkG+SeXktWuBUcAady8huArbi5FGJSIJtWRHXlERzJOvnytf/1hJITXZrj2Ua6kkhbrwwjftzKyduz+Hah+J5ITGBLIv27WHci2VpPCxmR0CvAAsMLP/AnZFG5aIRGH27FxHkH+yXXso11JJChOBT4HLgD8C64EJUQYlIntlckxAXUbNl+3aQ7nW1EV25gC/c/e/ZDekpqn2kRQys8R1dESSSfsiO8Ba4D/NrNrMrjczjSOIiLRxTZ289l/uPhoYC3wIzDOz1Wb2MzPrn8rGzWy8mb1pZuvM7MoEz08zs1ozqwpv32rxXyJSADQmIFFLOqbg7pvc/Xp3Pxb4V+AcUrjIjpkVAbcBpwEDgClmNiBB04fcvSy83dO88EWypzX0x7eGGPJRIZ2RnK5UTl7rYGYTzGwBwUlra4BJKWx7BLDO3Te4++fAgwSD1iJ5SdNB81O6Vz4rNE0VxDvZzO4FaoDpwBNAP3c/390XpbDto4DNccs14bqGJpnZq2b2iJn1akbsIlIg0vmlX2hnJKerqSOFHwPLgGPcfYK7L3D3fzRj25ZgXcN5E/8XKA4v2PMn4L6EGzKbbmaVZlZZW1vbjBBE0qPpoJmRzk493V/6hXZGcroiuxynmY0GKtz91HD5RwDu/otG2hcBH7p716a2qympkivpTgct1OmkDQvKQXDyV6pz/YuLE1+5rE+foAx11K9vKzIxJTVdy4FSMysxswOAycDi+AZmdmTc4lmkMIAtIvkl3e6bdH/pF9oZyemKLCm4+y5gJvAkwc7+YXdfaWbXmNlZYbNLzWylmb0CXApMiyoekXS1ZDqoKpSmv1NPt/ZQoZ2RnK7Iuo+iou4jyVeF2n2UbvdNut1PEmgN3Uci0kakM1CcbveNfulnV8pXXhOR9OTr2cjpXnksExeunzpVSSBb1H0kIk3S7J22Qd1HIpIRmudfWJQURKRJhXblsUKnpCAiTdI8/8JSEElBFRJFWk6zfwpLm08KqpAokv4Po6lTg0HlPXuCeyWEtqvNJwVVSJRCpx9G0hxtPilo5oQUOv0wkuZo80lBMyekXiHVG4qnH0bSHG0+KWjmhNQr1Cun6YeRNEebTwqaOSGFTj+MpDnafFIAzZwoZCpdrR9G0jyqfSQFo1BLV4uAah+JiEgLKClIwcjX0tUi2aSkIAUjn8cRVKpFskUX2RFp5dK9yI1Ic0R6pGBm483sTTNbZ2ZXNtHua2bmZpZ0EESk0OiMZMmmyJKCmRUBtwGnAQOAKWY2IEG7zsClwF+jikUkn+mMZMmmKI8URgDr3H2Du38OPAhMTNDuWuAG4LMIY5E2IJ/HBNKhM5Ilm6JMCkcBm+OWa8J1MWZ2LNDL3f8QYRzSRhRqmQqdkSzZFGVSsATrYqcOmVk74GbgB0k3ZDbdzCrNrLK2tjaDIYq0fjojWbIpyqRQA/SKW+4JbIlb7gwMApaYWTUwClicaLDZ3ee6e7m7l/fo0SPCkKW1UZmKgEq1SLZEmRSWA6VmVmJmBwCTgcX1T7r7Nnfv7u7F7l4MvASc5e6qYSExFRVBaYr68hT1j/MpKegcA8knkSUFd98FzASeBFYDD7v7SjO7xszOiup9RTItnZ26rnom+UYF8SRvVFRk/wih4YljEAzyptqnX1wcJIKG+vQJuoFEsiXVgnhKCiJNSHen3q5d4sqsZsH4gEi2qEqqSAake+KYzjGQfKOkINKEdHfqOsdA8o2SgkgT0t2p6xwDyTdKCpI1+TSNtF4mduo6x0DyiQaaJWt0OUyR3NFAs4iINJuSgqSsJd0/KlMhkl+UFCRlLalS2hrKVKjMhEjqlBRSoJ1K/lKZCZHmUVJIotB3Kpns/pk9O5ORpUaXshRpHs0+SkK1a/bKx9lDKjMhEtDsowzR9XHzm8pMiDRPQSWFlnR5aKeyVy66f9KlMhMizVNQSaEls2e0U9krV9NI0xnoV5kJkeYpqKTQEtqppC/XF6lRmQmR1LX5pJCJ2TPaqbRcujt1zR4Sya6Cmn2Uj7NnMikXVy7TRWpEWgfNPpL9tGRMJV26SI1IfimopJCPs2fynS5SI5JfIk0KZjbezN40s3VmdmWC52eY2WtmVmVmS81sQJTxFGIRtlx2sy4IAAAIfUlEQVQXpNNFakTyS2RjCmZWBKwBTgZqgOXAFHdfFdemi7tvDx+fBXzX3cc3tV1dT6HlcjWmsmBBMDD81lvBEcJ112mnLpJtqY4ptI8whhHAOnffEAb0IDARiCWF+oQQOhgo4GHgtmvqVCUBkXwRZffRUcDmuOWacN0+zOwSM1sP3ABcGmE8OZOpKqvpdvloTEVEkokyKViCdfsdCbj7be7eD7gC+EnCDZlNN7NKM6usra3NcJjRymSV1XRnDxXimIqINE+USaEG6BW33BPY0kT7B4GzEz3h7nPdvdzdy3v06JHBEJunJTvVtnDyla4nIVI4okwKy4FSMysxswOAycDi+AZmVhq3eAawNsJ40taSX+rpztPPxOyhXJeZEJH8EVlScPddwEzgSWA18LC7rzSza8KZRgAzzWylmVUB3wcuiCqeXEl3nn5paeIpnaWlids3pDITItIcBVXmoiUqKhIfIcyendqv9fqdcvyOtVOn1Ofap1smQmUmRARU5iJj0r3w/Nq1iX9pr02xoyzd7ieVmRCR5lBSiFi6SSXdnbLKTIhIcygpNEMu5vmnu1NWmQkRaQ4lhWbIxclj6e6UM7FT1/UkRAqHBppFRAqABppFRKTZlBRERCRGSUFERGKUFEREJEZJQUREYvJu9pGZ1QIJCje0Ct2BD3IdRBMUX3pae3zQ+mNUfOlJJ74+7p60zHTeJYXWzMwqU5nylSuKLz2tPT5o/TEqvvRkIz51H4mISIySgoiIxCgpZNbcXAeQhOJLT2uPD1p/jIovPZHHpzEFERGJ0ZGCiIjEKCk0k5n1MrPnzGx1eCnRWQnanGBm28ysKrz9LMsxVpvZa+F771c90AK3mNk6M3vVzIZlMbZ/iftcqsxsu5l9r0GbrH9+Znavmb1vZq/HrTvMzJ42s7Xh/aGNvPaCsM1aM8v4JWUbie1XZvZG+O/3mJl1a+S1TX4XIo6xwszejvt3PL2R1443szfD7+OVWYzvobjYqsPLAid6baSfYWP7lJx9/9xdt2bcgCOBYeHjzsAaYECDNicAf8hhjNVA9yaePx34b8CAUcBfcxRnEfAuwfzpnH5+wPHAMOD1uHU3AFeGj68Erk/wusOADeH9oeHjQ7MQ2ylA+/Dx9YliS+W7EHGMFcAPU/gOrAf6AgcArzT8/xRVfA2e/0/gZ7n4DBvbp+Tq+6cjhWZy93fc/eXw8SfAauCo3EbVbBOB+z3wEtDNzI7MQRwnAuvdPecnI7r7C8CHDVZPBO4LH98HnJ3gpacCT7v7h+7+EfA0MD7q2Nz9KXffFS6+BPTM5Hs2VyOfXypGAOvcfYO7fw48SPC5Z1RT8ZmZAV8HHsj0+6aiiX1KTr5/SgppMLNi4FjgrwmeHm1mr5jZf5vZwKwGBg48ZWYrzGx6guePAjbHLdeQm8Q2mcb/I+by86t3hLu/A8F/XODwBG1aw2d5EcGRXyLJvgtRmxl2cd3bSPdHa/j8vgK85+6NXTk9a59hg31KTr5/SgotZGaHAI8C33P37Q2efpmgS2QocCuwKMvhfdndhwGnAZeY2fENnrcEr8nqNDQzOwA4C1iY4Olcf37NkdPP0syuAnYBCxppkuy7EKU7gH5AGfAOQRdNQzn/LgJTaPooISufYZJ9SqMvS7Aurc9PSaEFzKwDwT/eAnf/fcPn3X27u+8IHz8BdDCz7tmKz923hPfvA48RHKLHqwF6xS33BLZkJ7qY04CX3f29hk/k+vOL8159t1p4/36CNjn7LMNBxTOBqR52MDeUwnchMu7+nrvvdvc9wN2NvHdOv4tm1h44F3iosTbZ+Awb2afk5PunpNBMYf/jr4HV7n5TI22+GLbDzEYQfM5bsxTfwWbWuf4xwYDk6w2aLQb+VzgLaRSwrf4wNYsa/XWWy8+vgcVA/WyOC4DHE7R5EjjFzA4Nu0dOCddFyszGA1cAZ7n7zkbapPJdiDLG+HGqcxp57+VAqZmVhEePkwk+92w5CXjD3WsSPZmNz7CJfUpuvn9Rjai31RswhuDw7FWgKrydDswAZoRtZgIrCWZSvAT8zyzG1zd831fCGK4K18fHZ8BtBLM+XgPKs/wZdiLYyXeNW5fTz48gQb0D1BH8+vom8AXgGWBteH9Y2LYcuCfutRcB68LbhVmKbR1BX3L9d/DOsO2XgCea+i5k8fP7Tfj9epVgB3dkwxjD5dMJZtysjyrGRPGF6+fXf+/i2mb1M2xin5KT75/OaBYRkRh1H4mISIySgoiIxCgpiIhIjJKCiIjEKCmIiEiMkoJIyMx2274VXDNWsdPMiuMrdIq0Vu1zHYBIK/Kpu5flOgiRXNKRgkgSYT39683sb+Htf4Tr+5jZM2HBt2fMrHe4/ggLrnHwSnj7n+Gmiszs7rBm/lNmdlDY/lIzWxVu58Ec/ZkigJKCSLyDGnQfnR/33HZ3HwHMAf5PuG4OQQnyIQQF6W4J198CPO9BQb9hBGfCApQCt7n7QOBjYFK4/krg2HA7M6L640RSoTOaRUJmtsPdD0mwvhr4qrtvCAuXvevuXzCzDwhKN9SF699x9+5mVgv0dPd/xm2jmKDufWm4fAXQwd1/bmZ/BHYQVINd5GExQJFc0JGCSGq8kceNtUnkn3GPd7N3TO8MglpUxwErwsqdIjmhpCCSmvPj7peFj/9CUNUTYCqwNHz8DPAdADMrMrMujW3UzNoBvdz9OeB/A92A/Y5WRLJFv0hE9jrI9r14+x/dvX5a6oFm9leCH1JTwnWXAvea2eVALXBhuH4WMNfMvklwRPAdggqdiRQBvzWzrgTVa292948z9heJNJPGFESSCMcUyt39g1zHIhI1dR+JiEiMjhRERCRGRwoiIhKjpCAiIjFKCiIiEqOkICIiMUoKIiISo6QgIiIx/x+OGo9WL9aroAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dpt_model_val_loss = dpt_model_hist.history['val_loss']\n",
    "\n",
    "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
    "plt.plot(epochs, dpt_model_val_loss, 'bo', label='Dropout-regularized model')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "여기에서도 기본 네트워크보다 확실히 향상되었습니다.\n",
    "\n",
    "정리하면 신경망에서 과대적합을 방지하기 위해 가장 널리 사용하는 방법은 다음과 같습니다.\n",
    "\n",
    "* 훈련 데이터를 더 모읍니다.\n",
    "* 네트워크의 용량을 감소시킵니다.\n",
    "* 가중치 규제를 추가합니다.\n",
    "* 드롭아웃을 추가합니다"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}