3.5-classifying-newswires.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.2.4'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 뉴스 기사 분류: 다중 분류 문제\n",
    "\n",
    "이 노트북은 [케라스 창시자에게 배우는 딥러닝](https://tensorflow.blog/케라스-창시자에게-배우는-딥러닝/) 책의 3장 5절의 코드 예제입니다. 책에는 더 많은 내용과 그림이 있습니다. 이 노트북에는 소스 코드에 관련된 설명만 포함합니다. 이 노트북의 설명은 케라스 버전 2.2.2에 맞추어져 있습니다. 케라스 최신 버전이 릴리스되면 노트북을 다시 테스트하기 때문에 설명과 코드의 결과가 조금 다를 수 있습니다.\n",
    "\n",
    "----\n",
    "\n",
    "이전 섹션에서 완전 연결된 신경망을 사용해 벡터 입력을 어떻게 두 개의 클래스로 분류하는지 보았습니다. 두 개 이상의 클래스가 있을 때는 어떻게 해야 할까요?\n",
    "\n",
    "이 절에서 로이터 뉴스를 46개의 상호 배타적인 토픽으로 분류하는 신경망을 만들어 보겠습니다. 클래스가 많기 때문에 이 문제는 다중 분류의 예입니다. 각 데이터 포인트가 정확히 하나의 범주로 분류되기 때문에 좀 더 정확히 말하면 단일 레이블 다중 분류 문제입니다. 각 데이터 포인트가 여러 개의 범주(가령, 토픽)에 속할 수 있다면 이런 문제는 다중 레이블 다중 분류의 문제가 됩니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 로이터 데이터셋\n",
    "\n",
    "1986년에 로이터에서 공개한 짧은 뉴스 기사와 토픽의 집합인 로이터 데이터셋을 사용하겠습니다. 이 데이터셋은 텍스트 분류를 위해 널리 사용되는 간단한 데이터셋입니다. 46개의 토픽이 있으며 어떤 토픽은 다른 것에 비해 데이터가 많습니다. 각 토픽은 훈련 세트에 최소한 10개의 샘플을 가지고 있습니다.\n",
    "\n",
    "IMDB와 MNIST와 마찬가지로 로이터 데이터셋은 케라스에 포함되어 있습니다. 한 번 살펴보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import reuters\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = reuters.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMDB 데이터셋에서처럼 num_words=10000 매개변수는 데이터에서 가장 자주 등장하는 단어 10,000개로 제한합니다.\n",
    "\n",
    "여기에는 8,982개의 훈련 샘플과 2,246개의 테스트 샘플이 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8982"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2246"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMDB 리뷰처럼 각 샘플은 정수 리스트입니다(단어 인덱스):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 245,\n",
       " 273,\n",
       " 207,\n",
       " 156,\n",
       " 53,\n",
       " 74,\n",
       " 160,\n",
       " 26,\n",
       " 14,\n",
       " 46,\n",
       " 296,\n",
       " 26,\n",
       " 39,\n",
       " 74,\n",
       " 2979,\n",
       " 3554,\n",
       " 14,\n",
       " 46,\n",
       " 4689,\n",
       " 4329,\n",
       " 86,\n",
       " 61,\n",
       " 3499,\n",
       " 4795,\n",
       " 14,\n",
       " 61,\n",
       " 451,\n",
       " 4329,\n",
       " 17,\n",
       " 12]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "궁금한 경우를 위해 어떻게 단어로 디코딩하는지 알아보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_index = reuters.get_word_index()\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "# 0, 1, 2는 '패딩', '문서 시작', '사전에 없음'을 위한 인덱스이므로 3을 뺍니다\n",
    "decoded_newswire = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'? ? ? said as a result of its december acquisition of space co it expects earnings per share in 1987 of 1 15 to 1 30 dlrs per share up from 70 cts in 1986 the company said pretax net should rise to nine to 10 mln dlrs from six mln dlrs in 1986 and rental operation revenues to 19 to 22 mln dlrs from 12 5 mln dlrs it said cash flow per share this year should be 2 50 to three dlrs reuter 3'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoded_newswire"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "샘플에 연결된 레이블은 토픽의 인덱스로 0과 45 사이의 정수입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 데이터 준비\n",
    "\n",
    "이전의 예제와 동일한 코드를 사용해서 데이터를 벡터로 변환합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.\n",
    "    return results\n",
    "\n",
    "# 훈련 데이터 벡터 변환\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# 테스트 데이터 벡터 변환\n",
    "x_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "레이블을 벡터로 바꾸는 방법은 두 가지입니다. 레이블의 리스트를 정수 텐서로 변환하는 것과 원-핫 인코딩을 사용하는 것입니다. 원-핫 인코딩이 범주형 데이터에 널리 사용되기 때문에 범주형 인코딩이라고도 부릅니다. 원-핫 인코딩에 대한 자세한 설명은 6.1절을 참고하세요. 이 경우 레이블의 원-핫 인코딩은 각 레이블의 인덱스 자리는 1이고 나머지는 모두 0인 벡터입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_one_hot(labels, dimension=46):\n",
    "    results = np.zeros((len(labels), dimension))\n",
    "    for i, label in enumerate(labels):\n",
    "        results[i, label] = 1.\n",
    "    return results\n",
    "\n",
    "# 훈련 레이블 벡터 변환\n",
    "one_hot_train_labels = to_one_hot(train_labels)\n",
    "# 테스트 레이블 벡터 변환\n",
    "one_hot_test_labels = to_one_hot(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MNIST 예제에서 이미 보았듯이 케라스에는 이를 위한 내장 함수가 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils.np_utils import to_categorical\n",
    "\n",
    "one_hot_train_labels = to_categorical(train_labels)\n",
    "one_hot_test_labels = to_categorical(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 모델 구성\n",
    "\n",
    "이 토픽 분류 문제는 이전의 영화 리뷰 분류 문제와 비슷해 보입니다. 두 경우 모두 짧은 텍스트를 분류하는 것이죠. 여기에서는 새로운 제약 사항이 추가되었습니다. 출력 클래스의 개수가 2에서 46개로 늘어난 점입니다. 출력 공간의 차원이 훨씬 커졌습니다.\n",
    "\n",
    "이전에 사용했던 것처럼 `Dense` 층을 쌓으면 각 층은 이전 층의 출력에서 제공한 정보만 사용할 수 있습니다. 한 층이 분류 문제에 필요한 일부 정보를 누락하면 그 다음 층에서 이를 복원할 방법이 없습니다. 각 층은 잠재적으로 정보의 병목이 될 수 있습니다. 이전 예제에서 16차원을 가진 중간층을 사용했지만 16차원 공간은 46개의 클래스를 구분하기에 너무 제약이 많을 것 같습니다. 이렇게 규모가 작은 층은 유용한 정보를 완전히 잃게 되는 정보의 병목 지점처럼 동작할 수 있습니다.\n",
    "\n",
    "이런 이유로 좀 더 규모가 큰 층을 사용하겠습니다. 64개의 유닛을 사용해 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 구조에서 주목해야 할 점이 두 가지 있습니다:\n",
    "\n",
    "* 마지막 `Dense` 층의 크기가 46입니다. 각 입력 샘플에 대해서 46차원의 벡터를 출력한다는 뜻입니다. 이 벡터의 각 원소(각 차원)은 각기 다른 출력 클래스가 인코딩된 것입니다.\n",
    "* 마지막 층에 `softmax` 활성화 함수가 사용되었습니다. MNIST 예제에서 이런 방식을 보았습니다. 각 입력 샘플마다 46개의 출력 클래스에 대한 확률 분포를 출력합니다. 즉, 46차원의 출력 벡터를 만들며 `output[i]`는 어떤 샘플이 클래스 `i`에 속할 확률입니다. 46개의 값을 모두 더하면 1이 됩니다.\n",
    "\n",
    "이런 문제에 사용할 최선의 손실 함수는 `categorical_crossentropy`입니다. 이 함수는 두 확률 분포의 사이의 거리를 측정합니다. 여기에서는 네트워크가 출력한 확률 분포와 진짜 레이블의 분포 사이의 거리입니다. 두 분포 사이의 거리를 최소화하면 진짜 레이블에 가능한 가까운 출력을 내도록 모델을 훈련하게 됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 훈련 검증\n",
    "\n",
    "훈련 데이터에서 1,000개의 샘플을 따로 떼어서 검증 세트로 사용하겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = x_train[:1000]\n",
    "partial_x_train = x_train[1000:]\n",
    "\n",
    "y_val = one_hot_train_labels[:1000]\n",
    "partial_y_train = one_hot_train_labels[1000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 20번의 에포크로 모델을 훈련시킵니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/20\n",
      "7982/7982 [==============================] - 1s 135us/step - loss: 2.5322 - acc: 0.4955 - val_loss: 1.7208 - val_acc: 0.6120\n",
      "Epoch 2/20\n",
      "7982/7982 [==============================] - 0s 48us/step - loss: 1.4452 - acc: 0.6879 - val_loss: 1.3459 - val_acc: 0.7060\n",
      "Epoch 3/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 1.0953 - acc: 0.7651 - val_loss: 1.1708 - val_acc: 0.7430\n",
      "Epoch 4/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.8697 - acc: 0.8165 - val_loss: 1.0793 - val_acc: 0.7590\n",
      "Epoch 5/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.7034 - acc: 0.8472 - val_loss: 0.9844 - val_acc: 0.7810\n",
      "Epoch 6/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.5667 - acc: 0.8802 - val_loss: 0.9411 - val_acc: 0.8040\n",
      "Epoch 7/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.4581 - acc: 0.9048 - val_loss: 0.9083 - val_acc: 0.8020\n",
      "Epoch 8/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.3695 - acc: 0.9231 - val_loss: 0.9363 - val_acc: 0.7890\n",
      "Epoch 9/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.3032 - acc: 0.9315 - val_loss: 0.8917 - val_acc: 0.8090\n",
      "Epoch 10/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.2537 - acc: 0.9414 - val_loss: 0.9071 - val_acc: 0.8110\n",
      "Epoch 11/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.2187 - acc: 0.9471 - val_loss: 0.9177 - val_acc: 0.8130\n",
      "Epoch 12/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.1873 - acc: 0.9508 - val_loss: 0.9027 - val_acc: 0.8130\n",
      "Epoch 13/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.1703 - acc: 0.9521 - val_loss: 0.9323 - val_acc: 0.8110\n",
      "Epoch 14/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.1536 - acc: 0.9554 - val_loss: 0.9689 - val_acc: 0.8050\n",
      "Epoch 15/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.1390 - acc: 0.9560 - val_loss: 0.9686 - val_acc: 0.8150\n",
      "Epoch 16/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.1313 - acc: 0.9560 - val_loss: 1.0220 - val_acc: 0.8060\n",
      "Epoch 17/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.1217 - acc: 0.9579 - val_loss: 1.0254 - val_acc: 0.7970\n",
      "Epoch 18/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.1198 - acc: 0.9582 - val_loss: 1.0430 - val_acc: 0.8060\n",
      "Epoch 19/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.1138 - acc: 0.9597 - val_loss: 1.0955 - val_acc: 0.7970\n",
      "Epoch 20/20\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.1111 - acc: 0.9593 - val_loss: 1.0674 - val_acc: 0.8020\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=20,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "손실과 정확도 곡선을 그려 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8VOXZ//HPBUQQWQWsCErArQIGiCmiqOD6c9+rIq7VorZWrU/7k8e9Kq27FuVni1VqS9wefVxqXYqVFlc0ICCIFFTQCEJA2QSVwPX74z4zGcIkmZCczCT5vl+v85qZc+5z5pqTybnm3Pd97mPujoiICECLbAcgIiK5Q0lBRESSlBRERCRJSUFERJKUFEREJElJQUREkpQUpF6ZWUszW2tmu9Rn2Wwys93MrN77bpvZYWa2MOX1PDM7MJOyW/FefzKzq7d2/Wq2e4uZ/bm+tyvZ0yrbAUh2mdnalJdtge+AjdHri9y9uDbbc/eNQLv6LtscuPue9bEdM7sQOMvdh6ds+8L62LY0fUoKzZy7Jw/K0S/RC9391arKm1krdy9viNhEpOGp+kiqFVUPPGFmj5nZGuAsM9vPzN4xs5VmtsTMxppZXlS+lZm5meVHrydGy18yszVm9raZ9a5t2Wj5UWb2HzNbZWb3mdmbZnZeFXFnEuNFZrbAzL42s7Ep67Y0s3vMbIWZfQwcWc3+udbMHq80b5yZ3R09v9DM5kaf5+PoV3xV2yo1s+HR87Zm9tcotjnAPmne95Nou3PM7Pho/t7A/cCBUdXc8pR9e2PK+hdHn32FmT1rZt0z2Tc1MbMTo3hWmtlrZrZnyrKrzWyxma02s49SPusQM5sezV9qZndk+n4SA3fXpAl3B1gIHFZp3i3A98BxhB8R2wI/AvYlnGn2Af4DXBqVbwU4kB+9nggsB4qAPOAJYOJWlN0BWAOcEC27EtgAnFfFZ8kkxueAjkA+8FXiswOXAnOAnkAXYEr4V0n7Pn2AtcB2KdteBhRFr4+LyhhwCLAeKIiWHQYsTNlWKTA8en4n8C+gM9AL+LBS2dOA7tHf5Mwohh9Eyy4E/lUpzonAjdHzI6IYBwJtgP8HvJbJvknz+W8B/hw93yuK45Dob3R1tN/zgH7AImDHqGxvoE/0/D1gRPS8PbBvtv8XmvOkMwXJxBvu/jd33+Tu6939PXef6u7l7v4JMB4YVs36T7l7ibtvAIoJB6Palj0WmOHuz0XL7iEkkLQyjPF37r7K3RcSDsCJ9zoNuMfdS919BXBrNe/zCTCbkKwADgdWuntJtPxv7v6JB68B/wTSNiZXchpwi7t/7e6LCL/+U9/3SXdfEv1NHiUk9KIMtgswEviTu89w92+B0cAwM+uZUqaqfVOdM4Dn3f216G90K9CBkJzLCQmoX1QF+Wm07yAk993NrIu7r3H3qRl+DomBkoJk4vPUF2b2QzP7u5l9aWargZuArtWs/2XK83VU37hcVdmdUuNwdyf8sk4rwxgzei/CL9zqPAqMiJ6fSUhmiTiONbOpZvaVma0k/Eqvbl8ldK8uBjM7z8xmRtU0K4EfZrhdCJ8vuT13Xw18DfRIKVObv1lV291E+Bv1cPd5wH8R/g7LourIHaOi5wN9gXlm9q6ZHZ3h55AYKClIJip3x/wj4dfxbu7eAbieUD0SpyWE6hwAzMzY/CBWWV1iXALsnPK6pi6zTwCHRb+0TyAkCcxsW+Ap4HeEqp1OwD8yjOPLqmIwsz7AA8AlQJdoux+lbLem7rOLCVVSie21J1RTfZFBXLXZbgvC3+wLAHef6O5DCVVHLQn7BXef5+5nEKoI7wKeNrM2dYxFtpKSgmyN9sAq4Bsz2wu4qAHe8wWg0MyOM7NWwOVAt5hifBK4wsx6mFkX4KrqCrv7UuANYAIwz93nR4taA9sAZcBGMzsWOLQWMVxtZp0sXMdxacqydoQDfxkhP15IOFNIWAr0TDSsp/EYcIGZFZhZa8LB+XV3r/LMqxYxH29mw6P3/jWhHWiqme1lZgdH77c+mjYSPsDZZtY1OrNYFX22TXWMRbaSkoJsjf8CziX8w/+R8Es5VtGB93TgbmAFsCvwPuG6ivqO8QFC3f8HhEbQpzJY51FCw/GjKTGvBH4JPENorD2VkNwycQPhjGUh8BLwl5TtzgLGAu9GZX4IpNbDTwLmA0vNLLUaKLH+y4RqnGei9XchtDPUibvPIezzBwgJ60jg+Kh9oTVwO6Ed6EvCmcm10apHA3Mt9G67Ezjd3b+vazyydSxUzYo0LmbWklBdcaq7v57teESaCp0pSKNhZkeaWceoCuI6Qo+Wd7MclkiToqQgjckBwCeEKogjgRPdvarqIxHZCqo+EhGRJJ0piIhIUqMbEK9r166en5+f7TBERBqVadOmLXf36rpxA40wKeTn51NSUpLtMEREGhUzq+nKfEDVRyIikkJJQUREkpQUREQkqdG1KYhIw9qwYQOlpaV8++232Q5FMtCmTRt69uxJXl5VQ19VT0lBRKpVWlpK+/btyc/PJwxOK7nK3VmxYgWlpaX07t275hXSaBbVR8XFkJ8PLVqEx+Ja3YpepHn79ttv6dKlixJCI2BmdOnSpU5ndU3+TKG4GEaNgnXrwutFi8JrgJF1HhdSpHlQQmg86vq3avJnCtdcU5EQEtatC/NFRGRzsSUFM9vZzCab2Vwzm2Nml6cpM9zMVpnZjGi6vr7j+Oyz2s0XkdyyYsUKBg4cyMCBA9lxxx3p0aNH8vX332d224Xzzz+fefPmVVtm3LhxFNdT3fIBBxzAjBkz6mVbDS3O6qNy4L/cfXp0u79pZjbJ3T+sVO51dz82riB22SVUGaWbLyL1r7g4nIl/9ln4Pxszpm5VtV26dEkeYG+88UbatWvHr371q83KuDvuTosW6X/nTpgwocb3+fnPf771QTYhsZ0puPsSd58ePV8DzKX6e+rGYswYaNt283lt24b5IlK/Em14ixaBe0UbXhydOxYsWED//v25+OKLKSwsZMmSJYwaNYqioiL69evHTTfdlCyb+OVeXl5Op06dGD16NAMGDGC//fZj2bJlAFx77bXce++9yfKjR49m8ODB7Lnnnrz11lsAfPPNN5xyyikMGDCAESNGUFRUVOMZwcSJE9l7773p378/V199NQDl5eWcffbZyfljx44F4J577qFv374MGDCAs846q973WSYapE3BzPKBQWx+y8CE/cxsppm9ZGb9qlh/lJmVmFlJWVlZrd575EgYPx569QKz8Dh+vBqZReLQ0G14H374IRdccAHvv/8+PXr04NZbb6WkpISZM2cyadIkPvywcsUErFq1imHDhjFz5kz2228/Hn744bTbdnfeffdd7rjjjmSCue+++9hxxx2ZOXMmo0eP5v333682vtLSUq699lomT57M+++/z5tvvskLL7zAtGnTWL58OR988AGzZ8/mnHPOAeD2229nxowZzJw5k/vvv7+Oe2frxJ4UzKwd8DRwhbuvrrR4OtDL3QcA9wHPptuGu4939yJ3L+rWrcZB/rYwciQsXAibNoVHJQSReDR0G96uu+7Kj370o+Trxx57jMLCQgoLC5k7d27apLDtttty1FFHAbDPPvuwcOHCtNs++eSTtyjzxhtvcMYZZwAwYMAA+vVL+zs2aerUqRxyyCF07dqVvLw8zjzzTKZMmcJuu+3GvHnzuPzyy3nllVfo2LEjAP369eOss86iuLh4qy8+q6tYk4KZ5RESQrG7/2/l5e6+2t3XRs9fBPLMrGucMYlIfKpqq4urDW+77bZLPp8/fz6///3vee2115g1axZHHnlk2v7622yzTfJ5y5YtKS8vT7vt1q1bb1Gmtjclq6p8ly5dmDVrFgcccABjx47loosuAuCVV17h4osv5t1336WoqIiNGzfW6v3qQ5y9jwx4CJjr7ndXUWbHqBxmNjiKZ0VcMYlIvLLZhrd69Wrat29Phw4dWLJkCa+88kq9v8cBBxzAk08+CcAHH3yQ9kwk1ZAhQ5g8eTIrVqygvLycxx9/nGHDhlFWVoa78+Mf/5jf/OY3TJ8+nY0bN1JaWsohhxzCHXfcQVlZGesq18U1gDh7Hw0FzgY+MLNES8zVwC4A7v4H4FTgEjMrB9YDZ7juDyrSaCWqZuuz91GmCgsL6du3L/3796dPnz4MHTq03t/jF7/4Beeccw4FBQUUFhbSv3//ZNVPOj179uSmm25i+PDhuDvHHXccxxxzDNOnT+eCCy7A3TEzbrvtNsrLyznzzDNZs2YNmzZt4qqrrqJ9+/b1/hlq0uju0VxUVOS6yY5Iw5k7dy577bVXtsPICeXl5ZSXl9OmTRvmz5/PEUccwfz582nVKrcGh0j3NzOzae5eVNO6ufVJRERy2Nq1azn00EMpLy/H3fnjH/+YcwmhrprWpxERiVGnTp2YNm1atsOIVZMf+0hERDKnpCAiIklKCiIikqSkICIiSUoKIpLThg8fvsWFaPfeey8/+9nPql2vXbt2ACxevJhTTz21ym3X1MX93nvv3ewisqOPPpqVK1dmEnq1brzxRu688846b6e+KSmISE4bMWIEjz/++GbzHn/8cUaMGJHR+jvttBNPPfXUVr9/5aTw4osv0qlTp63eXq5TUhCRnHbqqafywgsv8N133wGwcOFCFi9ezAEHHJC8bqCwsJC9996b5557bov1Fy5cSP/+/QFYv349Z5xxBgUFBZx++umsX78+We6SSy5JDrt9ww03ADB27FgWL17MwQcfzMEHHwxAfn4+y5cvB+Duu++mf//+9O/fPzns9sKFC9lrr7346U9/Sr9+/TjiiCM2e590ZsyYwZAhQygoKOCkk07i66+/Tr5/3759KSgoSA7E9+9//zt5k6FBgwaxZs2ard636eg6BRHJ2BVXQH3fUGzgQIiOp2l16dKFwYMH8/LLL3PCCSfw+OOPc/rpp2NmtGnThmeeeYYOHTqwfPlyhgwZwvHHH1/lfYofeOAB2rZty6xZs5g1axaFhYXJZWPGjGH77bdn48aNHHroocyaNYvLLruMu+++m8mTJ9O16+ZjdU6bNo0JEyYwdepU3J19992XYcOG0blzZ+bPn89jjz3Ggw8+yGmnncbTTz9d7f0RzjnnHO677z6GDRvG9ddfz29+8xvuvfdebr31Vj799FNat26drLK68847GTduHEOHDmXt2rW0adOmFnu7ZjpTEJGcl1qFlFp15O5cffXVFBQUcNhhh/HFF1+wdOnSKrczZcqU5MG5oKCAgoKC5LInn3ySwsJCBg0axJw5c2oc7O6NN97gpJNOYrvttqNdu3acfPLJvP766wD07t2bgQMHAtUPzw3h/g4rV65k2LBhAJx77rlMmTIlGePIkSOZOHFi8srpoUOHcuWVVzJ27FhWrlxZ71dU60xBRDJW3S/6OJ144olceeWVTJ8+nfXr1yd/4RcXF1NWVsa0adPIy8sjPz8/7XDZqdKdRXz66afceeedvPfee3Tu3Jnzzjuvxu1UN25cYthtCENv11R9VJW///3vTJkyheeff56bb76ZOXPmMHr0aI455hhefPFFhgwZwquvvsoPf/jDrdp+OjpTEJGc165dO4YPH85PfvKTzRqYV61axQ477EBeXh6TJ09mUbobsqc46KCDKI7uDTp79mxmzZoFhGG3t9tuOzp27MjSpUt56aWXkuu0b98+bb39QQcdxLPPPsu6dev45ptveOaZZzjwwANr/dk6duxI586dk2cZf/3rXxk2bBibNm3i888/5+CDD+b2229n5cqVrF27lo8//pi9996bq666iqKiIj766KNav2d1dKYgIo3CiBEjOPnkkzfriTRy5EiOO+44ioqKGDhwYI2/mC+55BLOP/98CgoKGDhwIIMHDwbCXdQGDRpEv379thh2e9SoURx11FF0796dyZMnJ+cXFhZy3nnnJbdx4YUXMmjQoGqriqryyCOPcPHFF7Nu3Tr69OnDhAkT2LhxI2eddRarVq3C3fnlL39Jp06duO6665g8eTItW7akb9++ybvI1RcNnS0i1dLQ2Y1PXYbOVvWRiIgkKSmIiEiSkoKI1KixVTM3Z3X9WykpiEi12rRpw4oVK5QYGgF3Z8WKFXW6oE29j0SkWj179qS0tJSysrJshyIZaNOmDT179tzq9ZUURKRaeXl59O7dO9thSANR9ZGIiCQpKYiISJKSgoiIJCkpiIhIkpKCiIgkKSmIiEiSkoKIiCQpKYiISJKSgoiIJCkpiIhIkpKCiIgkxZYUzGxnM5tsZnPNbI6ZXZ6mjJnZWDNbYGazzKwwrnhERKRmcQ6IVw78l7tPN7P2wDQzm+TuH6aUOQrYPZr2BR6IHkVEJAtiO1Nw9yXuPj16vgaYC/SoVOwE4C8evAN0MrPuccUkIiLVa5A2BTPLBwYBUyst6gF8nvK6lC0TB2Y2ysxKzKxEY7qLiMQn9qRgZu2Ap4Er3H115cVpVtni9k7uPt7di9y9qFu3bnGEKSIixJwUzCyPkBCK3f1/0xQpBXZOed0TWBxnTCIiUrU4ex8Z8BAw193vrqLY88A5US+kIcAqd18SV0wiIlK9OHsfDQXOBj4wsxnRvKuBXQDc/Q/Ai8DRwAJgHXB+jPGIiEgNYksK7v4G6dsMUss48PO4YhARkdrRFc0iIpKkpCAiIklKCiIikqSkICIiSUoKIiKSpKQgIiJJSgoiIpKkpCAiIklKCiIikqSkICIiSUoKIiKSpKQgIiJJSgoiIpKkpCAiIklKCiIiktRsksKGDfDMM+Bb3AFaREQSmk1SeOQROPlkePXVbEciIpK7mk1SOPts2GUXuPpqnS2IiFSl2SSF1q3hxhuhpASefTbb0YiI5KZmkxQgnC3suSdcey1s3JjtaEREck+zSgqtWsHNN8OHH8Kjj2Y7GhGR3NOskgLAKafAoEGhKun777MdjYhIbml2SaFFCxgzBj75BB5+ONvRiIjklmaXFACOPBIOOABuugnWr892NCIiuaNZJgUz+O1vYckSGDcu29GIiOSOZpkUAA48MJwx/O53sHp1tqMREckNzTYpANxyC3z1Fdx9d7YjERHJDc06KeyzD5x6Ktx1Fyxfnu1oRESyr1knBQiNzevWwa23ZjsSEZHsa/ZJYa+94Jxz4P77obQ029GIiGRXs08KADfcAJs2hTYGEZHmLLakYGYPm9kyM5tdxfLhZrbKzGZE0/VxxVKT/Hy46CJ46CFYsCBbUYiIZF+cZwp/Bo6soczr7j4wmm6KMZYaXXMN5OWF4S9ERJqr2JKCu08Bvopr+/Vtxx3h8svDQHkffJDtaEREsiPbbQr7mdlMM3vJzPplORZ+/Wvo0AGuuy7bkYiIZEc2k8J0oJe7DwDuA6q89Y2ZjTKzEjMrKSsriy2g7bcPieG552Dq1NjeRkQkZ2UtKbj7andfGz1/Ecgzs65VlB3v7kXuXtStW7dY47r8cujWLbQxJBQXh8boFi3CY3FxrCGIiGRNq2y9sZntCCx1dzezwYQEtSJb8SS0axcSwhVXwD//CV9+CaNGhQvcABYtCq8BRo7MXpwiInEwj+ku9mb2GDAc6AosBW4A8gDc/Q9mdilwCVAOrAeudPe3atpuUVGRl5SUxBJzwrffwh57wE47hZFUP/tsyzK9esHChbGGISJSb8xsmrsX1VgurqQQl4ZIChCuWbjwwqqXm4UL3kREGoNMk0K2ex/lrHPPhd13D9cupLPLLg0bj4hIQ8goKZjZrmbWOno+3MwuM7NO8YaWXa1awc03w4YNsM02my9r2zbc0lNEpKnJ9EzhaWCjme0GPAT0Bh6NLaoc8eMfw4AB0KlTODMwC20J48erkVlEmqZMk8Imdy8HTgLudfdfAt3jCys3tGgRBslbtiz0SNq0KTQuKyGISFOVaVLYYGYjgHOBF6J5VdS2Ny3HHAP77Rfuu7B+fbajERGJV6ZJ4XxgP2CMu39qZr2BifGFlTvM4Le/hS++gAceyHY0IiLxqnWXVDPrDOzs7rPiCal6DdUltbIjjoDp02H27DB4nohIY1KvXVLN7F9m1sHMtgdmAhPMrFnd7v7220P10f77w3/+k+1oRETikWn1UUd3Xw2cDExw932Aw+ILK/cMHAiTJ8OaNTB0qAbME5GmKdOk0MrMugOnUdHQ3OwMHgxvvx2G1z74YHih2e4JEWmqMk0KNwGvAB+7+3tm1geYH19YuWu33eCtt6BfPzjxRPjTn7IdkYhI/ckoKbj7/7h7gbtfEr3+xN1PiTe03PWDH4SqpMMOg5/+NHRXbWRDSImIpJVpQ3NPM3vGzJaZ2VIze9rMesYdXC5r1w7+9rcwRtINN8BFF0F5ebajEhGpm0yrjyYAzwM7AT2Av0XzmrW8PJgwAa6+Gh58EE4+ueK+CyIijVGmSaGbu09w9/Jo+jMQ7y3QGgmzMDjeuHGh4fmQQ2D58mxHJSKydTJNCsvN7CwzaxlNZ5EDd0nLJT/7GTz9NMycGbqsfvpptiMSEam9TJPCTwjdUb8ElgCnEoa+kBQnnQSvvgplZeEit/ffz3ZEIiK1k2nvo8/c/Xh37+buO7j7iYQL2aSSoUPhzTfDPRgOOggmTcp2RCIimavLndeurLcompi99grXMvTpA0cfDRObxdCBItIU1CUpWL1F0QT16AFTpsCBB8LZZ4exk3Qtg4jkurokBR3iatCxI7z0EpxxBlx1FVx2me7JICK5rdqkYGZrzGx1mmkN4ZoFqUHr1lBcDFdeCfffD/n5oQvr119nOzIRkS1VmxTcvb27d0gztXf3Vg0VZGPXogXcdRf861+wzz5w7bXhns+/+lW4eY+ISK6oS/WR1NKwYfDiizBjBhx/PNxzD/TuDRdcAB99lO3oRESUFLJiwIBQpbRgAYwaBY8+Cn37hmEydJ8GEckmJYUGUFwc2hJatAiPxcVhfu/eoZ1h0SK45ppQvTRkCAwfDi+/rN5KItLwlBRiVlwczgYWLQoH+UWLwutEYgDYYQe4+eaw7K67whnEUUfBoEHw2GMafVWksXOHefPgs89g06ZsR1M980b2c7SoqMhLSkqyHUbG8vPDwb6yXr1g4cL063z/fahSuu220NbQu3dolD7/fNh22zijFZH6snhxGPZm0qTw+OWXYX6bNrDrrrD77uGmXbvvXjHttFOoUYiDmU1z96IayykpxKtFi/TVQGY1/2LYtCncs+HWW+Gdd6BbN/jlL+HnPw+3BBWR3LFmDfz73xWJ4MMPw/xu3eDQQ8O0aRPMn18xffwxfPddxTa23bYiYVROGjvtFI4bW0tJIUdszZlCZe7w+uvw29/CK69Ap07hQrjLL4ftt6/PaEUkU+Xl8N57IQFMmhR+uJWXhzOBgw6Cww8Pd2csKKj61/+mTVBaunmiWLCgImF8/31F2bZtYfRouO66rYtXSSFHJNoUUm++07YtjB8PI0fWfnslJeHit2efDXd/+9nPwoVxP/hB/cUsIltyh//8p6I6aPJkWL06/HrfZ5+QAA4/PIyQ3KZN3d9v40b4/PPNE8WwYXDCCVu3PSWFHFJcHHoXffZZuGhtzJitSwipZs8OZw5PPBFGZP3pT+HXv4add66fmEWaiw0bwnD3S5fCsmXhsfK0bFloIygrC+v07l1xJnDIIdClS3Y/QyaynhTM7GHgWGCZu/dPs9yA3wNHA+uA89x9ek3bbYxJIU7z54c2h7/8JfxiOe+8cIrZp0+2IxOpP5s2hU4XK1eGg3h5+ZaP6ealLvv+e1ixYssD/4oqbhfWpk04A0+diopCMmiM/1+5kBQOAtYCf6kiKRwN/IKQFPYFfu/u+9a0XSWF9BYtCiOxPvRQ+Ac480z47/8Ow3iLNEZLloSqmn/8I1TXLF1a9222b7/5QX6HHbY88Cemdu3q1rCba7KeFKIg8oEXqkgKfwT+5e6PRa/nAcPdfUl121RSqN6SJeFahwceCCOynnJKqLoaODDbkYlUb9260KEikQg++CDM79atoqpmp50gLw9atdryMd28dGWaq0yTQjZ3UQ/g85TXpdG8LZKCmY0CRgHssssuDRJcY9W9O9x5Z6hCuvdeuO8+eOopOPbYcOYwZEh8/aBFamPTJpg1qyIJvP566J65zTbhPiS33QZHHFF97x2pf9lMCulOzNKetrj7eGA8hDOFOINqKrp2hVtuCRe9jRsXBt8bOhQ6dw6JYcgQ2G8/GDw43PdBpCGkVglNmhTq9wH69w/X3xx+eOjO2bZtduNszrKZFEqB1L4yPYHFWYqlyerUKVQfXX55OGN4883QnzoxtpJZGIwvkSSGDAntEPpl1ry5h942M2aEaebM0OMtcZOo1Lr2xPPKj5Xnff99xbU5O+wQEsARR1RUC0luyGabwjHApVQ0NI9198E1bVNtCvVj1apw4c3bb4ck8c478NVXYVmHDrDvvhVJYt99dZFcU7ZhQ+jZM3NmRRKYMWPzXjm77hqqcdq3D68Th42qHtPNMwvjeR1+uKqEsiHrDc1m9hgwHOgKLAVuAPIA3P0PUZfU+4EjCV1Sz3f3Go/2SgrxcA/dWxNJ4u23Q0NfYiiOPfcMyaGgAPr1C1PPng3XO+Orr8INiXbdVVULdbFqVcXBP/E4e3bFlbNt2oSqnIEDK6a999awKk1B1pNCXJQUGs7ateFsIpEk3n13826B7duHqqdEkkg839pksWJFxZWblR8Tty9t1Sr82tx//9BGsv/+0KNH/XzepiIxdMK8eeEK3MTjRx9tPuRKt26bH/wHDoQ99mjePXSaMiUFicXy5WGgrzlzwpR4nmgwhPCrsm/fzRNGv37h4P3VV1se8BPPU+9bbRau/k4dFKx793D28tZbIUEl6rd32WXzJFFQEP+BzT10oVy9OgyEtnp1zdM334R906VL6AhQ+THxPC8vsxhWrtzywD9vXtifiX0Dob/9HnuEs729965IADvu2LT64Uv1lBSkQS1fvnmSSEyJYQEgdDVMHeAr3YE/8dinD7RuXfX7bdgQqj/efDMkiTffrLjfddu2oaorkSiGDAm9rqqybl2Iv6qprCw8rlgRql8SB/lMxsXPywu9uzp0gO22Cwlk+fJwFlaV1MSRmjQ6dqxPqAXtAAAMlElEQVQ4A5g3b/N927JlGHphzz3DlEgCe+wRkqkO/qKk0ITEMXZSQykrq0gUn3wSepkkDvy9e9fPwGEJn39ekSTeeivUl2/cGJb17RuGKCgv3/Jgn/qrOpVZxQG5W7fwPHGAz2Rq377qxPbddyHJJGLJ5PnateFK29QDfuKxT5+QdEWqoqTQRNT3KKvNyTffhDaRRKJ4//2w7xK/wBNTt25bzuvaNXTnbdky25+iQnm56vtl6ykpNBH1cT8GEZFMk4J6Cue4zz6r3XwRkbpQUshxVQ31pCGgRCQOSgo5bsyYLS/Wats2zBcRqW9KCjlu5MjQqNyrV+gN06uXGplFJD7qy9AIjBypJCAiDUNnCiIikqSkICIiSUoKIiKSpKQgIiJJSgoiIpKkpCAiIklKCs1AcXEYQ6lFi/BYXJztiEQkV+k6hSau8iirixaF16BrH0RkSzpTaOKuuWbzYbchvL7mmuzEIyK5TUmhidMoqyJSG0oKTZxGWRWR2lBSaOI0yqqI1IaSQhOnUVZFpDbU+6gZ0CirIpIpnSmIiEiSkoKIiCQpKYiISJKSgmREQ2WINA9qaJYaaagMkeZDZwpSIw2VIdJ8KClIjTRUhkjzoaQgNdJQGSLNR6xJwcyONLN5ZrbAzEanWX6emZWZ2YxoujDOeGTraKgMkeYjtqRgZi2BccBRQF9ghJn1TVP0CXcfGE1/iise2XoaKkOk+Yiz99FgYIG7fwJgZo8DJwAfxvieEhMNlSHSPMRZfdQD+DzldWk0r7JTzGyWmT1lZjun25CZjTKzEjMrKSsriyNWEREh3qRgaeZ5pdd/A/LdvQB4FXgk3Ybcfby7F7l7Ubdu3eo5TGkIuvhNpHGIMymUAqm//HsCi1MLuPsKd/8uevkgsE+M8UiWJC5+W7QI3CsuflNiEMk9cSaF94Ddzay3mW0DnAE8n1rAzLqnvDwemBtjPJIluvhNpPGIraHZ3cvN7FLgFaAl8LC7zzGzm4ASd38euMzMjgfKga+A8+KKR7JHF7+JNB7mXrmaP7cVFRV5SUlJtsOQWsjPD1VGlfXqBQsXNnQ0Is2TmU1z96KayumKZomdLn4TaTyUFCR2uvhNpPFQUpAGMXJkqCratCk81jYhqEurSMPQ/RQk5+l+DiINR2cKkvPUpVWk4SgpSM5Tl1aRhqOkIDlP93MQaThKCpLz6qNLqxqqRTKjpCA5r65dWjX2kkjmdEWzNHm6olpEVzSLJKmhWiRzSgrS5NVHQ7XaJKS5UFKQJq+uDdVqk5DmRElBmry6NlTr4jlpTpQUpFmoy9hL9dEmoeonaSyUFERqUNc2CVU/SWOipCBSg7q2Saj6SRoTJQWRGtS1TULVT9KYKCmIZKAubRK5UP2kpCKZUlIQiVm2q5+UVKQ2lBREYpbt6qdcSCrSeCgpiDSAbFY/ZTupQN3PNHSm0nCUFERyXF2rn7KdVOp6ppEL1V/NKim5e6Oa9tlnHxdpbiZOdO/Vy90sPE6cWLt127Z1D4fUMLVtm/k2evXafN3E1KtX41i/rp+/rusntrG1f7/6WN/dHSjxDI6xWT/I13ZSUhCpvWwmFbP0B3WzhllfSSnINCnofgoiUqPi4tCG8NlnodppzJjM20Xqej+Luq7fokU4lFZmFtp44l4/258/QfdTEJF6U5eG8rq2iWS7TSXbbTINfT8QJQURiVVdu+TWdf3mnpRqLZM6plya1KYgIrWVzYZetSnETG0KItLY1KVNpj7Wh8zbFJQURESaATU0i4hIrcWaFMzsSDObZ2YLzGx0muWtzeyJaPlUM8uPMx4REalebEnBzFoC44CjgL7ACDPrW6nYBcDX7r4bcA9wW1zxiIhIzeI8UxgMLHD3T9z9e+Bx4IRKZU4AHomePwUcamYWY0wiIlKNOJNCD+DzlNel0by0Zdy9HFgFdKm8ITMbZWYlZlZSVlYWU7giItIqxm2n+8VfuatTJmVw9/HAeAAzKzOzNBd954SuwPJsB1GNXI8Pcj9GxVc3iq9u6hJfr0wKxZkUSoGdU173BBZXUabUzFoBHYGvqtuou3erzyDrk5mVZNLlK1tyPT7I/RgVX90ovrppiPjirD56D9jdzHqb2TbAGcDzlco8D5wbPT8VeM0b24UTIiJNSGxnCu5ebmaXAq8ALYGH3X2Omd1EuNz6eeAh4K9mtoBwhnBGXPGIiEjN4qw+wt1fBF6sNO/6lOffAj+OM4YGNj7bAdQg1+OD3I9R8dWN4qub2ONrdMNciIhIfDTMhYiIJCkpiIhIkpJCLZnZzmY22czmmtkcM7s8TZnhZrbKzGZE0/XpthVjjAvN7IPovbcYUtaCsdGYU7PMrLABY9szZb/MMLPVZnZFpTINvv/M7GEzW2Zms1PmbW9mk8xsfvTYuYp1z43KzDezc9OViSm+O8zso+hv+IyZdapi3Wq/DzHGd6OZfZHydzy6inWrHSMtxvieSIltoZnNqGLdWPdfVceUrH3/MrnpgqaKCegOFEbP2wP/AfpWKjMceCGLMS4Eulaz/GjgJcLFg0OAqVmKsyXwJdAr2/sPOAgoBGanzLsdGB09Hw3clma97YFPosfO0fPODRTfEUCr6Plt6eLL5PsQY3w3Ar/K4DvwMdAH2AaYWfn/Ka74Ki2/C7g+G/uvqmNKtr5/OlOoJXdf4u7To+drgLlsOXxHrjsB+IsH7wCdzKx7FuI4FPjY3bN+hbq7T2HLCydTx+Z6BDgxzar/B5jk7l+5+9fAJODIhojP3f/hYXgYgHcIF4hmRRX7LxOZjJFWZ9XFF423dhrwWH2/byaqOaZk5funpFAH0VDfg4CpaRbvZ2YzzewlM+vXoIGFoUL+YWbTzGxUmuWZjEvVEM6g6n/EbO6/hB+4+xII/7jADmnK5Mq+/Anh7C+dmr4Pcbo0qt56uIrqj1zYfwcCS919fhXLG2z/VTqmZOX7p6SwlcysHfA0cIW7r660eDqhSmQAcB/wbAOHN9TdCwnDlv/czA6qtDyjMafiFF3lfjzwP2kWZ3v/1UYu7MtrgHKguIoiNX0f4vIAsCswEFhCqKKpLOv7DxhB9WcJDbL/ajimVLlamnl12n9KClvBzPIIf7xid//fysvdfbW7r42evwjkmVnXhorP3RdHj8uAZwin6KkyGZcqbkcB0919aeUF2d5/KZYmqtWix2VpymR1X0YNi8cCIz2qZK4sg+9DLNx9qbtvdPdNwINVvG+2918r4GTgiarKNMT+q+KYkpXvn5JCLUX1jw8Bc9397irK7BiVw8wGE/bzigaKbzsza594TmiMnF2p2PPAOVEvpCHAqsRpagOq8tdZNvdfJaljc50LPJemzCvAEWbWOaoeOSKaFzszOxK4Cjje3ddVUSaT70Nc8aW2U51UxftmMkZanA4DPnL30nQLG2L/VXNMyc73L64W9aY6AQcQTs9mATOi6WjgYuDiqMylwBxCT4p3gP0bML4+0fvOjGK4JpqfGp8R7or3MfABUNTA+7At4SDfMWVeVvcfIUEtATYQfn1dQLi3xz+B+dHj9lHZIuBPKev+BFgQTec3YHwLCPXJie/hH6KyOwEvVvd9aKD4/hp9v2YRDnDdK8cXvT6a0OPm44aML5r/58T3LqVsg+6/ao4pWfn+aZgLERFJUvWRiIgkKSmIiEiSkoKIiCQpKYiISJKSgoiIJCkpiETMbKNtPoJrvY3YaWb5qSN0iuSqWG/HKdLIrHf3gdkOQiSbdKYgUoNoPP3bzOzdaNotmt/LzP4ZDfj2TzPbJZr/Awv3N5gZTftHm2ppZg9GY+b/w8y2jcpfZmYfRtt5PEsfUwRQUhBJtW2l6qPTU5atdvfBwP3AvdG8+wlDkBcQBqMbG80fC/zbw4B+hYQrYQF2B8a5ez9gJXBKNH80MCjazsVxfTiRTOiKZpGIma1193Zp5i8EDnH3T6KBy7509y5mtpwwdMOGaP4Sd+9qZmVAT3f/LmUb+YRx73ePXl8F5Ln7LWb2MrCWMBrssx4NBiiSDTpTEMmMV/G8qjLpfJfyfCMVbXrHEMai2geYFo3cKZIVSgoimTk95fHt6PlbhFE9AUYCb0TP/wlcAmBmLc2sQ1UbNbMWwM7uPhn4v0AnYIuzFZGGol8kIhW2tc1v3v6yuye6pbY2s6mEH1IjonmXAQ+b2a+BMuD8aP7lwHgzu4BwRnAJYYTOdFoCE82sI2H02nvcfWW9fSKRWlKbgkgNojaFIndfnu1YROKm6iMREUnSmYKIiCTpTEFERJKUFEREJElJQUREkpQUREQkSUlBRESS/j/SX1An4H9ifQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(loss) + 1)\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmcFNW5//HPwyYgCAgqCrJoXCGgOIJGMG4x4EZEDRByo6JBjUj0Z3KD4lVjIMk1Xq9xucbRxGviKBINBr2oUUJc4saADAhEQAQcQGQTWWV7fn+c6qan6Znpmelllu/79erXdFWdqn6mpqeeOudUnTJ3R0REBKBRvgMQEZHaQ0lBRETilBRERCROSUFEROKUFEREJE5JQURE4pQUZB9m1tjMNptZl0yWzScz+5qZZfz6azM7x8yWJkx/ZGYD0ilbjc96zMxure76Iuloku8ApObMbHPCZEvgK2B3NH2NuxdVZXvuvhtolemyDYG7H5OJ7ZjZ1cD33f2MhG1fnYlti1RESaEecPf4QTk6E73a3V8rr7yZNXH3XbmITaQy+j7WLmo+agDMbLyZPWNmT5vZJuD7Znaqmb1rZl+Y2Sozu9/Mmkblm5iZm1m3aPrJaPlLZrbJzN4xs+5VLRstH2RmC81so5k9YGb/NLMryok7nRivMbPFZrbBzO5PWLexmf23ma0zs4+BgRXsn9vMbGLSvIfM7N7o/dVmtiD6fT6OzuLL21apmZ0RvW9pZn+KYpsHnJTic5dE251nZhdF878OPAgMiJrm1ibs2zsT1r82+t3XmdnzZnZoOvumKvs5Fo+ZvWZm683sMzP794TP+Y9on3xpZsVmdliqpjozeyv2d4725xvR56wHbjOzo8xsevS7rI32W5uE9btGv+OaaPlvzax5FPNxCeUONbOtZta+vN9XKuHuetWjF7AUOCdp3nhgB3Ah4USgBXAy0I9QWzwCWAiMjso3ARzoFk0/CawFCoCmwDPAk9UoezCwCRgcLft/wE7ginJ+l3Ri/CvQBugGrI/97sBoYB7QGWgPvBG+7ik/5whgM7B/wrY/Bwqi6QujMgacBWwDekXLzgGWJmyrFDgjen8P8A+gHdAVmJ9U9rvAodHf5HtRDIdEy64G/pEU55PAndH7c6MYTwCaA/8D/D2dfVPF/dwGWA38GNgPOADoGy27BSgBjop+hxOAA4GvJe9r4K3Y3zn63XYB1wGNCd/Ho4GzgWbR9+SfwD0Jv8+H0f7cPyp/WrSsEJiQ8Dk3A5Pz/X9Yl195D0CvDP9By08Kf69kvZ8Af47epzrQ/y6h7EXAh9UoOxJ4M2GZAasoJymkGeMpCcv/Avwkev8GoRkttuy85ANV0rbfBb4XvR8ELKyg7IvA9dH7ipLC8sS/BfCjxLIptvshcH70vrKk8ATwy4RlBxD6kTpXtm+quJ//DSgup9zHsXiT5qeTFJZUEsOlwIzo/QDgM6BxinKnAZ8AFk3PBoZk+v+qIb3UfNRwfJo4YWbHmtn/Rc0BXwJ3AR0qWP+zhPdbqbhzubyyhyXG4eG/uLS8jaQZY1qfBSyrIF6Ap4Dh0fvvAfHOeTO7wMzei5pPviCcpVe0r2IOrSgGM7vCzEqiJpAvgGPT3C6E3y++PXf/EtgAdEook9bfrJL9fDiwuJwYDickhupI/j52NLNJZrYiiuF/k2JY6uGihjLc/Z+EWkd/M+sJdAH+r5oxCepTaEiSL8d8hHBm+jV3PwC4nXDmnk2rCGeyAJiZUfYglqwmMa4iHExiKrtk9hngHDPrTGjeeiqKsQXwLPArQtNOW+BvacbxWXkxmNkRwMOEJpT20Xb/lbDdyi6fXUlokoptrzWhmWpFGnElq2g/fwocWc565S3bEsXUMmFex6Qyyb/ffxKumvt6FMMVSTF0NbPG5cTxR+D7hFrNJHf/qpxykgYlhYarNbAR2BJ11F2Tg898EehjZheaWRNCO/VBWYpxEnCjmXWKOh1/VlFhd19NaOJ4HPjI3RdFi/YjtHOvAXab2QWEtu90Y7jVzNpauI9jdMKyVoQD4xpCfryaUFOIWQ10TuzwTfI0cJWZ9TKz/QhJ6013L7fmVYGK9vMUoIuZjTazZmZ2gJn1jZY9Bow3syMtOMHMDiQkw88IFzQ0NrNRJCSwCmLYAmw0s8MJTVgx7wDrgF9a6LxvYWanJSz/E6G56XuEBCE1oKTQcN0MXE7o+H2EcKacVdGBdyhwL+Gf/EjgA8IZYqZjfBiYBswFZhDO9ivzFKGP4KmEmL8AbgImEzprLyUkt3TcQaixLAVeIuGA5e5zgPuB96MyxwLvJaz7KrAIWG1mic1AsfVfJjTzTI7W7wKMSDOuZOXuZ3ffCHwLuITQsb0Q+Ga0+DfA84T9/CWh07d51Cz4Q+BWwkUHX0v63VK5A+hLSE5TgOcSYtgFXAAcR6g1LCf8HWLLlxL+zjvc/e0q/u6SJNY5I5JzUXPASuBSd38z3/FI3WVmfyR0Xt+Z71jqOt28JjllZgMJzQHbCZc07iKcLYtUS9Q/Mxj4er5jqQ/UfCS51h9YQmhWGAh8Rx2DUl1m9ivCvRK/dPfl+Y6nPlDzkYiIxKmmICIicXWuT6FDhw7erVu3fIchIlKnzJw5c627V3QJOFAHk0K3bt0oLi7OdxgiInWKmVV2Vz+g5iMREUmgpCAiInFKCiIiEqekICIicUoKIiISp6QgIpJlRUXQrRs0ahR+FhVVtkZm168KJQURqffyeVAuKoJRo2DZMnAPP0eNSn8bNV2/yvL96Leqvk466SQXkdx68kn3rl3dzcLPJ5+sO+s/+aR7y5bu4ZAaXi1bpr+Nmq7ftWvZdWOvrl1zs34M5TxWNfmV94N8VV9KCiJVV5cPqnX9oGyWen2z3Kwfo6QgUo/k86Ce74NqXT8o5/v3j0k3KahPQaSWq2mb8rhxsHVr2Xlbt4b56VhezoDU5c2vbet3Kefp3OXNz/T6EyZAy5Zl57VsGebnYv2qUlIQybKadnLm+6Ce74NqXT8ojxgBhYXQtSuYhZ+FhWF+LtavsnSqE7XppeYjqUtq2nTjnv/mi3z3CWRiH+a7o7w2QH0KIplRkwNCJtqD831Qj22jrl59JIGSgkgG1PSAmokrR2rDQV3qvnSTQp17HGdBQYHreQqSK926hY7dZF27wtKl2V8/pqgo9CEsXx7a0idMyGKbstRLZjbT3QsqK6eOZpEK1LSTNlNXjowYEZLInj3hpxKCZIuSgtR7Nbn6p6ZXvuT8yhGRGlJSkHqtptf4Z+JMX2f5UpcoKUi9VtNr/HWmLw2NOpqlXmvUKNQQkpmFM3eRhkIdzVJv5LNPQKShUVKQWq029AmINCRKClKrqU9AJLfUpyC1mvoERDJDfQpSL6hPQCS3lBSkVlOfgEhuKSlIraY+AZHcapLvAEQqM2KEkoBIrqimIFlX0yePiUjuZDUpmNlAM/vIzBab2dgUy7ua2TQzm2Nm/zCzztmMR3KvpvcZiEhuZS0pmFlj4CFgEHA8MNzMjk8qdg/wR3fvBdwF/Cpb8Uh+1PQ+AxHJrWzWFPoCi919ibvvACYCg5PKHA9Mi95PT7Fc6riaPo9ARHIrm0mhE/BpwnRpNC9RCXBJ9P5ioLWZtc9iTJJjus9ApG7JZlKwFPOS7039CfBNM/sA+CawAti1z4bMRplZsZkVr1mzJvORStboPgORuiWbSaEUODxhujOwMrGAu6909yHufiIwLpq3MXlD7l7o7gXuXnDQQQdlMWTJNN1nIFK3ZPM+hRnAUWbWnVADGAZ8L7GAmXUA1rv7HuAW4A9ZjEfyRPcZiNQdWaspuPsuYDTwCrAAmOTu88zsLjO7KCp2BvCRmS0EDgHUqCAikkcaJVVEpAHQKKmSMbojWaTh0NhHUqHYHcmxG9BidySD+glE6iPVFKRCuiNZpGFRUpAK6Y5kkYZFSUEqpDuSRRoWJQWpkO5IFmlYlBSkQrojWaRh0dVHUindkSzScKimICIicUoKIiISp6QgIiJxSgoiIhKnpCAiInFKCiIiEqek0ABolFMRSZfuU6jnNMqpiFSFagr1nEY5FZGqUFKo5zTKqYhUhZJCPadRTkWkKpQU6jmNcioiVaGkUM9plFMRqQpdfdQAaJRTEUmXagoiIhKnpCAiInFKCiIiEqekICIicUoKIiISp6RQB2hAOxHJFV2SWstpQDsRySXVFGo5DWgnIrmkmkItpwHt6oc9e0Iyd6/+NszCECWN6tCp3PLl8Npr4VVcDMccAwMGwOmnQ58+0KxZviOUZEoKtVyXLqHJKNV8yS73cCD/8kvYtKnsz6q+37w5c3G1agUHHBBerVun/75tWzjyyPA+WzZsgOnT9yaCRYvC/I4doV8/+Ne/4MUXw7wWLeCUU/YmiVNOgf33z15skh4lhVpuwoSyfQqgAe2qY8sWmDoV1q2r+ACfPG/Pnsq33aTJvgfi9u2he/ey8/bfv2Zn+Xv2hOSSKu7Vq8vO3727/O0cdhgcd1x4HXvs3vcdO4baSFVs3w5vv703CcycGeJs1QrOOAOuvx7OOQeOP37vtj//HN56C954A958E8aPD+s0aRJqD7Ek0b8/HHhgtXeXVJN5TeqzeVBQUODFxcX5DiOniopCH8Ly5aGGMGGCOpnTtWMHPPYY3HVXOHDGmKV3dp3O8ubNq34wzSb3cLBOTnLr18PChbBgQXj9619heUybNmWTROzVvTs0bhzK7NkDJSUhAbz6ajiob98elp9ySkgA55wTagVNm6YX75dfwjvv7E0S778PX30VlvXoEZJELFF07pzZfVUVX30F++2Xv8+vKTOb6e4FlZbLZlIws4HAb4HGwGPu/uuk5V2AJ4C2UZmx7j61om02xKRQG+zeDXPnhn+K2MGwVava2769Zw9MnAj/8R+wZEk4qPz85+Gg17p1OGuvTQfyfHCHlSv3JolYoliwAD77bG+5Zs3g6KPDAXnGjFDbgnDAjiWB00/PXLPU9u2h/yGWJP75z73Jq1u3skni6KOz93csLQ2fH3t9+GE4KYt9/oABIWnWle9R3pOCmTUGFgLfAkqBGcBwd5+fUKYQ+MDdHzaz44Gp7t6tou0qKeRecTFce21oGkiWqn27vDPqgw+Gb30rtG1nizu89BLcems4o+3dG371Kxg4sO7889YGGzbsTRCxn8uXw4knhiRw1lmhGSoXdu+GOXP2Jok33wxNUBC+U/377z1I9+4dmqGqyj3UohKTwCefhGWtW8M3vgEFBXvLxJJmhw5lP//EE6v3+RXFtWZNOLFZsiTEcPTR1dtWukkhm30KfYHF7r4kCmgiMBiYn1DGgdj5RRtgZRbjkSr64ovQbPXww3DIIfC734UDemXt8J9/Xnb+rl17t9msGQwaBMOGwYUXZrZj8e234ZZbwsHjiCPgqadg6NDaW5upzdq1g1NPDa98a9w4HGxPPBF+/ONwoFy0qGyS+MtfQtnYATx2kO7bNzTvJdu9O5w0JCaBWKI56KCw7pgxqRONO3z8cdnPf/75sKxVq7DPYp/fr1/oUK/I9u2wdOneA3/ya8uWvWXvv7/6SSFd2awpXAoMdPero+l/A/q5++iEMocCfwPaAfsD57j7PuejZjYKGAXQpUuXk5aluhxHMsY99GPcfDOsXQujR4c2+TZtqret7dtDgvj4Y/jzn+GZZ0KzRcuWcNFFMHw4fPvb1W+v/fDDkLymTAmdpbffDlddpcsdG5JUTT0QvgN9+4YD9EknwUcfVdwkNWBAuGy2qrXKlSvLdp7PnRu++02bwsknh+2eckq4UCD5oL9iRdlttWgRTmpSvbp3rzzJlKc2NB9dBnw7KSn0dfcbEsr8vyiG/zKzU4HfAz3dvdxrPtR8lF3z58OPfgSvvx7Och5+OJyhZdLu3eEf6Omn4dlnQxt127YwZEioQZx5ZnpV8KVL4Y474E9/CmeIP/tZOJPUZY2ybl048MeSxMyZe2usiZ3XAwbA4Ydn/vM3bAg111iSKC6GnTv3Lu/cee9BPvnAf8gh2WnqTDcp4O5ZeQGnAq8kTN8C3JJUZh5weML0EuDgirZ70kknuWTeli3uY8e6N2ni3q6d+yOPuO/enf3P3bHDfepU9x/8wL11a3dwP/hg9+uvd3/rrdQxrF7tPmaMe9Om7s2bu//0p+5r12Y/Vqm7Nm92f/fd/H1Ptmxxf+cd9wUL3Ldty08MQLGnc+xOp1B1XoT+iiVAd6AZUAL0SCrzEnBF9P44Qp+CVbRdJYXM++tf3bt2Dd+GK64IB9182LrV/bnn3C+7LBzswf3ww8NBf+ZM940b3W+/3b1VK/fGjd1/+EP3Tz/NT6widU26SSHbl6SeB9xHuNz0D+4+wczuioKbEl1x9CjQitDp/O/u/reKtqnmo8xZujQ0t0yZEqrUDz8cqtO1waZNIa6nn4ZXXglV/6ZNQxX8ssvgF78Ibb8ikp689ylki5JCze3YAffeGzqPzeDOO+HGG9O/2SjX1q8PV5fMmgUjR4bL8kSkamrDJalSC/3jH6EjecECuPhiuO++2j+O0oEHwtVX5zsKkYZBV3A3EJ99Bv/2b+HKnm3bwqBkf/lL7U8IIpJbSgr13I4dcM894YaXZ54J1/PPmwfnn5/vyESkNlLzUT328suhI3nhwpAE7r03+3dDikjdpppCPbR4cbhTeNCgcFfliy+GlxKCiFSm0qRgZqPNrF0ugpGa2bw5jP3To0d40Mndd4fb/dVUJCLpSqem0BGYYWaTzGygmcaarKqiojC+SqNG4WdRUWa3Hxur6Jhj4Ne/DkNFLFwIP/2pxv8RkaqpNCm4+23AUYRxia4AFpnZL83syCzHVi8UFYUnpy1bFg7ey5aF6Uwlhpkzw9C93/9+GMr4nXfgiSfg0EMzs30RaVjS6lOIbpH+LHrtIoxq+qyZ3Z3F2OqFcePKPkoTwvS4cTXb7po1IbmcfHLoQ/j97+G998JIjCIi1VXp1UdmNga4HFgLPAb81N13mlkjYBHw79kNsW5bvrxq8yuzcyf8z/+E0UG3bIGbbgpDRVdnWGsRkWTpXJLaARji7mUeYuDue8zsguyEVX906RKajFLNr6rXXguXmM6fD+eeG+5GPu64mscoIhKTTvPRVGB9bMLMWptZPwB3X5CtwOqLCRPCw2QStWwZ5qdj587wYJrTTw+Psty+PTzl6eWXlRBEJPPSSQoPA5sTprdE8yQNI0ZAYSF07RoGn+vaNUyPGFHxep9/DuPHh4dwfPe74clS994b7kYePFjPGxaR7Ein+cg8YSjVqNlId0JXwYgRlSeBmBkz4IEHwpAUO3aEZqLf/S7ciNa4cXbjFBFJ5+C+JOpsjtUOfkR4eI5kyFdfhSaiBx8MVxC1ahWuLLr+ejj22HxHJyINSTrNR9cC3wBWAKVAP2BUNoNqKFauDFcOde0aRjDdsAHuvz88yPuBB5QQRCT3Kq0puPvnwLAcxNIguIcHej/wADz3XHiI/fnnw+jRoSO5kUajEpE8Suc+hebAVUAPoHlsvruPzGJc9c7OnfDkkyEZfPBBuK9gzJjwwJsjdW+4iNQS6ZyX/okw/tG3gdeBzsCmbAZV32zfDt/5TniU5I4doeN4xQr4r/9SQhCR2iWdjuavuftlZjbY3Z8ws6eAV7IdWH2xbRsMGRLuK3joIbjuOl1OKiK1VzpJYWf08wsz60kY/6hb1iKqR7ZuDTWE116Dxx6Dq67Kd0QiIhVLJykURs9TuA2YArQC/iOrUdUDW7bAhRfCP/4Bjz8Ol1+e74hERCpXYVKIBr370t03AG8AR+Qkqjpu8+ZwRdFbb8Ef/xiGtRYRqQsq7Gh29z3A6BzFUi9s2gQDB8I//xmemaCEICJ1STpXH71qZj8xs8PN7MDYK+uR1UEbN8K3vx3uSn766fAENBGRuiSdPoXY/QjXJ8xz1JRUxhdfhIQwaxZMmgQXX5zviEREqi6dO5q75yKQumz9+jBw3Zw54S7liy7Kd0QiItWTzh3NP0g1393/mPlw6p516+Ccc8KDbyZPDh3MIiJ1VTrNRycnvG8OnA3MAhp8UlizJiSEjz6CKVNC85GISF2WTvPRDYnTZtaGMPRFg7Z6NZx9Nnz8Mbz4YkgOIiJ1XXUelrMVOCrTgdQlq1bBWWfB8uUwdSqceWa+IxIRyYx0+hReIFxtBOES1uOBSdkMqjZbsSIkhBUr4KWXwrOTRUTqi3RqCvckvN8FLHP30izFU6uVloZawWefwSuvwGmn5TsiEZHMSicpLAdWuft2ADNrYWbd3H1pViOrZVasgG9+E9auhVdfhVNOyXdEIiKZl84dzX8G9iRM747mVcrMBprZR2a22MzGplj+32Y2O3otNLMv0gs798aPD4/PVEIQkfosnZpCE3ffEZtw9x1m1qyylcysMfAQ8C3Cs51nmNkUd5+fsK2bEsrfAJxYleBzZfPmMI7R0KHQt2++oxERyZ50agprzCx+j66ZDQbWprFeX2Cxuy+JkspEYHAF5YcDT6ex3ZybODEMdHfNNfmOREQku9KpKVwLFJnZg9F0KZDyLucknYBPE6ZLgX6pCppZV6A78Pdylo8CRgF06dIljY/OrEcegZ491WwkIvVfOjevfQycYmatAHP3dJ/PnOqhk55iHsAw4Fl3311ODIVAIUBBQUF528iKWbOguBjuv1+P0RSR+q/S5iMz+6WZtXX3ze6+yczamdn4NLZdChyeMN0ZWFlO2WHU0qajRx+Fpk3hN7+BRo2gW7fQvyAiUh+l06cwyN3jVwVFT2E7L431ZgBHmVn3qGN6GOFxnmWY2TFAO+Cd9ELOnc2b4X//F/bsgU8/BXdYtgxGjVJiEJH6KZ2k0NjM9otNmFkLYL8KygPg7rsIT217BVgATHL3eWZ2V2LHNaGDeaK757RZKB0TJ8L27bA7qVFr61YYNy4/MYmIZFM6Hc1PAtPM7PFo+krgiXQ27u5TgalJ825Pmr4znW3lwyOPlL9s+fLcxSEikivpdDTfbWZzgHMInccvA12zHVi+xTqY27WDDRv2XZ6Hi6BERLIuneYjgM8IdzVfQniewoKsRVRLFBZC8+bw619Dy5Zll7VsCRMm5CcuEZFsKremYGZHEzqHhwPrgGcIl6TW+4GiE+9gHjUK9t8/9CEsXx5qCBMmwIgR+Y5SRCTzKmo++hfwJnChuy8GMLObKihfbzz9dEgMsTuYR4xQEhCRhqGi5qNLCM1G083sUTM7m9Q3pNU7hYW6g1lEGqZyk4K7T3b3ocCxwD+Am4BDzOxhMzs3R/HlXKyD+ZprdAeziDQ8lXY0u/sWdy9y9wsIdyXPBvYZBru+KCyEFi3g+9/PdyQiIrmX7tVHALj7end/xN3PylZA+bRp094O5rZt8x2NiEjuVSkp1HcTJ4YO5lGj8h2JiEh+KCkk0BDZItLQKSlEZs4ML3Uwi0hDpqQQefRRdTCLiCgpoA5mEZEYJQXUwSwiEqOkQOhg/vrX1cEsItLgk0Ksg3nUKHUwi4g0+KSgO5hFRPZq0Elh0yZ46il1MIuIxDTopBAbIlsdzCIiQYNOCoWF6mAWEUnUYJOCOphFRPbVYJOCOphFRPbVIJOCOphFRFJrkEkh+RnMIiISNMikEOtg7tcv35GIiNQuDS4pqINZRKR8DS4pqINZRKR8DSopqINZRKRiDSopqINZRKRiDSopxIbIVgeziEhqDSYpzJwJs2bpGcwiIhVpMEnh5ZdDB/OIEfmORESk9mowSWHcOFi4UB3MIiIVyWpSMLOBZvaRmS02s7HllPmumc03s3lm9lQ24+ncOZtbFxGp+5pka8Nm1hh4CPgWUArMMLMp7j4/ocxRwC3Aae6+wcwOzlY8IiJSuWzWFPoCi919ibvvACYCg5PK/BB4yN03ALj751mMR0REKpHNpNAJ+DRhujSal+ho4Ggz+6eZvWtmA1NtyMxGmVmxmRWvWbMmS+GKiEg2k0KqCz89aboJcBRwBjAceMzM9ukKdvdCdy9w94KDDjoo44GKiEiQzaRQChyeMN0ZWJmizF/dfae7fwJ8REgSIiKSB9lMCjOAo8ysu5k1A4YBU5LKPA+cCWBmHQjNSUuyGJOIiFQga0nB3XcBo4FXgAXAJHefZ2Z3mdlFUbFXgHVmNh+YDvzU3ddlKyYREamYuSc389duBQUFXlxcnO8wRETqFDOb6e4FlZVrMHc0i4hI5ZQUREQkTklBRETilBRERCROSUFEROKUFEREJE5JQURE4pQUREQkTklBRETilBRERCROSUFEROKy9jhOEalfdu7cSWlpKdu3b893KFKB5s2b07lzZ5o2bVqt9ZUURCQtpaWltG7dmm7dumGW6hlakm/uzrp16ygtLaV79+7V2oaaj0QkLdu3b6d9+/ZKCLWYmdG+ffsa1eaUFEQkbUoItV9N/0ZKCiIiEqekICJZUVQE3bpBo0bhZ1FRzba3bt06TjjhBE444QQ6duxIp06d4tM7duxIaxtXXnklH330UYVlHnroIYpqGmwdpo5mEcm4oiIYNQq2bg3Ty5aFaYARI6q3zfbt2zN79mwA7rzzTlq1asVPfvKTMmXcHXenUaPU57uPP/54pZ9z/fXXVy/AekI1BRHJuHHj9iaEmK1bw/xMW7x4MT179uTaa6+lT58+rFq1ilGjRlFQUECPHj2466674mX79+/P7Nmz2bVrF23btmXs2LH07t2bU089lc8//xyA2267jfvuuy9efuzYsfTt25djjjmGt99+G4AtW7ZwySWX0Lt3b4YPH05BQUE8YSW64447OPnkk+PxxR5/vHDhQs466yx69+5Nnz59WLp0KQC//OUv+frXv07v3r0Zl42dlQYlBRHJuOXLqza/pubPn89VV13FBx98QKdOnfj1r39NcXExJSUlvPrqq8yfP3+fdTZu3Mg3v/lNSkpKOPXUU/nDH/6Qctvuzvvvv89vfvObeIJ54IEH6NixIyUlJYwdO5YPPvgg5bo//vGPmTFjBnPnzmXjxo28/PLLAAwfPpybbrqJkpIS3n77bQ4++GBeeOEFXnrpJd5//31KSkq4+eabM7R3qkZJQUQyrkuXqs2vqSOPPJKTTz45Pv3000+L+wJdAAAPGUlEQVTTp08f+vTpw4IFC1ImhRYtWjBo0CAATjrppPjZerIhQ4bsU+att95i2LBhAPTu3ZsePXqkXHfatGn07duX3r178/rrrzNv3jw2bNjA2rVrufDCC4Fws1nLli157bXXGDlyJC1atADgwAMPrPqOyAAlBRHJuAkToGXLsvNatgzzs2H//fePv1+0aBG//e1v+fvf/86cOXMYOHBgyuv2mzVrFn/fuHFjdu3alXLb++233z5lYs1AFdm6dSujR49m8uTJzJkzh5EjR8bjSHXZqLvXikt+lRREJONGjIDCQujaFczCz8LC6ncyV8WXX35J69atOeCAA1i1ahWvvPJKxj+jf//+TJo0CYC5c+emrIls27aNRo0a0aFDBzZt2sRzzz0HQLt27ejQoQMvvPACEG4K3Lp1K+eeey6///3v2bZtGwDr16/PeNzp0NVHIpIVI0bkJgkk69OnD8cffzw9e/bkiCOO4LTTTsv4Z9xwww384Ac/oFevXvTp04eePXvSpk2bMmXat2/P5ZdfTs+ePenatSv9+vWLLysqKuKaa65h3LhxNGvWjOeee44LLriAkpISCgoKaNq0KRdeeCG/+MUvMh57ZSydalBtUlBQ4MXFxfkOQ6TBWbBgAccdd1y+w6gVdu3axa5du2jevDmLFi3i3HPPZdGiRTRpUjvOs1P9rcxsprsXVLZu7fgNRETqkM2bN3P22Weza9cu3J1HHnmk1iSEmqofv4WISA61bduWmTNn5juMrFBHs4iIxCkpiIhInJKCiIjEKSmIiEickoKI1AlnnHHGPjei3XffffzoRz+qcL1WrVoBsHLlSi699NJyt13Zpe733XcfWxNG+TvvvPP44osv0gm9TlFSEJE6Yfjw4UycOLHMvIkTJzJ8+PC01j/ssMN49tlnq/35yUlh6tSptG3bttrbq610SaqIVNmNN0KKkaJr5IQTIBqxOqVLL72U2267ja+++or99tuPpUuXsnLlSvr378/mzZsZPHgwGzZsYOfOnYwfP57BgweXWX/p0qVccMEFfPjhh2zbto0rr7yS+fPnc9xxx8WHlgC47rrrmDFjBtu2bePSSy/l5z//Offffz8rV67kzDPPpEOHDkyfPp1u3bpRXFxMhw4duPfee+OjrF599dXceOONLF26lEGDBtG/f3/efvttOnXqxF//+tf4gHcxL7zwAuPHj2fHjh20b9+eoqIiDjnkEDZv3swNN9xAcXExZsYdd9zBJZdcwssvv8ytt97K7t276dChA9OmTcvcH4Es1xTMbKCZfWRmi81sbIrlV5jZGjObHb2uzmY8IlJ3tW/fnr59+8aHn544cSJDhw7FzGjevDmTJ09m1qxZTJ8+nZtvvrnCQesefvhhWrZsyZw5cxg3blyZew4mTJhAcXExc+bM4fXXX2fOnDmMGTOGww47jOnTpzN9+vQy25o5cyaPP/447733Hu+++y6PPvpofCjtRYsWcf311zNv3jzatm0bH/8oUf/+/Xn33Xf54IMPGDZsGHfffTcAv/jFL2jTpg1z585lzpw5nHXWWaxZs4Yf/vCHPPfcc5SUlPDnP/+5xvs1WdZqCmbWGHgI+BZQCswwsynunjxy1DPuPjpbcYhI5lV0Rp9NsSakwYMHM3HixPjZubtz66238sYbb9CoUSNWrFjB6tWr6dixY8rtvPHGG4wZMwaAXr160atXr/iySZMmUVhYyK5du1i1ahXz588vszzZW2+9xcUXXxwfqXXIkCG8+eabXHTRRXTv3p0TTjgBKH947tLSUoYOHcqqVavYsWMH3bt3B+C1114r01zWrl07XnjhBU4//fR4mWwMr53NmkJfYLG7L3H3HcBEYHAl62RFpp8VKyL58Z3vfIdp06Yxa9Ystm3bRp8+fYAwwNyaNWuYOXMms2fP5pBDDkk5XHaiVMNUf/LJJ9xzzz1MmzaNOXPmcP7551e6nYpqJLFht6H84blvuOEGRo8ezdy5c3nkkUfin5dqKO1cDK+dzaTQCfg0Ybo0mpfsEjObY2bPmtnhqTZkZqPMrNjMitesWVOlIGLPil22DNz3PitWiUGk7mnVqhVnnHEGI0eOLNPBvHHjRg4++GCaNm3K9OnTWbZsWYXbOf300ymKDgIffvghc+bMAcKw2/vvvz9t2rRh9erVvPTSS/F1WrduzaZNm1Ju6/nnn2fr1q1s2bKFyZMnM2DAgLR/p40bN9KpUzg0PvHEE/H55557Lg8++GB8esOGDZx66qm8/vrrfPLJJ0B2htfOZlJIlc6SU+oLQDd37wW8Bjyx7yrg7oXuXuDuBQcddFCVgsjls2JFJPuGDx9OSUlJ/MlnACNGjKC4uJiCggKKioo49thjK9zGddddx+bNm+nVqxd33303ffv2BcJT1E488UR69OjByJEjywy7PWrUKAYNGsSZZ55ZZlt9+vThiiuuoG/fvvTr14+rr76aE088Me3f58477+Syyy5jwIABdOjQIT7/tttuY8OGDfTs2ZPevXszffp0DjroIAoLCxkyZAi9e/dm6NChaX9OurI2dLaZnQrc6e7fjqZvAXD3X5VTvjGw3t3bpFoeU9Whsxs1CjWEfT8P9uxJezMiDZ6Gzq47ajJ0djZrCjOAo8ysu5k1A4YBUxILmNmhCZMXAQsyHUSunxUrIlKXZS0puPsuYDTwCuFgP8nd55nZXWZ2UVRsjJnNM7MSYAxwRabjyPWzYkVE6rKs3rzm7lOBqUnzbk94fwtwSzZjiD0OcNw4WL481BAmTMjPYwJF6rra8nB5KV9NuwQaxB3N+XpWrEh90rx5c9atW0f79u2VGGopd2fdunU0b9682ttoEElBRGquc+fOlJaWUtXLwiW3mjdvTufOnau9vpKCiKSladOm8Ttppf7SKKkiIhKnpCAiInFKCiIiEpe1O5qzxczWABUPbJI/HYC1+Q6iAoqvZmp7fFD7Y1R8NVOT+Lq6e6XjBNW5pFCbmVlxOreR54viq5naHh/U/hgVX83kIj41H4mISJySgoiIxCkpZFZhvgOohOKrmdoeH9T+GBVfzWQ9PvUpiIhInGoKIiISp6QgIiJxSgpVZGaHm9l0M1sQPQvixynKnGFmG81sdvS6PdW2shjjUjObG332Po+ps+B+M1scPR+7Tw5jOyZhv8w2sy/N7MakMjnff2b2BzP73Mw+TJh3oJm9amaLop/tyln38qjMIjO7PEex/cbM/hX9/SabWdty1q3wu5DlGO80sxUJf8fzyll3oJl9FH0fx+YwvmcSYltqZrPLWTer+7C8Y0revn/urlcVXsChQJ/ofWtgIXB8UpkzgBfzGONSoEMFy88DXiI8R/sU4L08xdkY+IxwU01e9x9wOtAH+DBh3t3A2Oj9WOA/U6x3ILAk+tkuet8uB7GdCzSJ3v9nqtjS+S5kOcY7gZ+k8R34GDgCaAaUJP8/ZSu+pOX/Bdyej31Y3jElX98/1RSqyN1Xufus6P0mwlPlOuU3qiobDPzRg3eBtkmPRs2Vs4GP3T3vd6i7+xvA+qTZg4EnovdPAN9Jseq3gVfdfb27bwBeBQZmOzZ3/5uHpxsCvAtUf6zkDChn/6WjL7DY3Ze4+w5gImG/Z1RF8Vl4OMR3gacz/bnpqOCYkpfvn5JCDZhZN+BE4L0Ui081sxIze8nMeuQ0MHDgb2Y208xGpVjeCfg0YbqU/CS2YZT/j5jP/RdziLuvgvCPCxycokxt2JcjCTW/VCr7LmTb6KiJ6w/lNH/Uhv03AFjt7ovKWZ6zfZh0TMnL909JoZrMrBXwHHCju3+ZtHgWoUmkN/AA8HyOwzvN3fsAg4Drzez0pOWpHpuV02uTzawZcBHw5xSL873/qiKv+9LMxgG7gKJyilT2Xcimh4EjgROAVYQmmmR5/y4Cw6m4lpCTfVjJMaXc1VLMq9H+U1KoBjNrSvjjFbn7X5KXu/uX7r45ej8VaGpmHXIVn7uvjH5+DkwmVNETlQKHJ0x3BlbmJrq4QcAsd1+dvCDf+y/B6lizWvTz8xRl8rYvo07FC4ARHjUwJ0vju5A17r7a3Xe7+x7g0XI+O6/fRTNrAgwBnimvTC72YTnHlLx8/5QUqihqf/w9sMDd7y2nTMeoHGbWl7Cf1+Uovv3NrHXsPaFD8sOkYlOAH0RXIZ0CbIxVU3Oo3LOzfO6/JFOA2NUclwN/TVHmFeBcM2sXNY+cG83LKjMbCPwMuMjdt5ZTJp3vQjZjTOynuricz54BHGVm3aPa4zDCfs+Vc4B/uXtpqoW52IcVHFPy8/3LVo96fX0B/QnVsznA7Oh1HnAtcG1UZjQwj3AlxbvAN3IY3xHR55ZEMYyL5ifGZ8BDhKs+5gIFOd6HLQkH+TYJ8/K6/wgJahWwk3D2dRXQHpgGLIp+HhiVLQAeS1h3JLA4el2Zo9gWE9qSY9/B30VlDwOmVvRdyOH++1P0/ZpDOMAdmhxjNH0e4Yqbj7MVY6r4ovn/G/veJZTN6T6s4JiSl++fhrkQEZE4NR+JiEickoKIiMQpKYiISJySgoiIxCkpiIhInJKCSMTMdlvZEVwzNmKnmXVLHKFTpLZqku8ARGqRbe5+Qr6DEMkn1RREKhGNp/+fZvZ+9PpaNL+rmU2LBnybZmZdovmHWHjGQUn0+ka0qcZm9mg0Zv7fzKxFVH6Mmc2PtjMxT7+mCKCkIJKoRVLz0dCEZV+6e1/gQeC+aN6DhCHIexEGpLs/mn8/8LqHAf36EO6EBTgKeMjdewBfAJdE88cCJ0bbuTZbv5xIOnRHs0jEzDa7e6sU85cCZ7n7kmjgss/cvb2ZrSUM3bAzmr/K3TuY2Rqgs7t/lbCNboRx74+Kpn8GNHX38Wb2MrCZMBrs8x4NBiiSD6opiKTHy3lfXplUvkp4v5u9fXrnE8aiOgmYGY3cKZIXSgoi6Rma8POd6P3bhFE9AUYAb0XvpwHXAZhZYzM7oLyNmlkj4HB3nw78O9AW2Ke2IpIrOiMR2auFlX14+8vuHrssdT8ze49wIjU8mjcG+IOZ/RRYA1wZzf8xUGhmVxFqBNcRRuhMpTHwpJm1IYxe+9/u/kXGfiORKlKfgkgloj6FAndfm+9YRLJNzUciIhKnmoKIiMSppiAiInFKCiIiEqekICIicUoKIiISp6QgIiJx/x+YT0/k97OprwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.clf()   # 그래프를 초기화합니다\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 모델은 9번째 에포크 이후에 과대적합이 시작됩니다. 9번의 에포크로 새로운 모델을 훈련하고 테스트 세트에서 평가하겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/9\n",
      "7982/7982 [==============================] - 1s 66us/step - loss: 2.5398 - acc: 0.5226 - val_loss: 1.6733 - val_acc: 0.6570\n",
      "Epoch 2/9\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 1.3712 - acc: 0.7121 - val_loss: 1.2758 - val_acc: 0.7210\n",
      "Epoch 3/9\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 1.0136 - acc: 0.7781 - val_loss: 1.1303 - val_acc: 0.7530\n",
      "Epoch 4/9\n",
      "7982/7982 [==============================] - 0s 46us/step - loss: 0.7976 - acc: 0.8251 - val_loss: 1.0539 - val_acc: 0.7590\n",
      "Epoch 5/9\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.6393 - acc: 0.8624 - val_loss: 0.9754 - val_acc: 0.7920\n",
      "Epoch 6/9\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.5124 - acc: 0.8921 - val_loss: 0.9102 - val_acc: 0.8140\n",
      "Epoch 7/9\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.4124 - acc: 0.9137 - val_loss: 0.8932 - val_acc: 0.8210\n",
      "Epoch 8/9\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.3355 - acc: 0.9290 - val_loss: 0.8732 - val_acc: 0.8260\n",
      "Epoch 9/9\n",
      "7982/7982 [==============================] - 0s 47us/step - loss: 0.2782 - acc: 0.9371 - val_loss: 0.9338 - val_acc: 0.8000\n",
      "2246/2246 [==============================] - 0s 47us/step\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "model.fit(partial_x_train,\n",
    "          partial_y_train,\n",
    "          epochs=9,\n",
    "          batch_size=512,\n",
    "          validation_data=(x_val, y_val))\n",
    "results = model.evaluate(x_test, one_hot_test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1.0224982320678522, 0.7756010686194165]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "대략 78%의 정확도를 달성했습니다. 균형 잡힌 이진 분류 문제에서 완전히 무작위로 분류하면 50%의 정확도를 달성합니다. 이 문제는 불균형한 데이터셋을 사용하므로 무작위로 분류하면 19% 정도를 달성합니다. 여기에 비하면 이 결과는 꽤 좋은 편입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.182546749777382"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "test_labels_copy = copy.copy(test_labels)\n",
    "np.random.shuffle(test_labels_copy)\n",
    "float(np.sum(np.array(test_labels) == np.array(test_labels_copy))) / len(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 새로운 데이터에 대해 예측하기\n",
    "\n",
    "모델 인스턴스의 `predict` 메서드는 46개 토픽에 대한 확률 분포를 반환합니다. 테스트 데이터 전체에 대한 토픽을 예측해 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`predictions`의 각 항목은 길이가 46인 벡터입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(46,)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 벡터의 원소 합은 1입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(predictions[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "가장 큰 값이 예측 클래스가 됩니다. 즉, 가장 확률이 높은 클래스입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(predictions[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 레이블과 손실을 다루는 다른 방법\n",
    "\n",
    "앞서 언급한 것처럼 레이블을 인코딩하는 다른 방법은 다음과 같이 정수 텐서로 변환하는 것입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = np.array(train_labels)\n",
    "y_test = np.array(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 방식을 사용하려면 손실 함수 하나만 바꾸면 됩니다. 코드 3-21에 사용된 손실 함수 `categorical_crossentropy`는 레이블이 범주형 인코딩되어 있을 것이라고 기대합니다. 정수 레이블을 사용할 때는 `sparse_categorical_crossentropy`를 사용해야 합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy', metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 손실 함수는 인터페이스만 다를 뿐이고 수학적으로는 `categorical_crossentropy`와 동일합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 충분히 큰 중간층을 두어야 하는 이유\n",
    "\n",
    "앞서 언급한 것처럼 마지막 출력이 46차원이기 때문에 중간층의 히든 유닛이 46개보다 많이 적어서는 안 됩니다. 46차원보다 훨씬 작은 중간층(예를 들면 4차원)을 두면 정보의 병목이 어떻게 나타나는지 확인해 보겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/20\n",
      "7982/7982 [==============================] - 0s 61us/step - loss: 2.6576 - acc: 0.3773 - val_loss: 1.9684 - val_acc: 0.5280\n",
      "Epoch 2/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 1.6652 - acc: 0.6211 - val_loss: 1.5409 - val_acc: 0.6230\n",
      "Epoch 3/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 1.3324 - acc: 0.6706 - val_loss: 1.3951 - val_acc: 0.6790\n",
      "Epoch 4/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 1.1420 - acc: 0.7269 - val_loss: 1.3235 - val_acc: 0.6970\n",
      "Epoch 5/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 1.0109 - acc: 0.7457 - val_loss: 1.2704 - val_acc: 0.7010\n",
      "Epoch 6/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.9098 - acc: 0.7580 - val_loss: 1.2778 - val_acc: 0.7070\n",
      "Epoch 7/20\n",
      "7982/7982 [==============================] - 0s 39us/step - loss: 0.8304 - acc: 0.7742 - val_loss: 1.2595 - val_acc: 0.7080\n",
      "Epoch 8/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.7692 - acc: 0.7895 - val_loss: 1.2664 - val_acc: 0.7150\n",
      "Epoch 9/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.7130 - acc: 0.8002 - val_loss: 1.2920 - val_acc: 0.7110\n",
      "Epoch 10/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.6626 - acc: 0.8109 - val_loss: 1.3244 - val_acc: 0.7020\n",
      "Epoch 11/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.6172 - acc: 0.8190 - val_loss: 1.3556 - val_acc: 0.7110\n",
      "Epoch 12/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.5787 - acc: 0.8326 - val_loss: 1.3915 - val_acc: 0.7110\n",
      "Epoch 13/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.5408 - acc: 0.8444 - val_loss: 1.4515 - val_acc: 0.7080\n",
      "Epoch 14/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.5086 - acc: 0.8552 - val_loss: 1.4671 - val_acc: 0.7130\n",
      "Epoch 15/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.4773 - acc: 0.8667 - val_loss: 1.5340 - val_acc: 0.7110\n",
      "Epoch 16/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.4495 - acc: 0.8801 - val_loss: 1.5469 - val_acc: 0.7050\n",
      "Epoch 17/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.4245 - acc: 0.8822 - val_loss: 1.5952 - val_acc: 0.7140\n",
      "Epoch 18/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.4036 - acc: 0.8908 - val_loss: 1.6350 - val_acc: 0.7100\n",
      "Epoch 19/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.3843 - acc: 0.8943 - val_loss: 1.6489 - val_acc: 0.7090\n",
      "Epoch 20/20\n",
      "7982/7982 [==============================] - 0s 38us/step - loss: 0.3656 - acc: 0.8974 - val_loss: 1.7272 - val_acc: 0.7080\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f468a701898>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(4, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "model.fit(partial_x_train,\n",
    "          partial_y_train,\n",
    "          epochs=20,\n",
    "          batch_size=128,\n",
    "          validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "검증 정확도의 최고 값은 약 71%로 8% 정도 감소되었습니다. 이런 손실의 대부분 원인은 많은 정보(46개 클래스의 분할 초평면을 복원하기에 충분한 정보)를 중간층의 저차원 표현 공간으로 압축하려고 했기 때문입니다. 이 네트워크는 필요한 정보 대부분을 4차원 표현 안에 구겨 넣었지만 전부는 넣지 못했습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 추가 실험\n",
    "\n",
    "* 더 크거나 작은 층을 사용해 보세요: 32개 유닛, 128개 유닛 등\n",
    "* 여기에서 두 개의 은닉층을 사용했습니다. 한 개의 은닉층이나 세 개의 은닉층을 사용해 보세요."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 정리\n",
    "\n",
    "다음은 이 예제에서 배운 것들입니다.\n",
    "\n",
    "* N개의 클래스로 데이터 포인트를 분류하려면 네트워크의 마지막 `Dense` 층의 크기는 N이어야 합니다.\n",
    "* 단일 레이블, 다중 분류 문제에서는 N개의 클래스에 대한 확률 분포를 출력하기 위해 `softmax` 활성화 함수를 사용해야 합니다.\n",
    "* 이런 문제에는 항상 범주형 크로스엔트로피를 사용해야 합니다. 이 함수는 모델이 출력한 확률 분포와 타깃 분포 사이의 거리를 최소화합니다.\n",
    "* 다중 분류에서 레이블을 다루는 두 가지 방법이 있습니다.\n",
    "    * 레이블을 범주형 인코딩(또는 원-핫 인코딩)으로 인코딩하고 `categorical_crossentropy` 손실 함수를 사용합니다.\n",
    "    * 레이블을 정수로 인코딩하고 `sparse_categorical_crossentropy` 손실 함수를 사용합니다.\n",
    "* 많은 수의 범주를 분류할 때 중간층의 크기가 너무 작아 네트워크에 정보의 병목이 생기지 않도록 해야 합니다."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}