Week3_권지수_예습과제.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. 모델 훈련\n",
    "\n",
    "## 4.1 선형 회귀\n",
    "<img src='https://miro.medium.com/max/1920/1*GSAcN9G7stUJQbuOhu0HEg.png' width=50%>\n",
    "\n",
    "선형 회귀를 훈련시키는 방법\n",
    "- 직접 공식을 사용하여 파라미터 구하기\n",
    "- 경사하강법(GD)을 사용하여 반복적인 최적화 방식으로 파라미터를 조금씩 바꾸면서 비용 최소화"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1.1 정규방정식\n",
    "공식을 테스트하기 위해 선형처럼 보이는 데이터 생성"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "X = 2 * np.random.rand(100,1)\n",
    "y = 4 + 3 * X + np.random.randn(100, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`np.linalg` 넘파이 선형대수 모듈  \n",
    "`inv()` 역행렬  \n",
    "`dot()` 행렬곱"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3.83135638],\n",
       "       [3.03817169]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_b = np.c_[np.ones((100, 1)), X]\n",
    "theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)\n",
    "\n",
    "#정규방정식으로 계산한 값 확인\n",
    "theta_best"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3.83135638],\n",
       "       [9.90769976]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 예측\n",
    "X_new = np.array([[0],[2]])\n",
    "X_new_b = np.c_[np.ones((2, 1)), X_new] #모든 샘플에 x0 = 1 추가\n",
    "\n",
    "y_predict = X_new_b.dot(theta_best)\n",
    "y_predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAepElEQVR4nO3df5xcdX3v8dcnu1kgIRhJ+FVCiEUMAv4AVusQ1IVdakoRtFILCiKise3DK/gDa/RWbVFzH622sb231+YqtVRK2yvY2j6wwC4sYFmCAQMJ0CoFjIiYAPYCCWSS7Of+cWays5OdnTNzvnPOmTnv5+ORR3bP/DjfPXv2c77z+Xy/32PujoiI9L45WTdARETSoYAvIlIQCvgiIgWhgC8iUhAK+CIiBdGf5s4WL17sy5YtS3OXIiJd75577nnK3Q9J+j6pBvxly5axYcOGNHcpItL1zOzHId5HKR0RkYJQwBcRKQgFfBGRglDAFxEpCAV8EZGCUMAXESkIBXwRkYJQwBcRKQgFfBGRglDAFxEpiKYB38yuMrOtZrZ5hsc+bmZuZos70zwREQklTg//G8DK+o1mdhRwJrAlcJtERKQDmgZ8d78deGaGh/4U+ASgm+KKiHSBtnL4ZnYO8FN3vy/Gc1eZ2QYz27Bt27Z2diciIgG0HPDNbB7waeAzcZ7v7uvcfdDdBw85JPFyziIi0qZ2evjHAC8D7jOzx4AlwL1mdnjIhomISFgt3wDF3TcBh1a/rwT9QXd/KmC7REQksDjDMq8FJoDlZva4mV3a+WaJiEhoTXv47n5Bk8eXBWuNiIh0jGbaiogUhAK+iEhBKOCLiBSEAr6ISEEo4IuIFIQCvohIQSjgi4gUhAK+iEhBKOCLiBSEAr6ISEEo4IuIFIQCvohIQSjgi4gUhAK+iEhBKOCLiBSEAr6ISEEo4IuIFIQCvohIQSjgi4gUhAK+iEhBNA34ZnaVmW01s8012/7YzP7dzO43s2+b2cLONlNERJKK08P/BrCybtvNwInu/mrgh8DqwO0SEZHAmgZ8d78deKZu203uvrvy7V3Akg60TUREAgqRw38f8N1GD5rZKjPbYGYbtm3bFmB3IiLSjkQB38w+DewGrmn0HHdf5+6D7j54yCGHJNmdiIgk0N/uC83sYuBsYNjdPVyTRCRtExMwPg5DQ1AqZd0a6ZS2Ar6ZrQR+D3izu+8I2yQRSdPEBAwPQ7kMAwMwNqag36viDMu8FpgAlpvZ42Z2KfA/gQXAzWa20cy+2uF2ikiHjI9HwX7Pnuj/8fGsWySd0rSH7+4XzLD56x1oi4hkYGgo6tlXe/hDQ1m3SDql7Ry+iPSGUilK4yiH3/sU8EWEUkmBvgi0lo6ISEEo4IuIFIQCvohIQSjgi4gUhAK+iEhBKOCLiBSEAr6ISEEo4IuIFIQCvohIQSjgi4gUhAK+iEiLJiZgzZro/zy/Zz2tpSMi0oJO3D8grXsSqIcvItKCTtw/IK17Eijgi4i0oHr/gL6+cPcP6MR7zkQpHRGRFnTi/gFp3ZPA0rz/+ODgoG/YsCG1/YmI9AIzu8fdB5O+j1I6IiIFoZSOiEhOTUxUC7gL5od4v6YB38yuAs4Gtrr7iZVtBwN/DywDHgPe6e6/CNEgERGZPlQTjn1FiPeMk9L5BrCybtsngTF3PxYYq3wvIiKB1A7VBCzEezYN+O5+O/BM3eZzgb+ufP3XwNtCNEZEJPSM0zRmsHZC7VBNIMjomnZz+Ie5+88A3P1nZnZooyea2SpgFcDSpUvb3J2IFEGrM06rOe5GQxnTmsHaCaXX7GDsi5sY//YzfOr2H/0wxHt2vGjr7uuAdRANy+z0/kSke80047RRgI4TzFt5v8zt3g3f/370g4yNwZ13UiqXKc2dy6fYtT3ELtoN+D83syMqvfsjgK0hGiMixVZNY1SD+GwzTuME81beL3Xu8OCDMDoaBfjbboNnn40eO+kk+PCHoyvaG98IBx4YZJftBvzvABcD/6Py/z8FaY2IZK5ZmqSTWplxGieYd2oGa9vHaMuWqR782Bg8+WS0/Zhj4IILogB/+umweHGYhtZpOtPWzK4FhoDFwM+BzwL/CPwDsBTYAvymu9cXdvehmbYi+dZtOe80Lk71+2jpGD39NNx661SA/9GPou2HHhq9ychI9P/RR8/684Saadu0h+/uFzR4aDjpzkUkX7oq503Utk62b6bgPusx2rEDvve9qTTND37AhP8K4wNvYeh151D63SXRG554Ipg13Vfon00zbUVkrzRy3lmmjFrd/0zBffoxcoYWbYbP/1MU5CcmogfmzoVSiYlLv8bw31xMefccBu41xv4YSq9qvq+dO+Fzn4v+hTxGCvgislenV23MOmXUbP/1F4N9LoBvdkoHPcjYqs2M37iToS1XU/rgWNRbf+1ro0LryAicdhrMn8/4GijvjveJqbqvnTthcjK6ftxxR9TGUBTwRWSaTqZJsk4Zzbb/RheDsWueZPyaxxl69p8pvWMdPPkkJaB0zDFw0QgMf7BhobWVT0zVi+3nPhcF+8nJ8DdDUcAX6UFppU1a3U/WwyRn2/+0i8HOScYv/w6lZ66g9PDDlKBhoXU2pRKsXQvXXQfveEfzY1QqRQH/jjs6c4y0Hr5Ij0krbdLufnKXw9+xA+64g4mrf8Twte+n7H0MsIuxeedQOuOAqSB/wgn7FFrj7CvEMUptlI6IdJe00ibt7qfTI2ua7v91uylRmdG6eqrQWpo7l7HXPMj44vMY+s3FlC75blR8TSBvx0gBX6THpJU2CbGfVHr77vDAA1Nj4cfH4bnnpgqtl10WdcNPO43S/PmEbEbWKax6SumI9KC85vDrX9ux1NOWLVNj4W+5ZWpG68tfHu20wzNaa4X4XSilIyINpZU2abSfOEEuaOqpOqO1GuQffjjafthhUwE+ZqE1tKxTWLUU8EUkqLg990TpjkqhlbExGB1l4gf7M86bGZr3MKUzjoMPfShqRBuF1mZmuphlXYiOSwFfRIKK23NvaZJXdengag++ZkbrxAnvZ7j/K5Qn+xlwGPuUdXRdnfqLGXTP+kMK+CISVKuTjWYMjrWF1tHRaOngBoXW8T+bT3kT7Jns7KgkmPliBslSU2l+OlDAl1R0y0febpS3Y9v28gw//vH0pYN//vNo+8tfDu96VzQW/vTTYdGiaS9LcyRMo33Vblu0KLqlYpyfPe2lJhTwpeOyXj+ll+X12MYqVAYqtIZa/yfOhbPRvqrbFi2Cyy+P//tIe6kJBXzpuKzXT+llXXVst2+Plg6upmk2boxSNwsWRNEzQaE16UiYVi6cM+2rum3NmtZ+H2mP01fAl47L2+STXpLrY7tr1z73aGXXrmj26qmnwh/8QZSmGRycdUZrGimrUBfOVn8fnV6dtJ4mXkkq8pZn7iW5ObbVQmvtPVqrhdaTTppK0VSWDo4j7+sCNXqv0L8PTbySrpKnySd512rAyPTYVguto6PRjNbaQuu73z01o7Wu0BpXp1JW9cc4ZE87z+e6Ar5IjuS1CLvXU09Nv0drfaG1unTw0qVBdteJlFWjY5znQB2KAr5IjuSuCFsttFbTNDMVWkdG4Pjjg89ohc7kuHN3jFOUKOCb2UeA9wMObAIucfcXQzRMpIgyL8LWFlqr92jdtStqzKmnwh/+YdQ9ft3roD+d/mLonnfmxzhDbf/GzOxI4MPA8e7+gpn9A3A+8I1AbRPJnU4XSNMYtTHtZ3iDw+bN05cOfv75qULrRz4yVWidNy98YzKQ9siYPEl6ie4HDjCzXcA84InkTRLJp7Ty653MJU9MwPAZk5R3wsCcXYwd9BuUfnFD9OCxx8KFF0YpmqGhtgut9fvLY2AtQr5+Jm0HfHf/qZl9CdgCvADc5O431T/PzFYBqwCWBirkiGSha3O/1ULr6Cjj3zqW8ouXs4d+ynvmML70PZS+fF7QQmtV7gvQBZQkpfNS4FzgZcB/Af/XzC5092/WPs/d1wHrIBqHn6CtIpnqmtzv9u3Tlg5m48Zo+4IFDL36gwzc7ZQnnYGBfob+928R9BZPNbr2AtnDkqR0RoBH3X0bgJldD5wKfHPWV4l0oWpqYu3aaAmYXKUoqoXW2qWDawutV165d0Zrqb+fsZTSLF1zgSyQJAF/C/AGM5tHlNIZBjSNVnpO1qmJffLgXlNorS4d3EKhNc27YRW1OJpXSXL4683sW8C9wG7gB1RSNyK9JMvUxN6LzU5noG8PY0NXUrrvq7B1a/SEDhRaQypqcTSvEo3ScffPAp8N1BaRXMokNfHUU3DLLYx/aS7lF94aFVknnfGJ/SideyYTR5/P+O7TGHrbQgVUiU0zbaUteR1u1wmppCYaFFqH5g0zMOcsysxhYKCPoRtXM2E2lWL686kUU15/J3ltVxEp4EvL0s5p5yFgBE9N7NoFd989NeGpvtD6+c/D8DClwUHGvt8/7eefac11yOcQyGbnSh5+t0WigC8tSzOn3emLS2oBp1porV06uFpoPfnkWQut9RebmVJMeR0COVu7si6GF5ECvrQszZx2JwNZxwPOY49NXzq4ttB60UVTSwcffHBLb9soxRT3d5Jmr3q2cyWvF6lepoAvLUtzuF3Si8tswS14wKkUWvcG+UceibYffjiceebU0sFHHZVgJ5H6Xn/c30naverZ2qVx+ulTwJe2dMNY7mbBLXHAqRZaa5cOBjjooOjNLrssCvKvfGVHlg6uF+d3kkWvulG7NE4/fQr4knvtXlyaBbeWA05toXV0FO66a8ZCK4ODqS0d3Kq89ao1Tj9d+TwrRQKIE9xmDTjusGnT3pE0E7e8wPgLr2eI2yidUo4KrSMjsGJF1ywdrF51sSngS8+KE9z2yfE3KLROHPVOhnddT3lOPwP7GWN/bl0bLNWrLi4F/AIrwhjo2YJbtDa8Uy47A7abscPeRemJ66IHDz8cfvVXoxTN8DDj3zyK8u/DnkmNKJHupYDfIXkPpoUdA/3883tntI7/7dGUX/ydaNkCjPGXnEPp9940Y6E1b7lvkXYo4HdAFsG01QtMYcZAVwut1ZE0NYXWoRM/wMBTTnmPM7BfP0Nff0/DteHzlvvOe4dC8kkBvwPSDqbtXGB6tsc6OTl96eDbb58+o/WjH40O1ooVlObNi702fJ4CbGE/nUliCvgdkHYwbecCk7ceayKPPTbVg6+d0fqKV8BFFzFx1DsZ3/F6hs6at8/PGaeAmbcA2yufzvJ0ES0KBfwOSDuYtnuB6drRGtu27b1HK2Nj02a0Tpz0u4wfeDZDFy6h9LbDpgfrL7cXrPMWYHvh01neLqJFoYDfIWkG07QuMJn1yGoKrYyOwn33RdurM1ovvxyGh5n4r1cyPGJRELlh6pgkDdZ5C7BZfjoLdQ7k7SJaFIUO+L30kbLTF5hUe2SzFFpZsSKa0ToyAqecMm1G6/gMywaHCNZpBNhWz8UsPp2FPAfydhEtisIG/E4GsF66kFR1tEc2W6H1lFOmFVrrZ7TWHuuZgkioYN3JANst6Y2Q50BP1ZC6SGEDfoiTd6bA3i1/vK1K0iOb8QL46KNTN/8YG4vy8rC30Lr3Hq11SwfXvhfse6xnCiJ5r1V0S3ojdK8877+XXlTYgB9i2d2ZAnu3/PG2qtUeWTUwL1oUpdjL5cpNuFd+idLm/zNVaD3iCHjLW/bOaJ1t6eD6Y37xxfse69Wru+94d0t6Q73y7pco4JvZQuBrwImAA+9z94kQDeu0pCdvo8DeLX+87YjbI4sCs1Pe6RiTTE4ak/RR3uOM37iT0spXRVeBkRE47rjYSwfXH3NI71h3Mk3XTYFUvfLulrSH/xXgX939PDMbALpjycCKJCdvo8DeTX+8EDCQ7doF69dHSxb81eGUX7iUPfQzh0n6zDH2MDAwh6GbPw1vbO+0qz/m73lP9C+N0UmdTtMpkEoa2g74ZnYQ8CbgvQDuXgbKYZqVf7MF9m75400UyCYnpy0dzG23RTcEMWNo+XsZ6L+E8uQkA/v1sXat8fTTyYNyo2Pe6WPdq2k6KZ4kPfxfBrYBf2VmrwHuAS5z9+21TzKzVcAqgKVLlybYXf50U2Cf6cLUciB79NHpM1qrhdbly6OE+vAwDA1ROvjg2EsWtCqLY97LaTopFnP39l5oNgjcBaxw9/Vm9hXgWXf//UavGRwc9A0bNrTXUmnLbL34pj38bdum36P10Uej7UccMXV/1uFhWLIk9Z+rE2ZLb/XiUFvpHmZ2j7sPJn2fJD38x4HH3X195ftvAZ9s5Q30R9R5s/Xi90mRvOp5uOH2qTRNdUbrS14SPaE6Hr6FQmu3aHbx65ZPcyKzaTvgu/uTZvYTM1vu7v8BDAMPxn19r45Xz5tZ0xG7dlHas57SzlH4RGVG6+7dsN9+0SSnL3wh+iXVzWhtphsv5MrTSxEkHaXz34BrKiN0HgEuiftC/YGlY1ov/k2TlOZtgj+pmdFaKbRyyinw8Y9PzWg94IC29tetF3Ll6aUIEgV8d98ItJVXSvMPrBt7nME8+iilzaOU7huDP52h0Fqd0frSlwbZXVoX8tC/024bTivSjsxm2qa5wmM39jjbVi20VkfT1BZaV67sSKG12Xo2oXXqd6o8vfS6TJdWSOMPrJUeZ1d+Enj++Sg1Ux1Jc//90faUCq21wbe/Hy65BNauJci4+0aUDhRpT8+vpRO3x9k1nwTK5b0zWvcuHVxfaB0ZiW7n10KhtV21wXfPHvjLv4T99+/s8VO+XaQ9PR/w46aOcttrrJ3RWl9oHRwMUmhNohp8X3wR3KN/nT5+raQDu/JTm0iH9HzAh3ipoxC9xmDB5ZFHpi8d/NRT0fbly+G97907o7VRoTXNIFcNvldfDVddFV0w0+h1x/mdds2nNpGUFCLgx5G0iJwouGzdOjWjtVJoneANjC84h6EVx1M6/+jYhdYsglw1+DZbyCzt3nZuP7WJZKTQAb8+ACUpIrcUXKqF1upImtpC6+mnM/H2P2L4L36D8g5j4DZj7DNQijmoJssgN9vxy+JCpFy/yHSFDfihA9CswaW20Do6Gn1dW2j94hejxlQKreNroLyrvaCd1yCXxYVIY+tFpitswA8dgGac0frlSg9+pkLryAiceuqMhdYkQTuvQS6rC5HG1otMaXu1zHbkabXM4CmGaqF1dDTKx9cWWqsrS7Ywo7UXR5f04s8kkoZQq2UWNuBDwgBUW2gdHYXHHou2/9IvRcF9ZATOOKNnlg4WkezkYXnkrtfSx/0mhVY+9rEoyC9f3nNLB4tIb+jpgJ+oBz9bofW00/YptIqI5F3PRqqWc/STk1GvvToWvlponTNnaungWQqtIiJ517MBP9YonEcemX6P1mqh9bjjYs1oFRHpJj0b8OuHAS5aBGs+/TxD+01Q2vL3UZCvLbSeddbU0sFHHpll0zOjUTQiva1nA36pBGPf2c743/yERT+9j8t/+xzKvj8DrGBs/h9ROvO1UwuPqdCqdWdECqC3An610FpJ05TWr6e0ezdr+v47ZT+PPfRR7utjfPWNlD49J+vW5orWnRHpfd0d8GsLraOjcMcd0wutV1zBxBFvZ8v9J9F3dR/sgYEBY+iMYvfmZ5LXJRlEJJzuC/jNCq3Ve7QuXLjP3Zg+8IFoRUf1XPeV1yUZRCScxAHfzPqADcBP3f3s5E2qU53RWg3y1ULrkUc2LbTWpikAli7tzkCWVjFV686I9LYQPfzLgIeAgwK8Fzz33PQZrZs2RdsXLoxmtLZQaA2Vpshy9IqKqSISSqKAb2ZLgF8HvgB8tK03KZej+7JW8/B33z19RuuaNVMzWvv6WnrrEGmKrAOuiqkiEkrSHv5a4BPAgkZPMLNVwCqApUuXMvFvk4z/3ZMM+a2U/vObUW9+x46o0Do4CFdcMTWjdf/9EzYveZoi64CrYqqIhNJ2wDezs4Gt7n6PmQ01ep67rwPWAbzyoCN8+LQXKXMoA7ydsaP/hdL73jc1o3Xhwnab05Y4qZqsA66KqSISStvLI5vZGuAiYDewP1EO/3p3v7DRa5b0LfEnJ3/MHvro63OuvNJYvbqt3SfWSqpGM1BFJEuhlkdue/aRu6929yXuvgw4H7hltmAPsODYwxk4oI++vsp4+KF29z7dxESU6p+YiP+amVI1jZRKsHp1uGDfTntFRJJKdRz+/APDpyfaLapmlarJuggsIsUVJOC7+zgwHue5rRZRm6VT2i2qZpUbz7oILCLFleuZtnF6w0lv+J12sM26CCwixZXrgD9bb7i2599No1g06kZEspLrgN+oNzxTzz+r0T7t0BIGIpKFXAf8Rr1h5cFFRFqX64APM/eGlQcXEWld7gP+TNrNg1fz/osWwdNPK4cuIsXSlQEf2hveOTwMO3dG902ZMydan03j4EWkKApzn79q3n9yMvp+crL5DFsRkV6S+4AfahmCat5/TuUnnjNH+X8RKZZcp3RCLkNQm/dXDl9EiijXAT/08EuNfxeRIks9pdNKiqaaholW11T6RUQkiVR7+Nu3t74G/dq1Sr+IiISQasB/7rl4KRotISwiEl6qKZ0FC+KlaFq5OYmIiMST7g1Q5sebIaulE0REwkt9lE6ckTJaQlhEJLzcDsvUEEoRkbByP9NWRETCUMAXESmItgO+mR1lZrea2UNm9oCZXRayYSIiElaSHP5u4GPufq+ZLQDuMbOb3f3BQG0TEZGA2u7hu/vP3P3eytfPAQ8BR4ZqmIiIhBUkh29my4CTgPUzPLbKzDaY2YZt27aF2J2IiLQhccA3swOB64DL3f3Z+sfdfZ27D7r74CGHHJJ0dyIi0qZEAd/M5hIF+2vc/fowTRIRkU5IMkrHgK8DD7n7n4RrkoiIdEKSHv4K4CLgDDPbWPl3VqB2iYhIYG0Py3T37wEWsC0iItJBmmkrIlIQCvgiIgWhgC8iUhAK+CIiBaGALyJSEAr4IiIFoYAvIlIQCvgiIgWhgC8iUhAK+CIiBaGALyJSEAr4IiIFoYAvIlIQCvgiIgWhgC8iUhAK+CIiBaGALyJSEAr4IiIFoYAvIlIQCvgiIgWRKOCb2Uoz+w8ze9jMPhmqUSIiEl7bAd/M+oD/BfwacDxwgZkdH6phIiISVpIe/uuBh939EXcvA38HnBumWSIiElp/gtceCfyk5vvHgV+pf5KZrQJWVb7daWabE+wzLYuBp7JuRAxqZzjd0EZQO0PrlnYuD/EmSQK+zbDN99ngvg5YB2BmG9x9MME+U6F2htUN7eyGNoLaGVo3tTPE+yRJ6TwOHFXz/RLgiWTNERGRTkkS8L8PHGtmLzOzAeB84DthmiUiIqG1ndJx991m9iHgRqAPuMrdH2jysnXt7i9lamdY3dDObmgjqJ2hFaqd5r5P2l1ERHqQZtqKiBSEAr6ISEEECfjNlliwyJ9VHr/fzE6O+9qQYrTz3ZX23W9md5rZa2oee8zMNpnZxlBDpBK0c8jM/l+lLRvN7DNxX5tyO6+oaeNmM9tjZgdXHkvleJrZVWa2tdH8jxydm83amZdzs1k783JuNmtnHs7No8zsVjN7yMweMLPLZnhO2PPT3RP9IyrY/ifwy8AAcB9wfN1zzgK+SzR2/w3A+rivDfUvZjtPBV5a+frXqu2sfP8YsLgTbWujnUPAv7Tz2jTbWff8twK3ZHA83wScDGxu8Hjm52bMdmZ+bsZsZ+bnZpx25uTcPAI4ufL1AuCHnY6dIXr4cZZYOBe42iN3AQvN7IiYrw2l6b7c/U53/0Xl27uI5hakLckxydXxrHMBcG2H2tKQu98OPDPLU/JwbjZtZ07OzTjHs5FcHc86WZ2bP3P3eytfPwc8RLSCQa2g52eIgD/TEgv1jW70nDivDaXVfV1KdGWtcuAmM7vHouUiOiVuO0tmdp+ZfdfMTmjxtSHE3peZzQNWAtfVbE7reDaTh3OzVVmdm3FlfW7Glpdz08yWAScB6+seCnp+JllaoSrOEguNnhNreYZAYu/LzE4n+qM6rWbzCnd/wswOBW42s3+v9CKyaOe9wNHu/ryZnQX8I3BszNeG0sq+3gr8m7vX9rjSOp7N5OHcjC3jczOOPJybrcj83DSzA4kuOJe7+7P1D8/wkrbPzxA9/DhLLDR6TprLM8Tal5m9GvgacK67P13d7u5PVP7fCnyb6CNVJu1092fd/fnK1zcAc81scZzXptnOGudT95E5xePZTB7OzVhycG42lZNzsxWZnptmNpco2F/j7tfP8JSw52eAwkM/8AjwMqaKByfUPefXmV54uDvua0P9i9nOpcDDwKl12+cDC2q+vhNYmWE7D2dq0tzrgS2VY5ur41l53kuIcqnzszielX0so3GRMfNzM2Y7Mz83Y7Yz83MzTjvzcG5WjsvVwNpZnhP0/Eyc0vEGSyyY2W9XHv8qcANRtflhYAdwyWyvTdqmBO38DLAI+AszA9jt0Up6hwHfrmzrB/7W3f81w3aeB/yOme0GXgDO9+gsyNvxBHg7cJO7b695eWrH08yuJRo5stjMHgc+C8ytaWPm52bMdmZ+bsZsZ+bnZsx2QsbnJrACuAjYZGYbK9s+RXRx78j5qaUVREQKQjNtRUQKQgFfRKQgFPBFRApCAV9EpCAU8EVECkIBX0SkIBTwRUQK4v8D0YUjiG5ELq4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.plot(X_new, y_predict, 'r-')\n",
    "plt.plot(X, y, 'b.')\n",
    "plt.axis([0, 2, 0, 15])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 사이킷런에서의 선형 회귀 수행\n",
    "`intercept` 절편  \n",
    "`coef` 회귀계수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([3.83135638]), array([[3.03817169]]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(X, y)\n",
    "\n",
    "lin_reg.intercept_, lin_reg.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3.83135638],\n",
       "       [9.90769976]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin_reg.predict(X_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LinearRegression 클래스는 `scipy.linalg.lstsq()` 함수를 기반으로 하기 때문에 이를 직접 호출할 수 있음  \n",
    "∂ = X+ y  \n",
    "*X+은 X의 유사역행렬"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3.83135638],\n",
       "       [3.03817169]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "theta_best_svd, residuals, rank, s = np.linalg.lstsq(X_b, y, rcond = 1e-6)\n",
    "theta_best"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`np.linalg.pinv()` 유사역행렬 계산  \n",
    "유사역행렬은 특잇값 분해(SVD)라 부르는 표준 <u>행렬 분해 기법</u>을 사용하여 계산됨"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3.83135638],\n",
       "       [3.03817169]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.pinv(X_b).dot(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "정규방정식은 (n+1)x(n+1) 크기의 X'X 역행렬을 계산함  \n",
    "그러나 특성 수가 늘어나면 매우 느려진다!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 경사 하강법(Gradient descent)\n",
    "특성, 훈련 샘플이 매우 많아 메모리에 담을 수 없을 때 사용하며, 비용 함수를 최소화하기 위해 반복하여 파라미터를 조정함\n",
    "\n",
    "<img src='http://rasbt.github.io/mlxtend/user_guide/general_concepts/gradient-optimization_files/ball.png' width=50%>\n",
    "\n",
    "- 임의의 값에서 그레디언트가 감소하는 방향으로 진행(0이 되도록)\n",
    "- 학습 스텝 크기(학습률, learning rate)은 비용 함수의 기울기에 비례"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = 'https://www.researchgate.net/profile/Md-Saiful-Islam-15/publication/338621083/figure/fig4/AS:847811214069760@1579145353037/Gradient-Descent-Stuck-at-Local-Minima-18.ppm' width=50%>\n",
    "\n",
    "- 그러나 무작위 초기화 때문에 전역 최솟값이 아닌 지역 최솟값에 도달하여 멈출 수도 있음\n",
    "- 선형 회귀는 볼록 함수이기 때문에 지역 최솟값이 없고 하나의 전역 최솟값만 가지고 있음\n",
    "\n",
    "### 경사하강법을 사용할 땐 모든 특성이 같은 스케일로 만들어야 함! 그렇지 않으면 수렴에 오랜 시간이 걸림!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.1 배치 경사 하강법\n",
    "경사 하강법을 구하기 위해서는 **세타가 변경될 때 비용 함수가 얼마나 바뀌는지 계산**해야 함(=편도함수)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "해당 알고리즘 구현"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3.83135638],\n",
       "       [3.03817169]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eta = 0.1 #학습률\n",
    "n_iterations = 1000\n",
    "m = 100\n",
    "\n",
    "theta = np.random.randn(2, 1) #무작위 초기화\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    gradients = 2/m * X_b.T.dot(X_b.dot(theta)-y)\n",
    "    theta = theta - eta * gradients\n",
    "    \n",
    "theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "적절한 학습률을 찾기 위해서는 *그리드 탐색*을 사용하지만 오래 걸리는 모델을 막기 위해 <u>반복 횟수를 제한함</u>  \n",
    "반복 횟수를 크게 지정하고, 그래디언트 벡터가 아주 작아지면(허용오차보다 작아짐) 경사하강법이 최솟값에 도달한 것으로 보고 알고리즘 중지"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.2 확률적 경사 하강법\n",
    "- 매 스텝에서 **한 개의 샘플을 무작위로 선택**하고 하나의 샘플에 대한 그레디언트를 계산(배치 경사는 매 스텝에서 전체 훈련 세트를 사용) \n",
    "- 무작위라 배치  경사 하강법보다 불안정하지만 비용 함수가 불규칙할 때 전역 최솟값을 찾을 가능성이 높음\n",
    "- 학습률을 점진적으로 감소시키는 것으로 딜레마를 해결할 수 있음"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`학습 스케쥴` 반복에서 학습률 결정  \n",
    "학습 스케쥴을ㅇ 사용한 확률적 경사 하강법의 구현"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 50\n",
    "t0, t1 = 5, 50 #학습 스케줄 파라미터\n",
    "\n",
    "def learning_schedult(t):\n",
    "    return t0/(t+t1)\n",
    "\n",
    "theta = np.random.randn(2, 1) #무작위 초기화\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for i in range(m):\n",
    "        random_index = np.random.randint(m)\n",
    "        \n",
    "        xi = X_b[random_index:random_index+1]\n",
    "        yi = y[random_index:random_index+1]\n",
    "        \n",
    "        gradients = 2*xi.T.dot(xi.dot(theta)-yi)\n",
    "        eta = learning_schedult(epoch*m+i)\n",
    "        theta = theta - eta * gradients\n",
    "        \n",
    "# m: 반복(epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3.82377762],\n",
       "       [3.01868682]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**? 훈련 샘플 섞기**  \n",
    "확률적 경사 하강법을 사용할 때 훈련 샘플이 IID(independent and identically distributed)를 만족해야 전역 최적점을 향해 진행한다 볼 수 있음.  \n",
    "이를 위해 훈련 세트를 섞음(입력 특성과 레이블을 동일하게 섞어야 함)  \n",
    "만약 레이블 순서대로 정렬된 샘플을 그대로 사용한다면 한 레이블에 최적화될 것"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`SGDRegressor` SGD 선형회귀를 사용하기 위해 제곱 오차 비용 함수를 최적화하는 클래스 사용  \n",
    "- max_iter: epoch\n",
    "- tol: 한 에포크에서 파라미터보다 적게 손실이 줄어들 때까지 실행\n",
    "- eta: 학습률\n",
    "- penalty: 규제"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDRegressor(eta0=0.1, penalty=None)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import SGDRegressor\n",
    "\n",
    "sgd_reg = SGDRegressor(max_iter=1000, tol=1e-3, penalty=None, eta0 = 0.1)\n",
    "sgd_reg.fit(X, y.ravel())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([3.81883798]), array([3.02180601]))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_reg.intercept_, sgd_reg.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.3 미니배치 경사 하강법\n",
    "- `미니배치`라는 임의의 sample set에 대해 그레이디언트를 계산\n",
    "- 행렬 연산에 최적화된 하드웨어, GPU를 사용하여 얻는 성능 향상이 장점\n",
    "- 미니배치를 크게 하면 SGD보다 덜 불규칙하게 움직임\n",
    "- 지역 최솟값에서 빠져나오기는 **더** 힘들 수 있음"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 다항 회귀\n",
    "비선형 데이터에 사용"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2차방정식으로 비선형 데이터 생성하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = 100\n",
    "X = 6*np.random.rand(m, 1) - 3\n",
    "y = 0.5 * X**2 + X+2 + np.random.randn(m, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`PolynomialFeatures`를 사용해 훈련 데이터 변환  \n",
    "degree: 차수  \n",
    "주어진 차수까지 특성 간의 모든 교차항을 추가"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2.51850669])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "poly_features = PolynomialFeatures(degree=2, include_bias = False)\n",
    "X_poly = poly_features.fit_transform(X)\n",
    "X[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2.51850669, 6.34287595])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_poly[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "X_poly는 원래 특성 X와 제곱을 포함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([2.77555756e-17]), array([[1.00000000e+00, 1.82395305e-17]]))"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(X_poly, X)\n",
    "lin_reg.intercept_, lin_reg.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4 학습 곡선\n",
    "- 모델의 과적합을 판별하는 방법 중 하나(다른 하나는 교차 검증)\n",
    "- 훈련 세트와 검증 세트의 모델 성능을 훈련 세트 크기(반복)의 함수로 나타냄"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "def plot_learning_curves(model, X, y):\n",
    "    X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=10)\n",
    "    train_errors, val_errors = [], []\n",
    "    \n",
    "    for m in range(1, len(X_train)):\n",
    "        model.fit(X_train[:m], y_train[:m])\n",
    "        y_train_predict = model.predict(X_train[:m])\n",
    "        y_val_predict = model.predict(X_val)\n",
    "        train_errors.append(mean_squared_error(y_train[:m], y_train_predict))\n",
    "        val_errors.append(mean_squared_error(y_val, y_val_predict))\n",
    "\n",
    "    plt.plot(np.sqrt(train_errors), \"r-+\", linewidth=2, label=\"훈련 세트\")\n",
    "    plt.plot(np.sqrt(val_errors), \"b-\", linewidth=3, label=\"검증 세트\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZQU5bk/8O/DIsuwOwPDNgzqXETZmRBRMUQTA1yNVy8uMRqX3Iueo0Z/iUZjogwxnmiO8cYtIZ5fFDVeFJeo4ZJI4i+5Cm4MqwOIw87IACPLwDAyMDPP74+n61RPT+/T3dVd/f2cU6fr7aqufqan+6m33nrrLVFVEBFR7uvkdQBERJQaTOhERD7BhE5E5BNM6EREPsGETkTkE128euPCwkItLS316u2JiHLSypUrv1DVonDLPEvopaWlqKys9OrtiYhykojsiLSMTS5ERD7BhE5E5BNM6EREPsGETkTkE0zoREQ+wYROROQTOZ/Qjx0DPv0U4KCRRJTvcjqhNzUB48cDo0cD99/vdTRERN7K6YS+fDnw2Wc2v2CBp6EQEXkupxP6qlXufE0NcOCAd7EQEXktZkIXkeEi8g8R2Sgi60Xk9jDrTBeRehFZE5gy0gCyenXb8iefZOJdiYiyUzxjuTQD+JGqrhKR3gBWisjfVHVDyHrvqepFqQ8xsnAJ/Wtfy2QERETZI2YNXVVrVXVVYP4IgI0AhqY7sFgaG4FNm9o+t26dN7EQEWWDhNrQRaQUwEQAH4VZPFVE1orIX0TkzBTEFtW6dUBra/vniIjyVdzD54pILwCvAbhDVQ+HLF4FYISqNojILABvACgLs405AOYAQElJSdJBA+2bWwCgqsqSfKecPtVLRJScuFKfiHSFJfMXVfX10OWqelhVGwLzSwB0FZHCMOs9rarlqlpeVBR2fPa4BfdwcRw9Cmzb1qHNEhHlrHh6uQiAPwDYqKqPRlinOLAeRGRKYLv7UxloqOAaekGBO8+eLkSUr+KpoZ8D4FoA5wd1S5wlIjeLyM2BdWYDqBKRtQAeB3CVavouxj9xom3inj3bnWc7OhHlq5ht6Kq6DIDEWOdJAE+mKqhYNm4Ejh+3+ZIS66r43HNWZkInonyVk6cPg5tbJk0Cxo1zy2xyIaJ8lfMJfeJE4Iwz3J4t1dXWR52IKN/kZEIP7uEycSLQowdQFugkqQpsCL2GlYgoD+RcQm9tBdasccsTJ9pjcLML29GJKB/lXELfuhU4csTmCwuBoYFBCJjQiSjf5VxCD20/l0D/m7Fj3ed5YpSI8lHOJ3RHcA197Vreko6I8k9OJ/RJk9z5ESOA3r1tfv9+YM8em1cFmpszFx8RkVdyKqGrtu/h4ujUCRgzxi0vXQrcdx9QXGxt7cuWZS5OIiIvxD3aYjaorQX27bP5Xr2A005ru3zcOOCDD2z++uvbLps/Hzj33LSHSETkmZyqodfUAIMG2fz48e2HyQ0+MRpqy5b0xUVElA1yqoY+ZYq1jdfWAgcPtl8+bVrb8sSJbps7h9UlIr/LqYTuGDzYplDjxgEvvGD90GfPBiZPBrp3t5Oie/fakAA9e2Y+XiKiTMjJhB7NNde0LZeU2MVIALB9u437QkTkRznVhp6MkSPdeTa7EJGfMaETEfkEEzoRkU8woRMR+QQTOhGRTzChExH5hO8T+sCBbt/z+vrwFyQREfmB7xO6CFBa6pZZSyciv/J9QgfY7EJE+YEJnYjIJ5jQiYh8wndjuYQTLaG3tgJHjwJNTcDx48CJE8Dw4e2H5iUiynZ5kbYiJfSqKhu8q08foKgIGDrUTqCefjpw5EjGwyQi6pC8S+jbt7s3kH7gAeDzz9uvX10NvPZaRkIjIkqZvEjoffsC/fvb/LFjdpOMY8eAJUvcdfr3t9vaOaqqMhsjEVFH5UVCB9o3u/z970BDg5XLyoD9+4EFC9x1Pvkko+EREXVYzIQuIsNF5B8islFE1ovI7WHWERF5XEQ2i8g6EZmUnnCTF5zQt24FXn/dLV92mV2ANGaM+xxr6ESUa+KpoTcD+JGqjgZwFoBbRCT0vj8zAZQFpjkAfpfSKFMgOKFXVwNvvumWL7vMHk87DejWzeZ37wYOHMhcfEREHRUzoatqraquCswfAbARwNCQ1S4B8LyaDwH0E5Ewd/30TnBC/+Mf3WQ9bBhQXm7znTu3vUUda+lElEsSakMXkVIAEwF8FLJoKIBdQeUatE/6EJE5IlIpIpV1dXWJRdpBoU0ujksvbdvnnM0uRJSr4k7oItILwGsA7lDVw6GLw7xE2z2h+rSqlqtqeVFRUWKRdlBwQg/mNLc4mNCJKFfFldBFpCssmb+oqq+HWaUGwPCg8jAAuzseXuoEj7joOPlk4Nxz2z43dqw7z54uRJRL4unlIgD+AGCjqj4aYbW3AHwv0NvlLAD1qlqbwjg7rHt3YHBIq/4llwBdQgY/CK2ha7vjDCKi7BTPWC7nALgWwCcisibw3L0ASgBAVecDWAJgFoDNABoB3JD6UDtu5EigNmg3E9rcAthJ0r597WYYhw5Zb5eh7c4GEBFln5gJXVWXIXwbefA6CuCWVAWVLiNHAu+/b/O9ewMXXNB+Hac/+vLlVv7kEyZ0IsoNeXOlKACceqo7/6//as0w4fDEKBHlorxK6FdfDfTrB/ToAdx1V+T10pHQVYFVq4B33rEhe4mIUi0vxkN3jBoF1NTYBUSRaudA254uHUnoqsCKFcArrwCvvmojPQLAbbcBjz+e/HaJiMLJqxo6ABQURE/mAHDmme78+vVAS0ti76FqIzlOmAB89avAI4+4yRwAnnwSWLkysW0SEcWSdwk9HoWFQHGxzR871vbK0lg+/hg4/3xro1+3Lvw6qsDtt7NLJBGlVl41uSRizBgbNx2wZpeyMqutP/440NhowwWIWHt4XR2wdy+wb1/7G2b07AlccQVw+eXAiBHAxIl2m7vly4FFi4Arr8z830ZE/sSEHsHYsTZmOmBdF089FZg+HTh4ML7Xd+kCzJkD3HefW9sHgB/8APj1r23+xz8GLr7Ykj4RUUexySWC4J4uS5YA3/xmfMm8c2erkW/YADz1VNtkDliCd4ax2bnT2teJiFKBNfQIghP6R0FjS/bpA/zqV3Zi1WkDP/lkYNAgmwYOtG6RkfTtCzz4oNXeAeChh+zOSb172y3wRo0CZs605hwiokSIenRmrry8XCsrKz1573gcPdr2HqOA9ZD529+AqVM7tu2WFhuDfc2a8MvvvtsSPRFRKBFZqarl4ZaxySWCggLglFPccvfuwOLFHU/mgDXLPPGEPYbz8MO2nIgoEWxyieKyy6yN+6STgDfesJOiqXLuuXbR0QcfAEeO2PTuu8B779ny22+30SFnz07dexKRv7HJJYqmJrtUf8wYoKQk/e/X2GgDhn34oZW7dQOWLgXOOy/9701EuYFNLknq1g2YNSszyRyw7ot//jPwL/9i5aYm4NvfTuzCJiLKX0zoWaawEHj7bbe7Y3299aohIoqFCT0LlZYCL77oll94wW62QUQUDRN6lvr6191RHxsbgeee8zYeIsp+TOhZSgS49Va3/NRTHEediKJjt8Us9t3v2ngv9fVAdbVd1PStb2Xu/Zua7H0//dSmhgagf3+7SUj//nZrvjPOsHmidDp82DoNhN7Undrix5PFCgqAG24AfvMbKz/5ZPIJ/cgRYNMmoLnZyqo239Dg9oOvq7MeNc60Y0d8RwXFxTaG/De+Yf3now19QNnn4EFg82abDh92n1e1/39zszsVFgKnn25DVJx8cvjtqQK7dtn3rVMn6y3WrZt9n8vKgK5do8dz6JBVXj7+2EY6raqyG9P06GHfr5/+tP1V3F47dszirq+3qXdv+1szvQNiP/QsV13tdmMUAbZssZtdhzp82IbkbWiwoQVaWixJr1xp/drXr8/M+Ounngr8/vfhb8BNHdPUZEn30CEbmuLoUUseu3bZQG87d9owziKWSDt1suRZXGwXqQ0ZYlc819S462/dChw4kFw8hYXAsGE22FxRkY1zVF0NrF4deZvduwOTJtmNXyZObHuzma1bbSC85cuj31Rm8GC7mvq737W/MRZV++wOH7bP6/Bhm44dc6fmZhuHacgQO/Ls2xfYvx+orbWprs59bX29lWtqbNq1y353obp1s4rO+PHAgAHAl1/ae335pU0PPNB2zKh4ReuHzoSeA2bOBP76V5u/6y7rxtjaauO1/8//AH/6k10Adfx4at9XxHrcjB5ttbIBAyyZHDxoP9itW60ppqmp/Wuvuw742c8smRQU5PZgY42N9oN1kuixY22XO0c/n35qjw0NttM97TSb+vWzBLBvn01dutiwEqecYjvAAQPcnW1rK/DFF5YkamrsKGn9ehvCedOmxO+e5RedOrU/WhwyxD0adO5N0NLiPjrJ89ix7LyZzNKlNopropjQc9zixTZuOmDDEPTrZ7WHRH7cnTpZUu7Vy778IjaWTK9ednjYu7dtd+RIN9mUlsZuPmlpscS+dKkdCtfXt1+nWzc7PB861G7yUVpqjyNH2lRampox4VtbrYa6fbvtZPr2talPH/u8qqtt2rLFkuahQzYdPmw7ncJCi7NvX9tZbt9uU7I12FzRvbu78ykqarvz7dzZdkBdutjzn3/u7rhCd2zB+va1Xlpdu9r/oqnJdmo7d8YXU3k5cOGFdhvHMWNsx/fyyzZwXW1tx/7edOjSxX4/zvfNqcFH8+abduFgopjQc1xLi7XHbdsWfb0JE+yL37mzTV272knLr37VfiDpbnesrbU2zldeSfy1gwZZjau42KaioraDlznNCM7j8eOWpJ1p926rzYY7WvCTkSPtsyoosP9nr162oywpsWnIEPvcWlttamy0ndPu3TZ9+aU1k5SU2E7VeU08TRfBWlstYe3da8mrrs6O3EpKrCmltDT8UdnevdY2/uGHtnMNTj8FBdZdd8YM+xvDOXIE+OUvgUcfTex/3bWrVVqcnXzv3laJ6N7dJhGLbfdu22k1NNiRk9NcNXCgJew+fez1AwbY5zhsGDB8uHUMCP179++3I6tPPrGdX48eNnXvbo9Tp9q2E8WE7gP//d/WZhisTx9rn7v0UptKSz0JrZ0//9mahbZts5pwrifZrl3tsy4osKlHj7Y/3m7drHY7apRN/frZUYBzRNDYaAlh4EDbUTU12fItW+zoprHRtuNss39/N1EMHWrbHDvWaqrZdjLQK0eP2o5K1Z06dbKdmfPYrZubRCONbBpJc3P29qhhQveJmhqrBTlNAyed5HVEsalawqqrs3bhHTts2r7dEv62bXYY7vS+6agBA9wmnOCTYL162VFOWZkl3yFDLPE6ta6jR23n88UX9hkPHOg2Bw0alHgNlihdoiX0LN0HUTjOIV4uEXFrtqWlwLRp7ddpbnZ7E+zZY48HDrQ9HG9tdbvRqVqN6+ST3WnQIGtC6NMnY38aUdZhQifPdelizQvDh3sdCVFu44EkEZFPxEzoIvKMiOwTkaoIy6eLSL2IrAlM96c+TCIiiiWeGvoCADNirPOeqk4ITD/veFhERDmkoiJ6OUNiJnRVfReAzy+tIKK8k0jSjZSwVYH33wfmzQM++8y92m/evPhen2KpakOfKiJrReQvInJmirZJRBRZokkydHlw0o22rfp6W3fjRuvX2tpq5XnzrA/sOefYeqNGWf/YyZOt/B//Adx5J/CLX9i6y5fb61XbJ/wUiasfuoiUAlisqu2GkhGRPgBaVbVBRGYBeExVyyJsZw6AOQBQUlIyeceOHR0InYjylnPpZXD+EmlbrqiwSdVqz6efbuUNG+wCiBUrgIcestHCLryw/bY2bwYeewx49tm2o2+FG1gmEf3728UOSV4DFK0fOlQ15gSgFEBVnOtuB1AYa73JkycrEVHC9u9XHTbMLhDt1091wgTVSy+18q9/rbpokep771n5yitVBw0KvqA08tS/v2p5ueoVV8S3vjPdd589qqoeOqS6bJmVL7oovtfPnZvQnw+gUiPk1VTU0IsB7FVVFZEpAF4FMEJjbJhXihJRwioqUtNccf31wIIF8a170002JrST0k6csMu0ox0dRCqr2tVzQ4akpYYeT7fFhQA+ADBKRGpE5PsicrOI3BxYZTaAKhFZC+BxAFfFSuZEREmZMcOSozPQyt69NtLXwoVW/spXwr/u/kBvaiepPvts2zJglygvXw48/7xbVgXmz2+7rXB36Jg7N3rZIZLciFzxilR1T/eUU00uCR4SEVEanDihOnGipeB77nGbORyJloN/14msG64cS0dfHwRRmlyY0KNpaVF95pn2/1wvcedC6ZTN368ZM+y3OGKE6tGjsZNkrKQcbVkWfw5M6Ml64QXngEx1zx6vo1Gtrs6unQv5S2NjYkkw3YLfe/du97f45puJv95HoiV0juUSydy5wLXXuuXiYmv/8ugKMNTW2t0qAODBB+1OBUDWXKFGOW7zZuvWB9gg+7fdBixaZCcgNaQrYLB0ft/mzbNhNxctAq680p67+OL4b/OTj7+FSJk+3VPW19D/93+tNlBY6NYMfvpTb2KZO9eNIXiaPbt9jYo1+PyWTK301lvDf7+caeBA1Zkz7fsPqFZWqtbX22tT3dbseP756DH5tPYdD7DJJQn/9m/28dx/vz126mSPS5Z482X6+c/dL/O4cW2/3P36qZaWqk6ebOWWlszHR6kTKylGKyfaZHLLLaqDB9vrzjvPHq+/PnoydaaiInv88Y9VX35ZdfPm9t+/4HjiSfZOn+7Q6YILWFkJYEJP1JYtqiKqJ52kWltrX7wHH7SPa8AAb75Y//7v7pc70pc+XC0mkRNBlB3CHXW1ttp5nH/+08rPPKP66KOqFRVWvusu1TvusPlf/tJquO+80z7BOv/zlhbVDz5wvyvTp6s2NIR/7x/8IL7vW/DUs6dqcbHN33uv6htv2Py+fe4U+l533636rW/Z81272uORI5E/lzzFhJ6o22+3j+a669znWlpUZ81yv7DHj2c2prIye9+bb277PGBXzm3ZovrSS258vXqp/v73bX8EsXoBpBp3GIlpaFD92c/c796NN6p+//varukv0amgQPWss+y749R2e/d2l59/vvUaUY39HXHKLS2qO3Zom5p9MtPQodacc/fd7nOFhe6VnsH4fVJVJvTEHDrk1g5Wr3afj9SOnYkvWUODHTF06aLa1NR2WbgfnNO27kz33qu6YYPNHzqkunix1egA1b/+1Wp/zt+YSon8IPP9x/rDHyafFEOnqVOTe124/0G8CT643NJiNWsn4ScyjRmjunVr+PcmVWVCT8yjj9rHMn16+OVOzQFQffvtzMT04Yf2fmPHtl8Wrl3SafdPZLrqqvY/zljvFcnWrRYroPr006oHD9rz0RJ8PtfG5syxmiqgesop9vjMM/bZzZ9v5Z073R1vtKQabpmz8w6XwGP9z0Ml2vc7VmyRTsjm0/8/QUzo8WputpOLgLX5ReJ86QYOdNvY0+npp+39rrkmsdc1N0dO4Oeea4/OiS1neuwxt8998N/V2hrfj99pMgidRo+2x/nzVV99VXXpUiv/4heq//mfNn/TTaoPPeQ2HTkJLDSWVMhkwoiUBI8fV124sO3/pK4uelKMVU6mRp1KiZywTXcsPsWEHo9EmlTuu8/aHYHMnH2/5RZ7j1/9KvHXOrF9+aXNNzS0XfaTn4T/u087zd2JlJdbmzyg+sgj7omqcJ/NnXfaeiUl9ujsIJOZysrsBJ9zUUkiYiXsRI8Ikt0BrF5t77VokZ2kXLPGyldfbb2TnL/1e99TPXYs/HslWo62zMsjoURr9xQWE3q8XnvN/YHFsnt329ptOk2bpkk38cTbrOEk/Isusrb6WMl2+vT22/vOd+y5zp1V33/fXb5zp+oTT1jZGYsj3ORc2h06idjja6/Fn/Qi/a07d6r+13/Z8qoqtwdIrEQXrRYcau5cOzqqqHC7u8Y7pTvBZnNTRjbHlkWY0OP1yCPxJ+hMnSRtbVXt29e2XVvbsW3FU2uN9Hc5Ne/QE25XX23NJ8EnwB5+OPz7RTvEDrfs6qvDx+L0t1+50vo+O80UL7ygettt1qMDsG58Cxe6/aOdo47Q6dRT7bGiQvW3v3WbQe64w65HOPtsd/vbtsWX/EeMiD+Bs2ZKCWBCj5dzgubCC+N/zbPPuj/OF19MfULfvt22XVTUtk05HeJJwJFOuDq16BkzIl/YlEitN3jZnj1WnjAhviQZbera1e3T36dP8tu58UY7F3DokJWPHLGrix9+2F2nuFj173+PviMLVyaKggk9Xs4dRl5/PbHXOT9gp6kild56y7b5jW+kdrvxiKcm6jS9JHOkkmjbb6Sjh9Dp2mtjx+b8bbt2uU1tkfpTO9uLtiNLpAYe63MlioIJPV5nnmkfyapVib1u7ly3SQJQ/c1vop84TMQDD9g2f/jDjm0nGfGeWGxpUf3449TvzKLFkmitN56dUzzllhabd06Gh07hhl/IoaFZKfsxocejtdWuqAPcftPxilRzdLoGdsTll9s2nnuuY9tJBy97KXQ0YYdKpDdI8LKDB6385ZfxvxdRB0RL6Bw+17F/P3D0KNC3L9CvX2Kvde4s3tJi5XPOscdly+wx3nsXhrN2rT2OG5f8NtIldHjSSLfdSodYt/xKNLZY6wcvD17mfFe6d4//vYjSJK6bRKdD1t0kesUKYMoUGwt6zZrktyNiP+hwN7K9//72Y6pXVEQet7mxEejVC+jcGWhoALp1Sz4uSp9o/0OiFOvQTaLzxvbt9jhyZMe2M3euW2N3dpadAh/zpk3tE320O5hXVdk2Tj+dyTybMZlTlmBCdzgJvbS0Y9sJ9+NevNhq2i+/bOWzzwbGjgVGjLDyunXht+U8n43NLUSUdbp4HUDW2LbNHjtaQw81dy7w0UfWZOL44IO264wfb4+hTTJOQneWExFFwRq6I1U19FBO+6oqUF9vzy1bZic7q6vbrvvZZ22bYLL5hCgRZR3W0B3pSujB+vSxR6cXjOOPfwRuugl46SUrDxpkvW2cmFhDJ6I4sIYOWO05EwkdCN/drrraukw69u2z506csPKQIe17xxARhWC3RQDYuxcoLgYGDLD+6F5RtR4xtbXWPHPoEHDWWW5vGSLKe9G6LbLJBXBPiKa7dh6LiD0WF9tERJQANrkAqeuDngqxroAkIoqACR3IXPt5PELbydluTkRxYkIHsqfJhYioA2ImdBF5RkT2iUhVhOUiIo+LyGYRWScik1IfZpplU5MLEVGS4qmhLwAwI8rymQDKAtMcAL/reFgZlk1NLkRESYqZ0FX1XQAHoqxyCYDnA0P1fgign4gMTlWAadfa6iZ0Z2wVIqIclIo29KEAdgWVawLPtSMic0SkUkQq6+rqUvDWKbBnD3D8OFBUZANoERHlqFQkdAnzXNgrYVT1aVUtV9XyoqKiFLx1CrC5hYh8IhUJvQbA8KDyMAC7U7DdzGAPFyLyiVQk9LcAfC/Q2+UsAPWqWpuC7WYGe7gQkU/EvPRfRBYCmA6gUERqAMwF0BUAVHU+gCUAZgHYDKARwA3pCjYt2ORCRD4RM6Gr6ndiLFcAt6QsokxL140tiIgyjFeKsoZORD6R3wm9pQXYudPm2QediHJc/ib0igqgSxf3JhI9e/ImEkSU0/J3PPSKCmDHDmDBAivzJhJElOPyt4Z+5AjwyiteR0FElDL5W0N/5RW7j+e0acD553sdDRFRh+VvQn/2WXu84QabiIhyXH42uXz2GbBsGVBQAFx+udfREBGlRH4mdOdE6BVXcIRFIvKN/EvoLS3AE0/Y/I03ehsLEVEK5V9CX7oUaGgAysqAc87xOhoiopTJv4QefDJUwg3lTkSUm/InoVdUWAJ3+p7fey+vDCUiX8mfbosVFcCVVwJnnGFlXhlKRD6TPzV0AFi/3usIiIjSJr8S+oYN9nj22d7GQUSUBvmV0J0a+pw53sZBRJQG+ZnQzzzT2ziIiNIgfxL6iRN2yT8AjB7tbSxERGmQPwl982ZL6qWlNoYLEZHP5E9CZ3MLEfkcEzoRkU8woRMR+UT+JHSnD7pzpSgRkc/kR0JnDxciygP5kdCrqy2pjxzJHi5E5Fv5kdDZfk5EeSC/Ejrbz4nIx/IjoTsnRFlDJyIfiyuhi8gMEdkkIptF5J4wy6eLSL2IrAlM96c+1A5gkwsR5YGYN7gQkc4AngLwTQA1AFaIyFuquiFk1fdU9aI0xNgxx49bDxcR9nAhIl+Lp4Y+BcBmVd2qqscBvATgkvSGlULV1UBzs43h0rOn19EQEaVNPAl9KIBdQeWawHOhporIWhH5i4iEbdsQkTkiUikilXV1dUmEmwS2nxNRnognoUuY50JvyLkKwAhVHQ/gCQBvhNuQqj6tquWqWl5UVJRYpMli+zkR5Yl4EnoNgOFB5WEAdgevoKqHVbUhML8EQFcRKUxZlB3BhE5EeSKehL4CQJmIjBSRkwBcBeCt4BVEpFhEJDA/JbDd/akONinsg05EeSJmQlfVZgC3AngbwEYAi1R1vYjcLCI3B1abDaBKRNYCeBzAVaoa2iyTWRUV1rNl40Yrl5dbuaLCy6iIiNJGvMq75eXlWllZmd43efdd4Gtfs3mP9y9ERKkgIitVtTzcMn9fKbpihdcREBFlTH4k9Isv9jYOIqIMyI+E/sAD3sZBRJQB/k3o+/cDW7cCPXqwyyIR5QX/JnTnhOvEiUCXmEPWEBHlPP8mdKe55Stf8TYOIqIM8W9C//hje2RCJ6I84c+ErsoaOhHlHX8m9M8/B/bsAfr2BU47zetoiIgywp8J3amdl5cDnfz5JxIRhfJntmNzCxHlIX8n9ClTvI2DiCiD/JfQVd0+6KyhE1Ee8V9C37wZOHQIKC4Ghoa7Ux4RkT/5L6EHt59LuLvnERH5k78TOhFRHvFfQucVokSUp/yT0J1bzr3/vpVnzuQt54gor/hnGMKKCuDrXwemT7cybzlHRHnGPzV0wO4hSkSUp/yV0N97zx5nz/Y2DiIiD/gnoTc3u+3nTzzhbSxERB7wT0JfvRo4ehQoK7OLioiI8ox/ErrTfn7eed7GQUTkEf8kdH38nbEAAAYgSURBVKf9fNo0b+MgIvKIPxJ6a6ub0FlDJ6I85Y+EvnEjcOCADcZVWup1NEREnvBHQg9uP+eAXESUp/yR0Nl+TkTkg4Suyh4uRESIM6GLyAwR2SQim0XknjDLRUQeDyxfJyKTUh9qkOABt7ZvBz7/HBgwABg9Oq1vS0SUzWImdBHpDOApADMBnAHgOyJyRshqMwGUBaY5AH6X4jhdR44A8+YBdXU2/eUv9vy0aUCn3D/gICJKVjyjLU4BsFlVtwKAiLwE4BIAG4LWuQTA86qqAD4UkX4iMlhVa1Me8T2BA4SBA9s+z/ZzIspz8VRphwLYFVSuCTyX6DoQkTkiUikilXV1dYlF6ox3/tvfhl9+550c/5yI8lo8CT1cP8DQwcbjWQeq+rSqlqtqeVFRUTzxuSoq7ASoM865Mx9aZkInojwVT0KvATA8qDwMwO4k1iEiojSKJ6GvAFAmIiNF5CQAVwF4K2SdtwB8L9Db5SwA9WlpP3fMnRu9TESUh2KeFFXVZhG5FcDbADoDeEZV14vIzYHl8wEsATALwGYAjQBuSF/IaN+swmYWIqL47imqqktgSTv4uflB8wrgltSGRkREiWDHbSIin2BCJyLyCSZ0IiKfYEInIvIJUW13/U9m3likDsCOJF9eCOCLFIaTSowtOdkcG5Dd8TG25ORqbCNUNeyVmZ4l9I4QkUpVLfc6jnAYW3KyOTYgu+NjbMnxY2xsciEi8gkmdCIin8jVhP601wFEwdiSk82xAdkdH2NLju9iy8k2dCIiai9Xa+hERBSCCZ2IyCdyLqHHumF1hmN5RkT2iUhV0HMDRORvIlIdeOzvUWzDReQfIrJRRNaLyO3ZEp+IdBeRj0VkbSC2edkSW1CMnUVktYgszqbYRGS7iHwiImtEpDLLYusnIq+KyKeB793UbIhNREYFPi9nOiwid2RDbIH4/k/gd1AlIgsDv4+kYsuphB7nDaszaQGAGSHP3QPgHVUtA/BOoOyFZgA/UtXRAM4CcEvgs8qG+JoAnK+q4wFMADAjMI5+NsTmuB3AxqByNsX2dVWdENRPOVtiewzAX1X1dADjYZ+f57Gp6qbA5zUBwGTYEN9/yobYRGQogB8AKFfVMbAhyq9KOjZVzZkJwFQAbweVfwLgJx7HVAqgKqi8CcDgwPxgAJu8/twCsbwJ4JvZFh+AngBWAfhqtsQGu+PWOwDOB7A4m/6vALYDKAx5zvPYAPQBsA2BjhbZFFtIPBcCWJ4tscG9H/MA2HDmiwMxJhVbTtXQEefNqD02SAN3awo8DvQ4HohIKYCJAD5ClsQXaNJYA2AfgL+patbEBuA3AH4MoDXouWyJTQEsFZGVIjIni2I7BUAdgGcDTVX/V0QKsiS2YFcBWBiY9zw2Vf0cwCMAdgKohd3tbWmyseVaQo/rZtTkEpFeAF4DcIeqHvY6HoeqtqgdAg8DMEVExngdEwCIyEUA9qnqSq9jieAcVZ0Ea3a8RUTO8zqggC4AJgH4napOBHAU3jZLtRO4hea3AbzidSyOQNv4JQBGAhgCoEBErkl2e7mW0HPhZtR7RWQwAAQe93kViIh0hSXzF1X19WyLDwBU9RCAf8LORWRDbOcA+LaIbAfwEoDzReSPWRIbVHV34HEfrB14SpbEVgOgJnCkBQCvwhJ8NsTmmAlglaruDZSzIbZvANimqnWqegLA6wDOTja2XEvo8dyw2mtvAbguMH8drO0640REAPwBwEZVfTRokefxiUiRiPQLzPeAfak/zYbYVPUnqjpMVUth36//p6rXZENsIlIgIr2deVhba1U2xKaqewDsEpFRgacuALAhG2IL8h24zS1AdsS2E8BZItIz8Ju9AHYyObnYvDxBkeRJhFkAPgOwBcBPPY5lIazd6wSshvJ9ACfDTqhVBx4HeBTbubDmqHUA1gSmWdkQH4BxAFYHYqsCcH/gec9jC4lzOtyTop7HBmunXhuY1jvf/2yILRDHBACVgf/rGwD6Z1FsPQHsB9A36LlsiW0erEJTBeAFAN2SjY2X/hMR+USuNbkQEVEETOhERD7BhE5E5BNM6EREPsGETkTkE0zoREQ+wYROROQT/x+G7aFWN6Pp0gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lin_reg = LinearRegression()\n",
    "plot_learning_curves(lin_reg, X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "과소적합된 모델의 전형적인 모습. 두 곡선이 수평한 구간을 만들고 높은 오차에서 가까이 근접  \n",
    "cf) 모델이 훈련 데이터에 과소적합이라면 훈련 샘플을 더 추가해도 효과가 없음! 더 복잡한 모델 사용/더 나은 특성 선택으로 해결"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD4CAYAAADy46FuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAf9ElEQVR4nO3df7TcdX3n8ecruRDymyQECLmxiZJVgS0oKUJdLUsUUtcj7B7YjadK1NiwmLXaY48l9hxJ2sNWdm1RzhGOKVACUn40RcmqVNOg63YXwYvakgCBWDC5JpJbA0kICeEm7/3j+xnme4e5k3tn5s73e6+vxzlz5vv9fH/M+wZyX/n8mBlFBGZmZoMZV3QBZmZWbg4KMzNryEFhZmYNOSjMzKwhB4WZmTXUVXQB7XbSSSfF/Pnziy7DzGxUeeyxx/41ImbXOzbmgmL+/Pn09PQUXYaZ2agi6eeDHfPQk5mZNeSgMDOzhhwUZmbWkIPCzMwaclCYmVlDDgozM2vIQVESO3bAvn1FV2Fm9noOihL41rdg/nw47TTYubPoaszMBnJQlMD998PRo3DgQBYaZmZl4qAogUOHqtt79xZXh5lZPQ6KEnj11eq25ynMrGwcFCXgoDCzMnNQlMDhw9Xt/fuLq8PMrB4HRQm4R2FmZeagKAEHhZmV2TGDQtJtknZL2pxr+5+SnpL0z5K+LunE3LFVkrZJ2irpklz7uZIeT8dulKTUPkHSvan9EUnzc9csk/RMeixr1w9dNg4KMyuzofQobgeW1LRtBM6KiN8EngZWAUg6A1gKnJmuuUnS+HTNzcAKYGF6VO65HHghIk4HbgCuT/eaCVwLvAM4D7hW0ozh/4jll5+jcFCYWdkcMygi4gfAnpq270ZEf9r9IdCdti8F7omIVyLiWWAbcJ6kOcC0iHg4IgK4A7gsd826tL0eWJx6G5cAGyNiT0S8QBZOtYE1JuR7FJ7MNrOyacccxceAB9P2XGBH7lhvapubtmvbB1yTwmcvMKvBvV5H0gpJPZJ6+vr6WvphiuChJzMrs5aCQtKfAP3AXZWmOqdFg/ZmrxnYGLE2IhZFxKLZs+t+N3ip1Q49Rd2f0sysGE0HRZpcfj/we2k4CbJ/9c/LndYN7Ezt3XXaB1wjqQuYTjbUNdi9xpx8jyIi+8wnM7OyaCooJC0B/hj4QES8nDu0AViaVjItIJu0fjQidgH7JZ2f5h+uBB7IXVNZ0XQ58FAKnu8AF0uakSaxL05tY04+KMDzFGZWLl3HOkHS3cCFwEmSeslWIq0CJgAb0yrXH0bEf42ILZLuA54gG5JaGRFH0q2uJltBNZFsTqMyr3ErcKekbWQ9iaUAEbFH0p8BP0rn/WlEDJhUHytqg2LfPpgzp5hazMxqKcbYgPiiRYuip6en6DKGZcYMePHF6v6jj8Jv/VZx9ZjZrx9Jj0XEonrH/M7sEqjXozAzKwsHRQk4KMyszBwUBYsYuDwWPJltZuXioCjYkSOvb3OPwszKxEFRsNphJ3BQmFm5OCgK5qAws7JzUBSsdn4CHBRmVi4OioLV61F4MtvMysRBUTAPPZlZ2TkoCuahJzMrOwdFwdyjMLOyc1AUzEFhZmXnoCiYJ7PNrOwcFAXzHIWZlZ2DomD1ehQHD9ZvNzMrgoOiYIMFgoefzKwsHBQFqzf0BA4KMysPB0XBButReJ7CzMrCQVEwB4WZlZ2DomAOCjMrOwdFwQabo3BQmFlZOCgK5lVPZlZ2xwwKSbdJ2i1pc65tpqSNkp5JzzNyx1ZJ2iZpq6RLcu3nSno8HbtRklL7BEn3pvZHJM3PXbMsvcYzkpa164cuEw89mVnZDaVHcTuwpKbtGmBTRCwENqV9JJ0BLAXOTNfcJGl8uuZmYAWwMD0q91wOvBARpwM3ANene80ErgXeAZwHXJsPpLHCQ09mVnbHDIqI+AGwp6b5UmBd2l4HXJZrvyciXomIZ4FtwHmS5gDTIuLhiAjgjpprKvdaDyxOvY1LgI0RsSciXgA28vrAGvXyPYqsj5VxUJhZWTQ7R3FKROwCSM8np/a5wI7ceb2pbW7arm0fcE1E9AN7gVkN7jWm5IPixBOr2w4KMyuLdk9mq05bNGhv9pqBLyqtkNQjqaevr29IhZZFfuhp1qzqtiezzawsmg2K59NwEul5d2rvBeblzusGdqb27jrtA66R1AVMJxvqGuxerxMRayNiUUQsmj17dpM/UjHyPYp8ULhHYWZl0WxQbAAqq5CWAQ/k2pemlUwLyCatH03DU/slnZ/mH66suaZyr8uBh9I8xneAiyXNSJPYF6e2McVBYWZl13WsEyTdDVwInCSpl2wl0heA+yQtB7YDVwBExBZJ9wFPAP3Ayog4km51NdkKqonAg+kBcCtwp6RtZD2JpeleeyT9GfCjdN6fRkTtpPqo56Aws7I7ZlBExAcHObR4kPOvA66r094DnFWn/RApaOocuw247Vg1jmaDzVE4KMysLPzO7IIN1qPwZLaZlYWDomCNhp6i7hovM7POclAULD/0NHkynHBCtn3kSPaVqGZmRXNQFCzfozjuOJg6tbrveQozKwMHRcFqg2LatOq+5ynMrAwcFAVrFBTuUZhZGTgoCpafozj+eAeFmZWPg6Jg7lGYWdk5KArmyWwzKzsHRcEaDT15MtvMysBBUTAPPZlZ2TkoCuagMLOyc1AUzHMUZlZ2DoqCeXmsmZWdg6Jgfme2mZWdg6JgnqMws7JzUBSsdujJcxRmVjYOioK5R2FmZeegKJiDwszKzkFRME9mm1nZOSgKdPRo9k12FV1d2bfcSdn+gQMDj5uZFcFBUaDa3oQE48bBlCnVdvcqzKxoDooC1QZFhecpzKxMWgoKSX8oaYukzZLulnSCpJmSNkp6Jj3PyJ2/StI2SVslXZJrP1fS4+nYjVI2+CJpgqR7U/sjkua3Um/Z1C6NrfA8hZmVSdNBIWku8AfAoog4CxgPLAWuATZFxEJgU9pH0hnp+JnAEuAmSePT7W4GVgAL02NJal8OvBARpwM3ANc3W28ZuUdhZqNBq0NPXcBESV3AJGAncCmwLh1fB1yWti8F7omIVyLiWWAbcJ6kOcC0iHg4IgK4o+aayr3WA4srvY2xYChBsWdP5+oxM6un6aCIiF8AXwS2A7uAvRHxXeCUiNiVztkFnJwumQvsyN2iN7XNTdu17QOuiYh+YC8wq7YWSSsk9Ujq6evra/ZH6rjBgmL+/Or21q0dK8fMrK5Whp5mkP2LfwFwGjBZ0ocaXVKnLRq0N7pmYEPE2ohYFBGLZs+e3bjwEhlsjuKss6rbmzd3rh4zs3paGXp6D/BsRPRFxKvA/cBvA8+n4STS8+50fi8wL3d9N9lQVW/arm0fcE0a3poOjJnBmMF6FA4KMyuTVoJiO3C+pElp3mAx8CSwAViWzlkGPJC2NwBL00qmBWST1o+m4an9ks5P97my5prKvS4HHkrzGGPCUIJiy5bsjXlmZkXpavbCiHhE0nrgx0A/8BNgLTAFuE/ScrIwuSKdv0XSfcAT6fyVEVF53/HVwO3ARODB9AC4FbhT0jaynsTSZusto8GGnk4+GWbPhr4+ePlleO45eOMbO16emRnQQlAARMS1wLU1za+Q9S7qnX8dcF2d9h7grDrth0hBMxYN1qOArFfxve9l25s3OyjMrDh+Z3aBjhUUFZ6nMLMiOSgK5KAws9HAQVGgweYowEFhZuXhoChQox7FmWdWt596auC5Zmad5KAoUKOgmD4d5s2rnrdtW+fqMjPLc1AUqNHQE3j4yczKwUFRoEY9CnBQmFk5OCgK5KAws9HAQVGg/NCTg8LMyspBUaB8j6LeHMVb35p9jzZkk9kHD3amLjOzPAdFgY419DRxIpx+erZ99Gi2TNbMrNMcFAU6VlCAh5/MrHgOigIda3ksOCjMrHgOigINpUeRf4e2g8LMiuCgKJCHnsxsNHBQFOhYy2MBFi6sHtu+HfbtG/m6zMzyHBQFOtby2Er7woXVfX/mk5l1moOiQEMZegI46aTq9t69I1ePmVk9DooCDTUopk2rbjsozKzTHBQFGsryWMg+crzCQWFmneagKNBQexQOCjMrkoOiQA4KMxsNWgoKSSdKWi/pKUlPSrpA0kxJGyU9k55n5M5fJWmbpK2SLsm1nyvp8XTsRin7KDxJEyTdm9ofkTS/lXrLZijLY2FgUHh5rJl1Wqs9ii8Dfx8RbwHOBp4ErgE2RcRCYFPaR9IZwFLgTGAJcJOk8ek+NwMrgIXpsSS1LwdeiIjTgRuA61ust1SGsjwW3KMws2I1HRSSpgHvBm4FiIjDEfEicCmwLp22DrgsbV8K3BMRr0TEs8A24DxJc4BpEfFwRARwR801lXutBxZXehtjgYeezGw0aKVH8UagD/hrST+RdIukycApEbELID2fnM6fC+zIXd+b2uam7dr2AddERD+wF5hVW4ikFZJ6JPX09fW18CN1lpfHmtlo0EpQdAFvB26OiLcBB0jDTIOo1xOIBu2NrhnYELE2IhZFxKLZs2c3rrpEvDzWzEaDVoKiF+iNiEfS/nqy4Hg+DSeRnnfnzp+Xu74b2Jnau+u0D7hGUhcwHdjTQs2l4qEnMxsNmg6KiPglsEPSm1PTYuAJYAOwLLUtAx5I2xuApWkl0wKySetH0/DUfknnp/mHK2uuqdzrcuChNI8xJjgozGw06Grx+k8Cd0k6HvgX4KNk4XOfpOXAduAKgIjYIuk+sjDpB1ZGxJF0n6uB24GJwIPpAdlE+Z2StpH1JJa2WG+peHmsmY0GLQVFRPwUWFTn0OJBzr8OuK5Oew9wVp32Q6SgGYuGujx26lSQIAJeegmOHIHx4wc/38ysnfzO7AINdehp3LgsLCrcqzCzTnJQFGioQQFeImtmxXFQjJAvfxlWroRduwY/Z6jLY8ET2mZWnFYns62ORx+FT3862+7qykKjnuH0KBwUZlYU9yhGwNNPV7efemrw85oNCs9RmFknOShGwEsvVbdffLH+ORHuUZjZ6OCgGAH5oBjsl3p/f3V7/PhsZVMjDgozK4qDYgQMJSiG05sAB4WZFcdBMQL2769uDzb05KAws9HCQTEC8j2KQ4cGLoOtGM7SWPD7KMysOA6KEZAPCqj/i909CjMbLRwUI6A2KOoNP7USFF4ea2ad5KAYAUPpUQx36Mk9CjMrioNiBIx0j8JBYWad5KAYAZ6jMLOxxEExAhwUZjaWOChGQP59FFB/6Gm4cxS130dx9GhztZmZDZeDYgSMRI/iuONg0qRsOwIOHGi+PjOz4XBQtNnhwwNDANoTFODhJzMrhoOizWp7E9CeoSdwUJhZMRwUbVYvKNyjMLPRzEHRZkPtUTgozGy0aDkoJI2X9BNJ30z7MyVtlPRMep6RO3eVpG2Stkq6JNd+rqTH07EbJSm1T5B0b2p/RNL8VusdaUPtUeSHnhwUZlZm7ehRfAp4Mrd/DbApIhYCm9I+ks4AlgJnAkuAmySNT9fcDKwAFqbHktS+HHghIk4HbgCub0O9I6qZoaehzlH4E2TNrAgtBYWkbuA/ALfkmi8F1qXtdcBlufZ7IuKViHgW2AacJ2kOMC0iHo6IAO6ouaZyr/XA4kpvo6xq30MBIzP05A8GNLNOabVH8SXgs0D+7V+nRMQugPR8cmqfC+zIndeb2uam7dr2AddERD+wF5hVW4SkFZJ6JPX09fW1+CO1ZrAeRcTANs9RmNlo0XRQSHo/sDsiHhvqJXXaokF7o2sGNkSsjYhFEbFo9uzZQyxnZNQLiiNH4OWXB7Z5eayZjRZdLVz7TuADkt4HnABMk/Q14HlJcyJiVxpW2p3O7wXm5a7vBnam9u467flreiV1AdOBPS3UPOLqBQVkw0+TJ1f33aMws9Gi6R5FRKyKiO6ImE82Sf1QRHwI2AAsS6ctAx5I2xuApWkl0wKySetH0/DUfknnp/mHK2uuqdzr8vQar+tRlMlgQVH7i91BYWajRSs9isF8AbhP0nJgO3AFQERskXQf8ATQD6yMiCPpmquB24GJwIPpAXArcKekbWQ9iaUjUG9bNepR5Hl5rJmNFm0Jioj4PvD9tP0rYPEg510HXFenvQc4q077IVLQjBbN9CiamaPwqicz6xS/M7vNRnLoye+jMLMiOCjaLP8+ilm5hby1Q0+eozCz0cJB0Wb5HkV3bi1X7S/2diyPLfe0vpmNFQ6KNssHxdy51e12DD1NmJA9APr74eDB5mo0MxsOB0WbDdajaMfQE3j4ycw6z0HRZkPtUTSzPBYcFGbWeQ6KNhssKBr1KIY6RwFeImtmneegaLOhTmY3O/TkJbJm1mkOijZ69VV45ZVse9w4OPXU6rF2BYWHnsys0xwUbZTvTUyZMvCXeqOP8Gh26MlBYWad4KBoo3xQTJ0KJ55Y3XePwsxGKwdFG9X2KKZOre7v3599L0WFg8LMRgsHRRvVBsX48QPDIr9KqR3LY73qycw6wUHRRrVBAYMPP7Vjeax7FGbWCQ6KNqoXFIP9YvfyWDMbLRwUbXSsoMivfPIchZmNFg6KNhrO0JOXx5rZaOGgaKP8d1G4R2FmY4WDoo1q30cB7Z+jcFCYWac5KNqo2aEnL481szJzULRRs5PZw5mjmDgRurqy7UOHBgaOmdlIcFC0UbPvoxhOj0IaGD4vvDC8Gs3MhqvpoJA0T9L3JD0paYukT6X2mZI2SnomPc/IXbNK0jZJWyVdkms/V9Lj6diNkpTaJ0i6N7U/Iml+8z/qyOvE+yhg4KfS7to1vGvNzIarlR5FP/CZiHgrcD6wUtIZwDXApohYCGxK+6RjS4EzgSXATZLGp3vdDKwAFqbHktS+HHghIk4HbgCub6HeETfUoacjR+Do0Wxbyj7qYzjy33OxY8fw6zQzG46mgyIidkXEj9P2fuBJYC5wKbAunbYOuCxtXwrcExGvRMSzwDbgPElzgGkR8XBEBHBHzTWVe60HFld6G2U01KGn2t7EcH+iefOq2729w7vWzGy42jJHkYaE3gY8ApwSEbsgCxPg5HTaXCD/79/e1DY3bde2D7gmIvqBvcCsdtQ8EvLvo6i3PLbSo2hl2AncozCzzmo5KCRNAf4O+HRENFqwWe/fzdGgvdE1tTWskNQjqaevr+9YJY+Yoc5RNLs0tsI9CjPrpJaCQtJxZCFxV0Tcn5qfT8NJpOfdqb0XyP2KoxvYmdq767QPuEZSFzAd2FNbR0SsjYhFEbFo9uzZrfxILWlm6Gk4S2Mr8j0KB4WZjbRWVj0JuBV4MiL+MndoA7AsbS8DHsi1L00rmRaQTVo/moan9ks6P93zypprKve6HHgozWOUUr2gmDSpOll96FD2ndoeejKz0aSrhWvfCXwYeFzST1Pb54AvAPdJWg5sB64AiIgtku4DniBbMbUyIirf+XY1cDswEXgwPSALojslbSPrSSxtod4R1d+fBQFkk9MTJ1a3p0+HPakftHdv60FRO/QUMfwJcTOzoWo6KCLiH6k/hwCweJBrrgOuq9PeA5xVp/0QKWjK7sCB6vaUKQN/cZ944sCgyH8lajNDT1OnZt9LsW9fFk6/+hWcdFJzdZuZHYvfmd0m9YadKmpXPrXaowBPaJtZ5zgo2mSoQdGOoSfwPIWZdY6Dok3qfcR4RX7l04svtr48Frzyycw6x0HRJvW+tKiiUY+imTkK8NCTmXWOg6JNPPRkZmOVg6JNGgXFSAw9uUdhZp3ioGiToQbFzp3tGXpyj8LMOsVB0SaNguKcc6rbX//6wPdctGsyu7zvVzez0c5B0SaNguJ3fgfe8IZse88euP/+6rFmg2LatOwB2Zvu9rzuE7DMzNrDQdEmjZbHjhsHy5ZV97/xjep2s0EBHn4ys85wULRJox4FDAyKdsxRgCe0zawzHBRt0uh9FABvehO8+92vb3ePwszKzkHRJsfqUQB85COvb2slKNyjMLNOcFC0yVCC4oorYPLkgW2tDD35YzzMrBMcFG0ylKCYMgUuv3xgm4eezKzsHBRtMpSgAPjoRwfue+jJzMrOQdEmQw2Kd70LFiyo7rdr6GnHDr/pzsxGhoOiTRq9jyJv3DhYsaK6/6Y3Nf+aftOdmXVCK9+ZbTlD7VEA/NEfwdGjMH58NsHdiu5ueOKJbLu3F2bNau1+Zma1HBRtcOQIvPxyti3BxImNz+/qgs99rj2vnQ+KHTvg7LPbc18zswoPPbVB/kP+Jk/Ohpc6xRPaZjbSHBRtMJxhp3bzElkzG2keemrSSy/B5s1w8CA891y1vdNB4R6FmY20UdGjkLRE0lZJ2yRdM6Ivtnr1oPtHPr+Gf/gHuPJKOOUUuOACuOgi+NjHqqcPCIoG92rX/mA9iiOfXzPir930fplqcW1jv7ba/TLVMtK1tUnpexSSxgNfAd4L9AI/krQhIp5o5+v82bX97Nr+Kv23n0b/0wfp7xf9R+Dg/Wdz8AeHefkg/OyHH2fnMe7zlgWHoC99QuCaNbByZfVgK/sR2f4nPjHgePf//iSQLXV6+OFg4RuP8Mvd43jpwLVM/oujzJwRzDzxKFO2vIfDD7zK4VfhlcOCZ/4L0za8yrSpwbSpR5n4rdPRlkPVe69/E3ri4Gu7+tsFjNt6kHHK5mB012nws4OvlcZdc2FbdT/+Zh48fbD63o573gBbDyIBd89Dz1Tvzd90o3StBHytem+ov69/SfcCdMcp8NzL1ePrBu5r3cnw89zx26v7ETXnr2t8r1b3G9VyzP3hnNvp/TLVMpprbcO9Lj47uOw/inZSlPxdWpIuAFZHxCVpfxVARPx5vfMXLVoUPT09w36dt3bv46lfTBvWNf+GrZzGTo7nMMdzmDewnc/yP/gNtg/79Zu1j6lMZ1/HXs/Myu2PP9PPF744/D6ApMciYlG9Y6Nh6GkukJ+m7U1tr5G0QlKPpJ6+vr7hv8Lq1XT94udDOnU2u/kUX+Ix3s5TvIXvcRHfYQn/iw/wFf5bR0MCYBr7eQ8bO/qaZlZif/HFrGvexmGo0dCjuAK4JCI+nvY/DJwXEZ+sd36zPYq774YXX4Txn1hB161r6erK3u9wwu/9JyZ++34mTYKpF76df3v4x9XPZ5IGfm5GQfsHD8I/TnovXQ9t5NRT4dRTYfrMcex/8Sh79sALL8CBc9/F8T/8Pxx/PEyYAHHmmex/eAt798K+fXDoP38Y7rjztVvHlVfCujuy7YD4yEeI227n6NHsfSNctQK+urZaylW/D2v/qrq/4uPollteK5PlHyNuuS2738c/Dn91S/W1fj+7tvKjxVVXoa9+tfpz1rxWXHUVcfNXX6uNT1wNN91cPT+3HwGs/AR85abq8Zp9rcxd3+Bere4PpZaG+61cO9L7ZaplNNfahnu97f/dxAUXMGyNehSjISg6MvSUe8HBfzmXJBhGXW2jqdYy1eLafv1qbfe9h6FRUIxfPUKz5O2yZs2aXwKr16xZs2HNmjUvAzcC/3316tV1x5jWrl27ekX+w5SaceGFg+83Olb0fplqOdZ+mWpxbWO/ttr9MtUy0rUN0Zo1a3atXr16bb1jpe9RAEh6H/AlYDxwW0RcN9i5LfcozMx+DTXqUZR+eSxARHwb+HbRdZiZ/ToaDauezMysQA4KMzNryEFhZmYNOSjMzKyhUbHqaTgk9QFDe5t1fScB/9qmctrNtTXHtTXHtTVntNb2GxExu96BMRcUrZLUM9gSsaK5tua4tua4tuaMxdo89GRmZg05KMzMrCEHxevVfQt7Sbi25ri25ri25oy52jxHYWZmDblHYWZmDTkozMysIQdFImmJpK2Stkm6puBabpO0W9LmXNtMSRslPZOeZxRU2zxJ35P0pKQtkj5VlvoknSDpUUn/lGpbU5bacjWOl/QTSd8sU22SnpP0uKSfSuopWW0nSlov6an0/90FZahN0pvTn1flsU/Sp8tQW6rvD9Pfg82S7k5/P5qqzUFB9pcX+Arwu8AZwAclnVFgSbcDS2rargE2RcRCYFPaL0I/8JmIeCtwPrAy/VmVob5XgIsi4mzgHGCJpPNLUlvFp4Anc/tlqu3fR8Q5uXX2Zanty8DfR8RbgLPJ/vwKry0itqY/r3OAc4GXga+XoTZJc4E/ABZFxFlkX9GwtOnaIuLX/gFcAHwnt78KWFVwTfOBzbn9rcCctD0H2Fr0n1uq5QHgvWWrD5gE/Bh4R1lqA7rTX86LgG+W6b8r8BxwUk1b4bUB04BnSQtvylRbTT0XA/+3LLUBc4EdwEyyr5P4Zqqxqdrco8hU/lArelNbmZwSEbsA0vPJBdeDpPnA24BHKEl9aWjnp8BuYGNElKY2si/f+ixwNNdWltoC+K6kxyRVviKyDLW9EegD/joN2d0iaXJJastbCtydtguvLSJ+AXwR2A7sAvZGxHebrc1BkVGdNq8bbkDSFODvgE9HxL6i66mIiCORDQV0A+dJOqvomgAkvR/YHRGPFV3LIN4ZEW8nG35dKendRReUdAFvB26OiLcBByh2eO51JB0PfAD426JrqUhzD5cCC4DTgMmSPtTs/RwUmV5gXm6/G9hZUC2DeV7SHID0vLuoQiQdRxYSd0XE/WWrDyAiXgS+TzbXU4ba3gl8QNJzwD3ARZK+VpLaiIid6Xk32Tj7eSWprRfoTT1DgPVkwVGG2ip+F/hxRDyf9stQ23uAZyOiLyJeBe4HfrvZ2hwUmR8BCyUtSP86WApsKLimWhuAZWl7GdncQMdJEnAr8GRE/GXuUOH1SZot6cS0PZHsL8tTZagtIlZFRHdEzCf7/+uhiPhQGWqTNFnS1Mo22Vj25jLUFhG/BHZIenNqWgw8UYbacj5IddgJylHbduB8SZPS39nFZIsAmqutyAmgMj2A9wFPAz8D/qTgWu4mG1d8lexfVMuBWWQToc+k55kF1fbvyIbl/hn4aXq8rwz1Ab8J/CTVthn4fGovvLaaOi+kOpldeG1k8wD/lB5bKv//l6G2VMc5QE/67/oNYEaJapsE/AqYnmsrS21ryP6htBm4E5jQbG3+CA8zM2vIQ09mZtaQg8LMzBpyUJiZWUMOCjMza8hBYWZmDTkozMysIQeFmZk19P8B4w3bc5TmyTAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#10차 다항회귀모델의 학습 곡선\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "polynomial_regression = Pipeline([\n",
    "    ('poly_features', PolynomialFeatures(degree=10, include_bias=False)),\n",
    "    ('lin_reg', LinearRegression()),\n",
    "])\n",
    "\n",
    "plot_learning_curves(polynomial_regression, X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 훈련 데이터의 오차가 선형 회귀 모델보다 훨씬 낮음 \n",
    "- 두 곡선 사이의 공간은 과대적합 모델의 특징"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**편향-분산 트레이드오프**\n",
    "- 편향\n",
    "    - 잘못된 가정(ex. 데이터가 2년차인데 3년차로 가정)으로 일어남. 과소적합되기 쉬움\n",
    "\n",
    "- 분산\n",
    "    - 작은 변동에 모델이 과도하게 민감하기 때문에 일어남. 과대적합되기 쉬움\n",
    "    \n",
    "- 줄일 수 없는 오차\n",
    "    - 데이터 자체에 있는 잡음으로 발생, 잡음 제거로 오차를 줄일 수 있음\n",
    "    \n",
    "    \n",
    "모델의 복잡도가 커지면 분산이 늘어나고 편향이 줄어들며, 복잡도가 줄어들면 이와 반대!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.5 규제가 있는 선형 모델\n",
    "- 규제: 과대적합을 감소시키는 방법  \n",
    "- 다항 회귀의 경우 다항식의 차수를 감소시킴\n",
    "- 선형 회귀 모델에서는 보통 모델의 가중치를 제한함"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5.1 릿지 회귀\n",
    "- 규제항은 학습 알고리즘을 데이터에 맞추는 것 뿐 아니라 모델의 가중치가 가능한 작게 유지되도록 노력\n",
    "- 모델 훈련이 끝나면 성능을 규제 없는 성능 지표로 평가\n",
    "- α: 모델을 얼마나 규제할지, α가 커질수록 수평선이 됨"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "정규방정식을 사용한 사이킷런 릿지 회귀 적용 예시"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4.81333382]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "ridge_reg = Ridge(alpha=-1, solver = 'cholesky')\n",
    "ridge_reg.fit(X, y)\n",
    "ridge_reg.predict([[1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "확률적 경사 하강법을 사용했을 때  \n",
    "`penalty = 'l2'` 비용 함수에 규제항 추가(=릿지회귀)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4.79475725])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_reg = SGDRegressor(penalty='l2')\n",
    "sgd_reg.fit(X, y.ravel())\n",
    "sgd_reg.predict([[1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5.2 라쏘 회귀\n",
    "- 덜 중요한 특성의 가중치를 제거\n",
    "- 자동으로 특성 선택을 하고 희소 모델(sparse model)을 만든다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4.76333499])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "\n",
    "lasso_reg = Lasso(alpha = 0.1)\n",
    "lasso_reg.fit(X, y)\n",
    "lasso_reg.predict([[1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lasso 대신 SGDRegressor(`penalty='l1'`)을 사용할 수도 있음"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5.3 엘라스틱넷\n",
    "- 릿지와 라쏘의 절충 모델\n",
    "- 혼합 정도는 혼합 비율 r을 사용해 조절\n",
    "    - r = 0: Ridge\n",
    "    - r = 1: Lasso\n",
    "    \n",
    "- 릿지가 기본이 되지만 쓰이는 특성이 몇 개 뿐이라면 라쏘나 엘라스틱넷이 낫다!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4.76584464])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import ElasticNet\n",
    "\n",
    "elastic_net = ElasticNet(alpha = 0.1, l1_ratio = 0.5)\n",
    "elastic_net.fit(X, y)\n",
    "elastic_net.predict([[1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5.4 조기 종료\n",
    "규제의 다른 방식은 에러가 최솟값에 도달하면 훈련을 중지하는 것(조기종료)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`warm_start = True` fit()이 호출될 때 처음부터 시작하지 않고 이전 모델 파라미터에서 훈련을 이어함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'X_train' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-36-6f63474b5ac7>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      7\u001b[0m ])\n\u001b[0;32m      8\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 9\u001b[1;33m \u001b[0mX_train_poly_scaled\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpoly_scaler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit_transform\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     10\u001b[0m \u001b[0mX_val_poly_scaled\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpoly_scaler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_val\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'X_train' is not defined"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.base import clone\n",
    "\n",
    "poly_scaler = Pipeline([\n",
    "    ('poly_features', PolynomialFeatures(degree=90, include_bias = False)),\n",
    "    ('std_scaler', StandardScaler())\n",
    "])\n",
    "\n",
    "X_train_poly_scaled = poly_scaler.fit_transform(X_train)\n",
    "X_val_poly_scaled = poly_scaler.transform(X_val)\n",
    "\n",
    "sgd_reg = SGDRegressor(max_iter = 1, tol = np.infty, warm_start = True, penalty = None, learning_rage = 'constant', eta0 = 0.0005)\n",
    "\n",
    "minimum_val_error = float('inf')\n",
    "best_epoch = None\n",
    "best_model = None\n",
    "\n",
    "for epoch in range(1000):\n",
    "    sgd_reg.fit(X_train_poly_scaled, y_train)\n",
    "    y_val_predict = sgd_reg.predict(X_val_poly_scaled)\n",
    "    val_error = mean_squared_error(y_val, y_val_predict)\n",
    "    \n",
    "    if val_error<minimum_val_error:\n",
    "        minimum_val_error = val_error\n",
    "        best_epoch = epoch\n",
    "        best_model = clone(sgd_reg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'X_train' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-37-755069f3c94b>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      7\u001b[0m     ])\n\u001b[0;32m      8\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 9\u001b[1;33m \u001b[0mX_train_poly_scaled\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpoly_scaler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit_transform\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     10\u001b[0m \u001b[0mX_val_poly_scaled\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpoly_scaler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_val\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'X_train' is not defined"
     ]
    }
   ],
   "source": [
    "# 노트북\n",
    "from copy import deepcopy\n",
    "\n",
    "poly_scaler = Pipeline([\n",
    "        (\"poly_features\", PolynomialFeatures(degree=90, include_bias=False)),\n",
    "        (\"std_scaler\", StandardScaler())\n",
    "    ])\n",
    "\n",
    "X_train_poly_scaled = poly_scaler.fit_transform(X_train)\n",
    "X_val_poly_scaled = poly_scaler.transform(X_val)\n",
    "\n",
    "sgd_reg = SGDRegressor(max_iter=1, tol=-np.infty, warm_start=True,\n",
    "                       penalty=None, learning_rate=\"constant\", eta0=0.0005, random_state=42)\n",
    "\n",
    "minimum_val_error = float(\"inf\")\n",
    "best_epoch = None\n",
    "best_model = None\n",
    "for epoch in range(1000):\n",
    "    sgd_reg.fit(X_train_poly_scaled, y_train)  # 중지된 곳에서 다시 시작합니다\n",
    "    y_val_predict = sgd_reg.predict(X_val_poly_scaled)\n",
    "    val_error = mean_squared_error(y_val, y_val_predict)\n",
    "    if val_error < minimum_val_error:\n",
    "        minimum_val_error = val_error\n",
    "        best_epoch = epoch\n",
    "        best_model = deepcopy(sgd_reg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.6 로지스틱 회귀\n",
    "- 샘플이 특정 클래스에 속할 확률을 추정할 때 널리 쓰임\n",
    "- 확률이 50%가 넘으면 해당 클래스에 속한다고 예측(이진 분류기)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.6.1 확률 추정\n",
    "- 선형 회귀처럼 바로 결과를 출력하지 않고 결괏값의 `로지스틱logistic`을 출력\n",
    "- 로지스틱은 0과 1 사이의 값을 출력하는 `시그모이드 함수`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.6.2 훈련과 비용 함수\n",
    "- 양성 샘플(y=1)에 대해선 높은 확률을 추정하고 음성 샘플(y=0)에선 낮은 확률을 추정하는 모델의 파라미터 벡터 theta 찾기\n",
    "- 맞게 측정할수록 비용이 0에 가까워짐"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.6.3 결정 경계\n",
    "[iris data]  \n",
    "3종류의 붓꽃, 꽃잎/\n",
    "꽃받침의 너비와 길이"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "꽃잎(petal)의 너비를 기반으로 Versicolor 종을 감지하는 분류기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data',\n",
       " 'target',\n",
       " 'frame',\n",
       " 'target_names',\n",
       " 'DESCR',\n",
       " 'feature_names',\n",
       " 'filename']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "iris = datasets.load_iris()\n",
    "list(iris.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = iris['data'][:, 3:] #꽃잎 너비\n",
    "y = (iris['target'] == 2).astype(np.int) #맞으면 1, 아니면 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "log_reg = LogisticRegression()\n",
    "log_reg.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "꽃잎 너비가 0~3cm인 꽃에 대해 추정 확률 계산"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1756ff56c40>]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3gVxfrA8e+kEkhoJtQASQSlSECJEBQQBKVcr4CIFEUBFbmKil5REGwgCCoqXdGLiPUHClKkiIr0liC9S5FQE0oIpCfz+2MChBDIgZyTPeX9PM8+55ydPbvvuvhmzuzsjNJaI4QQwvV5WR2AEEII+5CELoQQbkISuhBCuAlJ6EII4SYkoQshhJvwserAwcHBOiwszKrDCyGES4qNjU3QWofkV2ZZQg8LCyMmJsaqwwshhEtSSh28Wpk0uQghhJuQhC6EEG5CEroQQrgJSehCCOEmJKELIYSbKDChK6WmKKVOKKW2XqVcKaXGKqX2KqU2K6XusH+YQgghCmJLDX0q0OYa5W2BGjlLH2BS4cMSQghxvQrsh661XqaUCrvGJu2BadqMw7tGKVVaKVVRa33UTjFe4ZdfYP168Pa+tPj5wUsvmfLff4fduy+VeXlB8eLQpYspX7UKjh69/PuBgdCsmSnfuhXOnzf79Pc3ryVKQMWKpjw11XzHxweUctRZCiHcidaatKw00rPS8fHyobhvcbsfwx4PFlUGDuX6HJez7oqErpTqg6nFU7Vq1Rs+4C+/wKQ8vwOKFbuU0KdOhW++ubw8OPhSQn//fZg9+/Ly8HDYt8+879/f/FHILTISNm0y75s3h7VrTTL38zNL06YmLoCOHeHwYfNH4MLSsCH897+mfPx4yMgwf0QCA6FMGQgLg5o1TXlWlvmDIYQoelprktKTOJN6hrNpZ69YElMTL75PSk8iJTOFlIyUAl9TM1PRmPknBt49kPdavWf32O2R0POro+Y7a4bWejIwGSAqKuqGZ9aYOBEmTDCJL/dywYQJ8MEHZl129uVlAGPGwNChl3/X1/dS+ahRcPw4pKdfWoKCLpX/5z/wwANmfVqaea1W7VJ55cqmFn/+PBw5Yl4DAy+VDxsGJ05cHlO3bvDdd+Z96dLmtWxZk+zLloXOnc1xtYZvv4Xy5aFCBfOroWxZ8ytECJG/bJ1N/Pl4DicdJu5sHHFn4zh+7jgJyQkkpCSY1+QE4s/Hk5CcQEZ2RoH7DPILItAvkOK+xQnwDSDAJ4AA3wDKFCtDpaBK5nPOuguvxXyK4e/tT6PQRg45T3sk9DigSq7PocARO+z3mpQyTR4++ZxByZJmuZrcyTc/DRpcu/yJJ65dPn78tcsPHTJJ/vx5SEqCM2cuxas1vPoqnDoFp0+b15Mn4dw5U37uHPTocfn+/P3NH6hXX4XkZHP8sDDzq+OWW6BUqWvHI4Sr01pzMuUke0/tZc/JPew9tZe9p/dy8MxB4s7GcSTpSL5JumxAWYKLBxNSPITw0uHcWelOgosHE1w8mLIBZSnpXzLfJdAvEC/lfLUoeyT0OUA/pdQPQCMg0ZHt5+7gQjNNmTJXlikFb7xx9e8WL27uDxw7Zu4DHD0KcXFQv74p378fXnvt8u+EhsInn0CnTpCYCNu3myakEiXsd05CFJVTKafYfHwzW45vYfPxzWw+sZldCbtITEu8uI1CUa10NcJKh9G0WlMqB1UmtGTopdeSlSlXohw+XpYNZ+UQBZ6NUup7oDkQrJSKA94CfAG01p8C84F2wF4gGejlqGCFaVuvUcMs+alTxyTtAwfMPYGdO2HbNqhUyZSvWGGai7y8zLZ33mmWTp0gJN/x24SwTnJGMrFHYlkTt4bVcatZd3gdh5MOXyy/KeAmIstH8mjdR6lxUw2ql61O9bLVCS8djr+Pv4WRW0NZNUl0VFSUltEWi96pUyapx8SYnkLr15smnW3boHZt+OMPU9aiBdxxh9ycFUUrJSOFFf+s4Ld9v/HHgT/YeGwjmdmZANxc5mYahTaifvn6RJaPJLJ8JBUCK6A8rKuZUipWax2VX5l7/d4QBSpbFh580Cxg2uwPHIALnY4WL4aRI837UqWgdWuzbbducuNVOMbfp/7m550/s2DvAlb8s4K0rDR8vXxpXKUxr971KtGh0USHRhNSQn5CFkRq6OIKR4/Cn3+arpvz5pkePrt3m/b9X3817fXlylkdpXBlW45v4acdPzFr5yw2H98MQN1ydbkv4j5aRbSiWbVmlPCTmzz5uVYNXRK6uKbsbNP1MjTUdO8MCYGzZ03NvUcPU3svbv/nI4QbOnH+BN9t+Y6vNn3FxmMbUSiaVG3CQ7UeokPNDoSVDrM6RJcgTS7ihnl5mWQOpj19xQr4+mvTF75bN9PdcuJEePRRa+MUzklrzZIDSxi3bhzzds8jMzuTqEpRjGs7js61O1M+sLzVIboVSejiutSuDe+9B8OHw9Kl8L//wa23mrKdO03PmrZtZUgET5eSkcK3W75l7NqxbDmxheDiwfRv1J+e9XtSp1wdq8NzW5LQxQ3x8jI9YVq0uLRu0iQYOxbq1TN96Tt2lBupnuZ8+nkmxUzi/ZXvE58cT73y9Zjy4BS61e1GMZ9iVofn9uR/N2E3H35oxtFJToaHHzYPL82aZXVUoigkZyQzetVoIsZGMGDxAG6veDtLnljCX8/8Ra/be0kyLyKS0IXd+PqaYRF27DBt7NnZpllGuC+tNd9u/pZbxt3CK4tfoV75eqzsvZJFjy2ieVhzj+sjbjVpchF25+0N3bub0S3T0sy6P/+EyZPNSJcXbrIK17Y2bi39F/VnTdwaGlRswHedvqNZtWZWh+XRpIYuHMbb+1KXxj17TPNL7dqmV0x2trWxiRt3Nu0s/5n3H6L/F82BMweY8uAU1j29TpK5E5CELorE00+b4QUaNYLnnjPjx2/fbnVU4not2LOA2ybexmexn9G/UX9299tNr9t7OeXIg55IroIoMhER5knTr74yXRwXLLA6ImGr8+nneXL2k7T7rh1B/kGsenIVH7f5mCD/oIK/LIqMtKGLIqUUPP44tGtnxpUB87DSrbfKaI/OauOxjXT9sSu7T+5mUJNBvHXPWx45kqErkBq6sERwsOmjnpZmbp5GRsLKlVZHJXLTWjNx/USiv4jmbNpZfnv8N0a0HCHJ3IlJQheW8vc3TS9BQeYhpc8+szoiAZCWmcZTc57iufnP0TKiJZv6buLe8HutDksUQBK6sFxkJKxbB61aQd++8MwzkJlpdVSe6/i549w77V6mbJzCm83eZG63uTJ0rYuQNnThFEqXhrlzzZABcXEysYZVthzfQrvv2nEy+STTH55O5zqdrQ5JXAdJ6MJpeHvDiBGmj7pSZn5UPz+oXNnqyDzD8oPL+ff3/ybQL5CVvVdye8XbrQ5JXCdpchFOx8vLzKT08MNw112mi6NwrLm75nL/N/dTIbACq55cJcncRUlCF05JKfj8c9ML5p57zENJwjG+3vQ1Hf+vI3XL1WVF7xVULVXV6pDEDZKELpzWHXeYwb28vaF5c9i82eqI3M83m7/hiZ+f4J6we/j98d8JLh5sdUiiECShC6d2660mqfv7w4ABVkfjXn7Y+gNP/PwELcJbMLfbXHnq0w3ITVHh9GrUgGXLTE8YYR8zts3gsZmP0aRqE+Z0nUNxX5kY1h1IDV24hIgIM1RASgr85z+ma6O4Mb/+/SvdZ3YnOjSaX7r/Qgm/ElaHJOxEErpwKfv2mckzWreGkyetjsb1bDi6gU7TO1E7pDa/dP+FQL9Aq0MSdiQJXbiUOnVgzhz4+2948EFITbU6Itex//R+/vXdvygbUJYFjy6gVLFSVock7EwSunA5zZvD11/DqlXw1FOmz7q4tpPJJ2n7bVtSM1NZ8OgCKgVVsjok4QCS0IVL6twZ3n0XFi6Ef/6xOhrnlpmdySM/PsL+M/uZ03UOtUNqWx2ScBBJ6MJlvf46bN0K1apZHYlzG/DrAP7Y/wefPfAZTas1tToc4UCS0IXLUgoqVDBjv4wYYZK7uNy0TdP4ZO0nvNDwBXrW72l1OMLBJKELl3fyJIwdC506wdmzVkfjPNYfXk+fuX1oEdaCD+//0OpwRBGwKaErpdoopXYppfYqpQbmU15KKTVXKbVJKbVNKdXL/qEKkb+QEJg+3fR86d1bbpICnEo5RafpnagQWIHpnafj6+1rdUiiCBSY0JVS3sAEoC1QG+imlMp7V+U5YLvWuh7QHBitlPKzc6xCXFWzZjBqFPz0E3z8sdXRWEtrTe/ZvTl27hg/PvKjjM/iQWypoTcE9mqt92mt04EfgPZ5ttFAkFJKAYHAKUDmnBFF6uWXTbPL4MFw7JjV0Vhn3LpxzN41m1GtRhFVKcrqcEQRsmUsl8rAoVyf44BGebYZD8wBjgBBQBetdXbeHSml+gB9AKpWlSE6hX0pBf/7H+zaZW6WeqINRzcwYPEAHrjlAfpH97c6HFHEbKmhq3zW5W2lbA1sBCoB9YHxSqmSV3xJ68la6yitdVRIiMxRKOyvVClo2NC837LF2liK2rn0c3T5sQvlSpRjavupmB/MwpPYktDjgCq5PodiauK59QJmamMvsB+oaZ8Qhbh+v/xiJp/+8UerIyk6A34dwN+n/ubbh77lpuI3WR2OsIAtCX09UEMpFZ5zo7Mrpnklt3+AlgBKqfLArcA+ewYqxPW4/35TU+/TxzNGZly0dxGfxn7Kfxv/l2bVmlkdjrBIgQlda50J9AMWATuA6VrrbUqpvkqpvjmbDQPuUkptAX4HXtNaJzgqaCEK4utrRmVMSzNJ3Z27Mp5OOc2Tc56kVnAtht07zOpwhIVsmuBCaz0fmJ9n3ae53h8B7rdvaEIUTvXqMHIkvPCCGczr8cetjsgxXlz4IsfOHePnrj9TzKeY1eEIC8mTosKtPfcctG1rhgdwR7N3zubrzV8zuOlg6aIoZAo64d68vMwNUnfs8HE27SzPzX+OyPKRDG422OpwhBOQhC7cnlKmDf2rr6BkSXjoIasjso/Bvw/mSNIRZnaZiZ+3PJgtJKELD5GdDRMnwsGDcO+9rj/h9Nq4tUxYP4Hn7nyOhpUbWh2OcBLShi48grc3fPopJCTAkCFWR1M4GVkZ9JnXh0pBlRjecrjV4QgnIgldeIw77jA3SSdOhNhYq6O5cR+v+ZjNxzczru04Svpf8UC28GCS0IVHGTYMypWDvn1ds+fLwTMHefvPt+lQswMda3W0OhzhZKQNXXiUUqVg8mRzo9TLBaszAxYPAGBMmzEWRyKckSR04XEefNDqCG7Mkv1LmLF9BkObD6VqKRmtVFzJBesoQtjHsGHw4otWR2GbzOxMXlj4AmGlw3jlrlesDkc4KUnowmOdPg3jxsHGjVZHUrBPYz5l64mtfHT/RwT4BlgdjnBSktCFx3rjDShb1sx05MyDdyUkJ/DGkjdoFdGKDjU7WB2OcGKS0IXHKlMG3nkHliyBOXkHhHYib/zxBklpSYxpM0YmrRDXJAldeLRnnoFateC115yzG+OO+B18vuFznr3zWWqH5J2bXYjLSS8X4dF8fGDKFAgIcM5ujIN+H0QJvxK80ewNq0MRLkASuvB40dGX3mvtPCMzLj+4nNm7ZjPi3hGElJA5eEXBnLBOIkTRy86GHj3g1VetjsTQWjNg8QAqB1XmxWgX6VspLCcJXQhMc4uPD4wdC//8Y3U08NOOn1h7eC1DWwyluG9xq8MRLkISuhA53nnHNLe89Za1caRnpTPwt4HUCanDE/WesDYY4VIkoQuRo2pVMxrjtGmwbZt1cXwW8xl/n/6b9+97H28vb+sCES5HEroQubz+OgQGwtCh1hw/KS2JocuG0iKsBW2rt7UmCOGypJeLELncdBNMnw716llz/DFrx5CQnMDIViPlISJx3SShC5FH69bm9cJwAEWVV0+nnObDVR/y4K0PyrRy4oZIk4sQ+YiLg2bNYPHiojvmR6s/IjEtkaHNLWrvES5PEroQ+ShXznRffPPNohm4KyE5gU/WfkLn2p2pV8Gi9h7h8iShC5EPPz8zmfTatbBggeOP9/7K90nOSOad5u84/mDCbUlCF+IqevaE8HDH19KPJh1l/LrxdK/bnVohtRx3IOH2JKELcRW+vmbM9NhYmDfPccd5b8V7pGel89Y9Fj/RJFye9HIR4hp69IC0NGjZ0jH7P5R4iM9iP6NX/V5UL1vdMQcRHkMSuhDX4OMDffs6bv8jlo9Aa82QZkMcdxDhMWxqclFKtVFK7VJK7VVKDbzKNs2VUhuVUtuUUkvtG6YQ1pozBzp0sO8kGEeSjjBl4xR61e9FtdLV7Ldj4bEKTOhKKW9gAtAWqA10U0rVzrNNaWAi8KDWug7Q2QGxCmGZc+dg9mz48Uf77XP0qtFkZWfxWpPX7LdT4dFsqaE3BPZqrfdprdOBH4D2ebbpDszUWv8DoLU+Yd8whbBWly5QsyYMG2afWnpCcgKfxn5Kt7rdiCgTUfgdCoFtCb0ycCjX57icdbndApRRSv2plIpVSj2e346UUn2UUjFKqZj4+Pgbi1gIC3h7w+DBsHUrzJ1b+P2NWTOG5IxkBjUZVPidCZHDloSe30gWeXvl+gANgH8BrYE3lFK3XPElrSdrraO01lEhITKllnAtXbtCRAS8+27h+qUnpiYybt04Hqr1kEz8LOzKll4ucUCVXJ9DgSP5bJOgtT4PnFdKLQPqAbvtEqUQTsDHBz76yHRjLMzcoxPXTyQxLZHXm7xu3wCFx7Mloa8HaiilwoHDQFdMm3lus4HxSikfwA9oBHxsz0CFcAbt8949uk7JGcl8vOZj2lRvQ4NKDewTlBA5Cmxy0VpnAv2ARcAOYLrWeptSqq9Sqm/ONjuAhcBmYB3whdZ6q+PCFsI658+bZpcVK67/u5/Hfk58cjyDmw62f2DC4yldFEPJ5SMqKkrHxMRYcmwhCiMlxYzxcttt8Ntvtn8vLTONm8fezM1lb2ZpT3lUQ9wYpVSs1joqvzIZy0WI6xQQAK+8Ar//DqtX2/69aZumcTjpsNTOhcNIQhfiBvTta6arGz7ctu0zszMZtXIUDSo24L6I+xwbnPBYktCFuAGBgfDSS/DLL/DXXwVvP33bdP4+/TeDmw6WuUKFw8jgXELcoH79TDL3KeD/omydzYjlI6gTUof2NQvZTUaIa5CELsQNKlXKtrFd5uyaw7b4bXzT8Ru8lPwoFo4j/7qEKKQDB2Dq1PzLtNYMXz6ciDIRdLmtS1GGJTyQJHQhCmn8eHjqKdi378qyxfsWE3MkhoF3D8THS34QC8eShC5EIb38shm8a9SoK8uGLx9O5aDKPF4v3/HqhLArSehCFFKlStC7t2l2OXz40voV/6xg2cFlDLhrAP4+/pbFJzyHJHQh7ODVVyErC0aPvrRu+PLhhBQP4ekGT1sXmPAoktCFsIPwcOjVCzIzzefYI7Es3LuQl6JforhvcWuDEx5D7tIIYSeTJ18aUve9Fe9Ryr8Uz975rLVBCY8iNXQh7ORCMp/+635++us3+jXsR6lipawNSngUSehC2NGePdCldTi+sS/SP7q/1eEIDyNNLkLYkXfwPqixC591AwjQgVaHIzyM1NCFsKP3V76Pzz2jSEkM5PPPrY5GeBpJ6ELYyeGzh/ly45c82b4m99wDH3xg5h8VoqhIQhfCTkavHk1Wdhav3f0agwebqeq2bLE6KuFJpA1dCDtISE7gs9jP6F63O+FlwglrBYcOQVCQ1ZEJTyI1dCHs4JM1n5CSkcKgJoMA04UxKAiysy8fDkAIR5KELkQhJaYmMn7deB6q9RC1QmpdVtatG9x/v0nsQjiaJHQhCmnC+gkkpiVerJ3n1qEDbN8OP/9sQWDC40hCF6IQkjOS+XjNx7Sp3oYGlRpcUf7II1C9uplMWmsLAhQeRRK6EIUwOXYyCckJDG46ON9yb28YOBA2bIBFi4o4OOFxJKELcYNSM1P5YNUHNA9rTpOqTa66XY8eUKXK1aepE8JepNuiEDdo6sapHEk6wrQO0665nZ8fLF4MERFFFJjwWFJDF+IGZGRlMHLFSKJDo7k3/N4Ct7/1VvD1hYyMIghOeCxJ6ELcgG+3fMvBxIMMaToEdWHc3AKsWwfVqkFMjIODEx5LEroQ1ykrO4sRy0dQv0J92tVoZ/P3ataElBQYMcKBwQmPJgldiOs0Y/sM9pzac121c4CSJeH552HWLNi2zYEBCo8lCV2I65Ctsxm+fDi1gmvRsVbH6/7+iy9CiRLw3nsOCE54PJsSulKqjVJql1Jqr1Jq4DW2u1MplaWUeth+IQrhPObsmsPWE1sZ3HQwXur660M33QR9+8L338PBgw4IUHi0Av9FKqW8gQlAW6A20E0pVfsq240C5PEJ4Za01ry77F1uLnMzXW7rcsP7+e9/zUNGVavaMTghsK0fekNgr9Z6H4BS6gegPbA9z3bPAz8Bd9o1QiGcxKK/FxF7NJYv/v0FPl43/ghHxYpmEcLebPnNWBk4lOtzXM66i5RSlYGOwKfX2pFSqo9SKkYpFRMfH3+9sQphGa01w5YNo0rJKvSo18MO+4PXXoMhQ+wQnBA5bEno+d3GzzvM0CfAa1rrrGvtSGs9WWsdpbWOCgkJsTVGISy3eN9iVh1axcAmA/Hz9iv0/pSCY8fgo49A6jbCXmxJ6HFAlVyfQ4EjebaJAn5QSh0AHgYmKqU62CVCISymteatP9+iSskqPHn7k3bb76BBkJoKn3xit10KD2dLQl8P1FBKhSul/ICuwJzcG2itw7XWYVrrMOBH4FmttYwALdzCwr0LWRO3hiHNhuDv42+3/dasCZ06wfjxcOqU3XYrPFiBCV1rnQn0w/Re2QFM11pvU0r1VUr1dXSAQlhJa82bf75JWOkwetbvaff9v/kmJCXBhx/afdfCA9l0q15rPR+Yn2ddvjdAtdY9Cx+WEM5h3u55xByJ4Yt/f2GXtvO86taFMWOgZUu771p4IBk+V4iruNB2HlEmgsfrPe6w4zz/vMN2LTyMPPovxFXM3jWbv479xZvN3sTX29ehxzpwALp3h8OHHXoY4eYkoQuRj2ydzVt/vkWNsjV4NPJRhx9Pa5gxQ0ZiFIUjCV2IfMzcMZPNxzfz5j1vFuqpUFuFh8NTT8Hnn8sYL+LGSUIXIo/M7EyG/DGEWsG16HZbtyI77uDB4OUFw4YV2SGFm5GELkQeX/71JbtO7mJEyxF4e3kX2XFDQ+GZZ8xk0nv2FNlhhRuRXi5C5JKckczbS9+mcWhj2t/avsiPP2iQmXv0ppuK/NDCDUhCFyKXcWvHcSTpCN93+v66ZiOylwoV5CEjceOkyUWIHKdTTjNy5Uja1WhHs2rNLI1l+XLo18/0fhHCVpLQhcgxauUoElMTea+l9fPDbdkCEybA/PkFbyvEBZLQhQAOnz3MmLVjeDTyUSLLR1odDk8/DTVqwKuvQmam1dEIVyEJXQjgjSVvkJWdxdDmQ60OBTA3Rt97D7Zvh6++sjoa4SokoQuPF3sklqkbp9I/uj/hZcKtDueihx6C6GgzIuP581ZHI1yB9HIRHk1rzUuLXiK4eDCDmw62OpzLKGVmNFq/3tTYhSiIJHTh0WbumMnyf5bz6b8+pVSxUlaHc4XGjc0ihC2kyUV4rNTMVAYsHkDdcnV58g77TS3nCN98Ay+9ZHUUwtlJQhcea8yaMew/s5+PWn9UJANwFcbOnWbu0ZUrrY5EODNJ6MIjHU06yvDlw/n3Lf+mVUQrq8Mp0KBBULkyvPACZGVZHY1wVpLQhUd6ZfErpGWlMfr+0VaHYpMSJcyQABs2wJQpVkcjnJUkdOFx/tj/B99t+Y6Bdw+kxk01rA7HZl26QNOm8PrrZmJpIfJy7oZDIewsPSudZ395logyEQxsMtDqcK6LUjB+vJmuLjDQ6miEM5KELjzK6FWj2XVyF/O7zyfAN8DqcK5bZKRZwLSlexfdcO3CBUiTi/AYB84cYNiyYTxU6yHa1mhrdTiFMnEiNGoE6elWRyKciSR04RG01jy/4Hm8lBeftP7E6nAKLTQUYmNl7HRxOUnowiN8v/V75u2ex9AWQ6lSqorV4RTagw+asV6GDoW9e62ORjgLSejC7R0/d5znFzxPo8qNeLHRi1aHYzdjx4K/P/TtKxNhCEMSunB7zy94nnPp55jSfkqRTvrsaJUrw8iRsGSJaX4RQhK6cGs/bf+JGdtn8PY9b1M7pLbV4djdM8/Apk0QFWV1JMIZSEIXbiv+fDzPzX+OOyrewSt3vWJ1OA7h5QW33Wber1kD2dnWxiOsJQlduCWtNU/NfYrTqaf5sv2X+Hq794Diq1ebYXbHjrU6EmElmxK6UqqNUmqXUmqvUuqKx+uUUo8qpTbnLKuUUvXsH6oQtvt8w+fM2TWHkS1HOsUcoY4WHQ0PPGAG8dq50+pohFUKTOhKKW9gAtAWqA10U0rlbYzcD9yjtY4EhgGT7R2oELbambCT/gv7c1/EfbwY7T69Wq5FKZg82Qzi1bUrpKZaHZGwgi019IbAXq31Pq11OvAD0D73BlrrVVrr0zkf1wCh9g1TCNukZ6Xz6MxHKe5bnKkdpuKlPKdVsWJFmDbN3CT973+tjkZYwZaxXCoDh3J9jgMaXWP7J4EF+RUopfoAfQCqVq1qY4hC2G7Qb4PYcHQDs7rMolJQJavDKXLt2sGQIeZJUuF5bEnoKp91+T7GoJRqgUnoTfIr11pPJqc5JioqSh6FEHY1Y9sMPlrzEf3u7EeHmh2sDscyw4Zdeq+1aY4RnsGW36NxQO5npUOBI3k3UkpFAl8A7bXWJ+0TnhC22Zmwk95zehMdGs3o1q4xaYWjTZ8OLVtKe7onsSWhrwdqKKXClVJ+QFdgTu4NlFJVgZlAD631bvuHKcTVnUs/R6fpnQjwCWBG5xn4eftZHZJT8Pc3T5E+84wMDeApCkzoWutMoB+wCNgBTNdab1NK9VVK9c3Z7E3gJmCiUmqjUirGYRELkYvWmqfmPMXOhJ183+l7QktK4/EF7dvD22+bG6VjxlgdjSgKSlv0pzsqKkrHxEjeF4Xz9p9v887SdxjVahSv3v2q1eE4nexs6NQJ5lXwdt0AAA2PSURBVM6FRYtME4xwbUqpWK11voM9eE6fLuF2vtvyHe8sfYee9Xsy4K4BVofjlLy8TA29Zk3T/CLcm0xBJ1zSqkOr6D27N82qNeOzBz5DSVeOqwoKMkMDBAVZHYlwNKmhC5ezM2En7X9oT2jJUH565Ce5CWqDC8l80yYzOca5c9bGIxxDErpwKf8k/sP9X9+Pl/JiwaMLCC4ebHVILuXwYZg/Hzp3howMq6MR9iYJXbiM+PPx3P/1/SSmJbLw0YXUuKmG1SG5nHbtYNIkWLgQevaErCyrIxL2JG3owiWcTjlN22/bcjDxIL8+9iu3V7zd6pBc1tNPw8mTZmRGb2/48kvzKlyfJHTh9E4mn+S+r+9jW/w2ZnWZRdNqTa0OyeUNHAiZmbBsmWl6kYTuHiShC6eWkJxAq2mt2Jmwk5+7/EzbGm2tDsltDBlikrqPDyQmQmCgJHZXJ23owmkdPnuYFl+1YNfJXczuOluSuQP4+EB6Otx3H3TpAmlpVkckCkMSunBKO+J3cNeUuzhw5gDzus2jdfXWVofktvz8zKQYP/1kbpomJVkdkbhRktCF01l1aBV3T7mbtMw0lvVcRssIeV7d0V5+2TxRunQptGgBR49aHZG4EZLQhVOZtmka9351L8HFg1n95GrpzVKEevSAOXPMnKQ9elgdjbgRclNUOIWMrAwGLB7AmLVjaBHWgumdp8tDQxZo1w5WrYJixczn7GwzHoxwDZLQheVOnD9B1x+7suTAEvo36s8H93+Aj5f807RKZKR51RqeeAJCQmDkSNPWLpyb/O0Vllq4dyGRkyJZdWgVX3X4io/bfCzJ3ElkZUHp0vDxx3DXXbB3r9URiYJIQheWSMtM46WFL9H227aElAghpk8Mj9d73OqwRC4+PjBuHMycCfv2we23w9dfy+xHzkwSuihyqw+tpsHkBnyy9hP63dmPdU+t47Zyt1kdlriKjh1h40aoXx9efBFOn7Y6InE1ktBFkUlKS+L5+c9z95S7SUxL5JfuvzCu3TgCfAOsDk0UoGpV+PNPWLECypY1N0tnzjSvwnlIQhcOl62z+WbzN9SeWJsJ6yfQr2E/tj+7nXY12lkdmrgO3t5Qu7Z5P3OmmdouOtr0ihHOQRK6cKjlB5fT6ItG9JjVg3IlyrGy90rGth1LkL9Mn+PKOnUyDyIdPgx33w3du8OhQ1ZHJSShC4fYcHQDHX7oQLOpzTiadJRpHaax/un1NK7S2OrQhB0oZR4+2rXLDPI1axZ06GB1VEISurCrtXFreeC7B2gwuQFLDy5laPOh7H5+Nz3q9cBLyT83dxMYCMOGmadLJ00y686ehQEDIC7O2tg8kfwfJgotIyuDGdtm0PTLpkT/L5rVcat5t8W7HHjxAG/c8wbFfYtbHaJwsGrVoGFD837ZMtN3PSzMDPq1erV0dSwq8gSHuGGHzx5m6sapTIqZxOGkw4SXDufD+z6kT4M+0kbuwR54wDyENH48fPEF/N//mWS/dOmlIQWEY0hCF9clKS2JmTtm8vXmr/lj/x9oNPdF3Mekf02iXY12eHvJDAnC1M4//BDeftvcPI2JuZTMx4yBOnXMqI4yoYZ9KW3Rb6GoqCgdExNjybHF9TmZfJL5e+YzZ/cc5u+ZT3JGMhFlIugR2YPHIh+jetnqVocoXERqqunTHh8P5crBgw/CQw/BvfeCv7/V0bkGpVSs1joqvzKpoYsrZOtsNh3bxO/7f2fu7rms+GcF2TqbioEVeaLeEzwW+RiNQxujlLI6VOFiihWDgwdh3jwzocYPP5hmmffeM/Ocnj8PJ05AeLjVkbomSeiCzOxMtp3YxtKDS1lyYAlLDyzldKp5vjuyfCSvN3md9jXbc0fFO6Sniii0gADo3NksaWnw++9Qt64pW7gQHn4Ybr4ZWraEJk3Mw0vVq5uukuLapMnFw2RlZ7Hv9D7WH1nP+sPrWX9kPX8d+4vkjGQAwkuH0yKsBS3CW9A8rDmhJUMtjlh4krg406f9t9/MUANnz5r1e/aYpL5unanB161rmm48Mclfq8lFErqbSs1M5eCZg+xI2MH2+O1sj9/Otvht7EzYSWpmKgABPgHcXvF27qx0J3dWupMmVZtQrXQ1iyMXwsjKgh07YP166NnTJO/eveHLL015qVImsdevD2PHmvKTJ6FkSfD1tTR0h5KE7maydTYnk09y9NxRjp07xqHEQ+w/s58DZw6w/8x+9p/ez9Fzl08KWbVUVeqE1KF2SG1qh9SmQcUG1ClXR8YeFy7l7FnYuhW2bIHNm82Sng5r15ry1q3hjz9ML5vq1c1rvXrQt68pP3TIjPEe5MK9agud0JVSbYAxgDfwhdZ6ZJ5ylVPeDkgGemqtN1xrn5LQjWydTWJqIqdSTnE69TSnU05ffL2w7lTKKY6dO8axc8c4eu4ox88dJ0tnXbYfL+VFlZJVCC8TTnjpcMJKhxFeOpxbg2+lVnAt6RcuPMKsWRAba/rB79ljbsDWrQtLlpjyWrXMU61BQVCxIgQHQ6tW8M47pnzyZDMzU0gIlCljtitf3vTIcRaF6uWilPIGJgD3AXHAeqXUHK319lybtQVq5CyNgEk5r05Ha022ziZLZ5GVnXXxNfe6zOxM0rPSSc9KJy0r7dL7zLR81+cuS8tKIzkjmfPp5zmfYZZz6ecufc7nVXP1P6r+3v6UDShLhcAKVAisQGT5SCoGVrz4uWJQRSoHVSa0ZCi+3m78O1MIG3TsaJbcMjMvvR82zEzWERcHx49DQoKp4V/wyiuQlHT593v1gilTzNOuZcuam7olS5olKAgeeQSeecbs54UXTHmxYuY1IMDc2G3c2HTZXLDA/LFo0sQx52/L7+2GwF6t9T4ApdQPQHsgd0JvD0zTprq/RilVWilVUWt99MrdFc7CvQt5edHL10zI11p3reRpL17KixK+JSjhV+Lia6BfIEF+QVQIrGDW+Zp1gX6BlAkoQ5liZS6+lg0oe/G9jBUuROH45MpyDz987W3j4kySj4+HxETTxBOa0y8gO9u04Z89e/mSlmbKU1Lg55/Na0oKZGSY9UOHmoQeH2/63DdqBGvW2P88wbaEXhnIPTBmHFfWvvPbpjJwWUJXSvUB+gBUrVr1emMFoJR/KeqUq4O38sbby/viqxdel32+uF552bTOW+Ws9/LGx8sHf29//Lz9Li7+Ppc+5y7LvT53ufTRFsL1XKh5R0RcWebtDaNHX/27pUrBsWOXPmdmmlr5hadhy5eHv/5y7ANUtiT0/DJT3mquLdugtZ4MTAbThm7Dsa/QuEpjZlSZcSNfFUKIIuPjY0ajvMDPz/TIcSRbnhKJA6rk+hwKHLmBbYQQQjiQLQl9PVBDKRWulPIDugJz8mwzB3hcGdFAoiPaz4UQQlxdgU0uWutMpVQ/YBGm2+IUrfU2pVTfnPJPgfmYLot7Md0WezkuZCGEEPmx6akSrfV8TNLOve7TXO818Jx9QxNCCHE9ZKQlIYRwE5LQhRDCTUhCF0IINyEJXQgh3IRloy0qpeKBgzf49WAgwY7hWEnOxTm5y7m4y3mAnMsF1bTWIfkVWJbQC0MpFXO10cZcjZyLc3KXc3GX8wA5F1tIk4sQQrgJSehCCOEmXDWhT7Y6ADuSc3FO7nIu7nIeIOdSIJdsQxdCCHElV62hCyGEyEMSuhBCuAmnTuhKqTZKqV1Kqb1KqYH5lCul1Nic8s1KqTusiNMWNpxLc6VUolJqY87yphVxFkQpNUUpdUIptfUq5a50TQo6F1e5JlWUUkuUUjuUUtuUUi/ms41LXBcbz8VVrksxpdQ6pdSmnHN5J59t7HtdtNZOuWCG6v0biAD8gE1A7TzbtAMWYGZMigbWWh13Ic6lOTDP6lhtOJdmwB3A1quUu8Q1sfFcXOWaVATuyHkfBOx24f9XbDkXV7kuCgjMee8LrAWiHXldnLmGfnFyaq11OnBhcurcLk5OrbVeA5RWSlUs6kBtYMu5uASt9TLg1DU2cZVrYsu5uASt9VGt9Yac90nADsycvrm5xHWx8VxcQs5/63M5H31zlry9UOx6XZw5oV9t4unr3cYZ2Bpn45yfZwuUUnWKJjS7c5VrYiuXuiZKqTDgdkxtMDeXuy7XOBdwkeuilPJWSm0ETgCLtdYOvS42TXBhEbtNTu0EbIlzA2aMhnNKqXbAz0ANh0dmf65yTWzhUtdEKRUI/AT011qfzVucz1ec9roUcC4uc1201llAfaVUaWCWUuo2rXXuezZ2vS7OXEN3p8mpC4xTa332ws8zbWaI8lVKBRddiHbjKtekQK50TZRSvpgE+K3WemY+m7jMdSnoXFzpulygtT4D/Am0yVNk1+vizAndnSanLvBclFIVlFIq531DzLU5WeSRFp6rXJMCuco1yYnxf8AOrfVHV9nMJa6LLefiQtclJKdmjlIqAGgF7MyzmV2vi9M2uWg3mpzaxnN5GPiPUioTSAG66pzb4M5EKfU9ppdBsFIqDngLc7PHpa4J2HQuLnFNgLuBHsCWnPZagNeBquBy18WWc3GV61IR+Eop5Y35ozNdaz3PkTlMHv0XQgg34cxNLkIIIa6DJHQhhHATktCFEMJNSEIXQgg3IQldCCHchCR0IYRwE5LQhRDCTfw/VwhHk7ZKdfEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_new = np.linspace(0, 3, 1000).reshape(-1, 1)\n",
    "y_proba = log_reg.predict_proba(X_new)\n",
    "\n",
    "plt.plot(X_new, y_proba[:, 1], 'g-', label = 'Iris virginica')\n",
    "plt.plot(X_new, y_proba[:, 0], 'b--', label = 'Not Iris virginica')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_reg.predict([[1.7], [1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "로지스틱 회귀의 규제 하이퍼 파라미터는 alpha가 아니라 그의 역수인 C  \n",
    "C가 높을수록 규제가 줄어든다"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.6.4 소프트맥스 회귀\n",
    "- 직접 다중 클래스를 지원하도록 일반화 될 수 있음! 이를 `소프트맥스 회귀` 또는 `다항 로지스틱 회귀`라고 함\n",
    "- sample이 주어지면 모델이 클래스 k에 대한 점수 s(x)를 계산하고, s(x)에 소프트맥스 함수를 적용해 클래스의 확률을 추정\n",
    "- 한 번에 하나의 클래스만 예측(다중 클래스, not 다중 출력)\n",
    "\n",
    "\n",
    "`argmax` 함수를 최대화하는 변수의 값 반환 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Logistic Regression은 클래스가 둘 이상일 때 일대다(OvA) 방식을 사용하지만 multi_class 매개변수를 `multinomial`로 바꾸면 소프트맥스 회귀 사용 가능  \n",
    "- 이를 사용하려면 solver 매개변수에 `lbfgs`와 같이 지원하는 알고리즘 지정  \n",
    "- 기본적으로 C를 사용해 조절할 수 있는 l2 규제가 적용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=10, multi_class='multinomial')"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = iris['data'][:, (2,3)] #꽃잎 길이, 너비\n",
    "y = iris['target']\n",
    "\n",
    "softmax_reg = LogisticRegression(multi_class = 'multinomial', solver = 'lbfgs', C = 10)\n",
    "softmax_reg.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "softmax_reg.predict([[5, 2]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[6.38014896e-07, 5.74929995e-02, 9.42506362e-01]])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "softmax_reg.predict_proba([[5, 2]])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}