皮肤癌图像分类.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 皮肤癌图像分类：准确率>85%(仅训练四个周期）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本实验基于pytorch,对[kaggle](https://www.kaggle.com/kmader/skin-cancer-mnist-ham10000) 上的一万多张皮肤癌图像进行学习和分类。该图片的标签分为7个等级，如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 'nv': 'Melanocytic nevi',黑素细胞痣\n",
    "* 'mel': 'meldermatofibroma',mel症\n",
    "* 'bkl': 'Benign keratosis-like lesions ',良性keratosis-like病变\n",
    "* 'bcc': 'Basal cell carcinoma',基底细胞癌\n",
    "* 'akiec': 'Actinic keratoses',日光性角化症\n",
    "* 'vasc': 'Vascular lesions',脉管疾病\n",
    "* 'df': 'Dermatofibroma'皮肤纤维瘤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本例将使用Densenet-121进行训练，由于GPU资源限制，我们仅训练了4个周期，准确率就达到了85%。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，输入相关的库。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ham10000_images_part_1', 'ham10000_images_part_2', 'hmnist_28_28_RGB.csv', 'hmnist_8_8_RGB.csv', 'hmnist_8_8_L.csv', 'hmnist_28_28_L.csv', 'HAM10000_images_part_1', 'HAM10000_images_part_2', 'HAM10000_metadata.csv']\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "# python libraties\n",
    "import os, cv2,itertools\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tqdm import tqdm\n",
    "from glob import glob\n",
    "from PIL import Image\n",
    "\n",
    "# pytorch libraries\n",
    "import torch\n",
    "from torch import optim,nn\n",
    "from torch.autograd import Variable\n",
    "from torch.utils.data import DataLoader,Dataset\n",
    "from torchvision import models,transforms\n",
    "\n",
    "# sklearn libraries\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "# to make the results are reproducible\n",
    "np.random.seed(10)\n",
    "torch.manual_seed(10)\n",
    "torch.cuda.manual_seed(10)\n",
    "\n",
    "print(os.listdir(\"../input/skin-cancer-mnist-ham10000\"))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第一步：数据分析与预处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "获取所有图像数据路径，将ham10000_metada .csv中的行信息与其对应的图像匹配。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = '../input/skin-cancer-mnist-ham10000'\n",
    "all_image_path = glob(os.path.join(data_dir, '*', '*.jpg'))\n",
    "#连接两个或更多的路径名组件\n",
    "imageid_path_dict = {os.path.splitext(os.path.basename(x))[0]: x for x in all_image_path}\n",
    "#os.path.splitext(“文件路径”)    分离文件名与扩展名\n",
    "#os.path.basename(x):返回path最后的文件名\n",
    "lesion_type_dict = {\n",
    "    'nv': 'Melanocytic nevi',\n",
    "    'mel': 'dermatofibroma',\n",
    "    'bkl': 'Benign keratosis-like lesions ',\n",
    "    'bcc': 'Basal cell carcinoma',\n",
    "    'akiec': 'Actinic keratoses',\n",
    "    'vasc': 'Vascular lesions',\n",
    "    'df': 'Dermatofibroma'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此函数用于计算整个数据集的平均值和标准偏差，用于输入的标准化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_img_mean_std(image_paths):\n",
    "    \"\"\"\n",
    "        computing the mean and std of three channel on the whole dataset,\n",
    "        first we should normalize the image from 0-255 to 0-1\n",
    "    \"\"\"\n",
    "\n",
    "    img_h, img_w = 200, 200\n",
    "    imgs = []\n",
    "    means, stdevs = [], []\n",
    "\n",
    "    #Tqdm 是一个快速，可扩展的Python进度条，可以在 Python 长循环中添加一个进度提示信息，用户只需要封装任意的迭代器 tqdm(iterator)。\n",
    "    for i in tqdm(range(len(image_paths))):\n",
    "        img = cv2.imread(image_paths[i])\n",
    "        img = cv2.resize(img, (img_h, img_w))\n",
    "        imgs.append(img)\n",
    "    \n",
    "    \n",
    "    imgs = np.array(imgs,dtype=np.float32)/255.0\n",
    "    #normalize the image from 0-255 to 0-1\n",
    "\n",
    "    for i in range(3):\n",
    "        pixels = imgs[:, :, i, :].ravel()  # resize to one row\n",
    "        means.append(np.mean(pixels))\n",
    "        stdevs.append(np.std(pixels))\n",
    "\n",
    "        \n",
    "    means.reverse()  # BGR --> RGB\n",
    "    stdevs.reverse()\n",
    "\n",
    "    print(\"normMean = {}\".format(means))\n",
    "    print(\"normStd = {}\".format(stdevs))\n",
    "    return means,stdevs    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "返回RGB通道的平均值和std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 20030/20030 [03:41<00:00, 90.53it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "normMean = [0.62485343, 0.62214255, 0.620066]\n",
      "normStd = [0.17797484, 0.1801317, 0.18257397]\n"
     ]
    }
   ],
   "source": [
    "norm_mean,norm_std = compute_img_mean_std(all_image_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在原始数据aframe中添加三列:path(图像路径)、cell_type(全称)、cell_type_idx(单元格类型的对应索引，作为图像标签)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lesion_id</th>\n",
       "      <th>image_id</th>\n",
       "      <th>dx</th>\n",
       "      <th>dx_type</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>localization</th>\n",
       "      <th>path</th>\n",
       "      <th>cell_type</th>\n",
       "      <th>cell_type_idx</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>HAM_0000118</td>\n",
       "      <td>ISIC_0027419</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>../input/skin-cancer-mnist-ham10000/HAM10000_i...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HAM_0000118</td>\n",
       "      <td>ISIC_0025030</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>../input/skin-cancer-mnist-ham10000/HAM10000_i...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HAM_0002730</td>\n",
       "      <td>ISIC_0026769</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>../input/skin-cancer-mnist-ham10000/HAM10000_i...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>HAM_0002730</td>\n",
       "      <td>ISIC_0025661</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>../input/skin-cancer-mnist-ham10000/HAM10000_i...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>HAM_0001466</td>\n",
       "      <td>ISIC_0031633</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>75.0</td>\n",
       "      <td>male</td>\n",
       "      <td>ear</td>\n",
       "      <td>../input/skin-cancer-mnist-ham10000/HAM10000_i...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     lesion_id      image_id   dx dx_type   age   sex localization  \\\n",
       "0  HAM_0000118  ISIC_0027419  bkl   histo  80.0  male        scalp   \n",
       "1  HAM_0000118  ISIC_0025030  bkl   histo  80.0  male        scalp   \n",
       "2  HAM_0002730  ISIC_0026769  bkl   histo  80.0  male        scalp   \n",
       "3  HAM_0002730  ISIC_0025661  bkl   histo  80.0  male        scalp   \n",
       "4  HAM_0001466  ISIC_0031633  bkl   histo  75.0  male          ear   \n",
       "\n",
       "                                                path  \\\n",
       "0  ../input/skin-cancer-mnist-ham10000/HAM10000_i...   \n",
       "1  ../input/skin-cancer-mnist-ham10000/HAM10000_i...   \n",
       "2  ../input/skin-cancer-mnist-ham10000/HAM10000_i...   \n",
       "3  ../input/skin-cancer-mnist-ham10000/HAM10000_i...   \n",
       "4  ../input/skin-cancer-mnist-ham10000/HAM10000_i...   \n",
       "\n",
       "                        cell_type  cell_type_idx  \n",
       "0  Benign keratosis-like lesions               2  \n",
       "1  Benign keratosis-like lesions               2  \n",
       "2  Benign keratosis-like lesions               2  \n",
       "3  Benign keratosis-like lesions               2  \n",
       "4  Benign keratosis-like lesions               2  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_original = pd.read_csv(os.path.join(data_dir, 'HAM10000_metadata.csv'))\n",
    "df_original['path'] = df_original['image_id'].map(imageid_path_dict.get)\n",
    "df_original['cell_type'] = df_original['dx'].map(lesion_type_dict.get)\n",
    "df_original['cell_type_idx'] = pd.Categorical(df_original['cell_type']).codes\n",
    "#pd.categorical.codes:得到原始数据的对应的序号列表\n",
    "df_original.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lesion_id</th>\n",
       "      <th>image_id</th>\n",
       "      <th>dx</th>\n",
       "      <th>dx_type</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>localization</th>\n",
       "      <th>path</th>\n",
       "      <th>cell_type</th>\n",
       "      <th>cell_type_idx</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>HAM_0000001</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HAM_0000003</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HAM_0000004</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>HAM_0000007</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>HAM_0000008</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     lesion_id  image_id  dx  dx_type  age  sex  localization  path  \\\n",
       "0  HAM_0000001         1   1        1    1    1             1     1   \n",
       "1  HAM_0000003         1   1        1    1    1             1     1   \n",
       "2  HAM_0000004         1   1        1    1    1             1     1   \n",
       "3  HAM_0000007         1   1        1    1    1             1     1   \n",
       "4  HAM_0000008         1   1        1    1    1             1     1   \n",
       "\n",
       "   cell_type  cell_type_idx  \n",
       "0          1              1  \n",
       "1          1              1  \n",
       "2          1              1  \n",
       "3          1              1  \n",
       "4          1              1  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this will tell us how many images are associated with each lesion_id\n",
    "df_undup = df_original.groupby('lesion_id').count()\n",
    "# now we filter out lesion_id's that have only one image associated with it\n",
    "df_undup = df_undup[df_undup['image_id'] == 1]\n",
    "df_undup.reset_index(inplace=True)\n",
    "df_undup.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lesion_id</th>\n",
       "      <th>image_id</th>\n",
       "      <th>dx</th>\n",
       "      <th>dx_type</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>localization</th>\n",
       "      <th>path</th>\n",
       "      <th>cell_type</th>\n",
       "      <th>cell_type_idx</th>\n",
       "      <th>duplicates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>HAM_0000118</td>\n",
       "      <td>ISIC_0027419</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>../input/skin-cancer-mnist-ham10000/HAM10000_i...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "      <td>duplicated</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HAM_0000118</td>\n",
       "      <td>ISIC_0025030</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>../input/skin-cancer-mnist-ham10000/HAM10000_i...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "      <td>duplicated</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HAM_0002730</td>\n",
       "      <td>ISIC_0026769</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>../input/skin-cancer-mnist-ham10000/HAM10000_i...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "      <td>duplicated</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>HAM_0002730</td>\n",
       "      <td>ISIC_0025661</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>../input/skin-cancer-mnist-ham10000/HAM10000_i...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "      <td>duplicated</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>HAM_0001466</td>\n",
       "      <td>ISIC_0031633</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>75.0</td>\n",
       "      <td>male</td>\n",
       "      <td>ear</td>\n",
       "      <td>../input/skin-cancer-mnist-ham10000/HAM10000_i...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "      <td>duplicated</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     lesion_id      image_id   dx dx_type   age   sex localization  \\\n",
       "0  HAM_0000118  ISIC_0027419  bkl   histo  80.0  male        scalp   \n",
       "1  HAM_0000118  ISIC_0025030  bkl   histo  80.0  male        scalp   \n",
       "2  HAM_0002730  ISIC_0026769  bkl   histo  80.0  male        scalp   \n",
       "3  HAM_0002730  ISIC_0025661  bkl   histo  80.0  male        scalp   \n",
       "4  HAM_0001466  ISIC_0031633  bkl   histo  75.0  male          ear   \n",
       "\n",
       "                                                path  \\\n",
       "0  ../input/skin-cancer-mnist-ham10000/HAM10000_i...   \n",
       "1  ../input/skin-cancer-mnist-ham10000/HAM10000_i...   \n",
       "2  ../input/skin-cancer-mnist-ham10000/HAM10000_i...   \n",
       "3  ../input/skin-cancer-mnist-ham10000/HAM10000_i...   \n",
       "4  ../input/skin-cancer-mnist-ham10000/HAM10000_i...   \n",
       "\n",
       "                        cell_type  cell_type_idx  duplicates  \n",
       "0  Benign keratosis-like lesions               2  duplicated  \n",
       "1  Benign keratosis-like lesions               2  duplicated  \n",
       "2  Benign keratosis-like lesions               2  duplicated  \n",
       "3  Benign keratosis-like lesions               2  duplicated  \n",
       "4  Benign keratosis-like lesions               2  duplicated  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here we identify lesion_id's that have duplicate images and those that have only one image.\n",
    "def get_duplicates(x):\n",
    "    unique_list = list(df_undup['lesion_id'])\n",
    "    if x in unique_list:\n",
    "        return 'unduplicated'\n",
    "    else:\n",
    "        return 'duplicated'\n",
    "\n",
    "# create a new colum that is a copy of the lesion_id column\n",
    "df_original['duplicates'] = df_original['lesion_id']\n",
    "# apply the function to this new column\n",
    "df_original['duplicates'] = df_original['duplicates'].apply(get_duplicates)\n",
    "df_original.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unduplicated    5514\n",
       "duplicated      4501\n",
       "Name: duplicates, dtype: int64"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_original['duplicates'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5514, 11)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now we filter out images that don't have duplicates\n",
    "df_undup = df_original[df_original['duplicates'] == 'unduplicated']\n",
    "df_undup.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1103, 11)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now we create a val set using df because we are sure that none of these images have augmented duplicates in the train set\n",
    "y = df_undup['cell_type_idx']\n",
    "_, df_val = train_test_split(df_undup, test_size=0.2, random_state=101, stratify=y)\n",
    "df_val.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    883\n",
       "2     88\n",
       "6     46\n",
       "1     35\n",
       "0     30\n",
       "5     13\n",
       "3      8\n",
       "Name: cell_type_idx, dtype: int64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_val['cell_type_idx'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8912\n",
      "1103\n"
     ]
    }
   ],
   "source": [
    "# This set will be df_original excluding all rows that are in the val set\n",
    "# This function identifies if an image is part of the train or val set.\n",
    "def get_val_rows(x):\n",
    "    # create a list of all the lesion_id's in the val set\n",
    "    val_list = list(df_val['image_id'])\n",
    "    if str(x) in val_list:\n",
    "        return 'val'\n",
    "    else:\n",
    "        return 'train'\n",
    "\n",
    "# identify train and val rows\n",
    "# create a new colum that is a copy of the image_id column\n",
    "df_original['train_or_val'] = df_original['image_id']\n",
    "# apply the function to this new column\n",
    "df_original['train_or_val'] = df_original['train_or_val'].apply(get_val_rows)\n",
    "# filter out train rows\n",
    "df_train = df_original[df_original['train_or_val'] == 'train']\n",
    "print(len(df_train))\n",
    "print(len(df_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    5822\n",
       "6    1067\n",
       "2    1011\n",
       "1     479\n",
       "0     297\n",
       "5     129\n",
       "3     107\n",
       "Name: cell_type_idx, dtype: int64"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train['cell_type_idx'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Melanocytic nevi                  883\n",
       "Benign keratosis-like lesions      88\n",
       "dermatofibroma                     46\n",
       "Basal cell carcinoma               35\n",
       "Actinic keratoses                  30\n",
       "Vascular lesions                   13\n",
       "Dermatofibroma                      8\n",
       "Name: cell_type, dtype: int64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_val['cell_type'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从以上每个类别的统计数据可以看出，训练数据存在严重的等级失衡。为了解决这个问题，我们可以从两个方面入手，一个是均衡采样，另一个是可以用来缓解训练中类别不平衡的损失函数，例如facal loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Melanocytic nevi                  5822\n",
       "Dermatofibroma                    5350\n",
       "dermatofibroma                    5335\n",
       "Vascular lesions                  5160\n",
       "Benign keratosis-like lesions     5055\n",
       "Basal cell carcinoma              4790\n",
       "Actinic keratoses                 4455\n",
       "Name: cell_type, dtype: int64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Copy fewer class to balance the number of 7 classes\n",
    "#把对应标签的样本相应的复制15,10.。。。倍\n",
    "data_aug_rate = [15,10,5,50,0,40,5]\n",
    "for i in range(7):\n",
    "    if data_aug_rate[i]:\n",
    "        df_train=df_train.append([df_train.loc[df_train['cell_type_idx'] == i,:]]*(data_aug_rate[i]-1), ignore_index=True)\n",
    "df_train['cell_type'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一开始，我将数据分为训练集、验证集和测试集三部分，由于数据量小，在实践中没有进一步对验证集数据进行划分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # We can split the test set again in a validation set and a true test set:\n",
    "# df_val, df_test = train_test_split(df_val, test_size=0.5)\n",
    "df_train = df_train.reset_index()\n",
    "df_val = df_val.reset_index()\n",
    "# df_test = df_test.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第二步：建立模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# feature_extract is a boolean that defines if we are finetuning or feature extracting. \n",
    "#如果feature_extract = False，则对模型进行微调，并更新所有模型参数\n",
    "#如果feature_extract = True，则仅更新最后一层参数，其余参数保持不变.def set_parameter_requires_grad(model, feature_extracting):\n",
    "    if feature_extracting:\n",
    "        for param in model.parameters():\n",
    "            param.requires_grad = False\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize_model(model_name, num_classes, feature_extract, use_pretrained=True):\n",
    "    # Initialize these variables which will be set in this if statement. Each of these\n",
    "    #   variables is model specific.\n",
    "    model_ft = None\n",
    "    input_size = 0\n",
    "    \n",
    "    if model_name == \"densenet\":\n",
    "        \"\"\" Densenet121\n",
    "        \"\"\"\n",
    "        model_ft = models.densenet121(pretrained=use_pretrained)\n",
    "        #pretrained：直接加载pre-train模型中预先训练好的参数\n",
    "        set_parameter_requires_grad(model_ft, feature_extract)\n",
    "        num_ftrs = model_ft.classifier.in_features\n",
    "        model_ft.classifier = nn.Linear(num_ftrs, num_classes)#修改最后一层分类器个数\n",
    "\n",
    "        input_size = 224\n",
    "        \n",
    "   \n",
    "        \n",
    "    else:\n",
    "        print(\"Invalid model name, exiting...\")\n",
    "        exit()\n",
    "    return model_ft, input_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "考虑到训练数据有限，我们使用ImageNet预训练模型进行微调。这样可以加快模型的收敛速度，提高精度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# densenet\n",
    "model_name = 'densenet'\n",
    "num_classes = 7\n",
    "feature_extract = False\n",
    "# Initialize the model for this run\n",
    "model_ft, input_size = initialize_model(model_name, num_classes, feature_extract, use_pretrained=True)\n",
    "# Define the device:\n",
    "device = torch.device('cuda:0')\n",
    "# Put the model on the device:\n",
    "model = model_ft.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# norm_mean = (0.49139968, 0.48215827, 0.44653124)\n",
    "# norm_std = (0.24703233, 0.24348505, 0.26158768)\n",
    "# define the transformation of the train images.\n",
    "train_transform = transforms.Compose([transforms.Resize((input_size,input_size)),transforms.RandomHorizontalFlip(),\n",
    "                                      transforms.RandomVerticalFlip(),transforms.RandomRotation(20),\n",
    "                                      transforms.ColorJitter(brightness=0.1, contrast=0.1, hue=0.1),\n",
    "                                        transforms.ToTensor(), transforms.Normalize(norm_mean, norm_std)])\n",
    "# define the transformation of the val images.\n",
    "val_transform = transforms.Compose([transforms.Resize((input_size,input_size)), transforms.ToTensor(),\n",
    "                                    transforms.Normalize(norm_mean, norm_std)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a pytorch dataloader for this dataset\n",
    "class HAM10000(Dataset):\n",
    "    def __init__(self, df, transform=None):\n",
    "        self.df = df\n",
    "        self.transform = transform\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.df)\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        # Load data and get label\n",
    "        X = Image.open(self.df['path'][index])\n",
    "        y = torch.tensor(int(self.df['cell_type_idx'][index]))\n",
    "\n",
    "        if self.transform:\n",
    "            X = self.transform(X)\n",
    "\n",
    "        return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the training set using the table train_df and using our defined transitions (train_transform)\n",
    "training_set = HAM10000(df_train, transform=train_transform)\n",
    "train_loader = DataLoader(training_set, batch_size=32, shuffle=True, num_workers=4)\n",
    "# Same for the validation set:\n",
    "validation_set = HAM10000(df_val, transform=train_transform)\n",
    "val_loader = DataLoader(validation_set, batch_size=32, shuffle=False, num_workers=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use Adam optimizer, use cross entropy loss as our loss function\n",
    "optimizer = optim.Adam(model.parameters(), lr=1e-3)\n",
    "criterion = nn.CrossEntropyLoss().to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第三步：模型训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this function is used during training process, to calculation the loss and accuracy\n",
    "class AverageMeter(object):\n",
    "    def __init__(self):\n",
    "        self.reset()\n",
    "\n",
    "    def reset(self):\n",
    "        self.val = 0\n",
    "        self.avg = 0\n",
    "        self.sum = 0\n",
    "        self.count = 0\n",
    "\n",
    "    def update(self, val, n=1):\n",
    "        self.val = val\n",
    "        self.sum += val * n\n",
    "        self.count += n\n",
    "        self.avg = self.sum / self.count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_loss_train, total_acc_train = [],[]\n",
    "def train(train_loader, model, criterion, optimizer, epoch):\n",
    "    model.train()\n",
    "    train_loss = AverageMeter()\n",
    "    train_acc = AverageMeter()\n",
    "    curr_iter = (epoch - 1) * len(train_loader)\n",
    "    for i, data in enumerate(train_loader):\n",
    "        images, labels = data\n",
    "        N = images.size(0)\n",
    "        #torch.Size([32, 3, 224, 224]) :32批次，3通道，及输入尺寸\n",
    "        # print('image shape:',images.size(0), 'label shape',labels.size(0))\n",
    "        #在torch中的Variable就是一个存放会变化的值的地理位置，里面的值会不断变化\n",
    "        images = Variable(images).to(device)\n",
    "        labels = Variable(labels).to(device)\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "        outputs = model(images)\n",
    "\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        prediction = outputs.max(1, keepdim=True)[1]\n",
    "        #torch.Size([32, 7])\n",
    "        #1所对应的是最大可能性的标签\n",
    "        train_acc.update(prediction.eq(labels.view_as(prediction)).sum().item()/N)\n",
    "        #eq() 判断预测和标签是否相等，再求和，平均，计算每一个批次32张图的准确率\n",
    "        train_loss.update(loss.item())\n",
    "        curr_iter += 1\n",
    "        if (i + 1) % 100 == 0:\n",
    "        #每100批次打印\n",
    "            print('[epoch %d], [iter %d / %d], [train loss %.5f], [train acc %.5f]' % (\n",
    "                epoch, i + 1, len(train_loader), train_loss.avg, train_acc.avg))\n",
    "            total_loss_train.append(train_loss.avg)\n",
    "            total_acc_train.append(train_acc.avg)\n",
    "    return train_loss.avg, train_acc.avg\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate(val_loader, model, criterion, optimizer, epoch):\n",
    "    model.eval()\n",
    "    val_loss = AverageMeter()\n",
    "    val_acc = AverageMeter()\n",
    "    with torch.no_grad():\n",
    "        for i, data in enumerate(val_loader):\n",
    "            images, labels = data\n",
    "            N = images.size(0) #\n",
    "            images = Variable(images).to(device)\n",
    "            labels = Variable(labels).to(device)\n",
    "\n",
    "            outputs = model(images)\n",
    "            prediction = outputs.max(1, keepdim=True)[1]\n",
    "\n",
    "            val_acc.update(prediction.eq(labels.view_as(prediction)).sum().item()/N)\n",
    "\n",
    "            val_loss.update(criterion(outputs, labels).item())\n",
    "    print('------------------------------------------------------------')\n",
    "    print('[epoch %d], [val loss %.5f], [val acc %.5f]' % (epoch, val_loss.avg, val_acc.avg))\n",
    "    print('------------------------------------------------------------')\n",
    "    return val_loss.avg, val_acc.avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[epoch 1], [iter 100 / 1124], [train loss 1.24811], [train acc 0.54219]\n",
      "[epoch 1], [iter 200 / 1124], [train loss 1.12128], [train acc 0.58516]\n",
      "[epoch 1], [iter 300 / 1124], [train loss 1.04036], [train acc 0.61500]\n",
      "[epoch 1], [iter 400 / 1124], [train loss 0.98004], [train acc 0.63492]\n",
      "[epoch 1], [iter 500 / 1124], [train loss 0.93528], [train acc 0.65144]\n",
      "[epoch 1], [iter 600 / 1124], [train loss 0.89797], [train acc 0.66609]\n",
      "[epoch 1], [iter 700 / 1124], [train loss 0.87116], [train acc 0.67558]\n",
      "[epoch 1], [iter 800 / 1124], [train loss 0.84955], [train acc 0.68371]\n",
      "[epoch 1], [iter 900 / 1124], [train loss 0.83332], [train acc 0.68990]\n",
      "[epoch 1], [iter 1000 / 1124], [train loss 0.81478], [train acc 0.69650]\n",
      "[epoch 1], [iter 1100 / 1124], [train loss 0.79827], [train acc 0.70196]\n",
      "------------------------------------------------------------\n",
      "[epoch 1], [val loss 0.43331], [val acc 0.86137]\n",
      "------------------------------------------------------------\n",
      "*****************************************************\n",
      "best record: [epoch 1], [val loss 0.43331], [val acc 0.86137]\n",
      "*****************************************************\n",
      "[epoch 2], [iter 100 / 1124], [train loss 0.58488], [train acc 0.78063]\n",
      "[epoch 2], [iter 200 / 1124], [train loss 0.57028], [train acc 0.78812]\n",
      "[epoch 2], [iter 300 / 1124], [train loss 0.57208], [train acc 0.78677]\n",
      "[epoch 2], [iter 400 / 1124], [train loss 0.56686], [train acc 0.78891]\n",
      "[epoch 2], [iter 500 / 1124], [train loss 0.56675], [train acc 0.78750]\n",
      "[epoch 2], [iter 600 / 1124], [train loss 0.56767], [train acc 0.78677]\n",
      "[epoch 2], [iter 700 / 1124], [train loss 0.56695], [train acc 0.78621]\n",
      "[epoch 2], [iter 800 / 1124], [train loss 0.56193], [train acc 0.78828]\n",
      "[epoch 2], [iter 900 / 1124], [train loss 0.55697], [train acc 0.78920]\n",
      "[epoch 2], [iter 1000 / 1124], [train loss 0.54864], [train acc 0.79222]\n",
      "[epoch 2], [iter 1100 / 1124], [train loss 0.54314], [train acc 0.79429]\n",
      "------------------------------------------------------------\n",
      "[epoch 2], [val loss 0.46306], [val acc 0.83917]\n",
      "------------------------------------------------------------\n",
      "[epoch 3], [iter 100 / 1124], [train loss 0.46239], [train acc 0.82156]\n",
      "[epoch 3], [iter 200 / 1124], [train loss 0.46400], [train acc 0.82297]\n",
      "[epoch 3], [iter 300 / 1124], [train loss 0.47831], [train acc 0.81719]\n",
      "[epoch 3], [iter 400 / 1124], [train loss 0.47006], [train acc 0.81984]\n",
      "[epoch 3], [iter 500 / 1124], [train loss 0.46270], [train acc 0.82306]\n",
      "[epoch 3], [iter 600 / 1124], [train loss 0.46266], [train acc 0.82344]\n",
      "[epoch 3], [iter 700 / 1124], [train loss 0.45779], [train acc 0.82554]\n",
      "[epoch 3], [iter 800 / 1124], [train loss 0.45831], [train acc 0.82387]\n",
      "[epoch 3], [iter 900 / 1124], [train loss 0.45300], [train acc 0.82510]\n",
      "[epoch 3], [iter 1000 / 1124], [train loss 0.44939], [train acc 0.82625]\n",
      "[epoch 3], [iter 1100 / 1124], [train loss 0.44553], [train acc 0.82801]\n",
      "------------------------------------------------------------\n",
      "[epoch 3], [val loss 0.50761], [val acc 0.81202]\n",
      "------------------------------------------------------------\n",
      "[epoch 4], [iter 100 / 1124], [train loss 0.42023], [train acc 0.83969]\n",
      "[epoch 4], [iter 200 / 1124], [train loss 0.41193], [train acc 0.84344]\n",
      "[epoch 4], [iter 300 / 1124], [train loss 0.39998], [train acc 0.84781]\n",
      "[epoch 4], [iter 400 / 1124], [train loss 0.39057], [train acc 0.85125]\n",
      "[epoch 4], [iter 500 / 1124], [train loss 0.38859], [train acc 0.85137]\n",
      "[epoch 4], [iter 600 / 1124], [train loss 0.38633], [train acc 0.85260]\n",
      "[epoch 4], [iter 700 / 1124], [train loss 0.38647], [train acc 0.85237]\n",
      "[epoch 4], [iter 800 / 1124], [train loss 0.38528], [train acc 0.85203]\n",
      "[epoch 4], [iter 900 / 1124], [train loss 0.38259], [train acc 0.85288]\n",
      "[epoch 4], [iter 1000 / 1124], [train loss 0.37820], [train acc 0.85478]\n",
      "[epoch 4], [iter 1100 / 1124], [train loss 0.37519], [train acc 0.85580]\n",
      "------------------------------------------------------------\n",
      "[epoch 4], [val loss 0.37837], [val acc 0.85321]\n",
      "------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "epoch_num = 4\n",
    "best_val_acc = 0\n",
    "total_loss_val, total_acc_val = [],[]\n",
    "for epoch in range(1, epoch_num+1):\n",
    "    loss_train, acc_train = train(train_loader, model, criterion, optimizer, epoch)\n",
    "    loss_val, acc_val = validate(val_loader, model, criterion, optimizer, epoch)\n",
    "    total_loss_val.append(loss_val)\n",
    "    total_acc_val.append(acc_val)\n",
    "    if acc_val > best_val_acc:\n",
    "        best_val_acc = acc_val\n",
    "        print('*****************************************************')\n",
    "        print('best record: [epoch %d], [val loss %.5f], [val acc %.5f]' % (epoch, loss_val, acc_val))\n",
    "        print('*****************************************************')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3wd5ZXw8d+5Vb3LVbYl04xtXGXj0J03EJvQy+IsCYEsy0IIkJDdFzabQjbJhgBvNiEhEIc1kCyhhA6hBIjBJAEsueKGMbZsy3KRrWL12573jxlJV/K90rWKb9H5fj73M/XOfWZGOvPMmZlnxBiDUkqp1OWIdwGUUkoNLw30SimV4jTQK6VUitNAr5RSKU4DvVJKpThXvAsQSVFRkSktLY13MZRSKmmsWrXqoDGmONK0fgO9iCwDLgAOGGOmR5h+NXCHPdgM3GSMWWdPqwKagCAQMMaUx1Lg0tJSKisrY5lVKaUUICI7o02LJXXzKLCoj+k7gLONMTOAHwJLe01faIyZFWuQV0opNbT6DfTGmBVAXR/T/26MqbcHPwBKhqhsRy0QDOELhOL180oplZCG+mLsPwGvhQ0b4M8iskpEbhji3+qhsc3P4l+8x6N/3zGcP6OUUklnyAK9iCzECvR3hI0+3RgzB1gM3CwiZ/Xx/RtEpFJEKmtra4/693PT3YzPT+eB5Z/S2Oo/6u8rpVSqGpJALyIzgIeBi40xhzrHG2Nq7O4B4HlgfrRlGGOWGmPKjTHlxcURLxz3645FUzjc7ufX72wb0PeVUioVDTrQi8hE4Dngy8aYrWHjM0Uku7MfOA/YMNjf68vJY3O4dPZ4Hvl7FXsa2obzp5RSKmn0G+hF5AngfeAkEakWkX8SkRtF5EZ7lu8BhcCvRWStiHTeFzka+KuIrANWAn8yxrw+DOvQw7fOOwmA/35zaz9zKqXUyNDvffTGmC/2M/164PoI47cDMwdetIEZn5fOtaeV8tv3tnP9mWVMGZNzrIuglFIJJSWbQPjaOceR7XXx09e2xLsoSikVdykZ6PMyPHxt4fEs/7iW9z891P8XlFIqhaVkoAe49rRSxuamcfdrm9G3aCmlRrKUDfRpbie3n3si66obefWjffEujlJKxU3KBnqAy+aUcNLobO59Ywv+oDaNoJQamVI60Dsdwh2LT6LqUCtPrNwV7+IopVRcpHSgB1h40ihOLSvgF299QnNHIN7FUUqpYy7lA72I8O/nn8yhFh/3v/1JvIujlFLHXMoHeoBZE/L4x1MnsnTFdp5fUx3v4iil1DE1IgI9wF0XTuMzkwu545mPqKiK2ry+UkqlnBET6D0uBw99aS4l+enc8LtKdh5qiXeRlFLqmBgxgR4gN8PNsmvnYYDrHq3QduuVUiPCiAr0AKVFmSz9cjm761q56fFVen+9UirljbhADzC/rIC7L5vB3z89xHdf2KBNJCilUlq/zRSnqsvnllB1qIVf/mUbk4szueGs4+JdJKWUGhYjNtADfPNzJ7L9YAs/eW0LJfkZnH/K2HgXSSmlhtyITN10cjiE/3flTGZPyOPrf1jNHz7UZhKUUqknllcJLhORAyIS8X2vYrlfRLaJyHoRmRM2bZGIfGxPu3MoCz5U0txOfv9Pp3L2icV8+/mPuOf1LYRCmrNXSqWOWGr0jwKL+pi+GDjB/twAPAggIk7gAXv6VOCLIjJ1MIUdLpleF7+9ppwvzp/Ir9/5lNufXosvoHfjKKVSQyzvjF0hIqV9zHIx8Dtj3brygYjkichYoBTYZr87FhF50p5302ALPRxcTgf/del0SvLTufeNj9l/uIOHvjyX3HR3vIumlFKDMhQ5+vHA7rDhantctPERicgNIlIpIpW1tbVDUKyjJyLcvPB4fn7VLCp31nHlQ39nT0NbXMqilFJDZSgCvUQYZ/oYH5ExZqkxptwYU15cXDwExRq4S2aP57GvzmdvYzuXPvA3NuxpjGt5lFJqMIYi0FcDE8KGS4CaPsYnhdOOK+LZm07D5RAu/fXfuP/tTzRvr5RKSkMR6F8CrrHvvlkANBpj9gIVwAkiUiYiHmCJPW/SOHF0Ni/fcgaLp4/lZ29u5aJf/ZV1uxviXSyllDoqsdxe+QTwPnCSiFSLyD+JyI0icqM9y6vAdmAb8FvgawDGmADwdeANYDPwtDFm4zCsw7AqzPJy/xdn8/A15dS3+rj013/jJ69upt0fjHfRlFIqJpKI7byUl5ebysrKeBfjCI1tfu5+bTNPrNxNaWEGd18+gwWTC+NdLKWUQkRWGWPKI00b0U/GHq3cdDc/uWwGf7j+VILGsGTpB/z7c+s50NQe76IppVRUGugH4LTji3jjG2dx/Rll/LGymnPufYefvblVXz6ulEpIGugHKMPj4jsXTOXN289m4UmjuP/tTzjn3uX87v0qbeNeKZVQNNAPUllRJg9cPYcXbj6d44qz+N6LGzn3Z+/yyvoabedeKZUQNNAPkVkT8njyhgU8cu08vC4nX//DGi745V/5w4e7aGrXVxYqpeJH77oZBsGQ4bnV1fzPX3ewZV8T6W4nX5gxliXzJjB3Uj4ikR4aVkqpgevrrhsN9MPIGMP66kaerNjNS2v30OILclxxJlfNm8Blc0ooyvLGu4hKqRShgT4BtHQE+NNHe3mqYjerdtbjEOvdtedNHcN500ZTkp8R7yIqpZKYBvoEs+1AEy+sqeHPm/axdX8zANPG5fD5aVbQP2l0tqZ3lIqVMeBvBROy+q2RPftDIQj5IeiHUKDnxxjo+n8Tu79XVxxh47CGTcj6hIJggmHdEAR9EOyAQGe3wxoX6Djy90MBu3wBcKfDmbcPaDNooE9gOw628OeN+/jzpv2s3lWPMVCSn86pZYXML8unvLSAyUWZGvhV6jHGCnwdTdBx2O42QXsjtDfYXfvTZg/7mq15fM3ga4GOZqs/esO4ySV7LHxry4C+qoE+SRxoauetTQdY/vEBVu2sp67FB0Bhpofy0nzmlRYwv6yAaeNycTo08Ksk0NEMtR/DgU1wYDPUboa67dBuB/ZQf3ekCaTlQFoueHPBmw3eLPBkgifLGvZkgScDHK7u70B3jRysaQ4nON12v9sadri6a+nG0H0m0LtL2BmDsfrFAeIER2fX2d11uMCVBi4vOD09uw43OF32b9vzOlzWcgZBA30SMsbwaW0LFVV1VFTVUVlVz666VsBqiuG04wo544Qizjy+mImFmt+PC2Os0/WQ3z5VD3SnB0zQCk5peWFpgQH+hr/Nqr36mrtTDSZEVxDq7A/6rRpyoP3ILliBxum2ux5w2V3ETh/4rfWJlN7oGhfsnjcY6E5RdP520Gd92uqhdgs07OpeF1caFJ8EhcdDer4dtLPBmxPWn21ts7RcO7jnDDoAjhQa6FPE/sPtfLD9EH/bdpD3PjnI3kbrH3hiQYYd9IuYX1ZAod7NM3yMgar3YNVjsOWV7iAajcMNGYWQWQyZdjej0AqYgTYriPvbrBxzZ9fXagf2liRIS0h3bTX8AOLNhqKTYNQUGDUViqdAfqlV21XDQgN9Cuqs8f/1k1r+uu0g7396iBaf1XTy8aOymFdawKllBcwrK2B8XnqcS5sCmmth7eOw+ndQ96lV25x+uZVTdbrt03E7LeD0WKf17Q3QchBaaq1uq93fWmcFPHeGdfHNnd7d70q30xJ2aqKr3/44PVZ5xHHkRcPO1EBnyqCr67WmB31WzTu8Fh70WWcEjs6yu8JSG+Hpjt7Dru6PXj9KCBroRwB/MMT66gY+3FFHxQ4r1dNkN7I2Pi+d048v5D++MFVfdh5NKNid5gj6utMeDbtgze9hy6tWumLiaTD3Wph6kRWYlUoQfQV6V6SRKvm4nQ7mTipg7qQCOMd6OnfLvsNU7KhjZVUdT1dWM6kwk5sXHh99IaGQVeM8vAcO11ifphorCHbVWl1htVe3lYuOlBMOdFhpDqe7Oy/scIX1O8MuZvXuF6tfHEd+TKj7drXevxf0deeQO29z6+wP+qyUiL/VvmOjtTtN4m+x5osmvQBO/ReYc42VY1YqyWiNfoS45IG/EQwZXr7ljJ4TWg7CizfD/k12UO8V8DpPzzsvMPbF6bHSBZ0pBCT6hcqhJA4r5eEMSy903c1gH0Ccbiv14c7oToN09nemTFyesDsl7JRHWh6UnWmvj1KJa9A1ehFZBPwCcAIPG2Pu7jX934Crw5Z5MlBsjKkTkSqgCQgCgWgFUcNr8fQx/OS1LVTXt3Y/hetvgyeWwN71MO0SyBkPOeN6djMKu+966Kwhhz944nDawd0b+90RoVDPB0y6HjoJHdnf+yNiB+OwgOzUE1Ol+tLvf4iIOIEHgHOBaqBCRF4yxmzqnMcYcy9wrz3/hcA3jTF1YYtZaIw5OKQlV0dlkR3oX9+wj+vPnGwF0+f+Gaor4R8eg6kX978QhwMcHsAzuMI4HIDDqmUrpYZdLFWw+cA2Y8x2Y4wPeBLoKyp8EXhiKAqnhs6kwkxOHpvDGxv3WSP+/F3Y/DJ8/sexBXmlVNKKJdCPB3aHDVfb444gIhnAIuDZsNEG+LOIrBKRG6L9iIjcICKVIlJZW1sbQ7HU0Vo0bQyVO+tpevd++OABOPVGWPC1eBdLKTXMYgn0kW6SjXYF90Lgb73SNqcbY+YAi4GbReSsSF80xiw1xpQbY8qLi4tjKJY6WotPGcN5UkHW8u/BlAvg8/+l90ArNQLEEuirgQlhwyVATZR5l9ArbWOMqbG7B4DnsVJBKg5O8G3mfs8DfOo5CS77rT6lqNQIEUugrwBOEJEyEfFgBfOXes8kIrnA2cCLYeMyRSS7sx84D9gwFAVXR+nQp8gTS2j1juKLzd+kIaB3qig1UvT7326MCYjI14E3sG6vXGaM2SgiN9rTH7JnvRT4szGmJezro4Hn7SZ2XcAfjDGvD+UKKJu/HZr3Q+uhnp+Wg1b30+VgDPsv+j21v9/Lm5v2c2X5hP6Xq5RKejFV64wxrwKv9hr3UK/hR4FHe43bDswcVAlHslDQbivlADQfsJ5abd5v9Tfts/vtT3tj5GWI07oXPmcsXPE/nFQym/F5DbyxcZ8GeqVGCD1/j5egH5r2QmO1/dltdRt2W00PdNbOI133dqVD9mjIGm21Clh2dvdwRhFkFlnBPaPQanwr7IKrAJ+fNob//XAnzR0Bsrz6J6BUqtP/8uHU0Qz1O6Buh93d3t3fWG23Ix4moxBySyB/EkyYB5mjIGuU1bRt1ih7uNhqo3sQd8ssPmUMy/62g+VbDnDhzHGDXEmlVKLTQD9U/G2w7yPrSdM9lVa3YWfPedILoKAMJpwKM66C3AlWYM+dALnjrXZXjoE5E/MpyvLy+oZ9GuiVGgE00A9Uwy7Y+T5Ur7SC+v4N3Q2C5YyH8XNhzpeh4DgruOeXQXpefMtsczqEz08bzfNr9tDuD5Lm1tsslUplGuhjYQwc3Ao7/259dr1v5dQBPNkwfjacdqsV3MfPtS58JrhF08fw+Ie7WLG1lvOmjYl3cZRSw0gDfTTNB2DbW/DJn2HHCvvCKFaefNJpcNotVnfU1KR88GjB5EJy0928vnGfBnqlUpwG+k6hENSsgU/esIJ7zRprfNZoOOE8K6hPOh0KJqdEswFup4PPnTyaNzftwxcI4XHpC5iVSlUa6PdvhJVLYfMr1js9xQEl8+Cz37EC/JgZKRHYI1k8fQzPrq7mg+2HOOtEbV9IqVQ1MgN9KAhb34APH7TSMq50OPkCOHERHPdZyCiIdwmPiTNOKCLT4+S1Dfs00CuVwkZWoG8/DGv+F1b+BuqrrLtjPncXzPnKiAnu4dLcThZOGcWbm/bxo0um43Sk5pmLUiPdyAj0HU3w7k+h8hHrxdATTrUC/JQLR/xr6BZNH8Mr6/dyz+tbOPvEYk4pySU7Td/8pFQqSf0ot/kVePXfrOYGTrnCetHG+DnxLlXC+OyUUcyZmMdvVmznNyu2IwLHF2cxc0IesybkMbMkjwyvE38whC9gf+x+f9CQ4XGSl+EmP8NDQaZH78lXKgGlbqBvrIZX/y98/CcYPR2u+j2U6HvJe8vwuHjua6dT3+Jj/Z5G1u5qYF11A3/ZcoBnVlUf9fLS3A4KMjzkZXhIc/e8k0fCLmqHjCEUMgSNIRA0hIwhELLGGcApgsMhOAQcIjgdgkOENLeD3HQP+Rlu8jM95GW4ybOH8zI8XQedvAy3HnSUsqVeoA8GrLto/vIjwMC5P4QFN+mLqPuRn+nh7BOLOdu+KGuMobq+jQ17GvEFQ3icDjwuB26763E5cDsctPoC1Lf6qW/1WZ8WnzXc4sMX7G7Lx4S1zWYwOMQK3C6HFdA7u04RRCAYMhhjdYP2QSFkDG3+INX1rWzYY/1mRyDUe1W6pLkd5KVbQb8wy8O43HTG56czPs/qluRlMCY3TW8tVSkvtQL9ntXwyjdg7zrr1sjz77MaCFNHTUSYUJDBhIKMeBelT+3+oH2A8dPQ6qOhzU9Dq5+GNp/VbbUOPAebO1jxSS0Hmjp6HHREYFS2l4JMLwWZbqub4e4anlCQwdknFvc4G1Eq2aROoG+rh0cvAG82XPkYTL04Ze9/V93S3E7G5qYzNjc9pvk7AkH2NbZTXd/Gnvo2qhva2NfYRl2Lj7oWHx/VN1DX4uNwe6DrO9/83Inc9rkThmsVlBp2qRPo0/Phqt9ZDzul5ca7NCpBeV1OJhVmMqmw75ZC/cEQ9a0+7n5tC//91lZKizK4eNb4Y1RKpYZWTMlJEVkkIh+LyDYRuTPC9HNEpFFE1tqf78X63SF1/Oc0yKsh4XY6GJWdxk8uO4X5ZQX82x/XU1lVF+9iKTUg/QZ6EXECDwCLganAF0VkaoRZ3zPGzLI//3mU31UqIXldTn7zpbmMy0vjht+vYteh1ngXSamjFkuNfj6wzRiz3RjjA54ELo5x+YP5rlIJIT/Tw7Jr5xEMGa57dCWNbf54F0mpoxJLoB8P7A4brrbH9fYZEVknIq+JyLSj/C4icoOIVIpIZW1tbQzFUurYmVycxW++PJddda187fFV+IPRb+tUKtHEcjE20q0rvd9YvRqYZIxpFpHzgReAE2L8rjXSmKXAUoDy8vKI8ygVTwsmF/Jfl57Cvz2znu++sIGfXHZKzLddtvmCbKxpZF11I5v3HibL6+q6n39cnnVvf1GWR2/jVMMilkBfDUwIGy4BasJnMMYcDut/VUR+LSJFsXxXqWRyZfkEqg618MDyTyktyuTLCyZZzUMErSYh/IEQ/mCIFl+QTTWHWV/dwLrqRrbubyIYsuovRVle2v1BmjsCPZbtdTkYn5fOmNw065NzZDfdE/1p33S3E5dTH/5SR4ol0FcAJ4hIGbAHWAL8Y/gMIjIG2G+MMSIyHysldAho6O+7SiWbb517ElUHW7n7tS3c/dqWPufNTXczoySXz518HDNK8phZksuonDQAGtv87KlvY09DGzUNVndPfRt7G9v4cHsd+w+3Ewgd3cltttdFboa7qymI3HSrP83lJGSspieMMV39IWMdYPIy3BRkesjPsD+Z1nBuupt0t1PPNJJcv4HeGBMQka8DbwBOYJkxZqOI3GhPfwi4ArhJRAJAG7DEGGOAiN8dpnVR6phwOIT/9w8zWTC5gDZ/ELfTbhrC6cDtEtxOB16XkxNHZzGxICNqkMxNd5Ob7mbquJyI00Mhw8GWDvY3drC3sY39h9ujNvlgDLT6gjS0+Wi0m6RosA8k9a0+/EGD2O0GdXYdYj0B3e4P0tQeiLhcAIdAptdFttdFptdFVpqLLG/3JzvNTVaaNT07zZo+NjeN2RPycWjT1wlBjEm8dHh5ebmprKyMdzGUGjECwRANbVYbRXWd7RW1Ws1ItHQEaLY/nf1N7fa49u5pvY3LTeOS2eO5bE4Jx4/KisNajSwissoYE7HlxtR5MlYpNWAup4OiLC9FWd4BfT8UMjT7ugP/ln1NPL+6mt+s2M6v3/mUmSW5XDanhAtnjqMg0zPEpVf90Rq9UmrYHGhq56W1NTy3eg+b9h7G7RROP76I44qzuu46Gp+XTkl+Ornpbr0WMAh91eg10CuljonNew/z/Jo9/GXLAarrW2n397zekOlxUpKfwfGjszhpdDYnjcnmpNHZTCzI0Fx/DDTQK6USijGG+tbOu45ardZEG9rYXdfK1v3N7Krrbmoi3e3khNFZnDg6m7KiTCYWZDCpMINJBZnkZuh7Jjppjl4plVBEhIJM6/WTp5Qc2RBhS0eATw408/G+w2zZ18TW/U28u7X2iLee5aa7mVRovTchy+PC63Z0vSTH63LaXQdetzWcFqHrEOl6R4Gxn+fsHHaI4HJaL8ZxOqw7qpz2i3K8LidpHuv3Ej3lpIFeKZVwMr0uZtnvLQ7X0hFgV10rOw+1squuxe62sqnmMC0dga73GXcEQl0PqA03p0NIdztJcztJ9zhIdzvJ9LrISXOTk+4mN727P8e+FTXT4yTd4yTD4yLD47Q/Vv9wvAJTA71SKmlkel2cPDaHk8dGfvYgXDBk7KAfpCMQosMfoj0Q7Nn1B7vaZOmsk3dWzq1XWkIwFCIQst5tbHVD+O1lt/uDtPmCtPmtT7svSKsvSIsvQEOrj111rTS2+Tnc5o/p4bfCTA+rvnvuwDZOHzTQK6VSktMhpNs153gz9vuOG9v8NLcHaLUPCG1+u78jSKsvgHOYLjproFdKqWEmInZqxgVxeDeStoCklFIpTgO9UkqluIS8j15EaoGdA/x6EXBwCIuTSnTbRKbbJTrdNtEl2raZZIwpjjQhIQP9YIhIZbSHBkY63TaR6XaJTrdNdMm0bTR1o5RSKU4DvVJKpbhUDPRL412ABKbbJjLdLtHptokuabZNyuXolVJK9ZSKNXqllFJhNNArpVSKS5lALyKLRORjEdkmInfGuzzxJCLLROSAiGwIG1cgIm+KyCd2Nz+eZYwXEZkgIstFZLOIbBSR2+zxI3r7iEiaiKwUkXX2dvmBPX5Eb5dwIuIUkTUi8oo9nDTbJiUCvYg4gQeAxcBU4IsiMjW+pYqrR4FFvcbdCbxtjDkBeNseHokCwLeMMScDC4Cb7b+Vkb59OoDPGmNmArOARSKyAN0u4W4DNocNJ822SYlAD8wHthljthtjfMCTwMVxLlPcGGNWAHW9Rl8MPGb3PwZcckwLlSCMMXuNMavt/iasf9zxjPDtYyzN9qDb/hhG+HbpJCIlwBeAh8NGJ822SZVAPx7YHTZcbY9T3UYbY/aCFeyAUXEuT9yJSCkwG/gQ3T6dqYm1wAHgTWOMbpduPwf+LxD+otuk2TapEugjNeKs942qqEQkC3gW+IYx5nC8y5MIjDFBY8wsoASYLyLT412mRCAiFwAHjDGr4l2WgUqVQF8NTAgbLgFq4lSWRLVfRMYC2N0DcS5P3IiIGyvIP26Mec4erdvHZoxpAN7Bus6j2wVOBy4SkSqstPBnReR/SaJtkyqBvgI4QUTKRMQDLAFeinOZEs1LwFfs/q8AL8axLHEj1luc/wfYbIz5WdikEb19RKRYRPLs/nTgc8AWRvh2ATDG/LsxpsQYU4oVW/5ijPkSSbRtUubJWBE5HyuP5gSWGWN+HOcixY2IPAGcg9WM6n7g+8ALwNPARGAXcKUxpvcF25QnImcA7wEf0Z1v/TZWnn7Ebh8RmYF1QdGJVQF82hjznyJSyAjeLr2JyDnAvxpjLkimbZMygV4ppVRkqZK6UUopFYUGeqWUSnEa6JVSKsW54l2ASIqKikxpaWm8i6GUUklj1apVB6O9MzYhA31paSmVlZXxLoZSSiUNEdkZbZqmbpRSKsUlZI1+wLb+GZwucGeCOx3cGeDJsPszwekGidRaglJKpa7UCvRPfxkC7dGnixM8nQeB9O4DgifDOihEHJfR66AR9vGETeucz+E8duurlFIxSK1A/9U3wN8G/ha72wa+lp7jfK3gb7XH2f2+Vmhr6Dne1wrBjqMvgyvtyODvyRzYOE+vA407A1xePStRKpkYAx1N0FYHrfanq/9Qz36XF67+45AXIbUC/bhZQ7u8UPDI4N/joNHS84DR17j2w9C0L2xcq7UcE+q/HOHE0c+ZRoSUVY+DSKRxvQ42elaiVGShELQ3hAXrQ1ECd33PIB7yR1mgQHoeZBRCeoHVHQapFeiHmsMJ3mzrMxyMgaCv10GkteeZRo8zlH7GNe878qwl0Hb05XJ6egX//lJW/Y3LBG+WdRDxZOmBRCWGoB/a6nsF696Bu9f49obolTOHyw7WdsAuPA4y5oUF8YIj+9Nyj8n/gwb6eBKxTtVcXkgfptdNhkJWsO+RsuqdxmrtdaYR6WDTCr5maKkNO2uxlxUKHF2Z3BlWwPd0HgCyex4IvNl9TO/sD5vfqX/GI56/LYaada8g3tHHawhcaWFBOR/GnBIlWBd093tzEjatqv8hqc7hsANi5vD9RtDfdxrL12IdJHzN0NEcob8Fmg+EjbPnj/XdMa70Pg4a9riu6Z0Hkj4OKnrgiB9jrH3fVxqkR78duP2t0ZfpybaCdWeALjwuerDuHO/JOHbrfAzoX7QaPKfbyjOm5w3dMkOh7rMIX4t1MSvqgSKsv7Pbegjqd/b8fswHjrReB4JYDxpRpjvdQ7ddkklnPrutPvKFx2gXJ/vLZ3cG5ZxxMHp69GDdOc7lOaarnYg00KvE5HBYwdKbNTTLM8Y6cMR6oOjd31YPjbvDzjiaYr+Q7vQcxYEiK3p/57wu79Bsk6PRlc/uL5cdFsTb6qNvI3H2DMoFk6GkPHouO73ACvJ6fWdANNCrkUEkLIU1evDLM8ZOS/V3oLAPCuFpqo4maG+Exj09p5tgbL/tcEc5aPRzoOjRn2Fdf+mrZh0+vqMxenmcXisYd9asR0+LHqw7UygJnM9ORYMK9CKyCPgF1ltpHjbG3N1rei7wv1hvYHEB9xljHhnMbyqVEESsYOnJAEYNfnnGWA/7HZGm6n2giHZQaYKmvT2/f7QXycE6CISnQQom9wrW+UcGcXeGBu0EN+BALyJO4AHgXKyXc1eIyEvGmE1hs90MbDLGXCgixcDHIvK4McY3qFIrlWpEup9zyCwa/PKMgUBHHweKFqt2382VwA0AABbCSURBVDuIxyMtpIbdYGr084FtxpjtACLyJHAxEB7oDZBtv5A5C6gDBlDNUEodFRFwp1mfzOF5CEclj8G0Xjke2B02XG2PC/cr4GSgButlzLcZE/nqjIjcICKVIlJZW1s7iGIppZQKN5hAHykp1/v+tc8Da4FxwCzgVyKSE2lhxpilxphyY0x5cXHEtvOVUkoNwGACfTUwIWy4BKvmHu464Dlj2QbsAKYM4jeVUkodpcEE+grgBBEpExEPsAR4qdc8u4D/AyAio4GTgO2D+E2llFJHacAXY40xARH5OvAG1u2Vy4wxG0XkRnv6Q8APgUdF5COsVM8dxpiDQ1BupZRSMRrUffTGmFeBV3uNeyisvwY4bzC/oZRSanD0nbFKKZXiNNArpVSK00CvlFIpTgO9UkqlOA30SimV4jTQK6VUitNAr5RSKU4DvVJKpTgN9EopleI00CulVIrTQK+UUilOXw6uVILz+/1UV1fT3t4e76KoBJCWlkZJSQlutzvm72igVyrBVVdXk52dTWlpKaIv4R7RjDEcOnSI6upqysrKYv6epm6USnDt7e0UFhZqkFeICIWFhUd9dqeBXqkkoEFedRrI34IGeqWUSnEa6JVSQy4rKwuAmpoarrjiiojznHPOOVRWVva5nJ///Oe0trZ2DZ9//vk0NDQMunx33XUX991336CXkyw00Culhs24ceN45plnBvz93oH+1VdfJS8vbyiKNqLoXTdKJZEfvLyRTTWHh3SZU8fl8P0Lp0WdfscddzBp0iS+9rWvAVZtODs7m3/5l3/h4osvpr6+Hr/fz49+9CMuvvjiHt+tqqriggsuYMOGDbS1tXHdddexadMmTj75ZNra2rrmu+mmm6ioqKCtrY0rrriCH/zgB9x///3U1NSwcOFCioqKWL58OaWlpVRWVlJUVMTPfvYzli1bBsD111/PN77xDaqqqli8eDFnnHEGf//73xk/fjwvvvgi6enpUddv7dq13HjjjbS2tnLcccexbNky8vPzuf/++3nooYdwuVxMnTqVJ598knfffZfbbrsNsHLlK1asIDs7e8Db/ljRGr1Sqk9Llizhqaee6hp++umnufLKK0lLS+P5559n9erVLF++nG9961sYY6Iu58EHHyQjI4P169fzH//xH6xatapr2o9//GMqKytZv3497777LuvXr+fWW29l3LhxLF++nOXLl/dY1qpVq3jkkUf48MMP+eCDD/jtb3/LmjVrAPjkk0+4+eab2bhxI3l5eTz77LN9rt8111zDT3/6U9avX88pp5zCD37wAwDuvvtu1qxZw/r163noIetV2Pfddx8PPPAAa9eu5b333uvzAJJItEavVBLpq+Y9XGbPns2BAweoqamhtraW/Px8Jk6ciN/v59vf/jYrVqzA4XCwZ88e9u/fz5gxYyIuZ8WKFdx6660AzJgxgxkzZnRNe/rpp1m6dCmBQIC9e/eyadOmHtN7++tf/8qll15KZmYmAJdddhnvvfceF110EWVlZcyaNQuAuXPnUlVVFXU5jY2NNDQ0cPbZZwPwla98hSuvvLKrjFdffTWXXHIJl1xyCQCnn346t99+O1dffTWXXXYZJSUlMW7F+NIavVKqX1dccQXPPPMMTz31FEuWLAHg8ccfp7a2llWrVrF27VpGjx7d7/3dkW4N3LFjB/fddx9vv/0269ev5wtf+EK/y+nrzMHr9Xb1O51OAoFAn8uK5k9/+hM333wzq1atYu7cuQQCAe68804efvhh2traWLBgAVu2bBnQso81DfRKqX4tWbKEJ598kmeeeabrLprGxkZGjRqF2+1m+fLl7Ny5s89lnHXWWTz++OMAbNiwgfXr1wNw+PBhMjMzyc3NZf/+/bz22mtd38nOzqapqSnisl544QVaW1tpaWnh+eef58wzzzzq9crNzSU/P5/33nsPgN///vecffbZhEIhdu/ezcKFC7nnnntoaGigubmZTz/9lFNOOYU77riD8vLypAn0mrpRSvVr2rRpNDU1MX78eMaOHQvA1VdfzYUXXkh5eTmzZs1iypQpfS7jpptu4rrrrmPGjBnMmjWL+fPnAzBz5kxmz57NtGnTmDx5MqeffnrXd2644QYWL17M2LFje+Tp58yZw7XXXtu1jOuvv57Zs2f3maaJ5rHHHuu6GDt58mQeeeQRgsEgX/rSl2hsbMQYwze/+U3y8vL47ne/y/Lly3E6nUydOpXFixcf9e/Fg/R1ChQv5eXlpr/7a5UaKTZv3szJJ58c72KoBBLpb0JEVhljyiPNr6kbpZRKcRrolVIqxWmgV0qpFDeoQC8ii0TkYxHZJiJ3RpnnHBFZKyIbReTdwfyeUkqpozfgu25ExAk8AJwLVAMVIvKSMWZT2Dx5wK+BRcaYXSIyarAFVkopdXQGU6OfD2wzxmw3xviAJ4GLe83zj8BzxphdAMaYA4P4PaWUUgMwmEA/HtgdNlxtjwt3IpAvIu+IyCoRuSbawkTkBhGpFJHK2traQRRLKRVvid5M8UgzmEAf6TUnvW/KdwFzgS8Anwe+KyInRlqYMWapMabcGFNeXFw8iGIppRLFSG+m2BhDKBSKdzEG9WRsNTAhbLgEqIkwz0FjTAvQIiIrgJnA1kH8rlIj12t3wr6PhnaZY06BxXdHnZyKzRS//PLL/OhHP8Ln81FYWMjjjz/O6NGjaW5u5pZbbqGyshIR4fvf/z6XX345r7/+Ot/+9rcJBoMUFRXx9ttvc9ddd5GVlcW//uu/AjB9+nReeeUVABYvXszChQt5//33eeGFF7j77ruPWD+AiooKbrvtNlpaWvB6vbz99tucf/75/PKXv+xqmO3000/nwQcf7LORt/4MpkZfAZwgImUi4gGWAC/1mudF4EwRcYlIBnAqsHkQv6mUOsZSsZniM844gw8++IA1a9awZMkS7rnnHgB++MMfkpuby0cffcT69ev57Gc/S21tLf/8z//Ms88+y7p16/jjH//Y7zb7+OOPueaaa1izZg2TJk2KuH4+n4+rrrqKX/ziF6xbt4633nqL9PR0rr/+eh599FEAtm7dSkdHx6CCPAyiRm+MCYjI14E3ACewzBizUURutKc/ZIzZLCKvA+uBEPCwMWbDoEqs1EjWR817uKRiM8XV1dVcddVV7N27F5/PR1lZGQBvvfUWTz75ZNd8+fn5vPzyy5x11lld8xQUFPS7zSZNmsSCBQv6XD8RYezYscybNw+AnJwcAK688kp++MMfcu+997Js2TKuvfbafn+vP4Nq1MwY8yrwaq9xD/Uavhe4dzC/o5SKr85mivft2xexmWK3201paemgmimuqKggPz+fa6+9dkibKQ5PEXW65ZZbuP3227nooot45513uOuuu7qW27uMkcYBuFyuHvn38DJ3HoD6Wr9oy83IyODcc8/lxRdf5Omnn+73gnUs9MlYpVS/Uq2Z4sbGRsaPt24SfOyxx7rGn3feefzqV7/qGq6vr+czn/kM7777Ljt27ACgrq4OgNLSUlavXg3A6tWru6b3Fm39pkyZQk1NDRUVFQA0NTV1tZ1//fXXc+uttzJv3ryYziD6o4FeKdWvaM0UV1ZWUl5ezuOPPx5TM8XNzc3MmDGDe+65J2IzxV/96lcjNlO8cOHCHssKb6b41FNP7WqmOFZ33XUXV155JWeeeSZFRUVd47/zne9QX1/P9OnTmTlzJsuXL6e4uJilS5dy2WWXMXPmTK666ioALr/8curq6pg1axYPPvggJ54Y8YbCqOvn8Xh46qmnuOWWW5g5cybnnntu11nB3LlzycnJ4brrrot5nfqizRQrleC0meKRp6amhnPOOYctW7bgcBxZH9dmipVSKon97ne/49RTT+XHP/5xxCA/EPqGKaWUSiDXXHMN11wTtRGBAdEavVJJIBFTrCo+BvK3oIFeqQSXlpbGoUOHNNgrjDEcOnSItLS0o/qepm6USnAlJSVUV1ejjf0psA78JSUlR/UdDfRKJTi32931VKZSA6GpG6WUSnFao1cqhYVChqb2AIdaOqhr8XGoxUddi4+GVj9lRRmUlxZQlOXtf0EqqWmgVyqJBIIhGtr8VtButoJ2XUtHVwA/1OKjrrm7v77VRzDU90XcyUWZzCstYF5ZAfNLC5hQkB6xDRaVvDTQKxVHHYEg9S3+rhp3eAA/ZAfx8Jp4Y5ufaDff5KS5KMzyUpDpYWJhBrMn5lGQ6enxKcz0UpDlISfNxdb9TazcUU9lVR2vbdjLU5XWC+NGZXu7gn55aT5TxuTgdGjgT2Ya6JUaIsYY2vzBsJp2d7DuXdPunN7cEYi4LIdAfkZ3gJ4yJtvu91J4RPD2kJ/pwe08uktucycVMHdSAXAcoZBh64EmKqrqqdhRR0VVHX9avxeAbK+LuaX5Vq2/tIAZJbmkuZ2D3VzqGNK2bpSKwhjD4fZAd3okQqDuqnU3W/0dgcivjXM7JWKgLsz0UJBldzO9FGS6Kcj0kpfuxhHnWnR1fSsVVXVdwf+TA80AeJwOZpTkMq+sgHml+cydVEBuujuuZVV9t3WjgV6NGMGQoaG1Z7Durml3HBHA61t8BKLkt9PdTitQZ/UK2uGBPKu7tp3tdSV93ruuxUdlVR2VO+tZuaOODXsaCYQMInDS6Oweef4xuUf3QI8aPA30KiX5AiHqW8Nz2h09a9q9xjf0k9/uTod4e9W0e+a3CzI8pHs0ddHqC7B2dwMVO+qp3FnHqp31tPqCAEwoSGfepAK71l/AccWZSX+gS3Qa6FVSaPUFogTqIy9K1jX7aIohv52f2R2su2va3h7jBpLfVkcKBENs2nu4R57/UIsPgIJMD+WT8plfVkB5aQHTxuXoNh9iGujVMRUKGVr9QZrbAzR3+GlsC8tzR7koeailg3Z/bPnt8OAdHsCtNIqX3HS33iWSAIwx7DjYQkVVHSt31FNRVceuulbASn3NnpjHvNIC5pcVMHtiHhkevTdkMDTQq5gEgiGaOwI0tQdo8QVobg/Q1BGwA3bvYX/XvM295mn2BaKmSCB6frs7gHt7XKhMhfy2suw/3E5FVR2VVVaef/O+wxgDTocwfVxOV56/fFI+hfog11HRQJ/CjDF0BEI9gm1X8O3w9xOsw+f3R61RhxOBLI+LrDQXWd7ubnbnsNdNltdpj3eTlWZNKwzLc2t+W3U63O5n9U6rtl+xo5611Q347DuXjivO7Lqlc35ZASX5+iBXXzTQJ6De6Y2mfgNxgJbw6XYQb+4I4A/2vw9dDiE7zUWmt1dgTnP3CtRW8M4+IohbQTvD7Yz7bX8qdXUEgnxU3chKu9ZfWVXH4XbrWszoHG9X0C+fVMBJY7I1RRemr0CvSbGjdKzSG53S3A6yvO4egbgkP51sb3aPWnV3YO4VuO2u1+XQ2pBKeF6Xk/JS64ItWBWij/c3UVlVx0r7Iu8rnQ9ypbkon5RPuR38Z5Tk4nXp2WIkI6JGn5DpjV615qxe/Z21b70zQaluxhiq69vsB7msh7m2dT7I5XIwqySP8tJ85pUVMHdSPjlpI+dBrhGTuvnmU2tpbPMPOr3RWTPO9rrI9Do1vaFUAjvU3EHlzvquWv/GsAe5pozJYX5pd61/dE7qPsg1YlI3nxxoQhBNbyg1ghRmefn8tDF8ftoYwH6Qa1cDK+1a/9OV1Tz2/k4AJhZkUF6az3z77p7JRSPjQa6UqtErpVRv/mCITTWHe6R76uwHuQozPVaqx767Z9q4HFxJmi4dMakbpZTqjzGGT2tb7FSPFfx317UBkOFxMmdifletf1YSPcg1bIFeRBYBvwCcwMPGmLujzDcP+AC4yhjzTH/L1UCvlDqW9jW296jxb7Ef5HI5hGnjc5lv1/rLSwsoyPTEu7gRDUugFxEnsBU4F6gGKoAvGmM2RZjvTaAdWKaBXimV6Brbwh7kqqpj3e5GfEHrjrvjR2XZqR4r+CfKg1zDdTF2PrDNGLPd/pEngYuBTb3muwV4Fpg3iN9SSqljJjfdzcIpo1g4ZRQA7f4gH+1pZKXdWNsr62p4YuUuAMbmpll39di3dZ44Kjvh7robTKAfD+wOG64GTg2fQUTGA5cCn6WfQC8iNwA3AEycOHEQxVJKqaGV5nZ2XbAF690GH+9r6qrxr9xxiJfX1QBWk9fl9rzzSvM5JQEe5BpMoI90yOqdB/o5cIcxJtjfqY0xZimwFKzUzSDKpZRSw8rpEKaOy2HquBy+clopxhh213U/yLWyqo6/bDkAgNflYOaEvK5Uz9xJ+WQf4we5BhPoq4EJYcMlQE2vecqBJ+0gXwScLyIBY8wLg/hdpZRKKCLCxMIMJhZmcPncEsB6kKuiqt5urbOOh97dzgPLP8XR+SBXWXetf9QwP8g1mIuxLqyLsf8H2IN1MfYfjTEbo8z/KPCKXoxVSo1ELR0B1uxq6Kr1r9nVQJvfeiPXpMKMrqB/5dwJA8rxD8vFWGNMQES+DryBdXvlMmPMRhG50Z7+0ECXrZRSqSbT6+KME4o444QiwHqQa2PNYSp2WKmetzfv5/1PD3HVvKG/RqkPTCmlVAIwxlDb1DHgNE5fNfrkfNZXKaVSjIgMW65eA71SSqU4DfRKKZXiEjJHLyK1wM4Bfr0IODiExYmnVFmXVFkP0HVJRKmyHjC4dZlkjCmONCEhA/1giEhltAsSySZV1iVV1gN0XRJRqqwHDN+6aOpGKaVSnAZ6pZRKcakY6JfGuwBDKFXWJVXWA3RdElGqrAcM07qkXI5eKaVUT6lYo1dKKRVGA71SSqW4pAz0IrJIRD4WkW0icmeE6SIi99vT14vInHiUMxYxrMs5ItIoImvtz/fiUc7+iMgyETkgIhuiTE+mfdLfuiTLPpkgIstFZLOIbBSR2yLMkxT7JcZ1SZb9kiYiK0Vknb0uP4gwz9DuF2NMUn2wWsr8FJgMeIB1wNRe85wPvIb1cpQFwIfxLvcg1uUcrOad417eftblLGAOsCHK9KTYJzGuS7Lsk7HAHLs/G6tZ8WT9X4llXZJlvwiQZfe7gQ+BBcO5X5KxRt/1rlpjjA/ofFdtuIuB3xnLB0CeiIw91gWNQSzrkhSMMSuAuj5mSZZ9Esu6JAVjzF5jzGq7vwnYjPUK0HBJsV9iXJekYG/rZnvQbX963xUzpPslGQN9pHfV9t7hscyTCGIt52fs07zXRGTasSnakEuWfRKrpNonIlIKzMaqPYZLuv3Sx7pAkuwXEXGKyFrgAPCmMWZY98tgXiUYL7G8qzaWeRJBLOVcjdWGRbOInA+8AJww7CUbesmyT2KRVPtERLKAZ4FvGGMO954c4SsJu1/6WZek2S/GmCAwS0TygOdFZLoxJvya0JDul2Ss0cfyrtpY5kkE/ZbTGHO48zTPGPMq4BaRomNXxCGTLPukX8m0T0TEjRUYHzfGPBdhlqTZL/2tSzLtl07GmAbgHWBRr0lDul+SMdBXACeISJmIeIAlwEu95nkJuMa+cr0AaDTG7D3WBY1Bv+siImNErLeri8h8rH126JiXdPCSZZ/0K1n2iV3G/wE2G2N+FmW2pNgvsaxLEu2XYrsmj4ikA58DtvSabUj3S9Klbkxs76p9Feuq9TagFbguXuXtS4zrcgVwk4gEgDZgibEvyycSEXkC666HIhGpBr6PdZEpqfYJxLQuSbFPgNOBLwMf2flggG8DEyHp9kss65Is+2Us8JiIOLEORk8bY14ZzhimTSAopVSKS8bUjVJKqaOggV4ppVKcBnqllEpxGuiVUirFaaBXSqkUp4FeKaVSnAZ6pZRKcf8fax+K8KGkzzgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(num = 2)\n",
    "fig1 = fig.add_subplot(2,1,1)\n",
    "fig2 = fig.add_subplot(2,1,2)\n",
    "fig1.plot(total_loss_train, label = 'training loss')\n",
    "fig1.plot(total_acc_train, label = 'training accuracy')\n",
    "fig2.plot(total_loss_val, label = 'validation loss')\n",
    "fig2.plot(total_acc_val, label = 'validation accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打印多类别预测的混淆矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    这个函数打印并绘制混淆矩阵。\n",
    "    可以通过设置‘normalize=True’来应用标准化.\n",
    "    \"\"\"\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, cm[i, j],\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVAAAAEmCAYAAAA0k8gFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd5wU9f3H8debOwEJoCCicAdSpSMdC2IXFRAsNFFBUNQQS4y/RGNPNBasEaMxJobYUGwUpSuiKFUQBQsoKHegFAsi4sHx+f0xc7iex95ybLu7z5PHPNiZnZ3Pd3b3Pvud73znOzIznHPO7bkKqS6Ac86VVp5AnXOuhDyBOudcCXkCdc65EvIE6pxzJeQJ1DnnSsgTqItK0r6SJkr6TtK4vdjOYEnT4lm2VJF0tKSPU10Ol3ryfqBlg6RzgKuA5sD3wBLgNjN7ay+3ex5wGXCkme3Y64KmOUkGNDWzlakui0t/XgMtAyRdBdwP/A04CKgP/APoE4fNHwJ8Uh6SZywkZaa6DC6NmJlPpXgC9gO2AP2irFOJIMGuDaf7gUrhc8cCOcAfgPXAOuCC8LlbgDxgexhjOHAz8GTEthsABmSG80OBzwhqwauAwRHL34p43ZHAAuC78P8jI56bBfwVmBNuZxpQazf7VlD+P0aUvy9wGvAJ8DXw54j1uwDvAN+G644GKobPzQ735YdwfwdEbP9PwJfAEwXLwtc0DmN0COfrAhuBY1P93fAp8ZPXQEu/I4DKwEtR1rkOOBxoBxxGkESuj3j+YIJEnEWQJB+SVMPMbiKo1T5rZlXN7N/RCiLpN8DfgVPNrBpBklxSxHo1gVfCdQ8A7gVekXRAxGrnABcAtYGKwNVRQh9M8B5kATcC/wLOBToCRwM3SmoUrpsP/B6oRfDenQD8FsDMuofrHBbu77MR269JUBsfERnYzD4lSK5PSaoCPA7818xmRSmvKyM8gZZ+BwAbLfoh9mDgL2a23sw2ENQsz4t4fnv4/HYze5Wg9tWshOXZCbSWtK+ZrTOzZUWs0xNYYWZPmNkOM3sG+AjoHbHO42b2iZn9CDxHkPx3ZztBe+92YCxBcnzAzL4P4y8D2gKY2SIzmxvGXQ38Ezgmhn26ycx+CsvzC2b2L2AFMA+oQ/CD5coBT6Cl3yagVjFtc3WBzyPmPw+X7dpGoQS8Fai6pwUxsx8IDnsvAdZJekVS8xjKU1CmrIj5L/egPJvMLD98XJDgvop4/seC10s6VNIkSV9K2kxQw64VZdsAG8xsWzHr/AtoDTxoZj8Vs64rIzyBln7vANsI2v12Zy3B4WeB+uGykvgBqBIxf3Dkk2Y21cxOIqiJfUSQWIorT0GZcktYpj3xMEG5mppZdeDPgIp5TdSuKpKqErQr/xu4OWyicOWAJ9BSzsy+I2j3e0hSX0lVJO0j6VRJd4WrPQNcL+lASbXC9Z8sYcglQHdJ9SXtB1xb8ISkgySdHraF/kTQFJBfxDZeBQ6VdI6kTEkDgJbApBKWaU9UAzYDW8La8aWFnv8KaPSrV0X3ALDIzC4kaNt9ZK9L6UoFT6BlgJndS9AH9HpgA7AG+B3wcrjKrcBCYCnwPvBuuKwksaYDz4bbWsQvk14FgrP5awnOTB9DeIKm0DY2Ab3CdTcRnEHvZWYbS1KmPXQ1wQmq7wlqx88Wev5mYIykbyX1L25jkvoApxA0W0DwOXSQNDhuJXZpyzvSO+dcCXkN1DnnSsgTqHPOlZAnUOecKyFPoM45V0JlemCEWrVqWf1DGqQkdnEdCxMplacFU7nfLvk+/3w1GzdujOvHnlH9ELMdv7rgq0j244apZnZKPOPviTKdQOsf0oDZb89PSezMjNRV7vN3pi6FZlTwFFqeHNW1U9y3aTt+pFKzYnuQAbBtyUPFXUWWUGU6gTrnSiOBSkfroidQ51x6EVAhI9WliIknUOdc+lHpaAryBOqcSzN+CO+ccyXnNVDnnCsB4TVQ55wrGZWaGmjpSPMJdumI4TSsdzBdOrT9xfJH/jGa9m1a0Ll9G67/858SXo41a9bQ48TjaNemBR0Oa8Xovz+Q0HiXjhhGg+yD6Ny+za5lX3/9Nb1PPZnDWh5K71NP5ptvvkloGQpMmzqFtq2a0ap5E0bddUdSYpbn2Mn+ru2xChmxTakuZqoLkA4GnzeElya8+otls2e9zisTJzB34RIWLH6fK678Q8LLkZmZyR133cOS9z/kjbfm8s9HHuLD5csTFm/weUN5eeLkXyy7d9QdHHv88by3/BOOPf547h2V+D/q/Px8rrx8JOMnTmbx0uWMG/tMQvfbYyf/u7ZnwpNIsUwplvoSpIFuR3enRo1f3oXhsX89wlVX/5FKlSoBcGDt2gkvR506dWjfoQMA1apVo3nzFqxdm7i7XBS1369MnMDgc4cAMPjcIUyaMD5h8QssmD+fxo2b0LBRIypWrEi/AQOZNDHxcctz7GR/1/aICA7hY5lSzBPobqxcsYK357zFcUcfwSknHseihQuSGv/z1atZsmQxnbt0TWrc9eu/4uA6dQA4uE4dNmxYn/CYa9fmkp1db9d8VlY2ubnJ+WMur7Ejpeq7FpXXQIsmaUsRy+pKej7ZZYlmx44dfPvtN7w2+21uvf1OhgweSLJG79+yZQuD+p/FqHvup3r16kmJmUpFva9KUu2ivMYukJ7fNT+E3yNmttbMzk51OSJlZWVxep8zkESnzl2oUKECGzcm/pY927dvZ1D/sxgwaDB9zzgz4fEKq137IL5ctw6AL9et48ADE990kZWVTU7Oml3zubk51K1bN8orPHY8pPq7FlUFxTalupiJ3LiklyUtkrRM0ohCz9WS9I6knpIaSPogXJ4haZSkBZKWSro44jV/lPS+pPckJfTsRq/T+/DGrNcBWLHiE/Ly8qhVK7EDv5gZl1w0nGbNW3DF769KaKzdOa1Xb556cgwATz05hp69T094zE6dO7Ny5QpWr1pFXl4e454dS89eiY9bnmOnw3dttwquhfez8Awzs45AJ+BySQdAcPtbgtu/3mhmrxR6zXDgOzPrDHQGLpLUUNKpBPc+72pmhwF3UQRJIyQtlLRw44YNMRXygvPO4YRjj2LFJx/TrHF9xjz+b84bMozVqz6jS4e2XHDeOfzzsccTfnj19pw5PP3UE7zx+mt07diOrh3bMWXyq8W/sISGnncOxx9zJCs++ZhDG9VjzOP/5qr/u4bXZszgsJaH8tqMGVz1f9ckLH6BzMxM7ntgNL179qBdmxac1a8/LVu1Snjc8hw72d+1PROfQ3hJzSQtiZg2S7pSUk1J0yWtCP+vEfGaayWtlPSxpB7FljSR7XqSbgbOCGcbAD2AN4AVwEgzeyNcrwEwycxah22hbYGt4ev2Ay4OX/uRmf0r1vgdOnYyHw80uXw80PLlqK6dWLRoYVw/9ArVs61S18tiWnfbjGsWmVmxg5JKygByga7ASOBrM7tD0jVADTP7k6SWwDNAF6AuMAM41Mzyd1vWmEpZApKOBU4EjghrjIuBysAOgvuJ7y67C7jMzNqFU0MzmxYu93swO1cexP8k0gnAp2b2OdAHGBMuH0NwZEu4fKyZ/WRmq4CVBMl0txJZTdoP+MbMtkpqDhweLjdgGNA8zP6FTQUulbQPgKRDJf0GmAYMk1QlXF6ziNc650q7WPuA7lmT2kCC2iXAQWa2DiD8v+BMaRawJuI1OeGy3UrktfBTgEskLQU+BuYWPGFm+ZIGAhMlbQYiG18eIzjcf1dBo+MGoK+ZTZHUDlgoKS98zZ8TWH7nXKrEfoKolqSFEfOPmtmjkStIqgicDlxbzLaKyshRj3oTlkDN7Cfg1CKeqho+n8cvD+Nbh8t3EiTGXyVHM7sDSO7Fys65JNuj8UA3xtAGeirwrpl9Fc5/JamOma2TVAcouFokB6gX8bpsYG20DadFP1DnnPuF+B7CD+Lnw3eACcCQ8PEQYHzE8oGSKklqCDQFop6F9uHsnHPpJY7jgYbnTE4i6MlT4A7gOUnDgS+AfgBmtkzSc8BygpPdI6OdgQdPoM65tBO/W3qY2VbggELLNhGclS9q/duA22LdvidQ51z6SYORlmLhCdQ5l37S4DLNWHgCdc6lF/ldOZ1zruT8EN4550om2eOilpQnUOdcWgnu6OEJNOVE6kZF2pYXtftYQlWuWDoa4J0rkij6oso0VKYTqHOuNBIVKvhJJOecKxE/hHfOuRLyBOqccyXhbaDOOVcyQl4Ddc65kvKTSM45V0JeA3XOuZIoRW2gpaOenGTTpk6hbatmtGrehFF3JfYOItu2beOE7ofTrWsHjujUlttvvRmAl198niM6taVm1X1Y/O7CqNuIVzm6HdGFLh0Oo8NhrfjrLTclPGakZL7nkS6+cBj169amY7vWSYsZKVX7nerYxZEU05RqnkALyc/P58rLRzJ+4mQWL13OuLHP8OHy5QmLV6lSJca/OoO35r3L7HcWMXP6VBbMn0uLlq3439PjOLLb0QmLXbgcU6a/xvx332PewiVMmzqFeXPnFv/COEj2ex7pvCFDGT9pSlJiFZbK/U5l7OIUnETyBFoKLZg/n8aNm9CwUSMqVqxIvwEDmTRxfPEvLCFJVK1aFYDt27ezffsOJNGseQuaHtosYXGLK8eO7duT9gVN9nseqdvR3alZMzV3yE7lfqcydiw8gZZSa9fmkp398435srKyyc3NTWjM/Px8jj68I4c2qMOxx59Ap85dExovWjm6dmxH/bq1Of7Ek+jSNTnlSMV7ng5Sud9p/Z4LVEExTamW0gQqqYGkD1JZhsLMfn0b6ET/0mVkZPDm3EUs++Rz3l20gOXLUvOWZGRkMG/RElauzmHhgvks+yA55UjFe54OUrnf6f6eew20lMrKyiYnZ82u+dzcHOrWrZuU2Pvtvz/djj6GmdOnJiXe7uy///50P+ZYpk1LTttgKt/zVErlfqf7ex6vBCppf0nPS/pI0oeSjpBUU9J0SSvC/2tErH+tpJWSPpbUo7jtp0MCzZQ0RtLScEerSOos6W1J70maL6mapAxJd0t6P1z3skQUplPnzqxcuYLVq1aRl5fHuGfH0rPX6YkIBcDGDRv47ttvAfjxxx+Z9fpMmjZLXttngQ0bNvBtRDlemzmDZs2aJyV2st/zdJHK/U7n9zzOJ5EeAKaYWXPgMOBD4Bpgppk1BWaG80hqCQwEWgGnAP+QFHVsyHToB9oMGG5mcyT9B/gdcAkwwMwWSKoO/AiMABoC7c1sh6QiW/4ljQjXpV79+ntcmMzMTO57YDS9e/YgPz+fIUOH0bJVq5LtWQy+/HIdvx0xjPz8fHbu3MkZZ53NKaf2YtKEl/nTH65g48YNDDjzdNq0PYwXJkxOXDnWreOiYUOCcthOzjq7P6f17JWweJGS/Z5HOv/cQbz5xiw2btxI4wbZ3HDjLQwdNjwpsVO536mMHZM4HJ2HuaM7MBTAzPKAPEl9gGPD1cYAs4A/AX2AsWb2E7BK0kqgC/DObmMU1RaSLJIaALPNrH44fzxwHVDZzI4qtO4LwCNmNj3W7Xfs2MnmzEt8H8qi+IDKrjw4qmsnFi1aGNfGyIq1m9iBZ42Kad21j5z5ObAxYtGjZvYogKR2wKPAcoLa5yLgCiDXzPYveIGkb8yshqTRwFwzezJc/m9gspk9v7v46VADLZzBNwOVilhPRazrnCuD9uBa+I1m1mk3z2UCHYDLzGyepAcID9d3o6gfgqg5Jx3aQOtLOiJ8PAiYC9SV1BkgbP/MBKYBl4SP2d0hvHOuDFCMU3Q5QI6ZzQvnnydIqF9JqgMQ/r8+Yv16Ea/PBtZGC5AOCfRDYIikpUBN4EFgAPCgpPeA6UBl4DHgC2BpuPycFJXXOZdg8TiJZGZfAmskFZyVPYHgcH4CMCRcNgQouIJgAjBQUiVJDYGmwPxoMVJ6CG9mq4GWRTy1ADi8iOVXhZNzroyKcx/Py4CnJFUEPgMuIKg4PidpOEGlrB+AmS2T9BxBkt0BjDSzqCcz0qEN1DnnfiFeCdTMlgBFtZGesJv1bwNui3X7nkCdc2knHS7TjIUnUOdc2kmHyzRj4QnUOZde5AnUOedKREApyZ+eQJ1z6SY9RlqKhSdQ51zaKSX50xOocy7NCCr4WXjnnNtzwhNouZfKEZE2ff9TymIfUK2ocWBcou3cmZpxdhIV1Q/hnXOuhPwkknPOlYS8BuqccyUS9AMtHRnUE6hzLs3ITyI551xJeQ3UOedKwttAnXOuZLwN1Dnn9kIpyZ9pcU+ktLJmzRp6nHgc7dq0oMNhrRj99weSGn/a1Cm0bdWMVs2bMOquOxIa69MVn3By9y67pub1D+Sxhx9k1G03c2K3TpzcvQvnnNmTL9dFva/WXrv4wmHUr1ubju1aJzROUbZt20a3I7rQpcNhdDisFX+95aakxk/m513YQw8+QKf2bejUrjWj/35/UmMXp0IFxTSlmifQQjIzM7njrntY8v6HvPHWXP75yEN8uHx5UmLn5+dz5eUjGT9xMouXLmfc2GcSGrtx00OZNns+02bPZ/Lr77BvlSqc0ut0LrnsKma8tZBps+dzQo/TuH/U3xJWBoDzhgxl/KQpCY2xO5UqVWLK9NeY/+57zFu4hGlTpzBv7tykxE725x1p2bIPePw/jzF7zjzmLlzC5FdfYeWKFUmJXSzF56ZyyeAJtJA6derQvkMHAKpVq0bz5i1YuzY3KbEXzJ9P48ZNaNioERUrVqTfgIFMmji++BfGwVtvvMYhDRqSXe8QqlWvvmv5j1t/SPgXtdvR3alZMzV3qZZE1apVAdi+fTs7tm9P2h9mKj/vjz/6kC5du1KlShUyMzM5unt3Jox/KSmxi1MwHmgsU7HbklZLel/SEkkLw2U1JU2XtCL8v0bE+tdKWinpY0k9itu+J9AoPl+9miVLFtO5S9ekxFu7Npfs7J9vS52VlU1ubnKS94QXx9HnrAG75u+89UY6t27MS+PGcvW1NyalDKmSn59P147tqF+3NsefeBJdupb9z7tly9bMefNNNm3axNatW5k6ZTK5OWuSErt4sdU+9+CH7jgza2dmBTeXuwaYaWZNgZnhPJJaAgOBVsApwD8kRR3UIm0SqKQGkj4oYvlqSbWKWL4lkeXZsmULg/qfxah77qd6RI0skcx+PTRDMmpDeXl5TJvyCr36nLlr2Z+u/wsLPviUM/oN5PF/PZzwMqRSRkYG8xYtYeXqHBYumM+yD371NUyIVH3eAM1btOCqq/9I79NOpm/vU2nTpi0ZmelzTjleNdDd6AOMCR+PAfpGLB9rZj+Z2SpgJdAl2obSJoGmk+3btzOo/1kMGDSYvmecWfwL4iQrK5uciFpAbm4OdevWTXjc12dMpU3bdhxY+6BfPdf37AFMnvhywsuQDvbff3+6H3Ms06Ylpz02VZ93gSEXDOfteYuYNvMNatSsSZMmTZMWuzh7UAOtJWlhxDSi0KYMmCZpUcRzB5nZOoDw/9rh8iwgshqeEy7brXRLoJmSxkhaKul5SVUKnpC0r6Qpki5KZAHMjEsuGk6z5i244vdXJTLUr3Tq3JmVK1ewetUq8vLyGPfsWHr2Oj3hcce/8Bx9zuq/a/6zT1fuejxt8is0btos4WVIlQ0bNvDtt98C8OOPP/LazBk0a9Y8KbFT9XkXWL9+PQBrvviCCS+/RL8Bg5IWOxppj87CbzSzThHTo4U2d5SZdQBOBUZK6h4tdBHLoo7Ylz519kAzYLiZzZH0H+C34fKqwFjgf2b2v2gbCH9lRgDUq19/jwvw9pw5PP3UE7Ru3YauHdsBcMutf+OUU0/b423tqczMTO57YDS9e/YgPz+fIUOH0bJVq4TG/HHrVmbPmskd943etez2W67ns5WfoAoVyK5Xn9vveTChZTj/3EG8+cYsNm7cSOMG2dxw4y0MHTY8oTELfLluHRcNG0J+fj47bSdnnd2f03r2SkrsVHzekQYPPJuvN20ic599uPeB0dSoUaP4FyVJvJoyzGxt+P96SS8RHJJ/JamOma2TVAdYH66eA9SLeHk2ELUPn4pqh0kFSQ2A2WZWP5w/HrgcaAd8B9xlZk9FrL/FzKpG22bHjp1szryFCStzuvIBlcufVA2o3O2Izry7aGFcG26r1WtuHa76d0zrzr6q26KIk0O/IOk3QAUz+z58PB34C3ACsMnM7pB0DVDTzP4oqRXwNEGSrUtwgqmpmeXvLn661UALfwsK5ucAp0p62tIl4zvnEiZONdCDgJfCbWUCT5vZFEkLgOckDQe+APoBmNkySc8By4EdwMhoybNgo+mkvqQjzOwdYBDwFtAeuBG4AfgHcGkKy+ecS7Q4DSZiZp8BhxWxfBNBLbSo19wG3BZrjHQ7ifQhMETSUqAmENl/5kqgsqS7UlIy51xSKP79QBMmbWqgZrYaaFnEUw0iHl8QsX7U9k/nXOmVkQbXuccibRKoc84VSIPKZUw8gTrn0opUBsYDlRT1+kUz2xz/4jjnHJSSI/ioNdBlBN2IInelYN6APe+l7pxzMSj1NVAzq7e755xzLlEEVCglCTSmbkySBkr6c/g4W1LHxBbLOVeeVVBsU6oVm0AljQaOA84LF20FHklkoZxz5ViMfUDT4TA/lrPwR5pZB0mLAczsa0kVE1wu51w5lga5MSaxJNDtkioQXpcu6QBgZ0JL5Zwrt0pTG2gsCfQh4AXgQEm3AP2BWxJaKrdXUjkiUqpGBSqQDndqTIVU7XeiopaS/Fl8AjWz/0laBJwYLupnZsm554FzrtwpGFC5NIj1SqQMYDvBYXy6DUDinCtjSsshfCxn4a8DniEYYDQbeFrStYkumHOu/FKMU6rFUgM9F+hoZlsBJN0GLAJuT2TBnHPlVzp0UYpFLAn080LrZQKfJaY4zrnyLjgLn+pSxCbaYCL3EbR5bgWWSZoazp9MMFK8c87FX5p0ko9FtBpowZn2ZcArEcvnJq44zjlXBs7Cm1lst8Vzzrk4Kk2H8LGchW8saaykpZI+KZiSUbhUuPjCYdSvW5uO7VqnJP60qVNo26oZrZo3YdRdd5Sb2N9++y2DB/ajfZsWdGjbknlz30lKXP+8UxO7OPG8Fl5ShqTFkiaF8zUlTZe0Ivy/RsS610paKeljST2K23YsfTr/CzxO8MNwKvAcMDamkpdC5w0ZyvhJU1ISOz8/nysvH8n4iZNZvHQ548Y+w4fLl5f52AD/94crOenkHix+/0PmLlxCs+YtkhLXP+/UfN7FiXM3pisIblhZ4Bpgppk1Jbj3+zUAkloCA4FWwCnAPyRlRNtwLAm0iplNBTCzT83seoLRmcqkbkd3p2bNmimJvWD+fBo3bkLDRo2oWLEi/QYMZNLE8WU+9ubNm5nz5myGXDAcgIoVK7L//vsnJbZ/3smPXRwp6Egfy1T8tpQN9AQei1jcBxgTPh4D9I1YPtbMfjKzVcBKoEu07ceSQH9SUFf+VNIlknoDtWN4ndtDa9fmkp398zjWWVnZ5ObmlvnYq1Z9Rq0DD+Tii4ZxRJcO/PaSC/nhhx+SEjuVyuvnHYsKFRTTBNSStDBiGlFoU/cDf+SXAyAdZGbrAML/C/JZFrAmYr2ccNnuyxnDvvweqApcDhwFXAQMi+F1cSXpZklXS2ouaUnYptE42eVIJLNfD8SRrO4cqYydv2MHSxa/y0UjLuGd+e9SpcpvuGdUerXJJUJ5/bxjEdxYrvgJ2GhmnSKmR3/ehnoB681sUaxhi1gWdXScWAYTmRc+/J6fB1VOpb7AeDO7KdUFibesrGxycn7+AczNzaFu3bplPnbdrGyysrPp3KUrAGeceTb3jLozKbFTqbx+3sURsR2ex+Ao4HRJpwGVgeqSngS+klTHzNZJqgOsD9fPASJvZZQNrI0WYLc1UEkvSXpxd9Ne7VaMJF0Xng2bATQDqgBXAhdKej0ZZUimTp07s3LlClavWkVeXh7jnh1Lz16nl/nYBx98MNnZ9fjk448BmPX6TJq3SM5JpFQqr593sWKsfRaXY83sWjPLNrMGBCeHXjOzc4EJwJBwtSFAQePvBGCgpEqSGgJNgfnRYkSrgY4ubj8TKbzv0kCgPUE53yW4Bv8RYIuZ3b2b140ARgDUq7/nNw49/9xBvPnGLDZu3EjjBtnccOMtDB02vIR7sWcyMzO574HR9O7Zg/z8fIYMHUbLVq3KfGyAu+/7O8OGnkteXh4NGzbikX/9Jylx/fNOzeddnAQ3J9wBPCdpOPAF0A/AzJZJeg5YDuwARppZftRyFtUWkg4kXQnUNLMbw/l7CarTVYmSQCN17NjJ5sxbmNiCul/wAZXLl6O6dmLRooVxfdNrN2ltA0aNi2nd0We2XGRmneIZf0/EOh5oqqRndnfOJYyAjFLyQ5jOgyPPBs6QtK+kakDvVBfIOZccpeW2xjHXQCVVMrOfElmYSGb2rqRngSUEQ+q9mazYzrnUCU4QpUF2jEGxCVRSF+DfwH5AfUmHARea2WWJLpyZ3Qbclug4zrn0kg61y1jEcgj/d6AXsAnAzN6jDF/K6ZxLvXh0Y0qGWA7hK5jZ54Wq1FFP7TvnXEmVtfvCrwkP4y0cmeQyoMwOZ+ecS72M0pE/Y0qglxIcxtcHvgJmhMuccy7uFONIS+kglmvh1xNcEeScc0lRSvJnTGfh/0URHdrNrPCwUc45Fxel5Sx8LIfwMyIeVwbO4Jdj5jnnXNyUqZNIZvZs5LykJ4DpCSuRc658E2Sk8zWSEUpyLXxD4JB4F8Q55wpoT+54lEKxtIF+w89toBWArwlvwuScc/FWmm5rHDWBhvdCOgwouFnKTkvX8e/STCrfpi3bdqQsdrV990lZbIAanX+Xsthfz38wZbE3/5iaz3xHgoYvLC0JNGpLQ5gsXzKz/HDy5OmcS7h43hc+kWJpqp0vqUPCS+Kcc/x8CF+qh7OTlGlmO4BuwEWSPgV+INg/MzNPqs65+FPpGVA5WhvofKADP9903jnnEq6snEQSgJl9mqSyOOccEJ9LOSVVJrizRSWCXPe8md0kqSbwLNAAWA30N7NvwtdcCwwnGHHucjObGi1GtAR6oKSrdvekmd0b+64451ysRIX49AP9CTjezIVLKdQAACAASURBVLZI2gd4S9Jk4ExgppndIekagm6Zf5LUkmDcj1ZAXWCGpEOj3Zkz2kmkDII7YFbbzeScc3En4nZfeDOzLeHsPuFkQB9gTLh8DD83U/YBxprZT2a2ClgJdIkWI1oNdJ2Z/SV6EcumaVOncPVVV5Cfn8/QYRfyf39M3nUDzZs2pFrValTIyCAzM5M5cxckNF5+fj4nHXM4depk8dS4l3l/6RL+eOXv2PbTNjIzM7nzngfp0KlzQstw8YXDmPzqJA6sXZtFSz5ISIymh9TmiTuH7ZpvmHUAf334FUY/PYtLBx7DJQO6syN/J1Pe/IDrHhhPZmYFHr5xMO2a1yMzowJPvTKfu/8zLSFly8/P56jDO1M3K4sXX56YkBgFOrVpStWqVcnIyCAjI5Npb8xlxNBz+HRlMMTvd999x3777cfMt1J4O/A9O8NeS1JkYR81s0d3bSoYw3gR0AR4yMzmSTrIzNYBmNk6SbXD1bOAuRHbygmX7VaxbaDlTX5+PldePpJXJk8nKzubbod3plev02nRsmXSyjB5+mvUqlUrKbEeffhBDj20Od9//z0Af7nhz1x9zfWccPIpzJg6mb/ceC0vvzqjmK3snfOGDOWS3/6OC4edn7AYKz5fz+ED7wCCe8d/OvU2Jrz+Ht07NaXXsW3o3P928rbv4MAaVQE468QOVKqYSef+f2Pfyvuw+IXreW7yQr5Y93Xcy/bQgw/QvHkLNn+/Oe7bLsoLk6ZzwAE/f78e/e/Tux7fdN0fqV69elLKsTt7eFvjjdHuCx8efreTtD/wkqTWxYT+1SaiBY92CH9CtBeWVQvmz6dx4yY0bNSIihUr0m/AQCZNHJ/qYiXE2twcZkydzOAhP9fMJPF9+Ie8efN3HHxwnYSXo9vR3alZs2bC4xQ4rkszVuVs4It13zCi39Hc/fh08rYHV/Js+CY44jOMKpUrkpFRgX0rVSRvez7f/7At7mXJyclhyuRXGTpseNy3vafMjIkvPc8ZZw9IdVGoEA6qXNwUKzP7FpgFnAJ8JakOQPj/+nC1HKBexMuygbVRyxklYPx/akuBtWtzyc7++T3MysomNzc3yiviSxK9T+vBkV078e/HHi3+BXvh+mv+wI1/uZ0KFX7+Gtx6593ccsO1tGvRiJuvv4brbr41oWVIhX49OvLclEUANDmkNke1b8zs/13NtMeuoGPL+gC8OGMxW7flsWr6bXwy+S/c/7+ZfLN5a9zL8sc//J5bb7/zF59BIgkxsO9pnNy9K088/tgvnpv79lvUOrA2jRo3TUpZoolHG6ikA8OaJ5L2BU4EPgImAEPC1YYABTWkCcBASZUkNQSaEnTn3K1SMmhU8hR1tWoyLxmbOest3pm/iJcnvsqjD/+Dt96cnZA40ya/Qq1atTms/S+vh/jvY4/yl9tHseTDz/jr7aO48ncXJyR+quyTmUHPY9rw4vTFAGRmVKBG9Sp0P/9u/nzfyzx5V1Ab79yqAfn5O2l08nW06HkTV5x3PA2yDohrWV59ZRIH1j6QDh06xnW70UycNovpb87nqRcm8vhjD/POnDd3PffS88+mRe1TBIkplqkYdYDXJS0FFgDTzWwScAdwkqQVwEnhPGa2DHgOWA5MAUZGOwMPJRvOrkzLysomJ+fn8aJzc3OoW7du0uIXxKpduza9+/Rl4YL5dDu6e9zjzJ/3NlMnT2Lm9Cls27aNLd9v5tILhzBtyivcdlfQQ+30M87m95ddEvfYqdSjW0uWfLSG9V8Hbb65X33LyzPfA2Dhss/ZudOoVaMq/U/txLS3l7Njx042fLOFd5Z8RseW9VmduyluZZn79hxemTSRqVMms23bNr7fvJlhQ87jP2OeiFuMwg6uE3y/DjywNqf26sPiRQs44qij2bFjB69OfJlpb8wtZgtJoPhUWsxsKdC+iOWb2E0TpZndBtwWa4y0q4FKaiDpQ0n/krRM0jRJLSTNL7TO0kTE79S5MytXrmD1qlXk5eUx7tmx9Ox1eiJC/coPP/yw62TODz/8wMwZ02nZKlqbd8ldf/NtvPfRKhZ9sIJHH3+Sbt2P4+HHxnDwwXV4+62g1vvmG6/TqHGThMRPlf6ndNp1+A4wcdZSju1yKABN6tem4j6ZbPxmCzlffs2xnZsBUKVyRbq0bcDHq7+Ka1n+ctvtrFy1ho9WrOJ/Tz7DMccdn9Dk+cMPP7Al4vv1xmszaN6yFQCzZ82kyaHNqJuVnbD4sRKQIcU0pVq61kCbAoPM7CJJzwEdgYqSGpnZZ8AAgqr2r0gaAYwAqFe//h4HzszM5L4HRtO7Zw/y8/MZMnQYLVu1KvGO7In1X33FwH5nArBjxw76DxzEyT1OSUrsAvc8+AjX/+kqduzYQeVKlbnngYcTHvP8cwfx5huz2LhxI40bZHPDjbck5KTKvpX34fiuzfndrc/sWjbm5Xf4582DWTjuz+Rtz+fCG4ME9sizs3n0lnNZ9Px1SPDE+Ll8sCLq+YS0t3H9V1xwbj8g+H6defZAjj+xBwAvv/AcZ5yV+sP3AqlPjbFRuo1QJ6kBQVtF03D+TwQdYHcSjEd6h6R3gQFmtiLatjp27GRz5qWmP5uPB5oaPh5ocp18zOG8t3hRXPNdo5Zt7dYnX41p3cEd6y2K1o0p0dLuED70U8TjfIKa8rNAf0mHElxkEDV5OudKq9jGAk2H8UDT9RD+V8zsU0n5wA0EydQ5VwYVnIUvDUpNAg09C4wiuLGdc66MSofaZSzSLoGa2WqgdcT83YUe313Ey5xzZYXK0H3hnXMumfwQ3jnn9oIfwjvnXAmVjvTpCdQ5l4ZKSQXUE6hzLr0EbaClI4N6AnXOpZk9G+szlTyBOufSTinJn55AnXPpxQ/hnXOupGIYbT5deAJ1zqUdT6DlXCo7Aqd6SLlU+mbB6JTFzt+ZuiEM96uSms88cw/uP7wn5Ifwzjm35wpGpC8NSsslp865ciROd+WsJ+n18BZByyRdES6vKWm6pBXh/zUiXnOtpJWSPpbUo7hyegJ1zqUdxfivGDuAP5hZC+BwYKSklsA1wMzwrhczw3nC5wYCrQjuH/8PSRnRAngCdc6lFQEVFNsUjZmtM7N3w8ffAx8CWUAfYEy42higb/i4DzDWzH4ys1XASqBLtBieQJ1zaSbW+qcAaklaGDGNKHKLwb3W2gPzgIPMbB0ESRaoHa6WBayJeFlOuGy3/CSScy69xFC7jLCxuJvKSaoKvABcaWabo/SQKeqJqF0rPIE659JKcAgfn7PwkvYhSJ5PmdmL4eKvJNUxs3WS6gDrw+U5QL2Il2cDUe9l7YfwRZg2dQptWzWjVfMmjLrrDo9dhmOvWbOGHiceR7s2LehwWCtG//2BhMa7dMQwGmQfROf2bXYte/GFcXRq15pqlTN4d1HybsOdys+7OIpxirqNoKr5b+BDM7s34qkJwJDw8RBgfMTygZIqSWoINAXmR4vhCbSQ/Px8rrx8JOMnTmbx0uWMG/sMHy5f7rHLaOzMzEzuuOselrz/IW+8NZd/PvJQQmMPPm8oL0+c/ItlLVu25ulnX+Coo7snLG5hqXzPYxKPDApHAecBx0taEk6nAXcAJ0laAZwUzmNmy4DngOXAFGCkmeVHC+AJtJAF8+fTuHETGjZqRMWKFek3YCCTJo4v/oUeu1TGrlOnDu07dACgWrVqNG/egrVrcxMWr9vR3alRo+YvljVv0YJDmzVLWMyipPI9j0U8ujGZ2VtmJjNra2btwulVM9tkZieYWdPw/68jXnObmTU2s2ZmNjna9sET6K+sXZtLdvbPzSBZWdnk5ibuD8pjpzZ2pM9Xr2bJksV07tI16bGTLV3e892JR0f6ZPCTSIWY/fqkW7Kua/fYyY9dYMuWLQzqfxaj7rmf6tWrJzV2KqTDex5NGhUlKk+ghWRlZZOT83NXsNzcHOrWreuxy2hsgO3btzOo/1kMGDSYvmecmbS4qZTq9zyaoHmzdGTQpBzCS7pT0m8j5m+WdJOkmZLelfS+pD7hc7+R9Iqk9yR9IGlAuLyzpLfD5fMlVUtEWTt17szKlStYvWoVeXl5jHt2LD17nZ6IUB47DWKbGZdcNJxmzVtwxe+vSkrMdJDK97xYMR6+p0MtNVk10LHA/cA/wvn+BNea3hd2bK0FzJU0IVy+1sx6AkjaT1JF4FlggJktkFQd+LGoQOGVCCMA6tWvv8cFzczM5L4HRtO7Zw/y8/MZMnQYLVu12uPtlITHTn7st+fM4emnnqB16zZ07dgOgFtu/RunnHpaQuINPe8c3pw9i00bN3Joo3pcd8PN1KhZk6t/fzkbN2zgrL69aNu2HeNfmZKQ+AVS+Z7HIg1yY0xUVFtIQgJJHwInAAcSJNJjgfuA7sBOoBnQEKgOTCXoTjDJzN6U1AZ4xMyO2pOYHTt2sjnzktevzpVvqRwPNCNB43IW56iunVi0aGFcg7ds296enPhGTOt2bLDfouKuREqkZLaBPg+cDRxMUCMdTJBMO5rZdkmrgcpm9omkjsBpwO2SpgEvU8wlVc65siKmkZbSQjK7MY0lGCrqbIJkuh+wPkyexwGHAEiqC2w1syeBu4EOwEdAXUmdw3WqSfITYM6VQfEajSkZkpaEzGxZeOInN7wG9SlgoqSFwBKCJAnQBhglaSewHbjUzPLCk0kPStqXoP3zRGBLssrvnEuiNEiOsUhqLc7M2kQ83ggcUcRqqwnaQAu/dgHBoKjOuTKutBzC+2Gwcy7tpEMXpVh4AnXOpZ1Skj89gTrn0ozS67LSaDyBOufSivBDeOecK7FSkj89gTrn0lApyaCeQJ1zace7MTnnXAl5G6hzzpWQJ1DnnCuB0jSgsifQMmhnCodVq5DiER5Sue+pGlIOYNv2qDePTJiEvN1xHCxZ0n+AXgQDF7UOl9UkGF+4AcGl4/3N7JvwuWuB4UA+cLmZ/eqy8kh+UznnXNqJz12NAfgvwSDtka4BZppZU2BmOI+klgQjxrUKX/MPSRnRNu4J1DmXfuKUQc1sNvB1ocV9gDHh4zFA34jlY83sJzNbBawEukTbvidQ51yaifWu8AKoJWlhxDQihgAHmdk6gPD/2uHyLGBNxHo54bLd8jZQ51xaKRhQOUYb43hLj6KiRm3l9Rqocy79xLERtAhfSaoDEP6/PlyeA9SLWC8bWBttQ55AnXNpZw8O4UtiAjAkfDwEGB+xfKCkSpIaAk2B+dE25Ifwzrm0E8duTM8Q3AG4lqQc4CbgDuA5ScOBL4B+sOu2Q88By4EdwEgzi9o/zGughVx84TDq161Nx3atUxJ/2tQptG3VjFbNmzDqrjuSGvuhBx+gU/s2dGrXmtF/vz+psVO13598/DGHd26/azq41n5J3fdk7ndOzhp6n3ICXdu35oiObXnkob8D8P7S9zj52KM4snM7Bp7Vh82bNye0HLGI1xG8mQ0yszpmto+ZZZvZv81sk5mdYGZNw/+/jlj/NjNrbGbNzGxycdv3BFrIeUOGMn7SlJTEzs/P58rLRzJ+4mQWL13OuLHP8OHy5UmJvWzZBzz+n8eYPWcecxcuYfKrr7ByxYqkxE7lfh/arBlzFyxm7oLFzJm7kH2rVOH0PmckJXay9zszI5Nbbx/FvMUfMG3WHB7758N89OFyrvjtxdz017/x9oIl9Dq9Lw/ed3fCyhCTcEDlWKZU8wRaSLeju1OzZs2UxF4wfz6NGzehYaNGVKxYkX4DBjJp4vjiXxgHH3/0IV26dqVKlSpkZmZydPfuTBj/UlJip3K/I73+2kwaNWpM/UMOSUq8ZO/3wXXqcFj7DgBUq1aNQ5s1Z93aXFau+Jgju3UH4NgTTmRikj733SkYUDmWKdU8gaaRtWtzyc7++SRgVlY2ubm5SYndsmVr5rz5Jps2bWLr1q1MnTKZ3Jw1xb8wDlK535GeHzeWfv0HJi1eKvf7i89Xs/S9JXTs3JXmLVsxedJEAMa/+HzSPvdoEnsSPn5KZQKVdKykSakuR7yZ/brLWbIOU5q3aMFVV/+R3qedTN/ep9KmTVsyMpNzjjGV+10gLy+PVydN5Iyz+iUtZqr2e8uWLZw/qD+333Uv1atXZ/Qjj/HYo//g2CO7sOX779mnYsWEl6E4paUG6mfh00hWVjY5Eb/+ubk51K1bN2nxh1wwnCEXDAfgphv+TFZWdlLipnq/AaZNmcxh7Tpw0EEHJS1mKvZ7+/btDDmnH/0GDqJ336Ct99BmzXlxYtDuv3LFJ0yb8mpCyxCL0jIaU8pqoJIaSPpI0mOSPpD0lKQTJc2RtEJSF0m/kfQfSQskLZbUJ1XlTYZOnTuzcuUKVq9aRV5eHuOeHUvPXqcnLf769UF/4jVffMGEl1+i34BBSYmb6v0GGPfcWPoNSN7hOyR/v82Myy69iEObtWDk5b/ftXxD+Lnv3LmTu+/8GxdceHHCyhCzUnIMn+oaaBOCPlgjgAXAOUA34HTgzwT9sV4zs2GS9gfmS5qRyAKdf+4g3nxjFhs3bqRxg2xuuPEWhg4bnsiQu2RmZnLfA6Pp3bMH+fn5DBk6jJatWiUlNsDggWfz9aZNZO6zD/c+MJoaNWokJW6q93vr1q28NnM6f3/okaTFhOTv99x35vDs00/SsnUbju7aEYAbbvkrn326ksf++TAAvfr0ZfD5QxNWhlhIe3QpZ0qpqHaYpASWGgDTwyGlkPQ/YKqZPSWpEfAiQWfWyuH/ADWBHsBBwNVm1quI7Y4gSMjUq1+/4yeffp7gPUk/Ph5oaqRy31M1HuhxR3Vl8bsL47rj7Tp0tOlvzItp3drV91kUx2vh91iqa6A/RTzeGTG/k6Bs+cBZZvZx5Isk7bahysweBR4F6NixU+r+mpxzJVdKaqDpfhZ+KnCZwlOTktqnuDzOuSQoJU2gaZ9A/wrsAyyV9EE475wr47wbUzHMbDXQOmJ+6G6e+9UpQTObBcxKYPGccymzVyMtJVWq20Cdc+4XCi7lLA08gTrn0o4nUOecKyE/hHfOuZJIkxNEsfAE6pxLK+nSRSkWnkCdc2knHQZLjoUnUOdc2ikl+TPtO9I758qheF2JJOkUSR9LWinpmniX0xOocy79xCGDSsoAHgJOBVoCgyS1jGcxPYE659JOnO4L3wVYaWafmVkeMBaI65jCZboN9N13F23cdx+VdDy7WsDGeJbHY6d17FTHL62x434HvsXvLppapaJqxbh6ZUkLI+YfDUdkA8gCIm/wlAN0jUcZC5TpBGpmB5b0tZIWpmqcQY+dGuV131P9vhdmZqfEaVNFVVHjOsSlH8I758qqHKBexHw2sDaeATyBOufKqgVAU0kNJVUEBgIT4hmgTB/C76VHi1/FY5eh2KmOX15jJ4yZ7ZD0O4KB2TOA/5jZsnjGSNk9kZxzrrTzQ3jnnCshT6DOOVdCnkBdkcKrOJxzUXgCjZEKDQ9TeL6skNRNUlUzy09mEpV0mqQzJaX8xKakcvd3Iale8Wu5wsrdF6UkJMnCs22SDpNUwRJ49i0yOUuqlKg4u3E+8Ekyk6ikJsAYYBkp7Bki6RxJTc1sZ6rKkAqSDgBGS7oi1WUpbTyBxiAieV4G3EhwiVhCFErWg4HBkvZJVLyIuBUAzGwE8BywOBlJVFJdgqtDHgF+S9hPL0VNCM2BC1LdfCHpQklHJjHkDwRdmY6WdGkS45Z6nkBjJKkHMBQYaWZrilm9xCKS5yXAH4HZZrY9UfEi4u4M4zY1syuBmcCiRCZRSdnANUBPgttYDwJeDsuTn4JmkjeAgwj/LlJxKC9pJDAS+DYJsQRgZtuAGcBjwCmeRGPnCTR2WcDbZvalpIxE/XFLqhAeUp0CDDSzlYlsF5TUKPxfYa3n/rAWfAnwGr9MovH+vuQCS4D9gOXAv4H9JQ2A4Mck0UlU0ulhZ2vMbCawL3BvOJ/UQ/nwc+8LnGlmywve70S8B4WOdA4GqprZFOBh4GRPorHxBFqEQm2QBYfPHxP8cbcws/zwj3ugpPPjGc/MdprZJuAboKWkTDPbEa53uKT99jZeQUxJlYFXJP01/GNaQ5DU9gnLcikwC1gt6TfxTCgRf8A7gCOAw4EvCK4YaSvpzLAMiWxrbgJsBi6RdLOkC4HrgZ2S4j7KUDFl2QfIB2qE/8PPf5/14x0vInleDfwLmCjpD8A8guaUEyT9Pt5xyxpPoEWI+HINB26UNAL4CfgO6CdphKRzgT8Dc+IY7wpJN4bX7X4OdAAKaogDgGuJ30mWCuGhWx+gp6TrCBLnZn7+A8bMLiYYlLZOnOIWbNfCNt7LCA7jlxEM9vB9WIZjJMV17MZIYa1zMsFgu2OAlcBhwIvA2cAxiYpdRFkGAaeY2bfAbGCUpJrhpYhDgf9J+k0C4vYFTjKz3gT7383MviFovvkf0EnS/vGOW5b4pZyFhGfYd4bJ8wLgSoK2sWHAe8BRQHdgJ3Cvmb0fp7iXEpwBv9DMloU1zbuB3wBVCGohQ81saTziRcStDDQgSCbvAgcQJO9vgMrACjO7J54xI2L/BfjezEaFPxq/BU4CXg9XGWNmGxIQ93SgF3BnGK8esI7gh+IMgsPou8zsg3jHLqIsI4ELgf5mtiI8nL4YOA94nqAp59x4lKXgux0xfxKwP8HJs25AbzPLk9QkbDr6jZn9sLdxyzJPoCFJXYClZrZNUlXgDuB+gsPLIcCpkSdzJFUys5/2Il5BolZYG/snwWCwiyRVMbOtYa2jLnAw8JmZ5e7NPoZxjwTqm9nY8I93BMFgCw0IarwfESTu2gSHk1PNbPXext1NWfoSnJi7rmCQB0lvA5OAR8zs6wTEzALeAWaY2TAF3cTOIGhC+Jwgie5IRvunpKbAk2H8LwlOpjUmSJyHEvRO+NzMPotz3L7AVoLKwGEE42aeFdZ4LwN6AP3M7Md4xi2LUt5pOR2EbZAnAGskrTezLZI+J2hQzzezE8P1rgU+NrMX9yZ5wi9OUDSQ9BXQBugILDKzreFzHczsTWDF3sQqpAZwu6QWBEnzDKApwR9sLYJ7x3Qws3vjGHN3ZgGdCe5V8xrBCZxNwOOJSJ4AZpYr6UqCfo8Dwx+S54BKQAvgN+FhbMKFNc63gaeBTwhq/98ANczspnjFKXTCaCBwH0G7Zw+CXgfPA6dLakDwgzbIk2eMzKxcT4S18PBxa2A+wR/TacBC4NjwubMJzhg328t4RxKcXYegu8q7BGd9JxOMX3h6+NxggjPTdRKwzycBHwBPhfMFyeNO4BzgTYIaqOIdu4iy1AV+R3DGfxrQJkmfe09gacRnUQGolqTYbYH24ePGwFVAo3D+EmB0HGNFfr8PAQYAjcP5PgTNUrPCuI8BLZLxHpSVKeUFSNmO/9x8USH8vx5QDfgP8BJB7fy3wBMEh7iz4/HHHf7hrgJuITh50ShMaH8gaPtbT9CdZwnQMoH734egtjMgYtkEoHuKPo/fEHSlSWbMUwl6HpydxJi/JzjxOBF4BqgS8dzw8Ee7dZxiRSbPkcDc8Ef5QqByuLxv+B50TsXnXtqnlBcgZTsODSMenwKMI+hCUxH4J0GH7n3CmkljoFYcYxeuAVaMqAH2Dmt/ByXhPegFfBYm857AYqBJqj+bJH8PTiqo/SUh1rkEtewKwA0EvQ1eIugH2wj4ezx+pIuI2yesXTYF7gIeAI4FMsPnBybrPShrU8oLkPQdDhrM9yW4C+FN4bKWwP0R6+xL0BfuncgaQpzLUVQN8OVk1obCmH0Jui2NBRqk+vMpqxNBW+9NBF21RoY1UIU/WtMI2j8rJSBuFkH/2sfC+crAX4EHwx+PzFS/N6V5Ko/9QGVBA3k34CJJfyLo37mlYIXw+asILm8r8Z09ozGz8QRdVW6XdIukngRdlRYnIl6UcrwMHA/8yRJ0tr28C09SHgMcYmY5BEcbT1mQ0cYCVYEM28sTk0WxoOfGlcBpkgZZ0Pf3FmA7wUmkivGOWZ6Uq25MkWcjw/lDCNqFVhPUBhcSXBmTQdCx+wUzyy9iU/EsU1/gBYImhGs8iZUtEV3SMgna0Z8gqHm2JzgKag1cGibWRJajJ3A7cLuZPROWp4YloJ9teVJuujEV6spxGdCKoB2yN0HyqkHwBetE0J1nYaKTJwQ1QEnHA6vN7PNEx3PJE36ux0haYGaTJN1M0Nf2DYIf6hMIfjQTmjwBzOwVSTuBRyXtMLNxgCfPvVSuaqAAkn5L0JVjMEE3lscIagaPEPw6P5TC4rkyRMFALccTNAf9i+Cw+SzgKjNbLCkjGT/Shcp0EvCpxblzfnlVrhKopOoEfS5vAPoRnHneBGwj6Kr0N4IrUjZZORtU1yWOpEMJznRXIhjPYBzBGfkdVp7+AMugcnMID2Bmm8PLF5sDZ5jZcQqGDPuGoBN7OzP7PqWFdGWOmX0i6U6Cts8fgecsCWO8usQrVwkUwMx+krQVyJTUhqAD/UTgVU+eLoHywtrmrakuiIufcnUIXyAcQOJK4ESCa4H7m9lHqS2Vc660KZcJFHYNYHswsNPiMMqRc678KbcJ1Dnn9lZ5vBLJOefiwhOoc86VkCdQ55wrIU+gzjlXQp5AnXOuhDyBlmOS8iUtkfSBpHGSquzFto6VNCl8fLqka6Ksu384JsGexrg5vI95TMsLrfNfSWfvQawGkhJ+V05XunkCLd9+NLN2ZtYayCO4L84uCuzxd8TMJpjZHVFW2Z/gdinOlWqeQF2BN4EmYc3rQ0n/ILjhXT1JJ0t6R9K7YU21KoCkUyR9JOkt4MyCDUkaKml0+PggSS9Jei+cjiS4ZXTjsPY7Klzv/yQtkLRU0i0R27pO0seSZgDNitsJSReF23lP0guFatUnSnpT0ieSeoXrZ0gaFRH74r19I1354QnUEQ6ueyrwfrioGfA/M2sP/ABcD5xoZh0IBp2+SlJlgiHaegNHE1zVVZS/A2+Y2WEEY2EuA64hGFKtnZn9n6STCe7XGEpxZwAAAi1JREFU0wVoB3SU1F1SR4JRjNoTJOjOMezOi2bWOYz3IcGN2go0IBgZvifwSLgPw4HvzKxzuP2LJDWMIY5z5W8wEfcL+0paEj5+k+BuoHWBz81sbrj8cIJ7Rs0J7kxBRYJ7RTUHVpnZCgBJTwIjiohxPHA+QDj25XeSahRa5+RwKridSVWChFoNeMnMtoYxJsSwT60l3UrQTFCVYJjCAs+FwxSukPRZuA8nA20j2kf3C2P/f3t3rxJXFEVx/L9iFcRWizR+BMFuel8iFhaDFqIgmUL0AfRBBAs78QGChJAuYbQRxUbtrCy0MyF22+LsgfGCjh7SzfqVd86dfavFuYeZva/fUMuGnAN0uP2LiFb/hQzJv/2XgB8R0W6sawH/63/AojSz3m3U2KqosQ98iYhzSSuU6ZM9ze+KrL0REf1Bi6TJd9a1IeRXeBvkGJiX9BnKjJ9sEHwJTEmayXXtF+7/CXTy3pFsav1A2V32fAdW+85WP0kap0wKWJD0UdIY5bhgkDHgNpvFLDU+W5T0IZ95GrjK2p1cj6RZSaNvqGPmHai9LiLucid3kG0AAbazSfA68E3SPfCLMiCtaZMyh2eNMj65ExFdSb/zZ0JHeQ46B3RzB/wHWI6IU0mHwBlwQzlmGGQHOMn1FzwP6ivKPKIJ4GtEPErao5yNnqoUv6OMejYbyN2YzMwq+RXezKySA9TMrJID1MyskgPUzKySA9TMrJID1MyskgPUzKzSE0SbxqrnRWecAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.eval()\n",
    "y_label = []\n",
    "y_predict = []\n",
    "with torch.no_grad():\n",
    "    for i, data in enumerate(val_loader):\n",
    "        images, labels = data\n",
    "        N = images.size(0)\n",
    "        images = Variable(images).to(device)\n",
    "        outputs = model(images)\n",
    "        prediction = outputs.max(1, keepdim=True)[1]\n",
    "        y_label.extend(labels.cpu().numpy())\n",
    "        y_predict.extend(np.squeeze(prediction.cpu().numpy().T))\n",
    "\n",
    "# compute the confusion matrix\n",
    "confusion_mtx = confusion_matrix(y_label, y_predict)\n",
    "#横坐标是真实值，纵坐标是预测值\n",
    "\n",
    "# plot the confusion matrix\n",
    "plot_labels = ['akiec', 'bcc', 'bkl', 'df', 'nv', 'vasc','mel']\n",
    "plot_confusion_matrix(confusion_mtx, plot_labels)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算各个分类类别的报告"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "       akiec       0.80      0.53      0.64        30\n",
      "         bcc       0.82      0.89      0.85        35\n",
      "         bkl       0.51      0.83      0.63        88\n",
      "          df       0.75      0.75      0.75         8\n",
      "          nv       0.98      0.87      0.92       883\n",
      "        vasc       0.73      0.85      0.79        13\n",
      "         mel       0.30      0.63      0.41        46\n",
      "\n",
      "    accuracy                           0.85      1103\n",
      "   macro avg       0.70      0.76      0.71      1103\n",
      "weighted avg       0.90      0.85      0.86      1103\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Generate a classification report\n",
    "report = classification_report(y_label, y_predict, target_names=plot_labels)\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "分类不正确的占比："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Fraction classified incorrectly')"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAWXUlEQVR4nO3de7BlZX3m8e9Dg4IXLoY2MlxsIASKZFSwRR1SRhANCIqKMWDQkUQJI3iJNYwdKxXHaGbQAsqxxthBBPESKQVUEIQQQB2DJt1NuIiCti2RBpEGQQQRaPjNH3sd3RxO77PO4ey9zznr+6nadfa67LV/TTX9nPW+73rfVBWSpO7abNwFSJLGyyCQpI4zCCSp4wwCSeo4g0CSOm7zcRcwU9tvv30tW7Zs3GVI0oKyZs2aO6pq6VTHFlwQLFu2jNWrV4+7DElaUJL8x6aO2TQkSR1nEEhSxxkEktRxBoEkdZxBIEkdZxBIUscZBJLUcQaBJHWcQSBJHbfgnix+PJatuHDcJTzKTScdOu4SJMk7AknqOoNAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknquGmDIMnJSX5vNhdPcnCSG5OsTbJiwHnPS/JwktfO5nskSbPX5o7gBuC0JP+a5Lgk27S5cJIlwEeBQ4C9gaOS7L2J8z4IXNK+bEnSXJk2CKrq9KraH3gjsAy4Nsk/Jjlgmo/uB6ytqnVV9SBwNnD4FOe9DTgXuH1GlUuS5kSrPoLmt/a9mtcdwDXAu5KcPeBjOwI3922vb/b1X3dH4NXAymm+/9gkq5Os3rBhQ5uSJUkttekjOBW4EXg58L+q6rlV9cGqegWwz6CPTrGvJm1/GHh3VT08qIaqOq2qllfV8qVLl05XsiRpBjZvcc53gL+uql9OcWy/AZ9bD+zct70TcOukc5YDZycB2B54eZKNVfWlFnVJkubAJoMgyb7N26uBvZp/rH+tqq6qqp8PuPYqYI8kuwK3AEcCr590jV37vu+TwFcMAUkarUF3BKcMOFbAgYMuXFUbk5xAbzTQEuCMqro+yXHN8YH9ApKk0dhkEFTVAQBJtqyqX/UfS7Jlm4tX1UXARZP2TRkAVfWmNteUJM2tNqOGrmy5T5K0AA3qI3gGveGeWyXZh9+MAtoaeNIIapMkjcCgPoI/At5Eb7TPKfwmCO4B3jPcsiRJozKoj+As4KwkR1TVuSOsSZI0Qm36CJ6bZNuJjSTbJfnAEGuSJI1QmyA4pKruntioqrvoPWUsSVoE2gTBkiRPnNhIshXwxAHnS5IWkDZTTHwGuCzJmfQeJPsz4KyhViVJGplpg6CqPpTkWuAgeiOH3l9Vrh0gadFbtuLCcZfwKDeddOhQrtvmjgDge8DGqvrnJE9K8tSq+sVQKpIkjVSbaajfApwD/EOza0fAieEkaZFo01l8PLA/vQfJqKofAE8fZlGSpNFpEwQPNEtNApBkcx67wIwkaYFqEwRfT/IeenMOvRT4AnDBcMuSJI1KmyBYAWwArgP+gt600n89zKIkSaMzcNRQs2j9WVV1NPDx0ZQkSRqlgXcEzaLyS5M8YUT1SJJGrM1zBDcB/5LkfOC+iZ1VdeqwipIkjU6bILi1eW0GPHW45UiSRq1NH8FTqurEEdUjSRqxNn0E+46oFknSGLRpGrq66R/4Ao/uIzhvaFVJkkamTRA8DbgTOLBvXwEGgSQtAm2moT5mFIVIksajzeyjOyX5YpLbk/w0yblJdhpFcZKk4WszxcSZwPnAf6I3BfUFzT5J0iLQJgiWVtWZVbWxeX0SWDrkuiRJI9ImCO5IcnSSJc3raHqdx5KkRaBNEPwZ8DrgNuAnwGubfZKkRaDNqKEfA68cQS2SpDFoM2rorCTb9m1vl+SM4ZYlSRqVNk1Dz6qquyc2quouYJ/hlSRJGqU2QbBZku0mNpI8jXZPJEuSFoA2/6CfAlyZ5Jxm+4+BvxteSZKkUWrTWfypJGuAA4AAr6mq7w69MknSSLRt4rkBuGvi/CS7NKOJJEkLXJtRQ28DfgpcCnwFuLD5Oa0kBye5McnaJCumOH54kmuTXJ1kdZI/mGH9kqTHqc0dwTuAPatqRk8TN6ubfRR4KbAeWJXk/EnNSpcB51dVJXkW8Hlgr5l8jyTp8Wkzauhm4OezuPZ+wNqqWldVDwJnA4f3n1BV91ZVNZtPprfOgSRphNrcEawDvpbkQuCBiZ1Vdeo0n9uRXohMWA88f/JJSV4N/G/g6cChU10oybHAsQC77LJLi5IlSW21uSP4Mb3+gScAT+17TSdT7HvMb/xV9cWq2gt4FfD+qS5UVadV1fKqWr50qROfStJcajN89H2zvPZ6YOe+7Z2AWwd8zzeS7J5k+6q6Y5bfKUmaoU0GQZIPV9U7k1zA1L/JTzcR3SpgjyS7ArcARwKvn/QdvwP8sOks3pfeXYdTXEvSCA26I/h08/Pk2Vy4qjYmOQG4BFgCnFFV1yc5rjm+EjgCeGOSh4D7gT/p6zyWJI3AJoOgqtY0P78+24tX1UXARZP2rex7/0Hgg7O9viTp8WvTWSxJWsQMAknqOINAkjpu0KihKUcLTWgxakiStAAMGjU0MVroNcAzgM8020cBNw2xJknSCA0aNfR1gCTvr6oX9R26IMk3hl6ZJGkk2vQRLE2y28RG84CY8zxI0iLRZtK5v6Q36dy6ZnsZ8BdDq0iSNFJt5hq6OMke/GadgBuq6oFBn5EkLRxtVih7EnAicEJVXQPskuSwoVcmSRqJNn0EZwIPAi9sttcDHxhaRZKkkWoTBLtX1YeAhwCq6n6mXmtAkrQAtQmCB5NsRfNwWZLd6VupTJK0sLUZNfRe4GJg5ySfBfYH3jTMoiRJo9Nm1NClSa4CXkCvSegdriAmSYvHJpuGkuzV/NwXeCbwE3pLTe7S7JMkLQKD7gjeBRwLnDLFsQIOHEpF0hgsW3HhuEt4lJtOOnTcJahDBgXBpc3PP6+qdQPOkyQtYINGDf1V8/OcURQiSRqPQXcEdya5Atg1yfmTD7oegSQtDoOC4FBgX+DTTN1PIElaBAatR/Ag8O0k/6WqNoywJknSCA1aqvLDVfVO4Iwkj1my0qYhSVocBjUNfbr5efKAcyRJC9ygpqE1zc+vT+xLsh2wc1VdO4LaJEkj0GY9gq8l2TrJ04BrgDOTnDr80iRJo9Bm9tFtquoe4DXAmVX1XOCg4ZYlSRqVNkGweZIdgNcBXxlyPZKkEWsTBH8LXAKsrapVSXYDfjDcsiRJo9JmGuovAF/o214HHDHMoiRJo9Oms/hDTWfxFkkuS3JHkqNHUZwkafjaNA29rOksPozewvW/C5w41KokSSPTJgi2aH6+HPhcVf1siPVIkkaszZrFFyS5AbgfeGuSpcCvhluWJGlUpr0jqKoVwAuB5VX1EHAfcPiwC5MkjUabpiGAHYEjkrwReC3wsjYfSnJwkhuTrE2yYorjf5rk2uZ1ZZJnty9dkjQXpm0aSvJe4MXA3sBFwCHAN4FPTfO5JcBHgZfS62ReleT8qvpu32k/Av6wqu5KcghwGvD8Wfw5JEmz1OaO4LXAS4DbquoY4NnAE1t8bj96D6Gta9Y2OJtJTUpVdWVV3dVsfhvYqXXlkqQ50SYI7q+qR4CNSbYGbgd2a/G5HYGb+7bXN/s25c+Br051IMmxSVYnWb1hg2vkSNJcajNqaHWSbYGPA2uAe4F/a/G5TLHvMQvcACQ5gF4Q/MFUx6vqNHrNRixfvnzKa0iSZqfNFBNvbd6uTHIxsHXL9QjWAzv3be8E3Dr5pCTPAk4HDqmqO1tcV5I0hwYtVbnvoGNVddU0114F7JFkV+AW4Ejg9ZOuswtwHvCGqvp+66olSXNm0B3BKQOOFXDgoAtX1cYkJ9CbuXQJcEZVXZ/kuOb4SuBvgN8C/j4JwMaqWj6D+iVJj9OgpSoPeLwXr6qL6A057d+3su/9m4E3P97vkSTNXpvZR49vOosntrdL8tZBn5EkLRxtho++parunthoxv2/ZXglSZJGqU0QbJamAR9+/cTwE4ZXkiRplNo8R3AJ8PkkK+l1Eh8HXDzUqiRJI9MmCN4NHAv8N3oPif0TvXH/kqRFoM0DZY8AK5uXJGmRaTsNtSRpkWrTNCTNyLIVF467hEe56aRDx12CNK95RyBJHddmYZrfBU4Entl/flUNnGJCkrQwtGka+gK9juKPAw8PtxxJ0qi1CYKNVfWxoVciSRqLNn0EFyR5a5Idkjxt4jX0yiRJI9HmjuC/Nj9P7NtXtFuuUpI0z7V5oGzXURQiSRqPNqOGtqA3vcSLml1fA/6hqh4aYl2SFhmfL5m/2jQNfQzYAvj7ZvsNzT4XlJGkRaBNEDyvqp7dt315kmuGVZAkabTajBp6OMnuExtJdsPnCSRp0WhzR3AicEWSdfSmoX4mcMxQq5IkjUybUUOXJdkD2JNeENxQVQ8MvTJJ0khsMgiSHFhVlyd5zaRDuyehqs4bcm2SpBEYdEfwh8DlwCumOFaAQSBJi8Amg6Cq3tu8/duq+lH/sSQ+ZCZJi0SbUUPnTrHvnLkuRJI0HoP6CPYCfg/YZlI/wdbAlsMuTNKm+ZSu5tKgPoI9gcOAbXl0P8EvgLcMsyhJ0ugM6iP4MvDlJC+sqm+NsCZJ0gi16SM4Lsm2ExtJtktyxhBrkiSNUJsgeFZV3T2xUVV3AfsMryRJ0ii1CYLNkmw3sdGsTtZmagpJ0gLQ5h/0U4Ark0wMGf1j4O+GV5IkaZTazDX0qSRrgAPozTX0mqr67tArE+AwQUnD16qJp6quT7KB5vmBJLtU1Y+HWpkkaSSm7SNI8sokPwB+BHwduAn46pDrkiSNSJvO4vcDLwC+3yxk/xLgX9pcPMnBSW5MsjbJiimO75XkW0keSPLfZ1S5JGlOtAmCh6rqTnqjhzarqiuA50z3oSRLgI8ChwB7A0cl2XvSaT8D3g6cPLOyJUlzpU0fwd1JngJ8A/hsktuBjS0+tx+wtqrWASQ5Gzgc+HVHc1XdDtyexB5ISRqTNncEhwO/BP4SuBj4IVOvUTDZjsDNfdvrm30zluTYJKuTrN6wYcNsLiFJ2oSBQdA073y5qh6pqo1VdVZVfaRpKppOpthXsymyqk6rquVVtXzp0qWzuYQkaRMGBkFVPQz8Msk2s7j2emDnvu2dgFtncR1J0hC16SP4FXBdkkuB+yZ2VtXbp/ncKmCPZjWzW4AjgdfPtlBJ0nC0CYILm9eMVNXGJCcAlwBLgDOaB9OOa46vTPIMYDW9xW4eSfJOYO+qumem3ydJmp1BK5TtUlU/rqqzZnvxqroIuGjSvpV972+j12QkSRqTQX0EX5p4k2SqdYslSYvAoCDoH/Wz27ALkSSNx6AgqE28lyQtIoM6i5+d5B56dwZbNe9ptquqth56dZKkoRu0eP2SURYiSRqPNlNMSJIWMYNAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4bahAkOTjJjUnWJlkxxfEk+Uhz/Nok+w6zHknSYw0tCJIsAT4KHALsDRyVZO9Jpx0C7NG8jgU+Nqx6JElTG+YdwX7A2qpaV1UPAmcDh08653DgU9XzbWDbJDsMsSZJ0iSpquFcOHktcHBVvbnZfgPw/Ko6oe+crwAnVdU3m+3LgHdX1epJ1zqW3h0DwJ7AjUMpur3tgTvGXMNMWfNoLLSaF1q9YM2z9cyqWjrVgc2H+KWZYt/k1GlzDlV1GnDaXBQ1F5Ksrqrl465jJqx5NBZazQutXrDmYRhm09B6YOe+7Z2AW2dxjiRpiIYZBKuAPZLsmuQJwJHA+ZPOOR94YzN66AXAz6vqJ0OsSZI0ydCahqpqY5ITgEuAJcAZVXV9kuOa4yuBi4CXA2uBXwLHDKueOTZvmqlmwJpHY6HVvNDqBWuec0PrLJYkLQw+WSxJHWcQSFLHGQQzNN20GfNNkjOS3J7kO+OupY0kOye5Isn3klyf5B3jrmk6SbZM8m9Jrmlqft+4a2oryZIk/9480zPvJbkpyXVJrk6yevpPjFeSbZOck+SG5u/0C8dd01TsI5iBZtqM7wMvpTf0dRVwVFV9d6yFDZDkRcC99J7g/v1x1zOd5snyHarqqiRPBdYAr5rn/40DPLmq7k2yBfBN4B3N0/LzWpJ3AcuBravqsHHXM50kNwHLq2rcD2e1kuQs4P9V1enN6MknVdXd465rMu8IZqbNtBnzSlV9A/jZuOtoq6p+UlVXNe9/AXwP2HG8VQ3WTJFyb7O5RfOa979hJdkJOBQ4fdy1LEZJtgZeBHwCoKoenI8hAAbBTO0I3Ny3vZ55/o/UQpZkGbAP8K/jrWR6TRPL1cDtwKVVNe9rBj4M/A/gkXEXMgMF/FOSNc3UM/PZbsAG4Mym+e30JE8ed1FTMQhmptWUGHr8kjwFOBd4Z1XdM+56plNVD1fVc+g9Hb9fknndDJfkMOD2qloz7lpmaP+q2pfezMXHN02f89XmwL7Ax6pqH+A+YF72KxoEM+OUGCPQtLOfC3y2qs4bdz0z0dz6fw04eMylTGd/4JVNm/vZwIFJPjPekqZXVbc2P28HvkivuXa+Wg+s77s7PIdeMMw7BsHMtJk2Q49D0/H6CeB7VXXquOtpI8nSJNs277cCDgJuGG9Vg1XVX1XVTlW1jN7f48ur6ugxlzVQkic3AwhomlheBszb0XBVdRtwc5I9m10vAebloIdhzj666Gxq2owxlzVQks8BLwa2T7IeeG9VfWK8VQ20P/AG4LqmzR3gPVV10Rhrms4OwFnNqLLNgM9X1YIYjrnA/Dbwxd7vCmwO/GNVXTzekqb1NuCzzS+O65in0+g4fFSSOs6mIUnqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQItekt9qZqu8OsltSW7p237CHH7PQUm+NIPzv5nkOcO6vtSWzxFo0auqO4HnACT5n8C9VXVy/znNg2ypqoU07440J7wjUGcl+Z0k30myErgK2DnJ3X3Hj0xyevP+t5Ocl2R1s/bAC2bwPe9Lsmriu5rQmfCmJN9q5thf3pz/lCSfbL7n35O8Yo7+yNKUDAJ13d7AJ5pJwW4ZcN5HgA9V1XLgdcxs6ub/U1XPA/4zsA2PnofoiVX1QuAdfdf8G+DiqtoPOBA4JcmWM/g+aUZsGlLX/bCqVrU47yBgz75f5rdLslVV3d/isy9JciKwJbA9vcV2vtoc+xxAVV2e5OnNrKsvAw7pWwFvS2CXdn8caeYMAnXdfX3vH+HRU433/xYeYL9mQaLWkjwJ+L/AvlV1S5IPTLru5DleqvmuV1XVDyddyzDQUNg0JDWajuK7kuyRZDPg1X2H/xk4fmJjBqN9tqIXMHc0M2ceMen4nzTXezHw06q6j96khm/v+659ZvhHkWbEOwLp0d4NXAz8mN6UwU9s9h8PfCzJMfT+v7mCvmDo80fNLK8TXg2cRW+65P/gsaut3ZPkSuCp/GZmyvcBH05yHb1f1tYyz5dE1cLm7KOS1HE2DUlSxxkEktRxBoEkdZxBIEkdZxBIUscZBJLUcQaBJHXc/weEkDw8lZKhMgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "label_frac_error = 1 - np.diag(confusion_mtx) / np.sum(confusion_mtx, axis=1)\n",
    "plt.bar(np.arange(7),label_frac_error)\n",
    "plt.xlabel('True Label')\n",
    "plt.ylabel('Fraction classified incorrectly')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 结论"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用Densenet-121时，验证集上7个类的平均准确率在4个epoch内可以达到85%。我们还计算了所有类别的混淆矩阵和每个类别的F1-score，这是一个更综合的指标，可以同时考虑分类模型的精度和召回率。我们的模型在F1-score指标上可以达到86%以上。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于时间有限，我们没有花太多时间在模型训练上。通过增加训练时间，调整模型超参数，尝试不同的网络，可以进一步提高模型的性能。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}