diff --git a/README.md b/README.md index 9c05f88..fc89ece 100644 --- a/README.md +++ b/README.md @@ -1 +1 @@ -# Sample-DL-Repo +# Tarasov Nikita diff --git a/Tarasov_hw3.ipynb b/Tarasov_hw3.ipynb new file mode 100644 index 0000000..d55c895 --- /dev/null +++ b/Tarasov_hw3.ipynb @@ -0,0 +1,1354 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.7" + }, + "colab": { + "name": "Tarasov_hw3.ipynb", + "provenance": [], + "collapsed_sections": [] + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "ucUIoKyJdAxb" + }, + "source": [ + "# Пишем свой фреймворк" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "o2DkIzfVdAxd" + }, + "source": [ + "# только numpy, только хардкор\n", + "import numpy as np" + ], + "execution_count": 1, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zL1UwDwNdAxi" + }, + "source": [ + "Мотивация: конечным пользователям вашего фреймворка не хочется думать, как они работают слои внутри. Им просто хочется объявить последовательность элементарных операций над входными данными, а о градиентах и прочем матане пусть позаботится сам фреймворк.\n", + "\n", + "**Module** — это абстрактный класс, от которого будут наследоваться слои нашей нейронной сети. Абстрактные классы нужны, чтобы можно было реализовывать не все методы, а только переопределить некоторые. Все в лучших традициях ООП.\n", + "\n", + "Модуль — это такая чёрная коробка, которая\n", + "1. Умеет принимать какие-то входные данные $X$ и возращать какие-то выходные данные $Y$ (`forward`)\n", + "2. Возможно, имеет какие-то параметры, которые можно изменять, (`parameters`, `grad_parameters`)\n", + "3. Будучи встроенной в вычислительный граф, умеет по градиенту относительно своих выходных значений вычислять градиент относительно входных данных, а также собственных параметров (`backward`)\n", + "4. Умеет переключаться в режимы обучения и инференса, если они отличаются (`train`, `eval`)\n", + "\n", + "Теперь поподробнее." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BPQOD218dAxj" + }, + "source": [ + "## Входные данные\n", + "\n", + "Современные нейросети оптимизируют различными вариантами стохастического градиентного спуска, и мы тоже будем его использовать. Его отличие от обычного в том, что на каждом шаге мы не считаем градиент на всем датасете (это было бы слишком долго), а оцениваем его, усреднив градиенты на его случайно выбранной малой части, которую называют батчем (`batch`). Если батч формировать случайно, и если его размер достаточно большой, то мы можем быстро получить немного шумную, но приемлемую для нас оценку градиента, и не прогонять через сеть все миллионы примеров ради одного маленького шага. Эта интуиция ограничивает размер батча сверху." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4--xRXAkdAxk" + }, + "source": [ + "Математик бы принял время прогона одного примера по всей сети за константу и пришел бы к выводу, что нужно считать по одному примеру и делать каждый раз один шаг, но маленький. Это верное заключение, но в реальности, если увеличить размер батча в $k$ раз, то он будет работать не в $k$ раз дольше, а намного меньше.\n", + "\n", + "Самая долгая операция в большинстве нейросетей — это перемножение матриц. Начиная с каких-то размеров для их перемножения имеет использовать алгоритм Штрассена, который работает уже быстрее, чем линейно. Проведём небольшой вычислительный эксперимент:" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "tfA1Fp7kdAxl", + "outputId": "e5aaa5df-6f34-48de-9e55-d672b1198b2b", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "A = np.random.randn(256, 2000)\n", + "B = np.random.randn(2000, 800)\n", + "\n", + "# помножить каждый вектор-строку на B и сконкатенировать\n", + "%time C = np.stack(np.dot(A[i].T, B) for i in range(256))\n", + "\n", + "# это то же самое, что использовать одно-большое матричное умножение\n", + "%time C = np.dot(A, B)" + ], + "execution_count": 2, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.7/dist-packages/IPython/core/magic.py:188: FutureWarning: arrays to stack must be passed as a \"sequence\" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.\n", + " call = lambda f, *a, **k: f(*a, **k)\n" + ] + }, + { + "output_type": "stream", + "name": "stdout", + "text": [ + "CPU times: user 319 ms, sys: 33.1 ms, total: 352 ms\n", + "Wall time: 193 ms\n", + "CPU times: user 59.3 ms, sys: 12.9 ms, total: 72.2 ms\n", + "Wall time: 40.2 ms\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Xoj5u-YHdAxp" + }, + "source": [ + "Такая чисто вычислительная причина ограничивает размер батча снизу. На практике, в большинстве случаев оптимальный размер батча — несколько сотен. В случае с CPU это несколько десятков, потому что выгода от распараллеливания вычислений не такая сильная.\n", + "\n", + "Вообще, почти все наши слои будут работать с векторами независимо, но из-за вычилсительных причин мы будем объединять их в матрицы. Вообще, более сложные нейросети работают с тензорами. «Тензор» это, вообще говоря, сложный математический объект, но в DL этот термин используется просто в занчении «многомерный массив». Например, картинки — это четырехмерный тензор: `[batch, channel, x, y]`." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yeECJUJvdAxq" + }, + "source": [ + "### Forward\n", + "\n", + "Эта функция просто принимает тензор (`numpy.ndarray`) и возвращает какой-то другой, над которым применили соответствующие операции.\n", + "\n", + "Важный нюанс: нам позже для реализации `backward` почти всегда будет нужно сохранять где-нибудь выход `forward` (это создает очень большую нагрузку на память при обучении; [в принципе это можно и не делать](https://arxiv.org/pdf/1604.06174.pdf), но так проще). Условимся сохранять его в `self.output`, сразу после того, как посчитали." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "s0i3gvWMdAxq" + }, + "source": [ + "### Параметры\n", + "\n", + "Параметр модели — это что-то, что можно поодгонять, чтобы функция потерь стала меньше. Он должен быть доступен оптимизатору, а оптимизатору не обязательно знать, как всё у слоя все внутри работает. Ему нужны просто градиенты — насколько ему нужно подвинуть параметры сети, чтобы стало лучше.\n", + "\n", + "Общаться с ним мы будем посредством двух функций: `params` и `grad_params`. Обе возвращают списки из тензоров — значения параметров и их посчитанных градиентов (см. `backward`) соответственно. Питон делает shallow copy, поэтому у оптимизатора так есть доступ на их изменение." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jcZHrwjZdAxr" + }, + "source": [ + "### Backward\n", + "\n", + "После того, как мы в вычислительном графе все последовательно посчитали и дошли до функции потерь, нам надо подогнать параметры так, чтобы на тех же данных при повторном прогоне она стала меньше — иными словами, нам надо сделать шаг против градиента функции потерь относительно параметров сети.\n", + "\n", + "Посчитать эти градиенты — нетривиальная задача. Мы могли бы рассмотреть каждый параметр по отдельности и как-нибудь посчитать градиент для него. Но это очень долго — параметров в современных сетях бывает по несколько миллионов.\n", + "\n", + "Вместо этого мы применим трюк, основанный на формуле для производной сложной функции:\n", + "\n", + "$$ f(g(x))' = f'(g(x)) \\cdot g'(x) $$\n", + "\n", + "Представьте, что часть сети от параметра до выхода — это всего две последовательно выполненные функции: $g$ и $f$. Тогда, согласно формуле, нам для этого параметра достаточно посчитать и перемножить две величины — $g'(x)$ (производная текущего слоя) и f'(g(x)) (производная относительно выхода текущего слоя).\n", + "\n", + "Какие-то другие параметры могли тоже зависеть от производной относительно выхода. и мы получаем выигрыш за счет того, что считаем её только один раз и запоминаем. Можно сказать, что мы применяем таким образом динамическое программирование на вычислительном графе, чтобы посчитать градиенты относительно всех его параметров.\n", + "\n", + "Обратный прогон (`backward`) определяется для каждого слоя и нужен как раз для подсчета градиентов, имея градиент относительно своих выходных значений (аналог $f'(g(x)))$.\n", + "\n", + "Он должен делать две вещи:\n", + "\n", + "1. Посчитать градиент относительно собственных параметров.\n", + "2. Посчитать и вернуть градиент относительно своих входных данных.\n", + "\n", + "Для лучшего понимания рассмотрите пример с `Linear` и `ReLU`." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UQAOtCLddAxs" + }, + "source": [ + "### train / eval\n", + "\n", + "Некоторые слои ведут себя по-разному во время обучечния и предсказания (`inference`). Обычно, это связано с разного вида регуляризацией — например, так ведут себя `BatchNorm` и `Dropout`.\n", + "\n", + "По сути, для таких слоев нужно просто написать два разных `forward`-а для обучения и инференса." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "oBVpDMN6dAxs" + }, + "source": [ + "class Module():\n", + " def __init__(self):\n", + " self._train = True\n", + " \n", + " def forward(self, input):\n", + " raise NotImplementedError\n", + "\n", + " def backward(self,input, grad_output):\n", + " raise NotImplementedError\n", + " \n", + " def parameters(self):\n", + " 'Возвращает список собственных параметров.'\n", + " return []\n", + " \n", + " def grad_parameters(self):\n", + " 'Возвращает список тензоров-градиентов для своих параметров.'\n", + " return []\n", + " \n", + " def train(self):\n", + " self._train = True\n", + " \n", + " def eval(self):\n", + " self._train = False" + ], + "execution_count": 3, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "s-137oXtdAxu" + }, + "source": [ + "Это **абстрактный класс** — от него наследуются другие слои, в которых эти функции будут реализованы." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TV9xeNAndAxv" + }, + "source": [ + "# Sequential" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "J-mgMM1MdAxw" + }, + "source": [ + "**Sequential** будет оборачивать список модулей и выполнять их последовательно.\n", + "\n", + "Это своего рода контейнер, внутри которого есть какой-то пайплайн.\n", + "\n", + "Можно даже засовывать один Sequential внутри другого." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qdWwOprDdAxw" + }, + "source": [ + "Многие не знают, но в питоне почти всегда для итерирования используется не **deep copy**, а **shallow copy**. Это делается для экономии памяти." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "0O968yWrdAxx" + }, + "source": [ + "class Sequential(Module):\n", + " def __init__ (self, *layers):\n", + " super().__init__()\n", + " self.layers = layers\n", + "\n", + " def forward(self, input):\n", + " \"\"\"\n", + " Прогоните данные последовательно по всем слоям:\n", + " \n", + " y[0] = layers[0].forward(input)\n", + " y[1] = layers[1].forward(y_0)\n", + " ...\n", + " output = module[n-1].forward(y[n-2]) \n", + " \n", + " Это должен быть просто небольшой цикл: for layer in layers...\n", + " \n", + " Хранить выводы ещё раз не надо: они сохраняются внутри слоев после forward.\n", + " \"\"\"\n", + "\n", + " for layer in self.layers:\n", + " input = layer.forward(input)\n", + "\n", + " self.output = input\n", + " return self.output\n", + "\n", + " def backward(self, input, grad_output):\n", + " \"\"\"\n", + " Backward -- это как forward, только наоборот. (с)\n", + " \n", + " Предназначение backward:\n", + " 1. посчитать посчитать градиенты для собственных параметров\n", + " 2. передать градиент относительно своего входа\n", + " \n", + " О своих параметрах модули сами позаботятся. Нам же нужно позаботиться о передачи градиента.\n", + " \n", + " g[n-1] = layers[n-1].backward(y[n-2], grad_output)\n", + " g[n-2] = layers[n-2].backward(y[n-3], g[n-1])\n", + " ...\n", + " g[1] = layers[1].backward(y[0], g[2]) \n", + " grad_input = layers[0].backward(input, g[1])\n", + " \n", + " Тут цикл будет уже чуть посложнее.\n", + " \"\"\"\n", + " \n", + " for i in range(len(self.layers)-1, 0, -1):\n", + " grad_output = self.layers[i].backward(self.layers[i-1].output, grad_output)\n", + " \n", + " grad_input = self.layers[0].backward(input, grad_output)\n", + " \n", + " return grad_input\n", + " \n", + " def parameters(self):\n", + " 'Можно просто сконкатенировать все параметры в один список.'\n", + " res = []\n", + " for l in self.layers:\n", + " res += l.parameters()\n", + " return res\n", + " \n", + " def grad_parameters(self):\n", + " 'Можно просто сконкатенировать все градиенты в один список.'\n", + " res = []\n", + " for l in self.layers:\n", + " res += l.grad_parameters()\n", + " return res\n", + " \n", + " def train(self):\n", + " for layer in self.layers:\n", + " layer.train()\n", + " \n", + " def eval(self):\n", + " for layer in self.layers:\n", + " layer.eval()" + ], + "execution_count": 4, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "50R6jFr8dAxy" + }, + "source": [ + "# Слои" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "guFtN_3FdAxz" + }, + "source": [ + "Приступим к реализации содержательной части — самих слоев.\n", + "\n", + "На вход всех слоев будет подаваться матрица размера `batch_size` $\\times$ `n_features` (см. описание `forward`)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jdBP7xZzdAxz" + }, + "source": [ + "Начнем с основного: линейный слой, он же fully-conected.\n", + "\n", + "$$ Y = X W + b $$\n", + "\n", + "Правильнее его называть афинным: после матричного умножения добавляется вектор $b$.\n", + "\n", + "`forward` у него трививальный, а `backward` уже сложнее: нужно посчитать градиенты относительно трёх вещей:\n", + "1. Входных данных. Автор добродушен и спалит вам ответ, а вам нужно его доказать: $\\nabla X = W^T (\\nabla Y)$.\n", + "2. Матрица весов $W$. Тут нужно подумать, как каждый вес влияет на каждое выходное значение, и выразить ваши мысли линейной алгеброй.\n", + "3. Вектор $b$. С ним всё будет просто.\n", + "\n", + "Не забудьте, что `grad_params` должен иметь такие же размерности, как и соответствующие параметры." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "4uEjKIasdAx3" + }, + "source": [ + "class Linear(Module):\n", + " def __init__(self, dim_in, dim_out):\n", + " super().__init__()\n", + " stdv = 1./np.sqrt(dim_in)\n", + " self.W = np.random.uniform(-stdv, stdv, size=(dim_in, dim_out))\n", + " self.b = np.random.uniform(-stdv, stdv, size=dim_out)\n", + " \n", + " def forward(self, input):\n", + " self.output = np.dot(input, self.W) + self.b\n", + " return self.output\n", + " \n", + " def backward(self, input, grad_output):\n", + " self.grad_b = np.mean(grad_output, axis=0)\n", + " self.grad_W = np.dot(input.T, grad_output)\n", + " grad_input = np.dot(grad_output, self.W.T)\n", + " return grad_input\n", + " \n", + " def parameters(self):\n", + " return [self.W, self.b]\n", + " \n", + " def grad_parameters(self):\n", + " return [self.grad_W, self.grad_b]" + ], + "execution_count": 5, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-jLyEmMEdAx7" + }, + "source": [ + "## Функции активации" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "H0ci6studAx7" + }, + "source": [ + "**ReLU** — одна из самых простых функций активации:\n", + "\n", + "$$\n", + "ReLU(x)=\n", + "\\begin{cases}\n", + "x, & x > 0\\\\\n", + "0, & x \\leq 0\\\\\n", + "\\end{cases}\n", + "$$\n", + "\n", + "`ReLU` это очень простой слой, поэтому автору не жалко её реализовать его за вас:" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "HrayJEANdAx7" + }, + "source": [ + "class ReLU(Module):\n", + " def __init__(self):\n", + " super().__init__()\n", + " \n", + " def forward(self, input):\n", + " self.output = np.maximum(input, 0)\n", + " return self.output\n", + " \n", + " def backward(self, input, grad_output):\n", + " grad_input = np.multiply(grad_output, input > 0)\n", + " return grad_input" + ], + "execution_count": 6, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xUTaMFaWdAx8" + }, + "source": [ + "У ReLU есть проблема — у него бесполезная нулевая производная при $x < 0$.\n", + "\n", + "[**Leaky Rectified Linear Unit**](http://en.wikipedia.org/wiki%2FRectifier_%28neural_networks%29%23Leaky_ReLUs) — это его модифицированная версия, имеющая в отрицательных координатах не нулевой градиент, а просто помноженный на маленькую константу `slope`.\n", + "\n", + "$$\n", + "LeakyReLU_k(x)=\n", + "\\begin{cases}\n", + "x, & x > 0\\\\\n", + "kx, & x \\leq 0\\\\\n", + "\\end{cases}\n", + "$$\n", + "\n", + "При `slope` = 0 он превращается в обычный `ReLU`. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "OMWPFjxldAx8" + }, + "source": [ + "class LeakyReLU(Module):\n", + " def __init__(self, slope=0.03):\n", + " super().__init__()\n", + " \n", + " self.slope = slope\n", + " \n", + " def forward(self, input):\n", + " self.output = np.maximum(input, input*self.slope)\n", + " return self.output\n", + " \n", + " def backward(self, input, grad_output):\n", + " grad_input = (input > 0) + self.slope * (input <= 0)\n", + " return grad_input" + ], + "execution_count": 7, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EorDi-TMdAx9" + }, + "source": [ + "**Сигмоида** определяется формулой $\\sigma(x) = \\frac{1}{1+e^{-x}}$.\n", + "\n", + "\n", + "\n", + "Когда-то она была самой часто используемой функции активации, потому что имела логичную вероятностную интерпретацию (вероятность наличия какой-то фичи), но потом перестали, потому что на очень больших или маленьких значениях её производные почти нулевые (см. проблема затухающего градиента).\n", + "\n", + "Также используют [гипреболический тангенс](https://ru.wikipedia.org/wiki/%D0%93%D0%B8%D0%BF%D0%B5%D1%80%D0%B1%D0%BE%D0%BB%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%B8%D0%B5_%D1%84%D1%83%D0%BD%D0%BA%D1%86%D0%B8%D0%B8), который на самом деле просто сигмоида, отнормированная так, чтобы значения были в $[-1, 1]$: $tanh(x) = 2 \\sigma(x) - 1$. Мы его отдельно реализовывать не будем.\n", + "\n", + "Давайте посчитаем её производную:\n", + "\n", + "$$\n", + "\\begin{align}\n", + "\\sigma'(x) &= (\\frac{1}{1+e^{-x}})'\n", + "\\\\ &= \\frac{e^{-x}}{(1+e^{-x})^2}\n", + "\\\\ &= \\frac{1+e^{-x}-1}{(1+e^{-x})^2}\n", + "\\\\ &= \\frac{1+e^{-x}}{(1+e^{-x})^2} - \\frac{1}{(1+e^{-x})^2}\n", + "\\\\ &= \\frac{1}{1+e^{-x}} - \\frac{1}{(1+e^{-x})^2}\n", + "\\\\ &= \\sigma(x) - \\sigma(x)^2\n", + "\\\\ &= \\sigma(x)(1 - \\sigma(x))\n", + "\\end{align}\n", + "$$" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "LUag-tekdAx-" + }, + "source": [ + "class Sigmoid(Module):\n", + " def __init__(self, slope=0.03):\n", + " super().__init__()\n", + "\n", + " def forward(self, input):\n", + " self.output = 1 / (1 + np.exp(-input))\n", + " return self.output\n", + " \n", + " def backward(self, input, grad_output):\n", + " grad_input = self.output*(1 - self.output)*grad_output\n", + " return grad_input" + ], + "execution_count": 8, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kBhjcx0XdAx_" + }, + "source": [ + "**Софтмакс** определяется так:\n", + "\n", + "$$ \\sigma(x)_k = \\frac{e^{x_k}}{\\sum_{i=1}^n e^{x_i} }$$\n", + "\n", + "Можно заметить, что сигмоида — это частный случай софтмакса. Его можно интерпретировать как вероятностное распределение: его выходы положительны и суммируются в единицу. Поэтому его используют как последний слой для классификации.\n", + "\n", + "Софтмакс — самый сложный с точки зрения написания `backward`. Как и все остальное, оно считается в 5 строчек кода, но [вывести их трудно](https://deepnotes.io/softmax-crossentropy). " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "QS-6BIe6dAx_" + }, + "source": [ + "class SoftMax(Module):\n", + " def __init__(self):\n", + " super().__init__()\n", + "\n", + " def forward(self, input):\n", + " self.output = np.subtract(input, input.max(axis=1, keepdims=True)) \n", + " esum = np.sum(np.exp(self.output), axis=1, keepdims=True)\n", + " self.probs = np.array(np.exp(self.output)) / esum\n", + " self.output = self.probs\n", + " return self.output\n", + " \n", + " def backward(self, input, grad_output):\n", + " grad_input = []\n", + " for b in range(self.probs.shape[0]):\n", + " eye = np.eye(self.probs.shape[1])\n", + " prob_matr = np.repeat(self.probs[b].reshape(1,-1), self.probs[b].shape[0], axis=0)\n", + " res = np.dot(grad_output[b].reshape(1,-1), prob_matr.T*(eye-prob_matr))\n", + " grad_input.append(res)\n", + " grad_input = np.array(grad_input).reshape(-1, self.probs[b].shape[0])\n", + " return grad_input" + ], + "execution_count": 9, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "h-5W8O3zdAyA" + }, + "source": [ + "## Регуляризация" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "V5bmlefZdAyA" + }, + "source": [ + "Самый популярный регуляризатор в нейросетях — [**дропаут**](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf). Идея простая: просто помножим поэлементно входные данные на случайную бинарную маску того же размера, как и сами данные. Сгенерировать маску можно через `np.random.binomial`.\n", + "\n", + "Дропаута обычно хватает как единственного регуляризатора. Если вы заметите, что сеть оверфитится — просто добавьте его побольше.\n", + "\n", + "**У дропаута разное поведение в режимах `train` и `eval`**. При `eval` он не должен делать ничего, а в `train` вместо применения маски нужно ещё домножить вход на $p$, чтобы скомпенсировать дропаут при обучении (так математическое ожидание значений будет такое же, как на трейне)." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "9nZsHZUDdAyA", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 70 + }, + "outputId": "482e37ac-9eb0-4093-ff9d-3b3bd6dff930" + }, + "source": [ + "'''\n", + "class Dropout(Module):\n", + " def __init__(self, p=0.5):\n", + " super().__init__()\n", + " \n", + " self.p = p\n", + " self.mask = None\n", + " \n", + " def forward(self, input):\n", + " if self._train:\n", + " mask = # ...\n", + " self.output = # ...\n", + " else:\n", + " # ...\n", + " return self.output\n", + " \n", + " def backward(self, input, grad_output):\n", + " if self._train:\n", + " # ...\n", + " else:\n", + " # ...\n", + " return grad_input\n", + "'''" + ], + "execution_count": 10, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'\\nclass Dropout(Module):\\n def __init__(self, p=0.5):\\n super().__init__()\\n \\n self.p = p\\n self.mask = None\\n \\n def forward(self, input):\\n if self._train:\\n mask = # ...\\n self.output = # ...\\n else:\\n # ...\\n return self.output\\n \\n def backward(self, input, grad_output):\\n if self._train:\\n # ...\\n else:\\n # ...\\n return grad_input\\n'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 10 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "r0YNMOOIdAyB" + }, + "source": [ + "`BatchNorm` -- относительно современный слой, сильно улучшающий сходимость. Всё, что он делает -- это нормирует свои входные значения так, что на выходе получаются значения со средним 0 и дисперсией 1.\n", + "\n", + "\n", + "\n", + "Почитать про вывод градиента для него можно тут: https://wiseodd.github.io/techblog/2016/07/04/batchnorm/\n", + "\n", + "BatchNorm тоже по-разному ведёт себя при обучении и инференсе. Во время инференса он использует в качестве оценки среднего и дисперсии свои экспоненциально усреднённые исторические значения. Это связано с тем, что батч может быть маленьким, и оценки среднего и дисперсии будут неточными (при батче размера 1 дисперсия вообще будет нулевая, и нам в алгоритме нужно будет делить на ноль)." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Bkl3--6RdAyB", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 70 + }, + "outputId": "1bbbbae8-d8c5-4d72-ff59-94c102695f64" + }, + "source": [ + "'''\n", + "class BatchNorm(Module):\n", + " def __init__(self, num_features, gamma):\n", + " super().__init__()\n", + " self.gamma = gamma\n", + " self.mu = # ...\n", + " self.sigma = # ...\n", + " \n", + " def forward(self, input):\n", + " if self._train:\n", + " # ...\n", + " else:\n", + " # ...\n", + " return self.output\n", + " \n", + " def backward(self, input, grad_output):\n", + " if self._train:\n", + " # ...\n", + " else:\n", + " # ...\n", + " return grad_input\n", + "'''" + ], + "execution_count": 11, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'\\nclass BatchNorm(Module):\\n def __init__(self, num_features, gamma):\\n super().__init__()\\n self.gamma = gamma\\n self.mu = # ...\\n self.sigma = # ...\\n \\n def forward(self, input):\\n if self._train:\\n # ...\\n else:\\n # ...\\n return self.output\\n \\n def backward(self, input, grad_output):\\n if self._train:\\n # ...\\n else:\\n # ...\\n return grad_input\\n'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 11 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IO_okIAhdAyC" + }, + "source": [ + "## Критерии" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5jZL4UjNdAyC" + }, + "source": [ + "Критерии — это специальные функции, которые меряют качество, имея реальные данные и предсказанные. Все критерии возвращают скаляр — одно число, усреднённое значение метрики по всему батчу.\n", + "\n", + "По сути это тоже модули, но мы всё равно создадим для них отдельный класс, потому что у них нет `train` / `eval`, а `backward` не требует `grad_output` — эта вершина и так конечная в вычислительном графе. Также нам не понадобится сохранять для них `output`." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "COX7VXtUdAyC" + }, + "source": [ + "class Criterion(): \n", + " def forward(self, input, target):\n", + " raise NotImplementedError\n", + "\n", + " def backward(self, input, target):\n", + " raise NotImplementedError" + ], + "execution_count": 12, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YF1c6q1adAyD" + }, + "source": [ + "В качестве примера реализуем среднюю квадратичную ошибку (`MSE`).\n", + "\n", + "Обратите внимание, что в критериях мы делим итоговое число на размер батча — мы не хотим, чтобы функция потерь зависела от количества примеров." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "RLZXaG3OdAyD" + }, + "source": [ + "class MSE(Criterion):\n", + " def forward(self, input, target):\n", + " batch_size = input.shape[0]\n", + " self.output = np.sum(np.power(input - target, 2)) / batch_size\n", + " return self.output\n", + " \n", + " def backward(self, input, target):\n", + " self.grad_output = (input - target) * 2 / input.shape[0]\n", + " return self.grad_output" + ], + "execution_count": 13, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tBCAlI7CdAyE" + }, + "source": [ + "Ваша задача посложнее: вам нужно реализовать кроссэнтропию — это стандартная функция потерь для классификации. Тут можно почитать про вывод её градиентов, а также софтмакса: https://deepnotes.io/softmax-crossentropy\n", + "\n", + "Напоминаем интуицию за принципом максимального правдоподобия: мы максимизируем произведение предсказанных вероятностей реально случившихся событий $ L = \\prod p_i $.\n", + "\n", + "Произведение оптимизировать не очень удобно, и поэтому мы возьмём логарифм (любой, ведь все логарифмы отличаются в константу раз) и будем вместо него максимизировать сумму:\n", + "\n", + "$$ \\log L = \\log \\prod p_i = \\sum \\log p_i $$\n", + "\n", + "Эту штуку называют кроссэнтропией. Такое название пошло из теории информации, но нам пока знать это не надо.\n", + "\n", + "Для удобноства вместо чисел — от 0 до 9 — будем использовать вектора размера 10, где будет стоять единица в нужном месте (такое кодирование называется one-hot)." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "3ckg7SdddAyF" + }, + "source": [ + "class CrossEntropy(Criterion):\n", + " def __init__(self):\n", + " super().__init__()\n", + " \n", + " def forward(self, input, target): \n", + " # чтобы нигде не было взятий логарифма от нуля:\n", + " eps = 1e-9\n", + " self.input_clamp = np.clip(input, eps, 1 - eps)\n", + " eye = np.eye(self.input_clamp.shape[1])\n", + " self.ohe = np.squeeze(eye[target], axis = 1)\n", + " self.res = np.log(self.input_clamp)*self.ohe\n", + " self.output = -np.sum(self.res, axis=1) / target.shape[0]\n", + " return self.output\n", + "\n", + " def backward(self, input, target):\n", + " self.grad_output = (self.input_clamp - self.ohe)/target.shape[0]\n", + " return self.grad_output" + ], + "execution_count": 14, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline" + ], + "metadata": { + "id": "9gCXyj1WS68A" + }, + "execution_count": 15, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "def SGD(params, gradients, lr=1e-3): \n", + "\n", + " for weights, gradient in zip(params, gradients):\n", + " #print(type(lr), type(gradient))\n", + " #print(lr, gradient)\n", + " weights -= lr * gradient\n", + " params = weights" + ], + "metadata": { + "id": "cKAE1bm5xPRD" + }, + "execution_count": 16, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "def loader(X, Y, batch_size): \n", + " n = X.shape[0]\n", + "\n", + " # в начале каждой эпохи будем всё перемешивать\n", + " # важно, что мы пермешиваем индексы, а не X\n", + " indices = np.arange(n)\n", + " np.random.shuffle(indices)\n", + " \n", + " for start in range(0, n, batch_size):\n", + " # в конце нам, возможно, нужно взять неполный батч\n", + " end = min(start + batch_size, n)\n", + " \n", + " batch_idx = indices[start:end]\n", + " \n", + " yield X[batch_idx], Y[batch_idx]" + ], + "metadata": { + "id": "s5aTjzA9xTgA" + }, + "execution_count": 17, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "\n", + "n = 1000\n", + "\n", + "X = np.random.randn(n, 10)\n", + "true_w = np.random.randn(10, 1)\n", + "Y = np.dot(X, true_w).reshape(n,1)# + np.random.randn()/5\n", + "print('best_possible_mse:', np.mean(np.power(Y-np.dot(X, true_w).reshape(n), 2)))\n" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "i-s8Z9pPxVBw", + "outputId": "86d35849-7d62-4922-bafd-fa515fc4bcd2" + }, + "execution_count": 18, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "best_possible_mse: 14.871417566238062\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "model = Sequential(\n", + " Linear(2, 2),\n", + " SoftMax()\n", + ")\n", + "\n", + "criterion = CrossEntropy()" + ], + "metadata": { + "id": "6ebfMiX6xWr5" + }, + "execution_count": 19, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "n = 500\n", + "\n", + "X1 = np.random.randn(n, 2) + np.array([2, 2])\n", + "X2 = np.random.randn(n, 2) + np.array([-2, -2])\n", + "X = np.vstack([X1, X2])\n", + "\n", + "Y = np.concatenate([np.ones(n), np.zeros(n)]).astype('int')\n", + "Y = Y.reshape(-1,1)\n", + "#Y = np.hstack([Y, 1-Y])\n", + "#print(Y.reshape(-1,1))\n", + "plt.scatter(X[:,0], X[:,1], c=Y)\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 265 + }, + "id": "mXDV2aWuzgBE", + "outputId": "3c75cd75-8b10-4a33-a43c-bf6d8bce0321" + }, + "execution_count": 20, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd5hT1daH333Sk+nDgKICdsWCV+AT67WhYu+KiorYrx3FelGvFeyKFXsXe+9iRVRsYEOxgKAC00t6zvr+2JmSyUkmM5MBBs77PPNATtl7JzNZe5+11/otJSLY2NjY2PRejOU9ABsbGxub7mEbchsbG5tejm3IbWxsbHo5tiG3sbGx6eXYhtzGxsaml+NcHp326dNHBg0atDy6trGxsem1fPnll5UiUtH++HIx5IMGDWLWrFnLo2sbGxubXotSar7Vcdu1YmNjY9PLsQ25jY2NTS/HNuQ2NjY2vRzbkNvY2Nj0cmxDbmPTy5DEP0j0CySxdHkPxWYFYblErdjY2HQekShSNwHC74Jyg0QR756o4qtQyv4qr8rYK3Ibm16CNNwA4feACEiD/jf8OtJ4x/Iems1yxjbkNja9ABGB0JNAuN2ZMAQf7dm+E0swG27GrD4Bs+Fm26WzAmI/j9nY9AoEpL0Rbz7V1HO9xn5Bqg8DiQJRiH6KBB+B8qdQzvV6rF+bzmGvyG1segFKGeAcbH3StUWP9Sv1lyUnimjySBSkEam/osf6tOk8tiG3seklqKJLAR/gSB5xgvKjii7ukf5EBGJfAu2riAlEP++RPm26hu1asbFZgRARiM6E2LfgWA28u6OUDwDl3gL6PI803Qexn8C1KSowDuUc0CNjUUohygMSsjjp7ZE+bbqGbchtbFYQRCJI9ViI/5A0ngrqLkBcw1FF56Fcm6Oc66CKr1p2g/IeCKGnaXWtAHjAd9CyG4NNh9iuFRubFQRpuh9i34EE0e4MU//EPkOqjkIiM5b9oFxDkuNoRoHrX6jCc5f9WGwyYhtyG5sVhdBzpIcXNhPO+wajmI2YjXdiVh6MWX0iEvkk9Xz0W6ifCMTbHHUAJiqDa0USlUj4dSTyCSKJvI7XJjO2a8XGZoWh/aZiOxK/IRLPSxanmI1I5Z5gVtJsqCX6GVJ4BkZgnH7ddB8QaXdnHGLfIvEFab55s3EKNN6ls04RUD4ofQjlWr/b47XJjr0it7FZUfDsTNa1lQrQGrHSeSSxGLPxdszaCcjS3cD8h9TVdggabkbMBv0ysQjLyUW5wFyS2nbkE2icSnN4ItIEZiVSMw4RM70Nm7xir8htbJYzIgmk/mIIvUqqP7otXvAfjVKqa31Ev0RqxoHESd24bIdyQ2wOeLYBz9YQ/wmItWssBs4NUg8FnwAsolukQbfnHtKlcdvkhr0it7FZzkjwcQi9jnZjtDPkKkBzlIgqOL1r7YsgtecmN1GzGHHQht4o1V37jwVVQMp6T/mg4ESUUZR6X/MqPg2lV+g2PYq9IrdZIRCzCQk+BZH3wChHBY5GuYcu72EtG4KPYLmaxQUl96Fc66OMwk41KRKC8DtgViGOgWBW5XajY3VwbgSAcvSBPi9qUa7Ix+AoRfnHgXdU+n3eURD7mrTNWkmA61+dGrtN58mbIVdKOYBZwCIR2Ttf7dqs/IjZiFQdCIl/0IZAIZH3kcILMAKjl/fwuoXE50P4TSAB3pHW+iQSzHC3gXL2z8mIS+JvJPQimDXgGAANN4IytRsEgw5X4gB4UGUPpLhvlGM1VPH/OrxT+Q9Aws9A7Bf0pGQAbiiaiDL8OfRt0x3yuSI/E/gRKOroQhubtkjwCUj8TWuEhKA33q5FfPv1WkNgNj0CDZOBBCDQeCdScAJGexeJZ2cIPUPqxiNg9AGjX4f9SHg6Unsm2i0TBZTuL2WfUrUez4RzoP49OFbvsM/2KOWBsse1rG74Hf1U5T8M5dq4023ZdJ68+MiVUmsCewH35qM9m1WMyDukh7kBygHx75b5cPKBJP5OGvEI2kAngDA0TkVic1OuVQWnJ/3SzbHZLsCHKr62w81NXWxivG67ZdVtZawFrc0SQOu1eED1ISUKJv4zUn0sEpvdqffa8j6UG+XbD6P0Noziy2wjvgzJ14r8ZmACkPEZUCl1InAiwIABPaMNYdNLMcoynEiAKu7x7kVMiH4KsR/AuRZ4dkYpd+brE5WgDFTGcaOr+GBlhGNI+E2Ua8OWI8pRAX1eR4JPQ+xzcAxE+Y/KTUMl+lXH1zTj2BBVdA4k/kGcg6HmSPQEkzJwpOFmVNn9ubdrs9zptiFXSu0NLBGRL5VSO2a6TkTuAe4BGDZsWAeZDzarEsp/dDL9vO2GnwFG/7Qwt3wjZhNSfRQk/gCJgPLoVWv5NJSjf+q1sZ+QunMh/gcgiGswqvgGa4ObdSWd/iCsjCJUwThgXOfegHLSYSIRoMMXD0B5ttMvE4uQTKGOcf3EIBKHyAcQ/xmcg8CzS9YJzmb5kY8V+bbAvkqpPdHPhkVKqUdF5Kg8tG2zCqA8WyOFZ0LDTTrZBBOM1VBl93YpblrE1PrduVzbeAvEf6HFLSFxkBBSdwGq7OHW68x6pPrIZIm1JLE5SPVoqJiebuA8uwLXWvToQvn2yDJ2gfCrOqvSrALPtqiC09MmldbrE3oCssSJdusY+t/Qs4ijH8q7GxjlkClRxzEQMWuRqsPAXJwsaOEGFUDKn8Vwpo9FYr8gwfsh/iu4h6H8x6AcHfv3bfKDEsnf4ji5Ij+3o6iVYcOGyaxZs/LWr83KgZgNEJut/cXOjTttxM3QK9BwPZh/gVEBgdNQ/sOztmMuHgFSbXHGieo7q2WjVYKPIfWTSQsTVAHty/bunt52cBq06KMIoKDgDIyCEzKPp3EKNE1tIx2rADeU3Ijy7NryXkRMpPYUiEy3aMUN3j21Dzz0IC2brQD4oHCCfs+Nd5DuWvGiSu9Gwi9D6EXSkoFQ4FgP/IckP1svEpmB1JyM3g8QwKF10sufT3laEYnoqBqjHKVcGT8Dm8wopb4UkWHtj9tx5DYrDMooBM+2XbpXwm9C3UW0xDGbS3XUCyYqcGS2O3M6J/GFWGcuxpIRN+kY/kMRzw4QfgtIaNdEFr+3mI3QeDepG7+iX9eeiXh2gZJb9NNG6AXt9rDCsSlGyWTMmtPRkSxt32MIGiYlXT/tjbgDiq7XT0i1p5JuxJPjSfwCDTcgoReQsqeg7gJS48cTIA1I1cGIdzfwH6+lcIOP6vuVEyk4DeUf2+VMVZtU8mrIReR94P18tmljkwvScBPpyoEhaLwN07svKvKW1gdxbQHuEa0GxLsnhKaRarQMcG0O8bmIWQfuLVHuIUjInx7zrZzg2jTjuJRjNQgcnX3skoDox0jkIzIHksUh8j6EXwffXjobNJOPO/G1/jf2TYZrotZuFeVBudbIOtZWInpfIfSsdr9YIbX6sw1No9XNg55XGm9BVDHKb+ua5wN7RW6zcpBYaH1camDpjkhz+J/ygnMTKHtAh8sVno1EP9UCUhIE/NpPH1+I1BwHGHrVXXAGONaA+Hxaw/w8ui1X1zNQxaxBqkYnfdHNoYqZiCANk/XkY7labmkVSfwDjv4ZjGympxADzGQ6vXcPCD1PZu0X9OcV+TRLe21p974kBE13gG3I84KttWKzcuDI5LJQQAMQBExtfGJzkKZH9FmjCNXnZVTxJAicCkWXAH6QpfpaaQQi0DgFCiZA4BgwVgNjTSg4SYfpxb7FrDkds/IgzIbrdXhijkj9lZBYkCxwnM2IJzEXQ2xWB4lCLkj8gyo4BR0z3hYvOIfQGrPelkSLuJUqnABG3w4G4wDnaliHWeZAYmnX7rNJwzbkNisFumJNe+PkwfpPPKxdAs33KhfKuztG4Vko5/pAHemrzDCEnsMoPA+j74cYfd/DKDgNCb+DVB8NkbcgPgeaHkQq90YSGdwNJEWswu9h1pwG4VfIyYC3YCKND+q494wY4FwX5dkRiiYmY/G9aPGtfaHsXnCuowWwmq/HC4WXtxSMUEYp9HlHP3FkfHB3ofyHgWckXTLmbWLpbbqH7VqxWSlQ3p2h5Eak4Tq9wjX6gf8oaLwVa0OZwR0gjVgbf9E+3+ZXEtMr7/rLSPXNR0HqkcY7UcWXWXdRfzGEX7Uuatz6jjKPMfE7mV0rbgiMbdFnMfwHIb79dAEJVdQahVM+DUKvIJF3wKjQESiujZHot0jwAb2B694OSqeioh8iTY/peHJM3YcCiq7W2jHFVyLV83XoYc6TkhdVeH6O19p0hG3IbVYalHdXlHfXltcigoSm6U25FLyZiwe7hiSFptrjA8/uus2m+6Hp9qRP2+raOEQ/smxeYj9A6BUyl3QDcIBrOMRmWpxToMqBedbnfAehCs5OPWwu1huluBDvriijTMe9+w9E+Q9svSz4EtRfQksYYewHCD0FfV7C8B2oY9zjPyHxX0FiKEdffUwFwLMvxDNNmoBrhH7P8QXg2hBVcA7K1ijPG7Yht+nViIhOnFH+NHEtpRSU3IJUj0kWVGje7ByMyhBJoowCpPBCHaJHGL0q9oFzIMp/oJbabbwVa9nZNiQ1vdOIfELWVavyaw1w57o6XT9ts9GhN3AtcaAKz0oJ6TMb7oamW9FLaAPqr0SKr8Pwpca9i8Sg4XJSJxgd9y2NU1FF5wMJpPEuLTWMQ2eGGquBcw2IfomlXg4AbjDKdCKUZ5e0GHIR0RE28Z/BuTa4huccliiJpUBcJ5CtwqGMtiG36bVI5BOk7uJk3UlBPLugiq9GGQUt1yjXxlDxPoTfaBN+uHXWL70ROAJxbaJD/MxK8OyG8u+nk1+a7qRDI44LvPtlaLxAn08z5k5wDUX5DwDvnkjtBKwjRlw6jtuy7TLt205iht+BphtJc9HUnYd4toLE30joDYjP024jsXpKiCWTjs5Hmh5K/r+NwTb/gOgf1uNpIQqR15Do+9rlVf4UyigBkhIJNWO1LICg49sda0LZoy3XWCHxBVrxMf4LOklpdSi5AeXarIOxrJzkNbMzV+zMTpvuIrFfkKqDSTWqbnBvidEmtT7fmP8MJjc/sAN8+6GKrkYpA0ksQRquTYppWU0EDlTFRyB12s8fmUF290sGPLuiCv4Dzo2QJUMzaJ17wf1/EP2MzKvoNriGYZQ/jrl0V73/kDNWfn4X+A7GKL4cALPuf8k487Z66U5wbandZM61wb0dutyBRiSKLN0pWSyjzWSnAqiK91Ims5WNTJmddtSKTa9EgveTboSiEP0aif+R377MaiQ+XyfuWBWGsCQBoVeR0LOIhPSkE36dzKt5BxL5AKncP+m66IIRB4i8i1QdgQQfyuDrB/05fUpORhwveHdHJNrB5mx7Mol5xSD8WuvL8IukF72IQ+xzpOE6pPYspHJPxGwjoxB5PzlBtXtikYQurrEKYhtym95J/HcsXQ/KDYm/8tKFmLWY1cchS3ZAKvdFlmwD7pFYx2BbEdFp6aHXwKwnPSW+LTFovJncjGvWUaPT8G8ic0igSfaEIgA/2jzoccmSEeAYlPswlIcUrfMU2pidjJMNum9pgsSfSItmDbqSlOV9YUgsyn2MKxG2IbfpnbiHARaSqhKxlL4VM4gEn8GsvxoJPYdY+oPb3VNzctL9EAVCepMxOBUKTiHnuGmzDol9j05IyoZK+vrzRYTcyrtlwLE22hAntDGVRojNIadJzFgDVfqgdt+kGXO3jmVvxvNvOjZDcQi/RYsb2DXEol30hveqUue1HbYht+mVKP8xyYSWtn/CPvAdrIsGt0ESfyGVu+osyuCDSP3/kKUjdRp7BiT+mw6/S1u5RiH8Hjmvys3FYDaRnmHZHjeozhVY7jkUJH4i/b2Hk4WZM70XN7i31X5q9xBU8bVahVIF0NWJ/DpRqeDM1p6KLk4WFuno82nz9OXaHNxDSf0dePQmqWeX3N7iSoYdtWLTK1GOCih/Hmm8SVd4N4rBfyzKf3jatVJ3KZjVtBgDCYJEkPqrUKW36UPx35DgY62JMI7+WnMlbeVugtSDMnKTGCEBkTe1q0EiWEeioFevZgTin2Vpy6C19mZnskE7i5DRDWQugbJHdWZs5F29UpegNtK+Q1CF41sigpRjdah4R0e6JP7Uk4B7mxSteOVYDfq8hYRe0mX9onOSUTltPycD3Nu3tqsUlN6lo2hCT+vQUt8+qMAJq6w8rh21YrNSIyLI4o2wtrpujNW+a1O8OIY2YD4wCpNREe0Nmhv8YyH0eGqRiWyoAiiYCA0TybyJ6UZLCmRq04Ciq1CebXXhidgvEP9R/yxTnHosrmFQcpMO9Qy/jASf0QbdsyP4x6ESPwIucG2eGnFi1mtXTYa4b0ks1hvD0tAqYmYEUOVPZyyusSph65HbrJJIfD6Zl84KkQRSdz6pBjYEZoh0z6MTjEJUwVits119CFnVAVsHAfGvOrg2SuYNSAUlUzCas1YD41o89GbV4RD7mhwfD/JA8kkgNgOWbo2oCpBKWia8+A/QdAeCTz+1KB9SfLM+13R3cs9BgVGMFF2J8uzQboXeDyrehvBrSGwuyrkB+PZEqY5cL6s2tiG3WalR0XeQTLolzg2S+iCZNgWbDa8CCsG3N6rgVJ3i7i7DLL4N6k6nQ2OuCpPhfh1tPloZYwc4N0El/kAa79Rx4q71W5suvhqpPARobHe/kXzdkwbeBMkkDhZKdt8ENRZVH80lUHuivsS5Aap4Esq1CYAW7vIdiG27c8c25DYrLZJYioRnZL7AuzMYgQwZjSktgaMIo70IVvgl9FeorYE2SI9vrodEprT6jkhAfA7S8L1+2XgHEjgOozCpp+IYpLNFm3XEW3DpzeA2Ql8rLPGfdTWh8jcxXJkrKPUUEp6ONE4BcxE4N0UVnoNyDc5vHxKF8JtIfC7KuQ54R+X1KcOOWrFZKZHYHKRyt6ReibV/XHn3AVVCbu6R1GQYMz4/ma7efpVt1VYE7XrIFFfdYefJ+xO6raYHkNgPiMT1Bq1ZQ/p7jIJ7BBiDutjnsiYBDZct817N4HNI7VlagtishuiHSNVoJPZd3vqQRBVSuTtS/19ouicZNbUrkseYd9uQ26yUSO35yWIN7Q2tAjxQeK6unyn1+nVWHODZWbcbm4NZuR9U7m7RdtYRob9ubrr/tYsg1cchiwdDw5VYb6AKRN7QWigtGEBFN/vuQWLfLNPuRMykOFr7jNUQ0nBD/vppuBoSi1vlEiQIZhVSNzFvfdiuFZuVDjGrITHf+qTyocpfai2CbPTVoXOSKaPSm9zgPFPHo1ePyaBfkgOqEFybZJS4zR0BqW79f86Y6KIZbrqVLNRTiAuJzNChpM7BPa9maFZn/l3mcUVO5B3Sw0VNiM5AJJES1dNVbENu0yOIRCH2rY6fdm6aEpnQ82SJJVYlLUZc4gu0JonRBxINtH7ZFODQ2aPuHVH+Q1BGIWbD5A5SyjtA6vJgxLtLlJbN24yhjt3BAcYAMH/vwr0NSO1pgKkn2NJ7WyfcnsAoImOGrmO1PHaUaUJSWc51DtuQ2+QdM/Q21E+gRf1OFUDp3XnfQAKQ8FtIw8064cToAwVnYvj3R9z/B9GZpMaBe8F/mL4vOgupHocO+Yujjb9TGxD3/6EK/oNyDkztLPYzHWuUZCOb1kozyUmkxxN+GnSBCqnKY7sutNRAV4x4MslJkpu2iQVIzVik4EKUoxxcW+R9ha6UG/EfDsEnSXVP+bSCZL7w7gmhF0n923GCZ6e8LXBsH7lNXpH4Aqgb36rPIU1gLkaqj9Wr9DxiBl9CasdDYh66CMIiqJ+AWX08FF0LjrV0erjyA17wbI0KjNPjrLsY7RttNpgxwAT3UIySyelGHJIaHx3507uDAuf24FxGeiF5NeKgP8PO/o6TBS/SXESmnpzrxmuDXjkSif+Zl1Gm9F44AfyHon+vXu3+KpyA8u6Rxz4uAOegNlIFAXCsjiq6PG992Ctym7wioWexXk3GIPIhtCnF1q1+RKBxMpZqgdGPUNFPoM8bEP1cqyG6BqNcG+l7zTpILLRo1dRjzIDyH4EEH066V3KIdOk0AvGZrJD+6x6jo1j3ZDx6YiFScxKq4rUs13YepZyookuQwnPBrAWjD0rl1ywqowjKX4box7oQhmMQeP6d135sQ26TX8wqLA25mPqLkjeiWdQCBWm6E+XbF+UZkX5aZVlVq0DmU45yKH8WaZikv5QCHVcL6iy5GvEA2h1gsmyyOh1t+umJSawjTG3M4/N0wec8o5Q3z37x9u0b4NlB//QAtmvFJq8ozw5JV0Z7zKSsab5wZzW6JBYgS7bVxY7boZQ3GU5oIbHqb81CFLMes+6/mIu3xFy8BWbteFABjNLbUX1nQZ5XbjmjysC5Pjg3p2e/wk5aN+MSaAO+PIx4MxEk0QsSnJYDtiG3yS+encG5MSmypMoH/kPzGoGglILAqVmuEJAapHosIhZPCN59SDdKUYh+gyT+RsREqo/UKn/NCn/hl5GlO2LGflvG+ibtkGqIfwPxr8ltA7WrxFn271GBsXqGcwKJrmykrvzYhtwmryjlRJU9BIUXgWs4uHdAFd+AKrwk/30FjgNPhiLHLUSTQk3taLwOSyMVfUtXAwq/DPEFpLuJIlC1B1J/oxbDsukGzRudzfig4Dwd5ZGJ0KtAUtUy+jUSfiOvGZK9FdtHbpN3lHKjAodB4LAe7kchid9oqWRjfRUSfg+pv1z71F2bQcHZOiIiE9IEwcfIWnYt/m2WPld1HODaEgLHQdMDSZmEdqgy8B0OGBB9W++fmA3QdC84sjy5mYt1IevqY8D8W98vMcS3D6roymWcr7DisGq+a5uVAon9kAw9zGJQJaSLDyT+0C6S6KdQPbqDluPJup/Zvh49YcRXlq9jQofbSVAnhVmtF6UagncDddqAm1VAMOk2+j5z09KI1Jye/H0Gk3HnEV3oOvhMT7yZXkG3/3KUUmsppaYrpX5QSn2vlDqz47tsbPJAYhHZhai86Mf39lokzRt3WTDKdSLTssSzFyvHQ7IHHAOh7mL0U00mF1QCgo8kqze1vSZGxoxHs0rvD6RNpCEIPdK9Yfdi8rEEiAPjRWQwMAL4j1Iq/yl8NjbtcW4MlklGBjg3haL/0eUMyfg8KLwEywLPeccBOCE6neW2gZpXIoDRiageK9Gv5opJ7cmyAZsm5bvq0G1DLiJ/i8hXyf83AD8Ca3S3XRubjlDONcE7itQivA4wSlFl9ycf17tKDIIPoSreBVdnC/q6wXsQuLfXm3eZCgs7h0DJVHRqe3N6+krid2+8vjXdvisoZ5Zi1FardRd4d+t6f72cvDrllFKDgH8BaWECSqkTlVKzlFKzli5dms9ubVZhVPG1UHh2Mh2/DHz7ocqfRxkluuxYd4jPQzn6oYqbdWNyxLkJRsk1qMJzIPwqGZOGpBZCT5F1U7XX0p14c4eOCMooIeBGT97NK34fGBWogpMQswYJvYCEXtQZvKsIeXPIKaUKgGeBs0Skvv15EbkHuAd08eV89WuzaqOUAxUYC4Gx6Sed69KxZKuR/LFwwTj6Ji8pp1NCVoYPif2EVI0mc7FlILFA/6wU7pR84QRVnMWI+yBwAnh3T25i/wnurVG+A5Hw21A/kdZ9kwRSfC2Gb69lNPblR15W5EopF9qIPyYiz+WjTRub7qJ8h4DKImkLgEsbhjT3hwLXVoiI1srwjCLndY9nD6TxNrIacaDna2quiGSRdHVvD30/60DMy9BFnKsOAhKoklsxAkfrAiH1E9FPN8HkTwTqLkASS/L8HlY88hG1ooD7gB9F5MbuD8nGJj8oRwWq7DGgKMtVDlTBmeDdv91xgfCLSOOtuq2Sq8C7F62bcBm+OsqP8h8AsdmdGWknru3tZJi4lBdVeB7KQlIhlSa0sQ5D8Gmk/mp9OPx6hrYFwm92ebS9hXysyLcFxgA7K6W+Sf5kSc2ysVl2KNdgKL0l8wXODXUSSfQdi5MRaJqKmE0o5cUouQ7Vdyaqz2vQdzb4xqAf45OG2BgA5W+hlEdro+dMUgt9lcWA4jtb1ClzN0thCD2DSCgZvWS1URxFmh5AVvKIFiWy7B/thg0bJrNmzVrm/dqsmogIUnUIxNuvkp2oskdA+ZGq/cm4WjQGoMrut9SKEbMa4r+26INIw/VaGZFEsmZoDhgVydC5fCsp9iZ84B0JBadD1f65f3Z4URVvglmHVB2KtTvLBZ5dMbJN6L0EpdSXIjKs/fGVJZXMxiYjSilU+dPgPwEt/2qAY11U6f0o91CI/0zWr4K5AKk6GLEo86aMMpR7OBh+pOpAXfBY6nMwRM2rcDd4DwCsFCN7G8l4+C4RgvArUHUoFF+Ljkpp3rR06mpGViiX1hB3bQT+0Vi7qWIQeQeRjvYsei+r8vOczSqEUgpVdB4UnZd+0jEQ/VXIlupfh4Sno3zWscrS9JiWA8g57M5s7S94T473rMgY4BsLoYe70YapJ8DoLKh4FxV5C8x6cG+D4EpKK7R5alE+KDgHldzQNoouxAxOQ/vRLZAQKK/1uV6OvSK3sXENAed6ZE/3F2i8FbNyb8za8Ujs59TTsa/pXDz4SpL404IbjM6Yk0wbvFEIPgyVoxAxUAWnoNxDMNyDUeVP6MgWVaL3NoonYwSOTL3duyOWZs2xhr5vJcU25DarPEopLb3rHZX9wsTP2g0TfhWpOgSJttnnca5L737AbY6nt8JHh1IFzkG6rF6uFY6cW5B54hTtnmq4VseGJ1GuwRhl92H0+xyjz8so7+5pd6qC8aCK2ozXADxQdHneizevSNiG3MaGZF1F947kpq1iAiGk/orW+/1jLGLWk1XlV3gM8O4Nzo10dSflBzyg+oBnV1Tx1VD+os6exSou36tDOB0Dyfn9ltwEri3I/nmHkMY7OvVOlHNNVJ/XwXcIKbkBtWci4ekZ7xOzEYnNRcy0XMZeQW9eQtjY5A2Jfgv1l9CpwsfxnzDjCyD0nM4wdGwM8e/Q6n1Gq+FLzO2ZQeeLgv9iFByJGXoHGq+FxD/gWAtVdAnKs62+Rkyk8AqIztBPJdEvgAgYpVBwPsq7C6IKIfwGHbqYXNtgOPtjFpwNNeOyX2v+0/n3Y5S2k0aIgLYENvkAACAASURBVESQ2jOgz8so56CWS0VMpOE6CD6qJ2KJIb4DUEUT816EuSfpPSO1selBpOl+Oq954obKvdGGu73POwGJ3/Iyth7HvRVm6FWou5CW8L3Er0jNKVB6OzgHI1UHgLkE/TSidMy87wAwisE5ELPqUIh933o+UyinZ09UyU2IRKDmZLJ/5gpcm3f67UjoSa1jk0YECT6BKrqw9drggxB8nGZjD0DoBUQVo4rGd7rv5YVtyG1sIFltprM5FXFWik3L2mPR/ur24XlhpGESEGi3MhYw50PTzegwwc6E9UV0lZ/qI4GOknREPwmY9dr1lStNT2U+1z5ztOk+0uP3wxB6FCk8p9f41W0fuY0N6GiITmuPrwRGHPRK2/zb+lz892SR50x0MjY7lizTlliY2/XRD5ClIzHN3FxeEvsFEj9lvsC5QerrTAqJEqQ3/X5tQ25jA6jAUdpNkPErocgentjbyVANSXVGaqAjHGD+DInf6dTTj9RA7Rm5Xdp0Z/a2C05Ife3axPo6x9q9ykduG3IbG0AZpajyl8B7MNaRF26WTbWg5YRzXdLftwJZnK8O0P7zLmqvRz9EzBzS9rOJbqnVof5SJPwmIjpxSxVehI5uaTaFCvCiiv7btXEuJ2xDbrPSIeHpmFUHYS7ZGrP6JCT2Y073KUc5RsmV4D+eNKPmHEz3iiUsbxxAcYZzzmS0TfuVrJC396y8Fu13BiO3ik/O9ckYAil/Q2Q6UjsBqZugh+UegiqfBp7dwDEAPDuhyh9rjdbpJfSeZwcbmxwwg89C/eW0+G6j7yNVM6H8Ca2E2AEiApH3SIu8iP8Ajv7JQhBtfacKvR7Klz/VKuIjH32Y2o0Qm2FxrrN1TbswHumuIJgTHKt1eJUqOAWJfEB2330Iwm8jsdko1+Yo14ao0lu7Ob7li70i70H++vUfvn5vDrVLV52SU8sTkQQ0TCL1Syzo6IscpfJjs5Mbf+1XonFwbaZT+ZUP7VNuFolyJv+foTZnzihwDSVV79wJKqBTzC1xgvewHNqWDEa8K7i1frtnD6wThKzoziTkhcKzUKpj15ZyDUaV3dtmZZ5pfBGIfNKNMa1Y2CvyHiDYEOKyA6/j+xlzcbmdRMMx9j5pJKfcdGyvCWfqlZhVGVZ+ArE5ObZRifX6JqGTXYyipAHzQ/hZUieNaPLerrojRMdiF5wJiV8g9hu4N0cFxkH0C6R+Yrv35wXfvqjAWCT8LJ1fWXeVCIRfhj7vQPzHHipXZ+jiy46+qMApKN/eOd+p3P+H6vMqIgkk+AQ0TCZ9he7Wv8uVBNuQ9wA3HH8n3338I7FInGhIh029du+7rLVRf/Y5OV0fojfRrF+/Qk5IRiYfMDk9lgM6AUUyhbpFtaEPP4+1wc6HeyUEkTcxyp9OOSrefSHxFzTenuzX1LUqiyYi1cflqe9OoNyo+I9QPg2puxgiVoU5uoMbo98X3WpBKQf49kQaJlud7Fhbpxdhu1byTKgpzKcvfUEskro6igQjPHfzq8tpVN2nrrKeq0bfxJ6+IxjlGc3E/SdRuSiHzScgEopwz4RHOLjfOPYvPYZrxtxK1d81ltdK9BvM2nMxq4/DbHqyUxrSSnnAfyg6SaUtPlTBqbm14agA/5ik+yQbPbjxmbDKSoxDZAYpRaCjnyLBlyH2Bcu+9qcJjj4ooxSj9A4oOBftEsrV1ZINpy6unAeUUYYqvVOv7lVB8qcQVXIHyijLSx8rAvaKPM+EG8Nk2jVvqO6d5aYSiQRnbXcJf/++hERMr/w+e/UrTvvyIh7+5Tbc3uy+y4v3uoYfZ/5MNKwLM3zw1Cd8O/07HvjpFnwFrQbTbHo0+RgcAQSiXyKhx6F8GipHHWlVeIEOLQs9AyhQbq1Z3QnDoAongGszpOFmMP/I+b7k3XTbqLo3Sz8WfjVZB7TZtWICYWi4qPv9dRqlKyI5NwX0U5pyDUE8oyD6Hkg8w5gMMNYC8y/0E4TVZGiAo39KGr3uI6x92hIBzzYoI3dJWuXZFvp+CtEv9QH30Jz87b0Je0WeZ0r6FlPaL/0R3zAUW47svG7EisCsN76h6u+aFiMOYCZMgnVBPnj606z3zv1iHnO/mNdixAEScZOmuiDvPPpRyzExG9psVDYbgRDE5yPB53Ieq1IujOJLUf0+R1W8ieo7EyNwRM736zYUeHYEM5cY6vaTdh6MauRjzMapLbHOABJ6DetScMvaiBvg2ARV9gBKKV1Gr/Z0pOZ4iLyg5WczjsnULo2yp7FeuTu02mKf11NWyxL9HFmyDVJ3LlJ/MbJke8xgljR8C5Ryozxb65+VzIiDbcjzjlKKs+4+CY/fjWHoL7nL7SRQEuC4qzpnUFYUFvy4KMUQNxNqDDP/+z+z3vvrt/OxKgsbboow9/NfWg/EvrGQgQUIQdN9mDWnYTbegSRyc+co5UM5Vu96dl7sC1AdZHJ6DwT3DugH2zzuGUgtNE5B6i9rPWYs71JwCtz/RvV5G6PiOZRjNSTxN1K5G0TeIudEn8QfUHs8BI4nNcrHCUYJqvh/LRV/AMQMIjUngTTq6kHSpPuqvwqJz8vf2+vl2Ia8Bxi++xbcOuNqdjlqBzYesQH7n7EnU+fcyGqD+i7voXWJAYPXxO1NN7K+Ai+DNk0vSAxgmiZN9UFWX6dvy4TWFo/PzYDBa7UeUEVk9Dubi7SxaLwTqdwdif/albfRSTrw9aoKVNGFOrVfuelwZWys2cn+QxB6rmXiUr7D6H54YzdQJajiq1FO/TsTEb3JmljQ+bYkhHKuiyqZrDXJHQPANxpV/mK63zryfoZGYkjo+c73vZJi+8h7iHU2H8iEB09LOx5sCFG3tJ6KtcpxunrHxz9s9yH0WaOcv3/9h3jSvWI4DAIlfnY4eETa9a/c/RYP/PdJmuqCeP0ePH4PkXAUM55Mi1bgdDvZY+xOrTe5NgNVmgyva28Umw18BCSK1F+GKnsk/2+0Le7hWGurKPDuq7W6jWIk9gN0tCGr/GBUgJmjUFTLfR4thesoR3m2RpwbQ/yrHG50kVITNOf+ypLyr2bacVXxWqqRjX/fRcVI9O84sQhVcFLHexcZ66AmwGzofN8rKfaKfBkRjcS4ftwdHNxvHCcOGc/Bfcfx8l1vLu9h5YTD4eDmj65gh0O3we114XQ72Wa/4dw285q0jc63H/mAu8Y/RH1lA4lYgqa6IKGGEAM2WgOHy4HhMNhg+Hrc/PGVFJUXttynlIEqu19voqmAji6wRHRMddJfEwlFqP6nBtPMbxSJUi5U6d3JKIcAOhLGA/5xGCXXoZpDHR1rdlDQ1w/OzXX1nc5+3SSaLE4BInFI5CI14AH31uDchM65e7xQdh8YZbRG/bhBBVBlU9NXyhnj7XNAeTOLVbXHsy2Ixe9W+VFe60LYqyJKrByYPcywYcNk1qxZHV+4EnH98Xcw/YlPWuLKQa9qdz1qB0664WiKygqz3J1OsCHE/Rc/zruPfYQZN9l6v+GcdN0YSvst3wKzY9b9D//8viTteFF5AU8svBsxBY/Pk/F+EVP7y806pO7spJxoe7zES7/k9jPu551HPgAgUOznlJvHstNh+dXIEAlB+D2QBnBv2+JaaD0fQZbuBGY1rSvHZEahewTKty9494T4PKTqUHKXffWAZweM0tt1P2YdsmQbdBELK4qh4GSUdyTKOUD7lusmJP3XbfFBS3RJCJQTcKFKbkJ5tkXMBiT0DES/Aue6KP/hKIsYfDGrkSU7YF1RyQEFZwEGNN5Kqv/cDc71UeXPolRuE4HZeCc03pnsy9RPOO5tUSVTVsx8hh5EKfWliAxLO24b8p4n1BjioL7jiFlsGCqlKCwLMOWza1l9nX45tScinLbVhfw+Z35LvLrD6aC8fykP/HRLh+GAPcko72ji0fQMQ6XgtfATnXInmXWXJ8MI2xkC3wFcd0ZfPnxmZsrE6PG7ufLlC9lip0278Q46j8QXaKMZm60PuDZFFU9OKSkGYAZfgPpL9SaqxNCGyer75wTfwaiii3VsPEmf9NJtkyvhdjg2RvV51nJj1wy9DY3XQGKRjqUOnIAKJKVc4z/qUEHXJp3eFBazCVm6g57g2uPaFlU6BWUEdC3Mxik6ExQFvv1QgVNRRqBz/UW/QULPgoRR3lHg2THniWBlIpMh7x1O2l5OXWUDDsOwXEuJCA01TUw54z6ueuWinNqb/cEP/PnTopSko0Q8QX11Ix9M+5SRR/87TyPvPGtt2J/f56RvgPVZs/N7Aqpogo5MiH2bNH4muDalKXE6H0w7i1gk9RONBKM8dtWzy9yQK+cAVPmTOoQSUIb105Xh3x/x7gaxbxHlgdrz9UZuS2q9C4z+0OdlDCPVXaOUQgougPoLSVuV+w/LaIgN30jwjUQkBjhTV7C5ujcskNALycnIgtjnSNVhiHsEyjMUVTgeVXRBl/sCUO4tUO4tutXGysyqN6UtByrWLMfpzmzExBS+eidHLRDgt9nzWzYd2xJuDPPzV8u3TuQJk8fg8aU+EXj8bk6YPKbTbSnlwyh/BFU+DVV0Bar8CYzyx6heHMXptg4NtHLrLDNUW7GrDJcYfpRnawz3lqg+08C3T7JyfUDrpvSZlmbEW5vfFku/d+NkJFOlm+Z7ld4AlfhCvUoWQcJvYFYfg1l1OGbTY0hGaQILop+S2U0Ug8TPEHoYqb0QqToAMXtnMlxvwTbkywCH08GJ143B48/sG3Z7ck9tXmP91S0Nmcvroqgs0ybhsmH47ltw+QsTWH/oOniT4YkXPX5Wt3zXyrURyrd3iwzt6mv3Rcx0l4RSsMHQdbrcT1cRswGz9hxk8b+QJcMwK/dGot92eJ8ySjGKJ2H0+waj39cYxdegjFIk+i1m3UTM2vOQ8PTWxKDwq1h/ZQXCr2ftywy+iCwZgVTtpf+t3A2pnaANcuwraJiEVI/Rm6q5oHKNaw/qpK6mqTleb9MVbEPeDhHhlXve5shBp7Cn7whOG3Ehcz7KrTBBNkaN24VRx+9sec7lcbHr0Tvk3NbQ3TantF8JDmeqMY9FYjx57fOM3+lSmupyqKbSCRprm5j39e/UV3cc8jV05BDu+GISL9c/wtTZN7DNvsPzOha3181REw9JmxhFYMaLs7jqiJsJNnRX/zp3pOZECL+FdnkkIP4zUnM0Eu9kuCFgNt6DVI+B0DQIv4jUnY3Uno5phpHoLCxXwRIFsz7z+CIzof6/IHXJcL4oJOa3aysM8blJLfYckM6ssCMQerkT19t0FtuQt+PJa5/n7nMeYsmCSmKRGHM/n8eFo67kp7ZZiO2o/Kuayr+qs7b7zfTveP3ed9OOG4Zio63W4/hrj8p5jM3hgMNH/QvVNtlGIBqO8ePMn7nh+Dtzbi8b9dUNTD52Cof1P4HxO13K6DVP4qaT7iYRX76FaQ87bz/OmXoy/ddN3SCOx+J88vznXHHoDWn3LPz5L+6/5AluO/1eZr31LfnY6JfYXC092z56Q2JI8FH938gnWgSsch/MhusR0/pvRRKLk1EeYVoiYCQIkY+hcucsyTFu8GyTeYxNd5FTtIwEkfDbHV8HkOikC6sX1b/sjeTFkCul9lBKzVVKzVNKdW9XYzkSjcR4/JrnCQdT040jwSgPTkzXdvjj+z85YfPxHL3uaRy97mmcsPl45v9gnbL+1HUvEgmm+yANp4MLHz0TXyA3UahmSvuVcMWL56cZMoBYJM7Ml78k1Nj1VWmoKcyVh9/IIf2O5+2HPyAajhGsDxENx3j3sQ95+LJpXW47X+w8ejs2HrFBWuZoLBJj9gc/8M8frcbmrYff56R/nce0yS/y0u1vcvlB13PZgdd1P/48sSCDkYpDfB5m08NIzakQ/ViveJseRCr3sTbm0U8ytBVKljmzMsZ+8I5EubJs8CYW5fZeoFVYqiPcQ8ld6dALvkNzH4NNp+m2IVdKOYDbgVHAYGC0UqrjmlorIDX/1IKF7xXg99nzU16HGkOc8++JzP9+AbFIjFgkxvzvF3D2DhMJNaV/4Sr/tNYIcbgcXDX6Jo5a+1Qm7j+JXzq5WdlUZxVnDShFqDF3Cdj2TDr6Nj59aRZmIt3QRYJRXpiS3Se7rFj489+YFr8zl8fFkgU6VK+pPsitp0wlGoq2PEmEm8J89c5sZrzYPc1rnBtmiN7waHXAhhtIFbuK6hj5pgfSb1F+MifxZHh6KL4SVWyht90W9zCss1QtMBfnpGejAmOTUr9tTYgPvAfoLFYVADz6GvdwVOCY3Pq36RL5WJH/HzBPRH4Tve39JLBfHtpd5pT0Lcr4uL3G+qunvP7wmZnEIrEUQSgRiEfjfPTMzLT7/7XLZjhd6V+mSFOE7z+Zy+L5S5n58izO3mEi332cu09+6G5bYDjSf42lqxV3OTmoZkkdn7/2taVQVjPB+lCXVrM1i2t5/rbXeWrSC/z12z9dGl9bNt1uI8uN31gkxsDBWt/k2+nf47D47MNNEaY/2b1yX8o5ADy7kKqBbiQN2JYZhLeiEPkw/bCns2GjDpR39w7jqVXgVAujmwlnTv5v5VgNVf4ceHYHVQKOgVB4Pqr4WlTF+zqOvnACquwRjLL7UoSwbPJPPgz5GkBbf8LC5LEUlFInKqVmKaVmLV26NA/d5h+Pz8N+p+2Rtonm8bs5+rLUR8Olf1aluWAAwsEISxemr2gOnbAf/iJfygZl+6w0EV2A4o6zH+xwrIlEglg0xtgrDidQ4sfl0Y/khsPA4/dw9t0ndznrrXZxLa4s4ZIAa282AMPo3J/PGw+8x+FrnsQdZ97PvRc+xjHrnc5NJ97VpTE2c/A5e+MNeFPcKx6/h71PGklxH13KK1Pop1KkhUp2BVVyHRScDEZfndLv2Q1V/pxOCMoUa21UWIzHhyq9p00BhOSq1mnlxnCAe3uUciHRrzGrT8RcOhKz9tw0UTHlXAtV/rzOMDX66vR99/ZYppEY/hZZgA7ft3MARuktGP0+x6h4GyNwBEopLW/gHYkKjEG5eqd0c29jme1AiMg9wD2gMzuXVb+dZdw1R+INeHjmxlcINYbpN7CCMRMPxu11Ubu0jpIKrbGxwbB1cHvdKZmFoI3IRv+3Xlq7RWUF3DbzGp658WW+fHs2pf2K+f6TuZZj+O3b+ZbHQbt0bj/jft574mPisQTrbjGIix87i9kf/sDsD39gzfVX56Bz9mHQJrl9Ga3ov95qWVfbHr+H/9x6XKfarF1ax43H35X2xPPave/yf6P+xbYHbNWlsfZZo5w7vpjEfRc9zjfvzaGgtICDz9mbvU4c2XLNFjtvajmpefwedm8r3NVFlHLpCkQWVYjEtUky47NtWJ8PFbD+/JR7uC6CEPlYi3F5tgGcSPVo7euWiI5XVyWo4iuQ8HtI7Vm0+M8TfyKRd6DsSZRro9Z2nQNRJa0FqCVRiVTtl4x2iaDXdG5U0VWrZMZkb6fbKfpKqa2By0Rk9+TrCwFE5JpM9/SGFH0RIdQU5tZTpvLhMzNxeVzEojFGjvk3x15xGOftcjkLflyUEs/s9rpYd4u1ufnjK1pWq4vnL+X6cXcw58MfANj835tw7n2n0HdABfuXHUNTbbqPu6i8kBF7D+W7j3+k74AKRl90IFvuoqvGnLfL5Xw/Y25KVqOvwMu9391I3wHpq7yu8vQNL/HQpdOItHnqUIZi+B5bMPbK0Sz9s4rHr3qWpQurGLz1Bhzzv8MZuHFmqdYnr32e+y563PLc2psN4J5v06NM8snsD3/gkn30n6QkBNM0OXj8Poy9YnSP9itmNVLzH4h9l9RbN6FgQqeLXYiYyQ3Tn7Ubw7Mj4ESW7phUIWyHezuMsvs7GFstEnwcIp+Ccy2U/xiUa8NOjctm2dJjWitK5wb/DOwCLAK+AI4Qke8z3dMbDDnA3ec9zEt3vJmm57H6Ov1YOPevtOzKDYevy9b7DeP5W16nobqRQZusxdKFVTTVBVs2DQ2HQWm/Yh7+9XaemvQCT01+McVYur1uRIREPNFyj8fv4fTbx7HR8PX4z/ALiLR7CjAcButtMYiDzt6b7Q4akZZc9Nev/zDljPv5+p3ZON1ORo75NydMPiqlzJoVHz33GU9Nep6qv2rYfMdNOOayQ+m/7mq8cvdb3DX+4ZZxK0Ph9XuY8vm1DNgozasGwF3jH+LZm16xPFfev5QnF96TdSz5INQU5rNXviRYH2LobkPoNzB/E19HSHyhjjxxbZhz2boO2zTrkSUjSF3tJ1EFGP1ykbztRH8ShdiPelPWud4qJ1i1ItCjollKqT2Bm9Fb4/eLyFXZru8NhlxE2LfoaMIWESjZaC5/lQ1fgZez7zmZHQ4ZwdTzHuHlu97G4TIw4yb91u7Lwrl/pUWLFJQGOGfqKdxw3O001VuHFfoKvJT0Lea2mVe3+IfrqxsYu+EZNNQ0tTw9uDwuNhy+Ljd9eAUAX779Lc/d8iq1S+rZdv/h7PefPQgUp4oaiQi/z1lAqCnMRaOuIthuDMpQbH/QCP771DmWY5v37R+c8q/zLM/tfOT2XPjIGVk/M5t0RGLI4qFYhiU6BmBU5K+yvRl6A+qT9UHFBMfqqNK7Uc6BeevDpmN6VDRLRF4DXstHWysKpmmmrJRzJZeJMRyM8Ne8f3A4HJx847EcfflhVC6qpmKtco7b6EzLkL94NE5haYCYhbJgM6HGMNFwlCmn38fFT5wNwBv3TycSjKa4gGKRGPO+/p2fv/yVb6Z/zyOXTWvZuP3juwW8cf907vxqMoEinYb9+3cLmLjfJGqXaD2PcFP65yKm8MkLn/P2Ix8wckx69MV6Qwax6XYb8d3HP6Ucd3mcHH/NkS2vo+Eo7z3xCZ++/AVl/UrY++TdWHfIoIzveVVGKRfiOwhCz5JqzH0QODFv/Uh8HtRNSO0j8TtSfTRUTLd96isA9m8gAw6Hg7U3ty5jFij2p2ZUdhKv38M6Q1pXMv5CHwM2WgNfwEvZ6qWW9yTiJutvuTYDO9jETMRN3n9qBscNPovf58xn3le/pbliQD85/PzFrzw08amU6JtoOEb13zW8es/bxGNxvnp3Nmdv/1/++X0J4aaIpRFv6TuWYPIxUzhy7VMsM11veP9yjrjoQApK/Lg8LrbYaROmzrmRijXLAV0k4vStL+L2M+5jxgtf8Nq973LmthfzdlJzPBeWhyzz8kQVXQjePWguAgFeCByH8h3S4b0Sm4tZdz5m1aGYDdchGbI1pekJ0nXQRRdajnYzDt8mL9iGvB2hxhDV/9QgIpw+5Xg8fk9LnLbhMPAWeDnvgf9QWBrIqmiYCZfbSd+BFaw/dG0eu+oZrj7iZp6/7TVql9axZMFSDj5nn7TwR7fXxXYHbsXsD39k/g+56Xf8+dMiztnxUgZuspZliF00EiMaibWELbYlEory7qMfcWj/E7hkn2szJx1lYMn8Si7e6+q044v/WMp6W67Dde9dxmuhx7nu3ctYY73W+PzX7n2XRb/83TJZmAmTSDDKradOJRLKPIGICC/d+SaH9T+B3ZyHcvT6p/Hx8591asy9FaXcGCWTUX0/QpU9ier7KUbhmR36ryXyMVJ1CIRe1IU8mh5EKvdC4haZyeZiMpaNM6uQxFIk9AoSeb9zCoo2ecMWQEjSVB/khuPvZOZLs8BQlPQp4ux7TmLKZ9fw1OQX+H32AtYfug6HTdiPNTfoz5AdN2Hq+Y/y5oPTSVhIyrbH4TTwFfrY6fBt2Wn0dhy38VnEInGdQPTsTO446wHcHjfKUAzeZgN+nPkLSini0Tgj9h7KOVNP5pwd/psW7piNeCyhJxyPM21VbiZM3p82g3jM2lUz/4c/ScS7nr6+8Oe/mP/jQgZuvCaJeIJJx0zhk+c/w+l2YiZMBg5ei2veuJjC0la1xg+f/tRaxsBh8NPn8xjyb2v97OdvfZX7L36yxRX296+LuXbMrfx32ni22nPLLr+H3oQySsGwfpprj4ggdZeQ6o6JgdQhDZNQpVNS2/bsiEQ+IjVDFa0nE/s+6XZxJZNSnVD2QHbJAJu8Y1cISjJh5OV893FqWJ/H7+bWGVezzubWGzqJeIKj1j6Vqr+q6ehjVApum3kNa28+kMNWP4HG2szqhA6ng4JSP9FInNUGVjD2ytFsvc8wDqwYS0NV7qpzhsPgmMsPw1/k486zH0zzvXsDHkr7FrN4QWXKOafbieEwOpw0PH63peEF8BV6GTPxUMKNYX6b/QefvfZVSiEMp9vJVntuyWXPtW6A/nffa5n5SrrWhzfg4aYPr2C9f62dds40TQ6qOI7GmvTPc90hg7jr6+uyvodVEUlU6rBFyzJtBqrfbJRqfYoTiSBVB0F8Pi3VmpQP3Dsn1RLbGXhViur7SaerDtl0TKbNzl7jWpn3ze9ccdiNHL/p2Uw65jb+nNsJIaAO+OvXf/h+xs9pFWdi4RhP3/BSxvscTgc3fvg/Nhi+nmWafFtE4M0Hp3PnWfdnNeKgJ4i6pQ2E6kP8PmcB/zvkBt544D0q1ii3vN7lcVm6edw+N4O33oB//lhiuYGaiCfY9eh/M2iTtfD43fiLfLh9LgpKAjmt/GPReJpgVTPhpggPXfYUD18+jY+f/zzFiIPevP3s1S9T/PP7nrq7pasnFo3Tfz3rMnihhhDhDJoyi37tvgTASonyY12ZHrS2eWqdT6U8qLJpug6nczNwb40qvi4p8GX12Udt3/kyplcY8m+mf8dZ213CR8/OZP4PC3nv8Y85ddgFnRaYysSSBZWWKemmKSz6xSLZog2rr92PKTOv4Yk/7+KSp87CcGb+SMNNEd56KPeNu2bi0Tg3nXQ383+yVlY88Kw92WDYuql6IkoXYFh78wEs+GGhpc/U5Xay9qYDuPub67n9i0mMv/cUHA5HS3RKR5hxE5fHhZU7VkwhkmVjFPTnGw23Thhb7Lyp5ZONYRi8fGf7IsIaX6EPX6F1PPwa66UXDbbRVYpwrp/hjMsyjwAAIABJREFUrCDNtUfb3WMUjMPo8yxG2UO6gr00Yi3mpZK65zbLil5hyG877d6UEDozYRJuCnPX+Ify0v6gTddKW42DNnSbbZ+bkOOfc//iurF3YmbwK7t9brY7cKus4YPZMOMmiah128/d8hoNNY2pvnqBhb/8zXEbn8VX786xjOZwuJyM2GcoAAM3XpNv3/+eqMXnkI14PMHD86bg8nZeFMnlcVFU1lrf8tdv/sBt0U4sEuODpz+1bMMwDMZcml5kwu11cewVh3d6TKsMgROx/vp7wWEdrdUe5d0TsJhEJQbu/BYTscnOCm/IY9EYf879y/LcT59lLvbQGUoqitn7pJF42xgDw1B4C7wceNZeiAjfffITr9/3Lt/PmGtpFG85ZWrGuHO3z8WIvYcyYu+hDMiSxt5VYuEYf/6Y7mqKhWPUVzZk3IwN1gdTfNxzv5iX08ZtWzYavh5zZ/1mqezYEZtss0HKa1+hL+MGa6A4c2mx/U8bxSk3HUOfNcp0lmnAQzyW4NL9J3P+blfw9++LOz22lR3l3U0LaLWXzVVulG/f3BrxjgLX5m3KvhmAF4ouyViA2qZnWOENucPpyKhQV1Cav/qUJ994LCffeAxrbtifkr5F7Hj4ttwxaxLegIfTtrqAC0ddxe1nPsAFu1/BaVtdSFN9a0heOBjJ6IJxOA3+98L5XPLk2RiGwRm365DG5jj0TsWj5zkjOhE3efjy1gIRgzYdYOnrd7gdHHLuvri97hafuOEw8AY8nHLzWKr/riEe7dwE4Pa6GTMxNdZ5wEZrUN4/XXrXG/Cw/2mjMrallGLwiA0YtvsWuDwuIqEoZsLETJh8894cztj6IkuN+FUZpVyo8ifBNQytrOgC58aossdRRlGObThRZQ+giieBdx/wHYEqfwrDf1iPjt0mnRV+W9kwDPY6aSSv3PlWSgidx+/h4LP3znrv3Fm/8sjl0/jjuz8ZuMlaHH3pIWw4PF2ZELQx2OvEkSmqeQA3jLuD32YvIN7GJfLb7Pncdc5DjL/3FEBnJzrdTssNwpK+JQwdOQTQm4tF5YVc8dL5vPXwB/wxZwEb/d96fDBtBg0WURdtcXmcKAXRcNdcM5n4/LWvOfWmsQAccu6+fDBtRkrSj9vrZvioLRh75eEUlRfw6j3vEGoIsdkOgxl75WhCjWFenfqOpWsqE0rBcVePZtPtNk45/t3HP1G1qCbt+j2O25lt9tOP6lV/1/Dt9O/wFfoYutsQ3J7/b++8w5yovj/83pkkk2QLvffeEaSD0psCwk9FFBtFUUEpXxQpioiIIoKgNBFQVBBRQCkiSpWO9N5772xLz/z+CKwsSXaz7C7Lsvd9Hp/HnU3unMmSM3dO+Rwj6+b/y/COY3DaXX5Dmb1eHXusg5Wz1vJY1yYh25gZEGp+RI4Z6N5owINQkq9fL4QBzC0Q5hapb6AkZO57Rw7QdXhHoi5Hs3L2OowmA26nm8dfacJT/wvuyHes2sOgVsNx2pzoOlw8eYkdK3czbMEAqjQKvcZ12czVCZw4+JKPK35aQ+vXmiEUQcmqxWjRuRFLvl2O05awfLFElaL0qjcIt9MdHyLyuD2UqlacjxcNIHvebMRF21g+c03A85vDzThtTrweHZPFCKnsyGNvVtBEX4th/8ZDtH6tGesXbuHc4fMYNCMtOzfi5Q+f4c1aAzh7+Dz2WAcGk8q/f26n5uMPM6HXt8nWo6napBJP9U74t9N1nREvfeVX724wqiiqghCCmcPn8ONHczCYVIQQqKrCsIUDGP3KpKBlkOBLMp8IEHqS+JBhkIxPhnDkRpORnhNepeIjZTl//BIN2tdJUn9jYp/vEny5fUMbnEzo/W3Ikqkej8evbO4WDpuTd5oOBV3HEmFm0E99uHHxBhsWbsGoGXHanRiMBrYt34UrwKSdPWsP0Kl0T0b/M5Soy4En0ysGheYvNWDJtytw2JzYon3hC6EIDEbVlzhNYRtA9PVYVv2yjpGdJyCEbweLrtPxvad4cXB7FEVh1ojfOH3wXPwTh9vpwe30MO7NKXeVvN2/6QhXLlxjz5r9rJy1jvPHL1KoTH6uXrju91q3y8PqORt59KnazBw+L36s3i0Gthrud6O9E0u4mRIPSXEnyYNLhmgI2r12P4NaDfd1pHl0PF4vT/dpTZePg2s6tzB2CFg7LYRgifvnkCQ49244SJ9H38PrSfozskZa+OnU18RF2zh/7CI7V+1hxsdzEuzQA6FZTGTPl5VzR/11LqyRFvKXyMvhbcf8fmc0G+k9qRtfdv8Gl8Md8FpDwWQxAsIvLCQUQa5COWn0bF02LNzKiT3+pY9CEX6hjFC5871C8YnqBaJQ2fyUrVmKpT+s8itPtISbcTpciSZpcxbIzvRDX2Eyp3wakESSnmTYhiCX08X7T3xKXJQNW7Qde5wDl93FvC//YPuK3UHfF5E9cCI0IntYyDrKLocLU4ijwHSvzuo5G8iZPzv5iudh7phFSTpx8GmeIESCEXC3cLs8XDgReCyewaiSs0AOCpUtkCKhKJfTHT+Q+HZ0r87FE5eYO2YRp4M0X6VEOOzOG0AwJ65ZNZ7o3hJbtC1o96wlLLi+txCCWq2qSScueaC57x35jpV7A+427bEOFk9dHvR97fv6i09pVo2n+rQJ+dzlapdGCVGi0+Xwlfq5nC561h3IjSuBwyV3ont13E4P5jAtQcWIyWzE6/ESfTV4S/64t6ZwdMeJu94Vg29aTmK7WZfDHbQkMDLIzTKlWCLMWMLNmMxG6j5RnTZvNKd++7qYwzS/19pi7MRGBU8U67rOrmQMs5ZIMiL3vSNPrBoiMb3w9m8/wRPdW6BZTFgizGgWE23eaM6z/duFfG6TZqTf9DfRLKb4Se0GzRBwJ6oaDVRuWIH18zf79FCS4VvDIi1M2DyChh3qkiN/NkpWLUZkjohEY7/FKhXm0qkrdx1SSQ2uX4xK9TUjsocxbMEAek3sxtfbP2fgzN6oqkr9p2tTrk5pzOG+3beiKr4bnwi+m79F9rzJr8YIlWsXrvPTJ3P5rNM4Fk3+W5Y5StKF+z7Z+VDDCgF3jOYwM407PhL0fYqi0O2zF3nxg/ZcOnWFXAWzJznaLBD12tVk0vaRfPbyOA5uPoLu8aIaVN84tpt2mcM0aj72MGWql2Dzku0J9ENCITx7GKf2n2HX6n1cPX+d2BtxiVZhAOxddzDZ15Ie1G9fh3+CdGXeiclipN2bj1G5vn83rWpQ+WTxINbP38za3zYRkS2MVbPXc/W8f4L0djSrxjNvh9jgkkwObztG34Yf4Ha5cdpdrJ6zgZnD5zJu06dky50lTc4pkQTivt+RWyMs9J78GiaLKT6ObA7TeKhRBer9X80k328JM/uGNtyFE7/F/PFLOLrzJB63F4/bi9vpRghBkfKFqFS/HD0nvMqgWb0BKFaxcMAQgFEzYtD84+BCEYRFWnm/7QgunbqCfrPuOb0GJBjNRhQ19TqPti3bxcRtIyhWqTBGzUiWnJFYI/3/FooqaPhMPZ5/7+mga6mqyiP/V4t3p79F9zFdyJ3IzE1zuBlzmEaXj5+lRsuqqXItd/JZp3HERdtw3qxKssc6uHLuGtMHz0qT80kkwbjvd+QATZ+vT9kaJfnr+1XEXIuhTpvqVGv+UPyk+rTk6rmr/D7+T78QhtvpJiyLhdErhyY4nrtITl8yVRAfXlGNKjkLZMftcnP59NUETtpoMnB427EUxblTC9Wg8vgrTVnx02pirsdhMhsxmgwYzUaunkt85xsMj9vDuB7TOHfsoi+P4HIHvNZsebPRd+obyfqbPtf///jk+bF+T0CKqqAogpHLhlC6Wom7sjspoq5GB5SO8Lg8rJ23id6TXkuT80okgbjvd+S3KFg6P12GPUfP8a9So2XVe+LEdV2nX/OPgsahzx1LWDK48Y+t9H7kPeKibQli5GVqlKTzx88xfPEgCpbJnyDm7rS7uHLWv5sxPfC4PSyY8CdRV2LQdR17nIPw7OEUKlvgrtd0uzwc3HIkXmo22A0rLiqOXf8knpR0u9z8OnoBncv24oXi3dm7/gAdBz2ZUPURn6iaLdrG6Fcm3bXdSWEwGghWRnM3AmISSUrIEDvy9GLP2v2cPx64/A+g1MPF4//f6/Uy6pWJAWPbe9cdYO/6A6BDniK5KFG1KAe3pI4Eb2rjvelobzncc0cucPHU5STfF6ym3O1wxa+ZFNduNgS5XW62/L2TmGuxVG5QPn6m59D2o9i6dGf8Zzx37CLyFM1NjnzZuHgyoY26Dif3n+Hq+Wtkzxva5JzkYI2wUKVRRbYt35WgqkezmHj8FSkFILm3SEeeCCf3n0UE8UGKQaHzbTKpF05cIvZGInop+n+vC1Ybfr/iCUEQK5AT16waRSoU5PCWo0k6c3uMg70bDmIOM/N51wm4HC50Xcft8vD0/1pT/+k6CZw4+EojL5+5ipZOO+B3vutB34YfcOWsb8arrkPl+uXo8G7olVESSWqQIR35lr938ONHv3Lu6AVKVSvByx8+Q8kq/mPAUkqR8gUJODVBwFO9W8ePHouNikNRlKBa5PcVt8Xu0xLFoKCqgifeaMFXPab4aajcia7rzBv7B/PG/uH3u3lj/8AeE7gSyB5jp2DpfNhi7PFJR/D92QqXLZDobvzA5iNMHfAjB7ccJWf+7Lzw/tM07FAvxCuE7HmzMXXvGHau2sv545coWbVomvw7lEiSIsM58hU/r2VU1wnxO7Or5zazbdkuRq/6MNUTW+XrlKZw+QIc23kioeaK7lNA3LN2P+N6TuPYrpMgQLNouF3Jmzh/OylpeQ8FzWqi4iPlOLnvNJdOX0lTh+51e4mLtjOh97e8+MHTfD/kF1SDihC+sEe2vFk5ezi0UWz2WAe7Vu8NKLFrMhup90QNDEYDJ/acwh7nwGzVMGoGBszsFXTNW6WDt3oRYq/H8XnXiVy7eIP/e+vxkK9TUZRkibBJJGlBhtBauYWu6zxboFvA2uGqTSrx2d+Dk1zD4/Gwf+Nh3C435WqXxqQl/lgeF23jk45j2LBoa4Ljt6bB354IVVQlRQ06Bcvkx+vxcP7oxZDjyqFiMBmo164GA2f2ZvbI35k6YGaqrh8MS7iZ3pO6UbVJJbb8vRPNqqF7vQzr8EWySiyFEIRnDyP2WmyCz8YcpvH94XFkyRXJ1qW7OPDvYXIXyskjT9VKtHV/UKvh/PvnNr98ZVgWK79enOpLZqaQw9uO8f2Hszmy/TgFy+TnxcHtqVivbIrXlWRegmmtZKgdefS1GKKCtKwf3Hwkyffv23iIwW1H4LA54vVWBvzYi9qtqwV9j2pQ2BmgxTtQ16XX48VkMeJxeYK2tSdGzLVYZp35mtGvTOKv71em6o7Z4/Lw0pAOKIoSdGxaWmCLsfPTp79RtUklmr5Qn70bDtKvyYfJrpPXdR1HrJPC5QtwfPdpwOfcvR4vm/7cRouXG1G9+UNUb/5QSOsd3HI0YNGJ2+Xh6vnr5C6UM1n23cneDQfp13QoTpvjpozyZfas2c/7v/Sl1uMPp2htieROMkz5IfgqBQKJSwFYI61cvxR8aLAt1s6AFsO4fvEGtmg7cVE24qJsDHt2dMCqjLNHzjN75O8M7ziWuBuhD5J1OdzUbl094OzJpMiRPxux1+M4tvtksp34rXLGYOi6To8a79KpTE8Ob/VXU0xLTu47Ta967+Fxe/hp+NwEA5eTg2pUuHruRvykJF3XcdpdfNVjCoe3J++a8hbLHfC4rutkyZlyfe6v+07HEedIcLNw2JyM7zktxWtLJHeSoRy5wWig9evN0Kz+SnZRl6PoWPgNJvT+NuBub/38zXgDiHJ4PV6W/vhPgmNzxy7k1Ur/49v3Z7Fu/r/JslH36mxYuBl3MmdfalaNNq83o2uFPhy6i9LEUEat2WODj6RLS7weL9cu3mB0t0nsWLUnqIphKOs44hx+NzmXw838CX8ma62XPmjv9+9Is/pKBzWLf2ducgkkPQxw4fjFZA+4lkiSIkM5coBXP32Bll0a++ZH3pb8cticuBwuFk9Zxh/fLPV7X/TVmICaLS6Hm6jL/4k/nTt2gakDZuK0u3zhk0ScjjlcCzhH0+P2Ji9WLiBnwewsnLw00aeKjIw9xs6yGauxRadMVCrQE5nX42Xx1OUMaj3cl8QNgRotq9L769fIlicLRs2AZtVo/XpzXh/1cvxN5+ncXXmu8Ov8+NEvuJzJc75ZcgWee6lZNYymDBXRlGQAMpwjVw0qb37Zle+PjgvY3WmPczBnzEK/41UaVwyoQ24ON1OteZX4n9fO2xRy/LZI+YJYI+5ewyUeHc4cPOcLeaR/p36akZhc7p2EZwtLeEDAK588H3wikQ6bl+ygZ92BIe94mz5fn1lnJjPr9GTmXf2W1z9/GafdSY8a7/LX9JXcuBzF5dNX+OnT3xjy5Och2w7w7LvtAsgom2jX87GQ9fAlklBJkSMXQowUQuwXQuwUQswTQqSdXuid6AQVdwo0yLhIuYI0ebF+AkErc5hGxXplqNascvyxxL5kqiHhx3Vg05Gg9c2SFCB8id8E6DD9g9m079smoCgZ+HbmsdfjWPfbppBPpSgKkTkiMJp8OY2lP672e3pz2pzsWLmboztPhLxumzda0P7tJ9CsWry2+mNdm/Dyh3LCvCT1Sekz3t/AAF3X3UKIEcAA4N2Um5U02fNmJTJnJJfveJRWVCV+av2d9J7YjZotq7J46jKcdhfNXmxA3mK56N/iI07sPUPhcgVo83rzgM7coBlu7pYThky8Xu89a7LJNAT5LG0xNgqVLcAHc95hQu9pnNrvL1pli7UHFLMKlb3rDmCP9b85CyE4vO0YxSuHNvtTCMHLQ56hQ7+2XD59hRz5s6VIgVMiSYwU7ch1Xf9L1/Vbz7obgIIpNyk0hBD8b/JraFYTys1BD0bNQFgWa4LW+TvfU69dTYYtGMBnfw8mR4HsDHjsY7Yt283Vc9fYvnw3n708jhadG8Ur/xlMBoyaAUuYOfighxCdeHg2691cquQmHreXiycvU735Q3Qd/jyWCP86cUu4mWKVCt/1OQqWyRdQ9EoogjxFg8vmBsNs1ShYOr904pI0JTWzLl2An4P9UgjRDegGULjw3X/RbqdGy6p8uW44v36xgDMHz1G5QXn+r+fjIYskTezzrZ/IlcPmZMeqvUzZ8wVr5m4CXWfVr+tDriSJzBlB1OXAY9705BWy3NeoBgWPx3tPn0QMRpUyNXzdu7VbVyNHvmyct1+MrxAyGFWy581GnTZ+/RIhU75OGb+pVEIR5CyQI+DAC4nkfiDJHbkQYqkQYneA/9re9ppBgBuYEWwdXdcn67peXdf16rlyJX9nEwyTxUTOAjko+XAxKjeoQNZkTGY5HmAyPMDJvafJWzQ37fu2odnLDTi89VjIVSjRV2MCxu5Vg5rsyof7GY/bS7Y89y4lAlC0YmGqNqkE+D7PsWs/psnzj2IJ9834bNzxUUYuG8yS71byUYfRTHp7OqeTUW7p8Xj49MWvAt6cugx7TiYpJfctSe7IdV1vmtjvhRCdgNZAEz0N+/3tcQ5W/LSGvesOULBsflp0asTmv3bwRbevb3ZSevhr+iqqNKrIkLlvo6qJN8gAhGUJI/Z6YMXC2BtxhGcNY84XCwNOmQ+G7tUDblI1qylJ4aiMhFEzUKVhBdYv2Bwwppza52rVrRldP3k+QaVSZI4I3p7Wg7en9QB84mVv1RrApdNXsMc6UI0qCyf9xeBf3qbmY0lPCdq3/iC2GP/mL92rs/LntTzyZK3UuyiJJBVJUWhFCNES6Ac00HX97tWikuD6pRv0qNmfqMvR2GMdmCwmZgybg9vlwXWb4p09xs725btY99u/PPpU7STXLVwmP/s2HvI7btQMbF6ynVqtqzE3gBrf3eBxe6jd6mH+XbID5x0OXVEUX9I0AxGeNYzuYztz8eRlDm45Gh+OuCWKFQyjZvDV2Xv/C8sYTCoelzdo2efjrzShx9guSdo0d8wiLpy4FK+C6HF58Lg8fNZpHD+fnZzkzd0W6wi66465kWb/vCWSFJPSOvJxQATwtxBiuxAiTUayTBv0E1fOXovf+TltTmzR9oDJR3usb+ceCsUfClyBYDAZEIrCmUPnMBgDf/mT85StGhTCIq2oRgMlqxbDYDJgCTfHDyEYsfT90Be7D8hbIjcTNo8ga64stHqtKV7vf08siTlx1aDQ6aNnmbRtZLyMQXjWMNq99TjPDXwycLWQUSVH/uwh2bXql3UJpGxv4bQ5Obn3dJLvr1C3TJBB31qy5G0lkntNinbkuq6XTC1DEmPtb5sCfsGCSb6aLP4t/Hdy5dw1ilcugsli8tshez1earSsgiPOEbRSJSJH8KTm7Siqgterc/1yFP/8sh5zmEaRcgXoO607RcoVxGT22RqRPZzoIIJg9xv5i+clZ4Ec6LrOpD7T8bhCe5rwuL1MG/gTP3z4C93Hduaj3/+rVHXYnfz21R/ERSUMbbhdHvauP4iu60nGqINVhng8XszhwZUQb2GNsNDjqy6Mf2saLqcbr8eLOUyjeOUiNHn+kRCuUCJJHzJEZ2dyWpo1q8ZjXYOP2nI5XQx/fiwvlujBtEE/4Xa6EYpAUQWqUcVgMtD/h55YIyxky5OVGi2r+glgaVYT7d5sGbQx5RbWSAsmsxHdq8cPnbDHOjh96By71+yPd+IAX6z+CMMd16moCibL/Tf/8ej248wa8RvLZ64JqkYZDI/bgz3Wwfie0zjw7+H445P7Tg+4mwbYvmI321fsjv/ZaXcyd+wietTsT5/677N85mp0XadtD/+/iaIICpXJT75ieUKy77EuTRi77mNadWtK/fZ16DP5dUat/DC+YUgiuR/JEHrk04f8zOzPfk/wRVcNKiWrFuXk/rOA7tMG9+o83ac1XT7uGHStr9/5nvkTlvjHqVWB16tjtmiUqVmS4YsHYdKM2GLtjH5lImt/24SiqmgWI93HdqHRs/X45PmxbFi4xTfFPcDHKFQBXgLGfsvVLsXIZR8QeyMOzaoRFmnl6vlr/D7+TzYt2srVizewxdhwOxPmAe4LBKiqgmpUcdruzjahCJq+UJ+n+rRmZOfxHNl+PNHXt33zMd78sgset4fej7zHsd0n40tHzWEaDZ6pS98pb/DVm1P489sV8SGxyBwRfL58CHmLBlY7lEgyEsH0yDOEI3c6XAx+4lN2rz2AwOcgs+fNxuhVH2KJsLBx4Rbiomw83Kxy0C+sx+3hp0/nMX1w0FL3eDSLiU4fPcvT/2sTfyz2RizR12LJVShHfNJM13X2bTzE1r93cvbIeVb+vM6vBjkUVINKteaV6ffdmxzYdJj3245I0YCKjELW3Fm4celGkmqIqkGlQ7+2dB72HKvnbOCzzuOxxyQU3zKZjUzcOpLCZQtw/vhF9m04RPa8WalUv1xATR6JJCOSoR35LQ5uOcLhbcfJWyw3VRpVSNYX9POuE1j589qAU+4DUbRiIb7ZOTrk9X/86FemD/n5rhtkVINC0YqFOXPoXJqX890PqAYFXdfxepL+wEwWE5O2fkahMgUY8/rXLJrsr26pWTW6f9GJx19NtFpWIsnQPBATgkpXK3FXczmvnr/G8plr7mq3HCpFyhfEEmbGdsdOUTWqWMLMxASpV7+Fx+3l1P4zQePEDxImsxGXwx2SyqRRM/DWuK4UKlMA8A3fMJoMfiqIiqokqxlMInmQyBTPnKcOnE3WxB7NYqL5yw2TfJ3H7SHqSjRer5c6T1QPuBn3uj10G/li6MY+oCiKQtY8WShVrTgvD302SSduMBl45p22zD43hZadG8cfb9GpEYoh8BDmGo9V8TsukWQGMtSO/G7JXyJvUI3q3EVyEn01Ft3rxRHnRAvTKF2tOG3ffCzoel6vl+lDZjNvzCLcLjeWCAsvDn46fiL77eg6/DraXx892LoGkyG4ONc9RjWpKELgcty9PYpBYcbxieS8oxZ8xU9rgk7RyZIzgql7x5Alp/9whtyFc/HBnHf45PmxuF0edK+XbHmyMPT3/rKyRJJpyVAx8pQw9JlRbFy0NUG1itmqMWHLCHLkz86q2eu4cvYaFeqVoUoj3xAKXdc5vucUtmgbJR8ujknzOYrpQ2bzy+fzEzjuxBxwWFYrjjhnkg46S64ILBEWLhy/FLRG/l7z+YoP+PjZMVy7cHeTi8rWKMFXGz/1O753w0HebTYUp931X2JXQL12NXnzq67sWXuAuWMWcuNKNHWfqE6Hfu0SOHaP28Ph7ccxmY0UrVBI6qBIMgUPRLIzJTgdLr7p9wOLp/i0yItVLkzP8a9SoW6ZgK8/c/gc77X+lMtnrvhGyunQ6aMOlKtThneaDEnWQInS1YpzbM+pkMoIDUYVhLhvduUlqxblqT5tGNl5fKKVNEIRCMB72w0oa+5Ivtk1mqy5AseuTx88yy+j5nN0xwlKVy/B033bkK9YHqYOnMEvoxYkaAKLzBHO9EPjCM8aFnAtiSQzkOkd+S10Xcfj9mAwBo8qeTweOhZ+g2vnr/vFcjVL8sWvek3sxtIf/2H/xoN43P85Q6H4dv0ploJNw8EWQoh4wa/EnhJMFiMd+rVj27JduBwumr7YgLY9Wia6U75w4hJf9viGzUt2oKgKjz5Zi5eGPEPXCn0C3jRadmlE3yndU+W6JJKMiHTkIbJj1R4+emYUNy4l3X5/J0IRfs7OZDbx66WpuOwuhrYfxb4NB1GNanzoJqXDiOGmNrg7/erOTRYjddrU4L1ZfUJ+jy3Gxsul3uL6pagEn5nBqMbri9+JNdLC79e/T7G9EklG5YEoP0xrLp2+wnutP7mrOm6T2UiZGiU5tPUo9lgHBpOKoqq8M607ljAzljAzny8fwuUzV4i6EkOhsvl5q/bAJDsak0IoIkE4Iz3QdXjh/aeS9Z4Vs9Zhi7H73fhhrT/aAAAcDElEQVSCOXEgTctHJZKMjHTkt7F46rJkTXq/Re7CORkwoxcV6pZh+4rdbFi4hfCsYTR9sT75iuUh6ko0mxZvQ1EVaj1elZwFcgDw/HtP81H7z5PsbAyGwahijbQQdSVxvZNATwqpidvp5vshsxn8y9shv+f4npPJvmEWKlsguaZJJJkC6chv48LxS36NJqFw43IUJrMRIQRVG1eiauNKXDhxib+mr2TXP3vZtWa/T/hLgNft5d0fevLok7UoWDofQlHQk9mOX7h8Qf43+TWy58vGx8+OIerK4URff7dOvHS14hzdeSLRXfKt9bct28XV89dCHrNXvHJRTBZjsrRaOg58MuTXBmPT4m1M6f8jZw+fJ0/R3HQd3pG6bWukeF2JJD3J0A1BsVFxTB/yM10r9KZHjXf589sVKRrQUKVRxaCKhoGaUG7htLtY+sM/8T+vX7CZruV7M+PjOWxbvhu3040txo4t2o7D5uTTF7/k+qUbXD59BaGEXjZ3S8e8//dvUaFuWfIVy0ONx6okq9kpVIQiGPRzH8KyhDYwOjbKxgvFejBt0EwObzvGgkl/sW7+v7hd/90YnXYnv41bTK96g1j64z8YtdDtVg1KisNQGxZuYejTn3Ns10kcNicn951m+PNjWPnz2hStK5GkNxk22em0O3mjWj/OHb0YHzvVrBoNO9Slz+TXOPDvEdB1ytQoiWpIeuxbgjWPXYwvFdSsGnWeqE79p+vw7aCZnD54zq+SRQho9Vozek3ohtPhon2ern662rdj1AyEZbFy/WJUkjaFZwmjQ/+2HPj3CEUqFKJ1t6bxoRmAqCvRdHuoL1GXoxM8TRg1Q4oaeaxZrCiKIC7aFi/BGxLC18VpMKqoRt+N54t/hpK7cE56P/Iex/ecite70aymkLVvAB55qhYfJCN8cyddyvXi1IGzfsfzFMnFj8cm3PW6tzi26wQLJy/l+oXr1G5dnQYd6sb3HkgkqcEDl+xcMWstF09eTpAAc8Q5WDbjH9bP/xe30xcOUI0qg3/pS5VGFZNc02Q28eX64fwyagErZ61Fs5ho80YLHnulMaqqUunRsjxf5A0/PRSj2USD9nVxu3yx4qRivy6HOyQnXrBMfj77+31yFcwZ8Pcx12P5pv+PxN6Iw+PxYo20ULRCIWq0rEKhMgVY8t1KNi/ZHpKmyZ3E3e1oM903mMPp8YLdhT3WztBnRtHhnXac2Hs6geNOjhPXrBqVHil3dzbd5OyRCwGPXzhxCa/XmyKVxGUzV/PFq5PiB1JsWryNeV/9wRf/DEWzJK5bL5GklAwbWtm6dGdAh+l2eoi6EkNctI24aBvRV2N4/4lPuX4ptM7EsEgrnT7swHcHvuTr7Z/T+rVm8bK1WXNloeMg/zit2+nGHudgYKvhzBu7KEUStEIRKAYFc5iZq2ev8XrVfuxavc/vdV6vl/81GMzSH/7BHuvA6/HiiHNw6fQVnnmnLQ2eqcurn72AZjUlayxdaqN7dU7uO8PquRvuWtVRNaiEZ7XSonOjFNmSs0DgkXHZ8mRJkRN32ByMeX0yDpsz/m9vj3Vwct8Zlny78q7XlUhCJcM68tyFcwWdp3knXq/OylnrUuW8J/ae9nOMXo+XkZ3Hs2/DoUTVC4UQvuadAJjDzbTs6hOH8rq92GPtxEXbiLoSzaBWw4m98Z964voFm3nj4X4c330qQQeox+0l+los//y6AfApMnYc+BTh2cPv8mpTB0UIwrOG+Tpk70BN4m8YlsVC804NmbB5BGGRocXrg/HSh8+gWROOATRbNV4Y3D5F6+7fdBglQK7DEeeQ8XfJPSHDOvJW3Zqi3tmdGcRJOu1OblxJOpQRCluX7gpYLhh9NcZv2MHtFCpbgLY9WlKxXtmAO2R7jJ0/py4PWGGi6zqr52wEYObwOQzvOIajO08EDJnYY+wcvpkUHN5xDDOHzyE6ifJES4QZa5bA8y4Tw2AyoBiS3u4bNAN/T18Z8EklPGsYj7/aJOjfTteh5/hXQq6GSYzmLzXk9dGdyJIrEtWgEJkjnC7DO9Lm9eYpWtcSbg5aGRRqslgiSQkZ1pHnLZqbD+e9Q9bcWTCHmzFZTBQolS/g4GWzVaNq40qpct7wIF9M3av7zdy8hVAFleuXo8eXXegxtgumZMZM3S430VdjiLkey4xhcxINUZjDzRQtX5AD/x5m46KtSYYzHn+1Cb9dm077vk+ENLT6FgbNgDlMw2BIPM2iKAJ7jCNwCaOA10e9TJ+vX6d09cA683FRNjqX6cmRHcdDti0xWndrxi/npzDv2nR+vTiN/+v5eIoFt0o9XJzInBF+N2hzmMYT3VukaG2JJBQyrCMHqNbsIX4+O5lxG4Yzdc8XfLf/S+q0qZ6ghNAcplGlcUUqPZqyRNktKj5SNuBxRRVBY9G6R2fJdysBKFm1GCP+ep9ytUth1AxEhBD2UA0qVZtU4vC2Y4mW7CmqgiVMo0GHumxfsSekqhWn3YWiKHTo15aHGpQPuSSwaIXC2KJtSQ7C8N7UtgnGheOXAAiLCP5EcP74Jfo2/IDoa8kb9BwMIQSWMHOqKSYKIfh40UCy5c2GNcKCNcKCUTPyZO9W1GhZNVXOIZEkRoatWrmFoigUKV8o/ueBM3uxctZaFk/zhSladGpE446PpNqXNiJHRGA7VJUmL9Tnz6nLA/7e7XQTdTWayOwRVKhbhi/XDQdg7thFTOzzXdDzmSxGHnmyNiWrFuPEvtMJ6rLv5OEmleg1qRuWMDOROcIxaoZEnSjAqtnr6TnhVSxhZob/MYgDmw/zyfNfcvFE4s1Rx3adCEnf5ZamTCA0i0ZkTt/n2eaNFuzbeCjoE4Tb5eav71ZQrHJRjCYD5euWjk9C3w8UKVeQmScnsmPlXqKvRFPx0XLkyJfycJBEEgoZ3pHfiaIoNO74KI07Ppom6xcuWwBzmObncIyakUfa1WTr0p1cPHHZ3y5VYf38zbTo1IhLp6/gsDkpUDIvpasVT/R8b0/tQYNn6uByuji+6yRhWcJwOVwJZl1qVhMj/nqfCnX/e1qo/3RtJv7vuySvR1EFGxZsodGz9QAoU70kk3eOYsGEP1kw6W/OHDoX8H2hSBkYNQNer47HG/i1QhE0eKYOAI88WYud/+xlwcS/At58HHFOpvSf4Qv/6GA0G/lofn/K1SqVpB33ClVVebhJ6oTwJJLk8MA58rSmYYe6TBs4E6fNGS9WpRpUsuaKpHrLKjzcpBJ/Tlvh9z6DycDVc9foXuNdTuw5hVAEEdnC6f9DT6yRVuKi/Ou2cxbMTr12Nbh+8QZv1RlI9JUYbDF2XzeowBceUARvftU1gRMHCMsSxieL32PIkyOxxdgDTi8Cn4OcM2Yh1kgLf3+/itgbscRG2Tiy/ThKIk8xQghMZmOikr5Z82QlImsYJ/ed9ouRR2QPZ8jcd4jMHhG/Xo+xXahUvzyfPj824NOA2+XB7brZaBVtY0DLYcw6MxmzVdZpSzI3GbazMz05d/QCo7tNYueqvQghqPlYVfpMfo1sebKyd8NB+jX50M/BGTUDkTkiuHb+egK1QnOYxsAZvRjafrRf2ESzmjAYDZSoUpQ9a/cnCGUoqkKVxhUZtiDxEWcej4eDm4+yZ90Bpgz4EY/Tf7erqAKhKMkSDIvIEU7FemXZ/NcO3C43usf/31HpGiX5ZPFARr8yiY1/bMXr8ZKnaC46DniSZi83CBga0XWdfk2HsnfDwfhpToqqBKx4sUSY6fvNGzR4pm7IdkskGZkHrrMzPclXPA8jl36A2+VGCJFAAqB87dK07NqYP6etwGlzoqgKqlGleaeGLPtxtZ/krMft4dC2Y3yzezTf9PuB9fP/jS9vdMQ5ceBk56q9fjZ4PV52rtqT5JxKVVUpV6sUZWuWZMfKPWxY4H8D9Xp08CRP9bF87dI8P+gptq/YHXTy0ZFtxzCaDAyZ+w4OmwO3001YlsQn/Agh+PiPgfw6aj6Lpy7H4/YSmTOCIwHme3pv1s1LJJkd6chTQLApQ29+2ZWmL9RnzdyNGDUjDZ+tx/6Nh1j6/Sq/17ocbs4fv0jBUvnIkT9bosnBO0nOMAmP20PeYrlCfn1iqAaFas0f4v22IxIfjCFA3OyY1CxayK3qJs1Ix4FP0XGgT+N84x9bGfbsF351+rquU7VJ0tILEsmDjnTkaUTZmqUoWzNhIi7QAAhzmEbVRr4EmS3GnuwhETHXY0OaYznqlYmsmr0+WWsHQ7NqFCpTgOirwcsBhSKoXL98qsSva7SsQoW6Zdizdn98ktkcptGyS2MKlMyX4vUlkoyOdOQp5PieU5w9cp5ilQqTr1ieoK8rUq4g9drWYN38zfGJR6NmwBxmZt+mQ5jMRuq0qc7ymWtC1moxmAzYYuxJOvI5XyxIILObHFSDiqIq6LqOoghyF8nFwBm9uH7xRqIDonVdT7XyO0VR+HjhAJb/tIblM1dj1Iw8/kpTarV6OFXWl0gyOqmS7BRC9AU+B3Lpuu5fe3cHGT3ZCT4t9PfbfMrBLUdQDSpup5s6T1Sn/w89g4ZcPB4Pi6csZ8HEJcRcj+Xq+esIReCyuzCHm8mSK4JLJ6/4OXJxM0Rx5/FchXIw4/jERGvkd6/ZxzvNhuIO0hxkCTfz1vhXGN9rGrpXR/fqOOIceHUd1aBSr11Neo5/JV7VL3chnxLj9pV7eKfxkEQ/I5PFxKStn1GojJzsI5GkBmmW7BRCFAKaAydTulZG4svuU9i/6VCC7skNC7Ywa8RvvPDe0wHfo6oqrV9rRqtuTelSrleCHa09xu4raQywGxeqgmY24XF7cNpdqAYFg8nI21O7J9noNO/LP4I6cYDS1UvQ7MUGNOxQl52r9uJ2eajcoDwmzehTYgyiCpirYHYMJkOiu3IBbFu2O1UdudfrxRZtwxxuvq8agiSS9CQ1QitfAP2A31NhrQyB2+Vm9a/r/WqdHTYnCyYuCerIb3H5zFUunvR/cAkWUjFpRjoN7YA9zsH2FXsoWCof7Xo+TuHbZljGXI9l1oh5/PPrBjSLT+Pj8VebcPVC4vK9VRv7koVGk5FqzR5K9LW3U6BkPko9XJyDm48E7R5VbgpTpRaLvvmbbwf9ROyNODSrRod32/Hsu+1SrWtXIsmopMiRCyHaAmd0Xd+R1JdJCNEN6AZQuHDhlJw23XE53XiCOF1bTNKa26pBCTpwWQgC/q7Zyw2JyBYeX8lxOw6bgzdr9ufiqcvxTwhfv/09u9fsp07rauzfcDCgaJVBM1CrVbUk7Q3G0N/78cH/jeTQlqMBJ9wrikLtNn5PgXfF8pmrmdhnenx+wX0jjhnD5qCoCh3eaZsq55BIMipJimYJIZYKIXYH+K8tMBAYHMqJdF2frOt6dV3Xq+fKlTplcOmFJcxM0QqF/I4riqBa08pJvj973mwUq1jIb16nZjXRonNjrJGW+P/Cs4bx0fz+RGQLvrNdPnMNV85dSxDmccQ5WDN3A1WbVCJnoRx+71EMCg/VL0/JqsWStDcYWXNlYeyaYUzbN4bXPn8JS4TPZkuEmay5s/DpkvdSrevy+yGz/bpTHXEOZn0y764mIEkkDxJ3newUQlQClgG3essLAmeBmrqun0/svQ9CsnPfxkP0u5lEdLvcmMxGNKvG+E2fkq948OqVW5w9cp4+9Qdji7HjdrpRDQoV65Vl6Px30XXYvWY/iiKo+EjZoMnTW3zywliWz1zjd/xWIrPuE9X5eeTv/PHNMmKvx2KJsPBE9xZ0HPRUqs6UdDpc7NtwEKPJQJmaJVM1ht3K+jxOu78cgKIqLIj+AZM5dAleiSSjkurJTl3XdwG5bzvBcaB6KFUrDwLlapXim52j+G3cYo7vPkX5OqVp80YLsuXOEtL785fIy4zjE9i4aCuXTl+hbM2SlKlRMj7emxzxpbxFcwdOPArfeLOwLGF0GdaRLsM6hrzm3WDSjDzUoEKarF24XH4Obzvudzxb7iwhS+9KJA8qqaa1khxHfj/uyD0eD4qiZMjE2cWTl+haoU8CRUZFVchdKCfTD3+VonmUoWCLtfPrqAUsm7Eag1GlVbemtHmjRZJPEslh69KdDG47IoGGjWY10Wfy6zRJI6VLieR+I9iOPNOLZu1avY+v3pzC8d0nMYeZafNGczoPey5VndC9YMfKPYx46Suirkbj9eiUqFKU93/uQ+7CaZuPcLvcvFlrAKf2n4kfMqFZNao0qsCwBQNS9Vzblu9iSv8ZnNp/hjxFc9Fp6LPUa1czVc8hkdzPSEcegGO7TvBWnUEJkmiaxUT99nXo992b6WjZ3aHrOueOXsBkMZEz/38T41fP3cgvn//O9YtRVG/xEB0HPknOAv4J0Lth9ZwNjOw8HtsdOiiaVWPUyg8pE2SEm0QiST7BHHmGHvWWUmZ+Ms8vgeawOVk5ex3XLyVef53aRF2NZseqPZw5HHiQQygIIchfIm8CJ/7TJ3P57KWv2LfhEOeOXuCPb5bxetV3uHr+WmqYza41+/ycOIDu9bJvw8EEx84eOc/YNybzVu0BjH1jcoquVSKR/EfGih+kMsd3nQw4/dykGTl/7CJZc4WWuEwJuq7z7Xs/8evohZjMRlxON2Wql+DD3/olWnIYCnHRNmYMm5Mgruxxe4iLsvHr6IV0++zFlJpP7kI5MVlM8drht1CNKjluu6Ec2nqU/zX8AJfd5ZPu3XqMpTNWM2rFEEpXk7t2iSQlZOodealqxVFU/4/A5XCRv2Tee2LDip/WMO/LP3A5XMTeiMNpc7Jv4yE+ffHLFK99fPdJDCb/e7XL6Wbr0p0pXh+g6Yv1Ue/4DIUQaBaN2q3/E7Ua99ZU7DH2+C5Qj9uDPcbOuLempoodEklmJlM78mf7/x8mc8LSNc2q0bxTo/gRZCll69KdDHvuCwa3G8Gq2evw3DHA4ZdR8/3mf7qdbrYt20XUlegUnTt7vmwBOy6FgDxFkp8E3fzXDga2+pju1d/lh6G/EH0thqy5sjDi78HkLZYbzWrCZDZSrFJhvvhnaIKhF/s3HQ64ZrDjEokkdDJ1aKVw2QJ8vuJDJvb+lgP/HiE8m5X/69WKDv1Sp+X7m/4/Mn/8n/GOetuyXfz9wyqG/v5ufElg1JXAmt6qqhJzPZbIHHd/Q8lbNDflapdmz7oDCWrMTRaNZ5LZ1j778/n8MGQ29puJ4RN7T7Hk2xV8vX0k5WqV4vvD4zh39AIGoxqwUsYSbib2hv9cUku4OZlXJZFI7iRT78gBylQvwZg1w1js+Ilfzk+l44AnU6Uj8dyxC/z25R8Jdtv2WAc7Vu1ly9//hTVqtKySYFTcLbQwjTxFU146+MGct6naqCJGzYgl3Ex41jB6f92NCnXLhLxG7I1Ypg/+Od6JAzjtLq5duM78CUuA/xKtwcodW7/eHM2SsPvSZDHR+vXmd3FVEonkdjL1jjwt2bZ0V/yYs9uxx9hZv2AzNVpUAeCF959mzdyNxEXbcDncCMU3nb7XxG6pckOJyBbO8MWDuHbxBtFXY8hfIk+ya+QPbT2GUTP4Vfg47S42LNrKcwOeTHKNTkM7cPHkZdbM3YjJbMRpd1G3bQ06De2QLFskEok/0pGnEdZIC4rq3yVqMKpEZPtvok/OAjn4ZvcXzBu7iG3Ld5OvWG6e7tsm1Ss5suXOErJ8wJ1kyRWJJ4B6ohCQI39oU4AMRgMDZ/Ti8pkrnDl0ngKl8qZaLbtEktmRjjyNqN2mesB2f9Wg0vzlhgmOZcudhS4fp60OSkooVrEw+Uvm5fieUwk0000WE0/2apWstXIWyCEduESSymT6GHlaYbZqDP9jEBHZw+MlaTWrxv+mvJ4hBwZ//MdASjxUBM1qwhppwRxupvuYzlSsVza9TZNIMj2ZukX/XuB2udn5zz5cDheVG5THEnb/V2kc2XGcCb2/Zd+Gg1gjLLR98zE6DnwS1aBy+uBZoq5EU6JKUTRL6miNSySS0JBaK5KQOHf0Aq9VeTtB271mNdGwQz3ento9HS2TSCRSa0USErNH/u6vPxPnZPnMNVy7cD2drJJIJIkhHbkkAQe3HMXj9p9HajIbOXXgbDpYJJFIkkI6ckkCilcqHFx/pkTSI+wkEsm9RzpySQLav9PWb46nyWKizhPVZdmgRHKfIh35fcy5oxf4bvAsxnb/hvULNvsJbqUFhcsWYMTf71OyajGEAHOYRuvXmtFv+ltpfm6JRHJ3yKqV+5TVczYw4qWv8Lg9uF0ezOFmytUqyfA/Bt2zMXQZeY6pRPIgIqtWMhBOu5ORXcbjsDlx32yNt8fY2bfhEMtnrrlndqiqKp24RJIBkI78PmTPugMBHag91sGyGavTwSKJRHI/Ix35fYhRMxIs4qVZTYF/IZFIMi3Skd+HlKtdyk+7G3yJx8dfaZoOFkkkkvsZ6cjvQ1RVZdiC/oRnDcMa4RPbMpmNPNa1CbVaPZz0AhKJJFMhZWzvU8rUKMmsM1+zcdFWYq7FUqVxRfKXuDcDoSUSScZCOvL7GM2iUf/pOulthkQiuc+RoRWJRCLJ4EhHLpFIJBkc6cglEokkgyMduUQikWRwpCOXSCSSDE66iGYJIS4BJ+75if8jJ3A5Hc+fXmTG686M1wzyuh9Uiui6nuvOg+niyNMbIcTmQApiDzqZ8boz4zWDvO70tuNeI0MrEolEksGRjlwikUgyOJnVkU9ObwPSicx43ZnxmkFed6YiU8bIJRKJ5EEis+7IJRKJ5IFBOnKJRCLJ4GR6Ry6E6CuE0IUQOdPblrRGCDFSCLFfCLFTCDFPCJE1vW1KS4QQLYUQB4QQh4UQ/dPbnnuBEKKQEGKFEGKvEGKPEKJXett0rxBCqEKIbUKIhelty70mUztyIUQhoDlwMr1tuUf8DVTUdb0ycBAYkM72pBlCCBUYDzwGlAeeE0KUT1+r7gluoK+u6+WB2kCPTHLdAL2AfeltRHqQqR058AXQD8gUGV9d1//Sdd1988cNQMH0tCeNqQkc1nX9qK7rTmAW0DadbUpzdF0/p+v61pv/H43PsRVIX6vSHiFEQaAVMCW9bUkPMq0jF0K0Bc7our4jvW1JJ7oAi9PbiDSkAHDqtp9Pkwkc2u0IIYoCVYGN6WvJPWEMvk2ZN70NSQ8e6AlBQoilQKD5aIOAgfjCKg8UiV2zruu/33zNIHyP4DPupW2Se4cQIhyYA/TWdT0qve1JS4QQrYGLuq5vEUI0TG970oMH2pHruh5w5LwQohJQDNghhABfiGGrEKKmruvn76GJqU6wa76FEKIT0Bpooj/YTQRngEK3/Vzw5rEHHiGEEZ8Tn6Hr+tz0tuceUA94QgjxOGAGIoUQP+q6/kI623XPkA1BgBDiOFBd1/UHWTUNIURLYDTQQNf1S+ltT1oihDDgS+g2wefA/wU66rq+J10NS2OEb2cyHbiq63rv9LbnXnNzR/62ruut09uWe0mmjZFnUsYBEcDfQojtQohJ6W1QWnEzqfsmsARfwm/2g+7Eb1IPeBFofPNvvP3mTlXyACN35BKJRJLBkTtyiUQiyeBIRy6RSCQZHOnIJRKJJIMjHblEIpFkcKQjl0gkkgyOdOQSiUSSwZGOXCKRSDI4/w9ZjbdXY9JNkgAAAABJRU5ErkJggg==\n" + }, + "metadata": { + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "code", + "source": [ + "epochs = 10\n", + "batch_size = 10\n", + "learning_rate = 1e-1\n" + ], + "metadata": { + "id": "Kchi-W9kxZaY" + }, + "execution_count": 21, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "history = []\n", + "import time\n", + "\n", + "for i in range(epochs):\n", + " for x, y_true in loader(X, Y, batch_size):\n", + " # forward -- считаем все значения до функции потерь\n", + " y_pred = model.forward(x)\n", + " loss = criterion.forward(y_pred, y_true)\n", + " \n", + " #print(y_pred, y_true)\n", + " #print('SUM OF SQUARES:', np.mean(np.power(y_pred-y_true, 2)))\n", + " \n", + " # backward -- считаем все градиенты в обратном порядке\n", + " grad = criterion.backward(y_pred, y_true)\n", + " model.backward(x, grad)\n", + " #time.sleep(1)\n", + " # обновляем веса\n", + " SGD(model.parameters(),\n", + " model.grad_parameters(),\n", + " learning_rate)\n", + " \n", + " #print(model.layers[0].W[0][0])\n", + " #print(loss)\n", + " \n", + " history.append(loss)\n", + "\n", + " \n", + "plt.title(\"Training loss\")\n", + "plt.xlabel(\"iteration\")\n", + "plt.ylabel(\"loss\")\n", + "plt.plot(history, 'b')\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 295 + }, + "id": "C6e3QHvUxaqA", + "outputId": "e8878109-714e-4765-a3ed-a91a9b3cce17" + }, + "execution_count": 22, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2de/gcRZWw35OE+x2SVS7RhCWKUdZbuKmLKIrAsgRXVFC5CCzoJ8oKqKAIiKIgiyi74BoNiBBugkhAIHKRW0BIghAIEPIjEEhITEggJARCLuf7o3v8TSY9M909Vd3VM+d9nnlmpru66nRVdZ2uOqeqRFUxDMMwjEYGlC2AYRiGESamIAzDMIxETEEYhmEYiZiCMAzDMBIxBWEYhmEkYgrCMAzDSMQUhGE0QURuEZHDXYfNKMOeIjLbdbyGkYZBZQtgGC4RkaV1fzcElgOr4v/Hquq4tHGp6r4+whpGVTAFYXQVqrpx7beIPAccraq3N4YTkUGqurJI2QyjatgQk9ET1IZqROQ7IjIPuEREthCRm0RkgYi8HP/eru6au0Tk6Pj3ESJyn4j8dxz2WRHZN2fY4SJyj4gsEZHbReRCEbk85X28K07rFRGZJiIH1J3bT0SeiOOdIyInxccHx/f2iogsEpF7RcSefaMtVkmMXuKtwJbA24FjiOr/JfH/twGvA//b4vpdgenAYOCnwFgRkRxhrwAeArYCzgAOTSO8iKwD3Aj8Gfgn4OvAOBF5ZxxkLNEw2ibAe4A74+MnArOBIcBbgO8CtsaO0RZTEEYvsRo4XVWXq+rrqrpQVa9T1WWqugQ4C/hoi+tnqeqvVXUVcCmwNVGDmzqsiLwN2Bk4TVXfVNX7gPEp5d8N2Bg4O772TuAm4JD4/ApgpIhsqqovq+rDdce3Bt6uqitU9V61RdiMFJiCMHqJBar6Ru2PiGwoIr8SkVki8ipwD7C5iAxscv282g9VXRb/3Dhj2G2ARXXHAF5IKf82wAuqurru2Cxg2/j3Z4D9gFkicreI7B4fPxfoA/4sIjNF5OSU6Rk9jikIo5dofGs+EXgnsKuqbgrsER9vNmzkgrnAliKyYd2xoSmvfREY2mA/eBswB0BVJ6nqaKLhpz8C18THl6jqiaq6PXAAcIKI7NXhfRg9gCkIo5fZhMju8IqIbAmc7jtBVZ0FTAbOEJF147f8f095+YPAMuDbIrKOiOwZX3tVHNcXRWQzVV0BvEo0pIaI7C8iO8Q2kMVEbr+rk5MwjH5MQRi9zM+BDYCXgL8CtxaU7heB3YGFwI+Aq4nma7REVd8kUgj7Esl8EXCYqj4VBzkUeC4eLvtKnA7ACOB2YCnwAHCRqv7F2d0YXYuYrcowykVErgaeUlXvPRjDyIL1IAyjYERkZxH5ZxEZICL7AKOJbAaGERQ2k9owiuetwB+I5kHMBr6qqn8rVyTDWBsbYjIMwzASsSEmwzAMIxGvQ0zx+OovgIHAb1T17IbzexB5kvwLcLCqXhsffx/wS2BTIpe8s1T16lZpDR48WIcNG+b8HgzDMLqZKVOmvKSqQ5LOeVMQ8WzUC4FPEo2zThKR8ar6RF2w54EjgJMaLl9G5L43Q0S2AaaIyARVfaVZesOGDWPy5MlO78EwDKPbEZFZzc757EHsAvSp6sxYiKuIvDX+oSBU9bn43BqTdlT16brfL4rIfKKFxpoqCMMwDMMtPm0Q27LmGjOz6V8zJjUisguwLvBMwrljRGSyiExesGBBbkENwzCMtQnaSC0iWwOXAV9uWKAMAFUdo6qjVHXUkCGJQ2iGYRhGTnwqiDmsuQjZdvGxVIjIpsCfgO+p6l8dy2YYhmG0waeCmASMiHfPWhc4mJTr3sfhrwd+V/NsMgzDMIrFm4KI9/s9DpgAPAlco6rTROTM2jaJ8ZIDs4HPAr8SkWnx5Z8jWnr5CBF5JP68z5eshmEYxtp0zUzqUaNGqbm5GoZhZENEpqjqqKRzQRupjWqjChMnRt9ls2AB3HtvvmvvuANmzHArTyvmzYO/Ora6LVsGy9suKG4Ya2IKIlBU4bDD4M4724cNlXHj4CMfgSuuKFsS+PCHYY892odL4hOfgHe8w608rdhpJ9h99/bhsrDRRvDe97qNsyweewx22w2WLs123Rln5H9JKIIrroC5c8uWYk1MQQTMZZfBXhXeGLL21t3XV64cUGwPoFNeeslPvNOn+4m3aL79bXjwweyN/Q9+kP8lwTcvvwxf/CLsu2/ZkqyJKQjDMIySWbky+p6TeiJAMZiCCJQQxu0Nw+htTEEYhmEYiZiCiFm9Gv74x3De3EORwzCM3sUURMyYMfDpT8PFF5ctiWEYRhiYgoiZPTv6fvHFcuUwqsXKlfD662VLkY2nnoJnny1bimzcfTe8853F5/X8+fDAA37TmDYt3BEDUxANiJQtQUQIFeaNN2DUqGiymyuWLYvyeNw4d3GWyX77wYYb+ol7yhQ/8b7rXbD99n7i9sV//Rc8/XSk3Ipk553hQx/yF/+DD8J73gP/93/+0ugEUxANhNAwZ2HJEpg500/cTz4ZNVLHHecuzpob3xlnuIuzTG67zV/ct97qL24jHc8/7zf+2rP7yCN+08mLKQhHLFsWvZndd1+x6e6xB/zzPxebplEMVXtZKQrLl+IwBdFA3iGmxx6Lur8nnuhGjrQPQahvHoZhVB9TEDH2VuIPy9t8hGIPCw3Ll+IwBdFAKJXPGlXDMMrGFESMNcj+CEXpgpWzYWSh5xXESy/BW94CU6eWLUlvYA20YVSHnlcQq1dHk2FWrChbkjWxhtQPlq+GkZ6eVxC14Q9rOPxheZsPyzejbExBSOv/ZWGNgx8sXw0jPT2vIGpYw+GPeqUbigI2DKM9Pa8grMHqLexFwDDSYwrCsQ0itHgMwzDyYgoiUBtEN1Gv7EzxGUY/oT8PPa8gauQpqCVLIjfZelwpmNArTlWxfDWM9HhVECKyj4hMF5E+ETk54fweIvKwiKwUkYMazh0uIjPiz+H+ZGz9vxmvvAKbbgqnneZepm7DemWGkUzoz4Y3BSEiA4ELgX2BkcAhIjKyIdjzwBHAFQ3XbgmcDuwK7AKcLiJb+JEz+q69WaZ9w1y4MPq+8kr3Mhn+sB6EYaTHZw9iF6BPVWeq6pvAVcDo+gCq+pyqTgUaBmr4FHCbqi5S1ZeB24B9fAgZqgbv1oYs1Pw2DGNtfCqIbYEX6v7Pjo85u1ZEjhGRySIyecGCBbkFXTNOJ9F4ZeHCaPvFqhCSkbrs9A2jSlTaSK2qY1R1lKqOGjJkSK44qujmOnJktIG7YRjdQagvLj4VxBxgaN3/7eJjvq/NRBXdXOfPLy/t555b23OrHVXIU8Mog1AVQw2fCmISMEJEhovIusDBwPiU104A9haRLWLj9N7xMedY45We6dNh+HD4yU/KliQ/oT+QhhES3hSEqq4EjiNq2J8ErlHVaSJypogcACAiO4vIbOCzwK9EZFp87SLgh0RKZhJwZnzMG6E1HKHJA/D889H3XXe5jffnP4d11nEbp2FUgdBfUAf5jFxVbwZubjh2Wt3vSUTDR0nXXgxc7FM+yF9AITbgodIur775zWLkACs3w8hCpY3ULgh1P4jQ5KmnE9lCf2NyyQ9/CDfcULYUBoT9PIWM1x5EFaiikbpq1Odp2Q9qkenXZtnnTbNTWZ99NnIq+NjH+o+tWtVZnIZbyn4e2tHzCqJG6AVlGFnZfvvou75un3JKMWlPnw4bbghDh7YPa4RLzysI1z2GIhfr6zalpmo9uHp85MUEL76Aa7PjjtF3KHU0FDkaCXWIu4bZIAKdKJeGqgwXhFT5Q5LFCJMzz4SddlrzmI96s3AhrFzpPl6XWA8iUBtEmgpZdOXynTfWgzB8kaWBP/10f3LUUIXBg+FDH/KfVif0fA+iRhXfLKvSg0hq9MtSBFUq5yrJ2u34Kov77/cTryt6XkGEPgbYirIUhK+8qmIZGEYeqlLXTUEEOqQR4hBTN1CVB9Nwi5V7PkxBBKog0lCVIaakhzPtMcMIAdd1syp1vecVRI1agYWiMNJUoKooiJCoyoMJ1ZI1dELPy1Dl63kF4coGUYZiqR9iCrWCQXojdcj3YPQ21oPoUUJ1c02D9SAMw/BJzysIX/z61zB2bP7re3GIqYi3qqq8uRluCa3cQ5OnGaYgOqTZENUxx8DRR/f/v+wydyt71tIseojJlkY3epVercM9P5MaoobPdwU47LDoO206rcINGBD1HqwHEWYarqiSrKETWl6GJk8zrAdBMQrCJQMHRt/1CiJk+c0gHQ6W7/no1XwzBcGaDVgVjNQ1BdFtE+WsB2H0ClWph6YgAqVVBapaDyKJKihio3sI/fkIVT5TEFRviCmk9aOee669HM3OP/88/O1v7cMZRtnYPIgexoWCcP1GHPKGQbV0p0yB4cPhoovyxfP2t8MHPuBOrjRU5cGEaskaOpaX+TAFQfVsEEmU4eY6fXr0fd996a8r+0GdO7fc9I1qYj2IHqcqBVZPVWR24drril128Z9GyFT1BcgoB1MQhPnQZG0sq6IsymbZsrIlaE39BjJWpu7oNC999yBCLWtTEORTECEUaAgyQLb8ayVzKPdTJrNmlS1BP+PGwbx5/uL/znfC31Gt1/GqIERkHxGZLiJ9InJywvn1ROTq+PyDIjIsPr6OiFwqIo+JyJMicopfOZufe/ZZuOCC9nGUPUZpjWv3UWaZvvQSfOlLsP/+/tL46U/hwx/2F/+iRdDXF/0O7fkITZ5meFMQIjIQuBDYFxgJHCIiIxuCHQW8rKo7AOcD58THPwusp6o7AR8Ejq0pD1802w/i4x+H44+HV17xmXo+qlLJ0hqpq3I/PgklD1asiL7nzClXjk7YcUcYMcJNXKGUS9H47EHsAvSp6kxVfRO4ChjdEGY0cGn8+1pgLxERQIGNRGQQsAHwJvCqL0FbubnWFENVK8iqVbD55nDJJeXJUNW8K5sQbWNVYsGCsiVww513wsUXl5O2TwWxLfBC3f/Z8bHEMKq6ElgMbEWkLF4D5gLPA/+tqosaExCRY0RksohMXtBBbQjxQcw6D6JZ+GXLYPFi+MY33MjVKq1Or62yInEle4h1sRsI3Ujdir32gqOOcpt+WkI1Uu8CrAK2AYYDJ4rI9o2BVHWMqo5S1VFDhgzJnVj9Q7l0aedxFEWnlfaWW+C889zIYhhG9+FTQcwBhtb93y4+lhgmHk7aDFgIfAG4VVVXqOp8YCIwyqOs/2hsv//9bOF9UYSRer/94KSTsl/XSZrd2oPwwerVZUvQPYRWt8zNFSYBI0RkuIisCxwMjG8IMx44PP59EHCnqirRsNLHAURkI2A34Clfgnby9l/mkEColaqRXhg2cVUWVSnTXqNXy8WbgohtCscBE4AngWtUdZqInCkiB8TBxgJbiUgfcAJQc4W9ENhYRKYRKZpLVHWqL1nzrMUUWoUJTZ56QppJbRghUJW67nVHOVW9Gbi54dhpdb/fIHJpbbxuadJxX7R6wy17QbxOwxSBq4lyVaZb78sXRedXaEbqqhCqkTp4erXC1FP25EDDqCpVqeumIGg9xBTy+HkaN1eXuMiLqjwYNcqUt2p5lYW899bNeRIipiBorSBcHc9Kq3iKUlo+ffufecZfemVQZdl7gdCGmKpSX0xBEHYvoRVF9yCapdUu/5LkrIoLZ9EPclXrYlaq0kD2OqYg2tDsgfU9JBXCA2TeR+kxN9dsFD3EFHoPItRyNwVBtDDZkiWdxVFGAZfVg/BFN9yDYXQTXt1cq8KSJe0VRFU0fiOhGa+rkm81zEjth6oZqX33IEIdWrQeREraVZAyjFhpehC+FtbLQtrKX+UGscqy9wKhl0+o8vW8gsi7z0OoBeqTvG85VR4Kq5q8VaHX87Uq99/zCuLNN9c+tmTJ2nsXp+1BzJzpRi5XM6l99iB61fUviZUro703QqfKeQzdY6SuCj2vIJLeijfdFIYPj35nHbpZuNCNXGkI5c28m20Qadl2W9h667KlqA7dWg/SUpX773kjdbPGbf78Nf9XpUAbqZLcIcqaViYfW9KGmB+u6HUjddHx56XnexDtqCmQdl5MoRqpfcuQhlA9NIzeIbQGODR5mtHzCiLLLOAq4lP+PBPpWl0TYl6bm6sfqtaD6FVMQeR0waxKDyIEN1fDP6++Ch/8IDz+uJv4uq3sQzNSVyV/TUG0URDNhpjKpGxvGV9KJ6Q8rhp33AEPP5x+y9yysbKuBqYgUg4xqcKxx8JDD615vDFcEbz2Wvo0i+hBdLONoRuGmNLG873vwX/+p9u0m1G1ISbrQRgtWbwYxoyBT36ymPSyVqCQG7J6BWI9iPQUrXh//GP4zW+i391WFt12P0XR8woi6xBTSENOvnsQjbiYSV01Qla8RcfjEutBuI3PF6YgHHkxlV2BynRzdTVRrioPjU9CyYNQ5HBFnvspMg9CzW9TEI68mMpg1qz2YcqQ88Ybo3x9/fXi0zbWJoS62kiSTDfcAFdcUbwsZRBimSRhM6kdrTRaRoF/61vt0y/DzfXss6PvBQuKSc8nRcuU1l7jmzLSPvDA6PsLX2gepqy3+hDrZhH0fA8iLbUKsnRp5E4YgpfHihXu0qt5RqWh3rMrDaE0er1KiHletEyh5UFo8jSj5xVE1h7EqlXw9a83P9/4u2zSyPLoo7DxxnD11fnS6ObZ6FWWvRNCeAFyeV2naalGKzx/4Qswb577tEJyfqnHFIQHzxwXhVzkWkwPPxx933prdhmyYkbq4rF8dZMHV14Zfb73PffypJFv7lxYtKjztLPgVUGIyD4iMl1E+kTk5ITz64nI1fH5B0VkWN25fxGRB0Rkmog8JiLr+5DRlZdSnsb6iSeSwz70EPzP/6SLox0uXWGbhVu1Khp6M8InlN39qqC0ypCxVflssw1stVVxskBKBSEix4vIphIxVkQeFpG921wzELgQ2BcYCRwiIiMbgh0FvKyqOwDnA+fE1w4CLge+oqrvBvYEHI6497N6devzaVdzzRInwC23wLvfDZdfvva5XXfNriBcVObGyplWKd54I2yySbo0qtaDKHOsvMx5ECGWBXTvPIhQ8zttD+JIVX0V2BvYAjgUOLvNNbsAfao6U1XfBK4CRjeEGQ1cGv++FthLRCROZ6qqPgqgqgtV1csKRCtXtj6f1SCbNuwTT0TfjzySPt48uJS72flXX02fhmFAccOwnaRXlUbcJ2kVRO3dcj/gMlWdVnesGdsCL9T9nx0fSwyjqiuBxcBWwDsAFZEJcW/l24lCiRwjIpNFZPKCrD6VMWne9pPCtao8aeN0SUhurqGn95e/pPfa6oZGwXoQ4aXr43n1QVoFMUVE/kykICaIyCaAz2ZwEPAR4Ivx96dFZK/GQKo6RlVHqeqoIUOG5EoobWPerOCSxgxDK+S0FOGNFELefPzj8OUvly1Fb1PFHkQ3L0rZjLQK4ijgZGBnVV0GrAO0e8TmAEPr/m8XH0sME9sdNgMWEvU27lHVl+L0bgY+kFLWTHRLD6JZmiE0yGnxJev48WsfmzbNT1qdEopXXahG6irV51Y0Uz6hKaG0CmJ3YLqqviIiXwJOJRoOasUkYISIDBeRdYGDgcZHdTxwePz7IOBOVVVgArCTiGwYK46PAk+klDUTeXoQ99/fb0NI4tOf7kymVlx/ffLxIlxSs6YRSu9qdKPlKwUvvQTLl3eHkdrIR2NZ+CyPUMs6rYL4JbBMRN4LnAg8A/yu1QWxTeE4osb+SeAaVZ0mImeKyAFxsLHAViLSB5xA1EtBVV8GfkakZB4BHlbVP2W6s5Ssu266cI0FeOSRzc/fdltnMrXiP/4jW3gXbq7z5hXzFur6Ibn++v45HlnTGjIE9t03f9pFP/Ct3jzz9GhD7UHkjSO0Bjg0eZqRdi2mlaqqIjIa+F9VHSsiR7W7SFVvJhoeqj92Wt3vN4DPNrn2ciJXV69svnm6cGUMG2XBhXzN3FznzYOLL4Zhw9Y8nociH4ysyrSRv/zFjRxFUJUGp1PKNFIXMfwTWjmm7UEsEZFTiNxb/yQiA4jsEJUnbaG3a4BDK9garuS6665qvFWmxfeEsZDqg/UgwrPDVMWFNq2C+DywnGg+xDwig/O53qQqkG5xd3RhpC5iSCj0fGzFWWeVLUE/kyfDAw/0/3f9dhuqgiizB+Ez7UobqWOlMA7YTET2B95Q1ZY2iKrw97+nCxd6w+bbSC3ixkjdiuuvh3POyXZNXvIsL3Lqqe7jzxvHzjvDhz7kNj1jTawHkdIGISKfI+ox3EU0Qe5/RORbqnqtR9kKIa+bayNlF7CLiTetGvQ8bzZZ8+SEE6Lv73wne1rdiKs6lXWIacQIGD7cTdrNqMIQUyOhvd0XQVoj9feI5kDMBxCRIcDtRMtjVJpBKXPAlwJYuNBNPCG6uaaJs1sJyYupnjRy9fVFnxApMl99url220zqATXlELMww7VBk1ZB+OpBXHop3HtvvmvTbMTjws3V9XVVIvR7rNnQWsnpwsNt+fLO46jHRb5ed12x6dXopZ5E2kb+1nhdpCNE5AjgTzS4r1aVdVL6YvlsKPK+rbmeVNUrS22AnxnLecJ1yuzZ/tO4/npYf/1oY6myqc/X3/2us/ufPRtOPDGdg4fr8syydE+ZpDVSfwsYA/xL/Bmjql0xUuxKQXRSgSZOhGs7HKzz0WVtNFI30q4yh1bZ68nb0J/dbg3jNqR1ishKq7zutHG76aboe/LkzuKpx1WD20kP+bDD4Gc/i1ZGcJ1mPYsXRw4FM2Zkuy4EUg8Tqep1qnpC/Gmy4EP1SKsgfE6UGzsWPps4XTA9vsc0672YanmRR2mG0oPIy29+ky5c0n1eeim89a0waVL+OPKELTvPly9fe4iqzCVMatT2dPf97Nx0U+SSfMYZbuIrkpYj8CKyBEjKJgFUVTf1IlWBhDDEVDZpH5Bnn42+n3vOqzhdy113Rd+PPx65qRZF2XV3ww2jPc8XL46GU1esgM02yxeXj7k3WXshLnrGzRxAyi6rRloqCFVNuU9YdXHVgyijYOsrahGrudb2w007ubAbqIoBP+8QU+0tOu1zkIfVq/s3lRoxIvouwnZST1IeFGFzqzpp3Vy7loED04U7ea0dtcvH5fCRz/h9x2nkH2IaPDgq/1decS+TD/LUnw9+EDbYwF26Pp01QrPb9byCaFUgF1zQX5BTp7aOp+yGrwgbRBHXhEZVehB5KWu72CLzJ++2vlUpQ590xVyGTmjViB1/fH8XPM/1RdKJET3Ng+BqJrU9dOkpayZ1yNx0U3ojvy+SVj2+4AKYPz85fBJVeQ56XkG0IzS/96x0stSGT9febqeovKmlE8qLSlry5s8vfuFu9YF2sqQdVpo2LXqZPOQQePppeOgh97KUhSmICvP66/2/2xmpV67sPD0Xb6KhPABVV/yN+J5J7ZoQ8tXFZMm774addop+L1wI73wn7Lpr8nXLlvU/sy6WsZk+Ha68Mvt1WTAF0Qbf+wYUxRtv9P++4441N8OpyrowVcLyrvqk6UHUz4tpV+Z//GP7XQqzKK0dd4QvfCF9+Dz0vJG6CIpoLLK4uX7iE8lhXa/mGvKwR1q30NAb+jT7CIR4DyHKlJZmeZ2mp3b33dF3Ve7fehBdQicV7vTT28eR9FB060zqUGYld8tM6qqS5kUhzVykNIT6MmU9CEeE+oCmSXvevLWPjRmz9vpQvdLQlLkoYa/MPQlRpk5xodBDyxdTEAVQRKG7TuPYY9uHWbYse7yhPAB5jdShvulVbYipCmR1//btal4GNsTkiLJ7EC7eSLLaII4/Pn3cVaLMHkTedMquf1kpcwJnI1nXYmpGFgXRzIsptBcQUxBtCPHhSiLELmseu0UIFGWkVoXnn/cXf8i4ytdO4nGxFlOaTbuqjCmINqTdSavsN7jGNFasaD8LPAv1y33nlSkkfA/JpInjvPPg7W+PJlolyeWq9xfiPAhXFNlTa7fgXyc9iNB6DjVMQbShKsMNjWlsuilsvbXbtF98sfM4QlEavhR6X1/6LWRrc1HaLZ9+0UXwmc+0DuMrX0OP13cPLwvdNJG0hlcFISL7iMh0EekTkbXWQxWR9UTk6vj8gyIyrOH820RkqYic5FNO3xSpIKZOhd//PpoYl3VJglZvMcuXR55NRfGlL8FVV2W/7nOfg0suyZ/u/fevuXNa1rIbMQL22GPN6xYt6l8qPYl2w4Nf+xr84Q/JYdK8eXZS/958E1at6jweXxQpU7tdFTvxYqrlcWh4UxAiMhC4ENgXGAkcIiIjG4IdBbysqjsA5wPnNJz/GXCLLxldkrZybLFF/jRaVaLa28t73xs1klnlakfRwxTjxkVr29SoDfW99BIsWdL8ut//Ho48Mn+6H/4wfOQjzc/nyc+ttoo+jbi00fgaolCFv/0t+j1litt409LqZaeT+j1nTuvzWW1RLryYFi3qn0wXAj57ELsAfao6U1XfBK4CRjeEGQ1cGv++FthLJKrqInIg8CwwjYpTX7k6WXf/pIr1o1w1gBMmwPrrR4ugDRkCw4evHWbVqkh5uCak+QxlpVdbP6h+7a9OySLvqafCCy90Hk8jfX3ZwrezQbiaBxHS1qQ+FcS2QH2xzo6PJYZR1ZXAYmArEdkY+A7wA4/yOaUII3WzYYZWlOlO6CrtW2+NvidOjL6T3iZPPDFSHt2ED/fLIoyhrjcfSprIWQRl9CBCI1Qj9RnA+aq6tFUgETlGRCaLyOQFCxbkSiitl1InuCr8ViuyFrHlqAt8yZNVea5YES1aWOPQQ9PZLjpZvNGHK3KI/vP1Q4PNCMlInTaOdjaITryYOsHnM+5zJvUcYGjd/+3iY0lhZovIIGAzYCGwK3CQiPwU2BxYLSJvqOr/1l+sqmOAMQCjRo3KlU2LF+e5am2aFdLYsdGqiy5wsWR3kRTZCGRtIGfOjBYtrMV9+eXRJ0/aeXE5yauVnEXbj2bOdBtfWbPEm01ma4arfM7jTu7rBcGngpgEjBCR4USK4GCgcXHa8cDhwAPAQcCdqqrAv9YCiMgZwNJG5eCKtHtS5+Xoo93FlcZI3UhIM1bB3wM9IGNf2HfPqt14dRryeNRXYMEAABitSURBVCi5mPxVBCH1ILKmU5/H1oPIiaquFJHjgAnAQOBiVZ0mImcCk1V1PDAWuExE+oBFREqkUFwpiCIqaisF4SL9kIYpaowZAzNmtA/n04vHN81kz5N2lqGtbqCMeRDNjpc1IbGSCgJAVW8Gbm44dlrd7zeAz7aJ4wwvwsVkffMsg1dfhQULWg8xhbjUhou00ywaGCJFjY1nDROKsnDh8ZM1nk5oNozjyovJhR3KBz2/mqvvISYX7L47PPFE5OrZjGaVZP/9/ciUF1+Vuajejw/5s8ZZP/O68dqk2e4hutyWNV7fSdx5bBBnngn33Zc9rSyYgvBIFYaYnngi+s5jpHaxPEYV8NUTLHMexO9/D//2b2seu+EGOPDA5uk89FC0DPuGG7qVxTXdYoOoJ0lB1DbjahVfqzjT4DMPKjDA4pdOG5Y333QjRxryDDG5IgQjtestUUOkPn/mz4fDDlvz/MMPNw9f47zzmofpxK07bdkV2YNwQdbh2WbKoqxegCkIj3S64mmt0S57LZUQ3hLLfOiLMlJ3Mg/CB0nptHppefnlztO79dbO769qPQhoX/bd6MXU8wpiacupeOn5+tfdxJOXEBTEhAlr/k96YHy9KVWhB9FM2bQyIrtwW60vh073Tf71r2HffeGyy5qHS7MkRzs5Jk2CW1KswlaUDUK1veuyq7qdZx6EL8wG4cgGMX++m3jy4qKSPP1083NpelpFDrc1UgUF4YOs5d5pL2/WrOi72dpIafn731uf32WX6Lvd/ZXxYpTFBtGMqngx9XwPwoVxs3HMtx0+CtTF8E5tf4Ik8gzFFdloN6ZV9hBG3olyjft8u+hBhOjFlGUFg2ee6SytvNSv3tPsDd/VntSdYArCIy56EFlXWVV1t8RHyNQq7u239+9f3UmD1qqxbFT0IQy55eGrXy1bgmKoL59rrmkd9q9/TRePa97//mzhXc3tyIopCI+UMVFu9er+NfbraTbOWXUmToQLLkgfPk8eVKEH0UiSDaJZmHbptIojJI+hGvXyfv7zbuIpIw5XPYhGGebN6x/OKxNTEB3mwBtvZL+mWYX813/NL0+IiiXvsg8uFERasqY1c6bbfRGy0mzsOqTyr9pEueeeay9P3nkQaeJLYuutYdgwN3F1gimIDnMgzyY1q1cn72lQ2+8gDyE0EK4Mb3kaj7w9iDwyfutbfuJNwqcNIqtSDaGO+eDoo+Hcc/Nd69vO9uij7cOYgvBIGUNMqvDAA27jLHIYQTXZnz6Ni2ZRPYg0D1aatJLO5/Xgqe1L3Yn9qZkibHUfnTYgPhRJSD0IiGagF5FOs/iaxX9wiuVLTUF4pCwbRNWor4RLl8KWW2a7Jm9aaWlsxBpnHLtMK2+8tSVTpk6NvtPYINKm41NBhOp1B93Tqwl1HkTPK4gy/OdDfuA6SctV7yBP/uT1Ysoy9pw1bhfXuTBSuzLCunxWWsk0p3FbsZzxFEFRtq9G7rnHXVyt6HkFUcbbfBV7EHkosyEte0mMLPH62MY0ryxF0UqmESPcxOMSV3YcV9TbME1BeKSMbTx9FGgIjUDNTbfm2dVutmyreLJSpJE6L502Jj6HkVzG26ksWbzEyq73ZfUgfMXViCmIEhSEjx6Eqt+eSdpKeOml/eP/d96ZLx4XE+XqZ8HmSSvt+WZ85Svtw6RpXBrX+snj5uq7EX3ySTjooGxLrZQ9TyW0dLLaIDqxWWTBFESX9CCmT/e7omzaSuvCO8tFD+LUU9On5cMoOH589viSXJ/bLQYXwhDT0UfDdddFC+ylTa/s1Y9d0UzJ77ln6+usB1EROl3uOw8+3vRPOqkaD10nDZov437Rb9/NGvrBg1tft3p1OjfXKuzP7aMH8cwz4Qy13X13/rTy9Cx90fMKYsiQ4tP8znf8xJtGQfg0HE+cmDyslJU8CrSTGehFOQ20evCb5W99D7fVEFP9vJS0PvZp8TEPwrWCmDQJdtgBLrqos/iyjii4skG4cHX2Qc8riDI8isaO9RNvmnsZPdpP2gA//zn09bUO46vxyLPkSS2trOnNmBHtieCSZmVXP1O/1RDTjBlu5clDlnx0rSBqS9Xff3/+uObOhXXW6VymTglpHkTP7wfRTS6naXoQN96YL+4ivUXyGKl9KYikc9OnR588aTU7nmZuiWo6b63GMJ3W8XZln+ctOsSJcjNnZk8nbw/ikUfSxZ9GHlMQHuk1BZGXEBSEr7SKdpVMauTbLXldC9fsWBENhkh/3KeeCptsAt/4Rv/5iRNho42S5WskRAVRZFpf+5q7uGyIySPdpCBqSzn4wEUl3GijMLxu6gnBSH3PPXDooe2vW7QonzydbjPajB/+cM3/J58MH/jAmjJ+8IPuZaqnG91cs6ZrCsIjZU+0cUknq8HWk2Wpgyw07pbWjGaNR6thnU6M70XVgWazvdMu3jd0KLz66prHivRiyhN30r4n4E9BFL3trY+8DskG0fMKopt6EK7uZdQoN/EkccIJ7cM0u49f/tKtLFCsF5MLmimIVnQy/JTH26bMeRDXXpvfk87XpEkXaTfSFW6uIrKPiEwXkT4ROTnh/HoicnV8/kERGRYf/6SITBGRx+Lvj/uSsUqNQztc3cu8eWsfc1UJ0yyVXfTS5UUPMXWShq8GO0/aWamfc+RapvoGs9UWpZ2kk8YhwAVp4izKSO1NQYjIQOBCYF9gJHCIiIxsCHYU8LKq7gCcD5wTH38J+HdV3Qk4HLjMl5zdpCB8VpQQVov1QRlG6iQZOr22fviu0dXY5TyIrI1XPeuu2/+7ajaI+rTKWqCvGZVUEMAuQJ+qzlTVN4GrgEYv/NHApfHva4G9RERU9W+q+mJ8fBqwgYis50PIblIQeXa3aySNsdQ3ecok70PSzkjtkzwNTTM31/q5NY22mjT3t3x5uvTTvEUXOcRUlA2m2Rt7nrTa2Una5d8998CXv5w+fCf4VBDbAvUDCrPjY4lhVHUlsBjYqiHMZ4CHVXWtKiwix4jIZBGZvCDt6mwNdJOCaDdJLQ2XX558vGjPoqx0MqN1/vzW513TeH9Z0njttfbXzp695v/6IZclS5LjdfFykQWfPQhfddVVvJ/9bGdxf//7a9qiqqogOkZE3k007HRs0nlVHaOqo1R11JCca2Z0k4IoY+FBHxTZg1CF/fbLd60rssh+113tr21cX6xeKXSat67ezqs8xFRPnvzIspBjGqqqIOYAQ+v+bxcfSwwjIoOAzYCF8f/tgOuBw1T1GV9CdpOC+POfy5bADXkeoE4URKtlKsq2TzTSOGM8Sb7GF4VO7mHlymiL2Vo8rozioRjO08ZVf+8huLa6urYdPhXEJGCEiAwXkXWBg4HGR388kREa4CDgTlVVEdkc+BNwsqo68u5PppsUhE+KzKfzzisurXb3de+97tKqHy+fOzea+OYDlw3G0qWRrK5xbYMo0nBc1qQ5V+Gz4G2pDVVdKSLHAROAgcDFqjpNRM4EJqvqeGAscJmI9AGLiJQIwHHADsBpInJafGxvVW0xWpwPUxDpCO1NupFOehCtePHF1uezUBvnV4X3vKdfQfh+e0xTx9M2sPXhmu38VvbsX1+z34tyc81KJRUEgKreDNzccOy0ut9vAGuZbFT1R8CPfMpWo5cURKjd2EZCehtsnJjWCTXvFdU1ew+d5O3q1fm3dq0nbaNeH+6112DWrHzpue5BFEGoSm3cONh0UzjxRHfy1AjaSF0EpiDSUfTchFDSaubVleZaVzK0YvVqOPLIzmVIU77NDOJ53FxDM1KnmUWdNJzl42Um60rBt98ezSD3gSmIHlIQzz6b/9rQexBZNrqvp919vfJK83N5vcY6ycvG9axWr17bdVUVbrklW3ppn4M0ZZNX2eQhTzzNhg3T7CPemKYPBZG09WwrVq2CgQPdywGmIHpKQeywQ/5rQ7dB5C3H//f/8qeZVUHUJqNlWS6jsQFqHJppdm29666riWtJ4/DTpq0dLs2ijI37IXz/+83TbCdTVh58cO1jIu1XQw51iMkUhEd6SUF0Quj51GwCWDtuvTV/mlkVRK0RdtnQuHpbTxNPUpgf/zhfeo35/qMmFkcfCiIPU6cmDzGF8OI0ebK/OVCmIAJv+EIhhAehGffdl19BtKPVEELjhLS8cbbzv29FUv1Nu2xGu3iSZHG9W12aNNudv+02uO46v3Ice+zasvzud/mN9I10+ny5dKaox3aUMwWRiqLX2W/H6tUwIH69OeuscmRwZYPotHFovD6PXGmeg0WLYPvts1/XCe3iV4W99177WCs6WZal/vvww5uHLRpfnn8934MYPLhsCapBJ5vB+6C+ESyrdxOCgkjTQD/9dPswad6Eb7997WO+10LKo4Cq+NLXLM/S5qXZIDzxlreULYHRSJrucv3wTp4hlbS0ekDzDjFloV1jl6YxvOGG9mH22ad9mGZDTD7nrbQznpexWJ+P+23mWZX2Xnz18HteQRjh0cq1tEYRjTO0fkDz9iA6Wc21kbTrI7miMS1fO8PVaJfHSffuqwfhM5+bubamTTOvm3c7et4GYVSTEFauLWMehGv7RVYa3Vp9D+e0exFI2vP6jjv8bJsbsqOG2SAMo476hsNnI9Uq7ry9GNc2iCLH3BtX2l29GiZ6XE5z8eLW508/fe1jEyfCAQf4kScEkoZUB3hqyU1BGJXknHhz2nvu8efiB62HUPIqiMYG/Q9/yBcPwJ13rr0EeJGsWuV3mKnVZk5FU0YPIinNrEtxdIIpCPxpX8Mfv/hF9P3Rj649K7co8hoGXdogfvtbeOqp/Nd3iu/eS5pZ2UXx2GPR97RpsNdexaSZ1sbiqw0zGwSw8cZ+30KN7qS2kU5WOlkTKwlfBso0+DZS++odbbpp9mvGjYu+i3T5TlIGRbZV9u5M2MYnI1xOOSXfdVOnupWjTHz3IFwsZZ6Er5n39Xz3u53HkVZOs0F4pCiXScMtl15abvqTJpWbfgj4VhAuljJvJEnmG290n85PftJ5HL57aO0wBUG+7qZRPkccUbYExuzZZUuQnaRGN9SXxLTDmObm6pH99y9bAsMwiiKEOTRpSSurKQiPDBlStgSGYRRFqL2FTsi6yVBaTEEA73tf2RIYhlEUtTk03YQvzyZTEMBuu5UtgWEYRdFu57gqYkNMHhk2DE44oWwpDMMwwsIURMynPlW2BIZhGPnwNaHQFETM3ntHO2ZttVXZkhiGYYSBKYg6ttgCvv3tsqUwDMPIhq8Ji14VhIjsIyLTRaRPRE5OOL+eiFwdn39QRIbVnTslPj5dRAobABo2rG0QwzCMnsCbghCRgcCFwL7ASOAQERnZEOwo4GVV3QE4HzgnvnYkcDDwbmAf4KI4Pu987GPR97nnwtVXw4EH9p/bYIMiJDAMwwgDnz2IXYA+VZ2pqm8CVwGjG8KMBmor6lwL7CUiEh+/SlWXq+qzQF8cn3eGDIkW7zvpJPjc5+D666Nlnb/xDXjmGRg7Nvk6WzLcMIxuw+dy39sCL9T9nw3s2iyMqq4UkcXAVvHxvzZcu21jAiJyDHAMwNve9jZngjeyzjr9+w8ceSQceigMGhStQzNgAGyzTbTL04QJMHcuvPWtsOWWcPfdcNhh8E//FO1ZMG8evPYabLQRzJoVHfvYx+Dpp+Gqq+Bb34rW9r/rLnjXu6KN5J97DgYPjq65445oc5lvfhOuuCKaHLPZZlH8ixdHNpRBgyJFNnAg7LgjrLtuJOfcuZGnw/LlUU/o/e+HGTOi1TJ32w0efzw6P3RodHzQoGivhUcfhe22i5aoXrIkygvVyJi/cGF0/6pR+FWrovX7118/Or5yZaRca2FE+r9r61+9/nok04AB0XW1pas33DCKa8CANdfOGTCgf7x1nXX6Z8UOGgSbbBLtZ13zCW82LrveelGaNXka4+2EWtqNKwTXy5p0L83i8rXScJ64s17jU35jTQZ6Gl+p9H4QqjoGGAMwatSowqriOutE30OH9h9bf30Y3dA/2mOP/t+77946zrPOap/uEUfAZZdFv3/0o/bhDcMwOsHnwMgcoK4JZbv4WGIYERkEbAYsTHmtYRiG4RGfCmISMEJEhovIukRG54YtzxkPHB7/Pgi4U1U1Pn5w7OU0HBgBPORRVsMwDKMBb0NMsU3hOGACMBC4WFWniciZwGRVHQ+MBS4TkT5gEZESIQ53DfAEsBL4mqqWvHWGYRhGbyHaJVakUaNG6eTJk8sWwzAMo1KIyBRVHZV0zpwzDcMwjERMQRiGYRiJmIIwDMMwEjEFYRiGYSTSNUZqEVkAzOogisHAS47EqQp2z91Pr90v2D1n5e2qOiTpRNcoiE4RkcnNLPndit1z99Nr9wt2zy6xISbDMAwjEVMQhmEYRiKmIPoZU7YAJWD33P302v2C3bMzzAZhGIZhJGI9CMMwDCMRUxCGYRhGIj2vIERkHxGZLiJ9InJy2fK4QkSGishfROQJEZkmIsfHx7cUkdtEZEb8vUV8XETkgjgfporIB8q9g/yIyEAR+ZuI3BT/Hy4iD8b3dnW8/DzxcvJXx8cfFJFhZcqdFxHZXESuFZGnRORJEdm928tZRL4Z1+vHReRKEVm/28pZRC4Wkfki8njdsczlKiKHx+FniMjhSWk1o6cVhIgMBC4E9gVGAoeIyMhypXLGSuBEVR0J7AZ8Lb63k4E7VHUEcEf8H6I8GBF/jgF+WbzIzjgeeLLu/znA+aq6A/AycFR8/Cjg5fj4+XG4KvIL4FZV3RF4L9G9d205i8i2wDeAUar6HqLtBA6m+8r5t8A+DccylauIbAmcTrTd8y7A6TWlkgpV7dkPsDswoe7/KcApZcvl6V5vAD4JTAe2jo9tDUyPf/8KOKQu/D/CVelDtPvgHcDHgZsAIZphOqixzIn2Ktk9/j0oDidl30PG+90MeLZR7m4uZ/r3st8yLrebgE91YzkDw4DH85YrcAjwq7rja4Rr9+npHgT9Fa3G7PhYVxF3qd8PPAi8RVXnxqfmAW+Jf3dLXvwc+DawOv6/FfCKqq6M/9ff1z/uOT6/OA5fJYYDC4BL4mG134jIRnRxOavqHOC/geeBuUTlNoXuLucaWcu1o/LudQXR9YjIxsB1wH+p6qv15zR6pegaP2cR2R+Yr6pTypalQAYBHwB+qarvB16jf9gB6Mpy3gIYTaQctwE2Yu2hmK6niHLtdQUxBxha93+7+FhXICLrECmHcar6h/jw30Vk6/j81sD8+Hg35MWHgQNE5DngKqJhpl8Am4tIbXvd+vv6xz3H5zcDFhYpsANmA7NV9cH4/7VECqOby/kTwLOqukBVVwB/ICr7bi7nGlnLtaPy7nUFMQkYEXs/rEtk6BpfskxOEBEh2vP7SVX9Wd2p8UDNk+FwIttE7fhhsTfEbsDiuq5sJVDVU1R1O1UdRlSWd6rqF4G/AAfFwRrvuZYXB8XhK/WmrarzgBdE5J3xob2I9nLv2nImGlraTUQ2jOt57Z67tpzryFquE4C9RWSLuOe1d3wsHWUbYcr+APsBTwPPAN8rWx6H9/URou7nVOCR+LMf0djrHcAM4HZgyzi8EHl0PQM8RuQhUvp9dHD/ewI3xb+3Bx4C+oDfA+vFx9eP//fF57cvW+6c9/o+YHJc1n8Etuj2cgZ+ADwFPA5cBqzXbeUMXElkY1lB1FM8Kk+5AkfG994HfDmLDLbUhmEYhpFIrw8xGYZhGE0wBWEYhmEkYgrCMAzDSMQUhGEYhpGIKQjDMAwjEVMQhpGAiNwffw8TkS84jvu7SWkZRmiYm6thtEBE9gROUtX9M1wzSPvXBEo6v1RVN3Yhn2H4xHoQhpGAiCyNf54N/KuIPBLvQTBQRM4VkUnxuvvHxuH3FJF7RWQ80axeROSPIjIl3rfgmPjY2cAGcXzj6tOKZ8GeG+9x8JiIfL4u7rukf8+HcfEMYsPwyqD2QQyjpzmZuh5E3NAvVtWdRWQ9YKKI/DkO+wHgPar6bPz/SFVdJCIbAJNE5DpVPVlEjlPV9yWk9R9Es6LfCwyOr7knPvd+4N3Ai8BEorWH7nN/u4bRj/UgDCMbexOtefMI0fLpWxFt0gLwUJ1yAPiGiDwK/JVowbQRtOYjwJWqukpV/w7cDexcF/dsVV1NtGzKMCd3YxgtsB6EYWRDgK+r6hoLnsW2itca/n+CaKOaZSJyF9GaQHlZXvd7FfbsGgVgPQjDaM0SYJO6/xOAr8ZLqSMi74g36GlkM6JtLpeJyI5E277WWFG7voF7gc/Hdo4hwB5Ei8sZRinYW4hhtGYqsCoeKvot0f4Sw4CHY0PxAuDAhOtuBb4iIk8Sbf/417pzY4CpIvKwRsuR17ieaKvMR4lW4v22qs6LFYxhFI65uRqGYRiJ2BCTYRiGkYgpCMMwDCMRUxCGYRhGIqYgDMMwjERMQRiGYRiJmIIwDMMwEjEFYRiGYSTy/wEhhUxIWXkIrwAAAABJRU5ErkJggg==\n" + }, + "metadata": { + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "code", + "source": [ + "import os\n", + "from sklearn.datasets import fetch_openml\n", + "from sklearn import preprocessing\n", + "\n", + "mnist = fetch_openml(\"mnist_784\")\n", + "X = mnist.data / 255.0\n", + "y = mnist.target\n", + "np.savez('mnist.npz', X=X, y=y)" + ], + "metadata": { + "id": "ho7-9TlgCY2l" + }, + "execution_count": 23, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "enc = preprocessing.OneHotEncoder(sparse=False)\n", + "y = np.array(y).reshape(-1,1).astype(int)\n", + "X = X.to_numpy()\n", + "from sklearn.model_selection import train_test_split\n", + "X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=42, train_size=0.8)" + ], + "metadata": { + "id": "CtWLnShlxcRw" + }, + "execution_count": 24, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "X.shape" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "GgCpm5jsUjSb", + "outputId": "8577bc8c-f941-4d0e-aa20-877ab3ebc828" + }, + "execution_count": 25, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(70000, 784)" + ] + }, + "metadata": {}, + "execution_count": 25 + } + ] + }, + { + "cell_type": "code", + "source": [ + "def accuracy(model, loader):\n", + " total = 0\n", + " correct = 0\n", + " for X, y in loader:\n", + " y_pred = model.forward(X)\n", + " y = y.ravel()\n", + " res = y_pred.argmax(axis=1)\n", + " total += res.shape[0]\n", + " correct += (res == y).sum()\n", + " return correct / total" + ], + "metadata": { + "id": "0k2t5KAbIibA" + }, + "execution_count": 26, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "def SGD(params, gradients, lr=1e-3):\n", + " res = []\n", + " for i in range(len(params)):\n", + " params[i] -= lr*gradients[i]\n", + " return params" + ], + "metadata": { + "id": "oylN3N0R9kOq" + }, + "execution_count": 27, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "epochs = 50\n", + "batch_size = 64\n", + "learning_rate = 1e+2\n", + "\n", + "d = 28*28\n", + "\n", + "model = Sequential(\n", + " Linear(28*28, d*2),\n", + " ReLU(),\n", + " Linear(d*2, 10),\n", + " SoftMax()\n", + ")\n", + "\n", + "criterion = CrossEntropy()\n", + "\n", + "history = []\n", + "import time\n", + "l = loader(X_train, y_train, batch_size)\n", + "for i in range(1, epochs):\n", + " for x, y_true in loader(X_train, y_train, batch_size):\n", + " y_pred = model.forward(x)\n", + " loss = criterion.forward(y_pred, y_true)\n", + " grad = criterion.backward(y_pred, y_true)\n", + " model.backward(x, grad)\n", + " SGD(model.parameters(),\n", + " model.grad_parameters(),\n", + " learning_rate)\n", + " history.append(loss)\n", + " print(accuracy(model, loader(X_train, y_train, batch_size)), accuracy(model, loader(X_test, y_test, batch_size)))# accuracy(model,loader(X, Y, batch_size)))\n", + "\n", + " \n", + "plt.title(\"Training loss\")\n", + "plt.xlabel(\"iteration\")\n", + "plt.ylabel(\"loss\")\n", + "plt.plot(history, 'b')\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "3n08VcpeDVFY", + "outputId": "f4440496-5d1e-4576-cbc5-2bc98bd9584c" + }, + "execution_count": 33, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "0.7759821428571428 0.7693571428571429\n", + "0.9672857142857143 0.9574285714285714\n", + "0.9770714285714286 0.9648571428571429\n", + "0.9844285714285714 0.9715714285714285\n", + "0.9881071428571429 0.9746428571428571\n", + "0.9903214285714286 0.9730714285714286\n", + "0.9923214285714286 0.975\n", + "0.9935714285714285 0.9772142857142857\n", + "0.995 0.9776428571428571\n", + "0.9952321428571429 0.977\n", + "0.9961964285714285 0.9772857142857143\n", + "0.9966964285714286 0.9778571428571429\n", + "0.9966964285714286 0.9770714285714286\n", + "0.9971964285714285 0.9780714285714286\n", + "0.9975357142857143 0.9795\n", + "0.9977142857142857 0.9791428571428571\n", + "0.997875 0.9784285714285714\n", + "0.9980892857142857 0.9792857142857143\n", + "0.9981964285714285 0.9790714285714286\n", + "0.9983392857142858 0.9791428571428571\n", + "0.998375 0.9790714285714286\n", + "0.9984285714285714 0.9788571428571429\n", + "0.9985714285714286 0.9792857142857143\n", + "0.9986071428571428 0.9799285714285715\n", + "0.9986428571428572 0.9798571428571429\n", + "0.9986785714285714 0.9793571428571428\n", + "0.9987142857142857 0.9795\n", + "0.9987321428571428 0.9802857142857143\n", + "0.9987321428571428 0.9799285714285715\n", + "0.9987321428571428 0.9796428571428571\n", + "0.99875 0.9805714285714285\n", + "0.9987857142857143 0.9797857142857143\n", + "0.9988035714285715 0.9794285714285714\n", + "0.9988392857142857 0.9800714285714286\n", + "0.998875 0.9797857142857143\n", + "0.9989285714285714 0.9802857142857143\n", + "0.9989285714285714 0.9807857142857143\n", + "0.9989642857142857 0.9803571428571428\n", + "0.9989821428571428 0.9800714285714286\n", + "0.9990178571428572 0.98\n", + "0.9990535714285714 0.9802142857142857\n", + "0.9990535714285714 0.9802142857142857\n", + "0.9990714285714286 0.98\n", + "0.9990892857142857 0.98\n", + "0.9990892857142857 0.9800714285714286\n", + "0.9990892857142857 0.9800714285714286\n", + "0.9990892857142857 0.9798571428571429\n", + "0.9991071428571429 0.9801428571428571\n", + "0.9991071428571429 0.98\n" + ] + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7wcdX3/8dfHYNAKCoEUMQESamyJVLkcQGpFHpViAE1spRp+XCsVUVL5VWwJYkFi2wdC9SeWgISCgkADcmuAQEBuwUtCThICBIichEBuQCAXAklOcpLP74+ZNXv2zO7O7M7szu6+n4/HeZzd79y+Ozs7n+98v9/5jrk7IiIipd7V7AyIiEg+KUCIiEgkBQgREYmkACEiIpEUIEREJJIChIiIRFKAECnDzO43s9PTnjdhHo42s+Vpr1ckjp2anQGRNJnZ20Vv/wjoBbaF77/m7jfHXZe7H5fFvCKtQgFC2oq771J4bWZLgX9w91+VzmdmO7l7XyPzJtJqVMUkHaFQVWNm55vZq8DPzGx3M7vXzFab2drw9fCiZR4zs38IX59hZr82s/8M533JzI6rcd6RZjbTzDaY2a/MbLKZ3RTzcxwQbmudmS00s7FF0443s+fC9a4ws2+H6XuGn22dma0xsyfMTL99qUoHiXSSDwJDgP2AswiO/5+F7/cFNgFXVlj+CGARsCdwGXCdmVkN894CPAnsAXwPODVO5s3s3cA9wIPAHwP/CNxsZn8aznIdQTXarsCBwCNh+nnAcmAosBfwHUBj7EhVChDSSbYDF7t7r7tvcvc33f0Od9/o7huAfwc+XWH5l939WnffBtwA7E1wwo09r5ntCxwGXOTuW9z918C0mPn/BLALcGm47CPAvcBJ4fStwGgze7+7r3X3eUXpewP7uftWd3/CNQibxKAAIZ1ktbtvLrwxsz8ys2vM7GUzewuYCexmZoPKLP9q4YW7bwxf7pJw3g8Ba4rSAJbFzP+HgGXuvr0o7WVgWPj6i8DxwMtm9riZHRmmXw70AA+a2RIzmxhze9LhFCCkk5SWms8D/hQ4wt3fDxwVpperNkrDKmCImf1RUdo+MZddCexT0n6wL7ACwN3nuPs4guqnu4HbwvQN7n6eu+8PjAW+ZWafqfNzSAdQgJBOtitBu8M6MxsCXJz1Bt39ZaAb+J6ZDQ5L+Z+PufhsYCPwL2b2bjM7Olx2ariuk83sA+6+FXiLoEoNM/ucmX04bANZT9Dtd3v0JkR2UICQTvZj4L3AG8As4IEGbfdk4EjgTeDfgFsJ7teoyN23EASE4wjyfBVwmru/EM5yKrA0rC47O9wOwCjgV8DbwO+Aq9z90dQ+jbQtU1uVSHOZ2a3AC+6e+RWMSBK6ghBpMDM7zMz+xMzeZWZjgHEEbQYiuaI7qUUa74PAnQT3QSwHvu7u85ubJZGBVMUkIiKRVMUkIiKR2qaKac899/QRI0Y0OxsiIi1l7ty5b7j70KhpbRMgRowYQXd3d7OzISLSUszs5XLTVMUkIiKRFCBERCSSAoSIiERSgBARkUgKECIiEkkBQkREIilAiIhIJAUIwKzy36uvVl+HdI716+Htt+tbx4YNcPPN/dPmzYM5c6ovO3duMN/MmfD889Hz3HzzwDzOnw+zZw+cd9MmuOEGKB11Z8sW+PnPB6YD3HsvrFhRPa8F11wDt9wyMH3tWrj11srLvvACPP54/7S77473u1y+HO67D26/Hd58s/x8vb3lPysE+/q55/qnRe2D114L8lbJ66/DnXf2T6u0rzdtghtvHDjNPUjftKny9uri7m3xd+ihh3otvvQl92BXl//bZZeaVi0NduWV7ueem/12khwT4P7Vrw5MP/nkYNrs2f3nhXjrLP4rNWtWkH7KKdHLlTr33CB9+vT+6RdeGKTfdlt0HoYNq55Xd/d168pve8yYIL2np/zypctu3hy8P+CA6tvea68dyx91VPn5zj8/mOeOO+LloZBWug8OPDBI37ix/LYOPTSYZ+3aHWmV9vU55wTTZszon/7gg0H6N75RfltxAN1e5rza8VcQcUohmUboFjRsGPzt39a3jquuggUL0slPwYQJcMUV6a6znCRXENdeOzCtUPJ855108lOskLeVK+PNv2pV8P+tt/qnv/Za8H/duujl4l5B9PWVn/bKK8H/zZvLz1Nqe/gsvJdeqj5v4TMUb6vSfOvXx88HDNwHS5YE/7dXeF5fId/btg3cftS+Lnw/Gzb0Ty98X1nWcHR8gLAYTx+u9GV3opUr4a676lvHOefAQQelk5+09fUF1R4a6Fg6nQJElo+nl0Qefjj4PtK+sojDHS64AJ55Bi69FMaPD+qtRTpZxwcIyY9C497MmY3f9ltvBYHhU5/aUW3wxhvxl7/jjqDRUqSdtM1oriJpqLVa6cQT61teJI90BSEiIpEUIGJwhzVrmp0LEZHG6vgAEbdK4NvfzjYfUl5vL2zc2OxciHQeBYiYAWLr1mzz0U4+/3n40Y/SW9/IkfC+96W3vjzJss1C7SFSr44PEFLezJlw003Jl7v3XjjvvPTyUbhRqJ1k2b1aXbclLR0fIFTKKu/Tn4ZTT212LvpbtgxefDG79et4kFaT5TGbaYAwszFmtsjMesxsYsT0s83sGTN7ysx+bWaji6ZdEC63yMw+m2U+JV8qHfD77gsf+Uj621SpW6C1CgiNOGYzCxBmNgiYDBwHjAZOKg4AoVvc/c/d/SDgMuBH4bKjgfHAR4ExwFXh+lLXSgdEu2vEAf/UU8F2Hnww+21JfM3+HaqAEC3LK4jDgR53X+LuW4CpwLjiGdy9eHiw9wGFw2QcMNXde939JaAnXF/qmn1gSmM98UTw/557mpsPCejEnG9ZBohhwLKi98vDtH7M7BwzW0xwBfHNhMueZWbdZta9evXq1DKelQ99CM48s/p8P/tZ8MNJYxTZ6dOhp6f+9YhI52l6I7W7T3b3PwHOB76bcNkp7t7l7l1Dhw7NJoMpWrUKrr+++nwXXxz8TyPmnXACjBpV/3pEpPNkGSBWAPsUvR8eppUzFfhCjcvWTPc3iIhEyzJAzAFGmdlIMxtM0Og8rXgGMysu254AFDowTgPGm9nOZjYSGAU8mUUm9awHaSbdKNc42h/JZTaaq7v3mdkEYAYwCLje3Rea2SSCR9xNAyaY2THAVmAtcHq47EIzuw14DugDznH3bZEbqlPcRrI8BpLNm4MroF13bXZO2kejTiK6UU5aQabDfbv7dGB6SdpFRa/PrbDsvwP/nl3uCtuJN9/vfpdtPmpx4IGweLFKRmnQSVVkoKY3UreK5cubnYOBFi9udg6yoYAnkg8dHyDinozyWMXUbhpZilcQan2t/h0mzX8zPm/HB4i4tmXSAtI8s2fD0qXNzkXjqSqp9bX6d5g0/9XmzzJw6JGjGdptN9h7b3j++WTLNaKk8IlP7NjW7NlBAPyLv8h+u62k1UuoklwrfectPRaTwPr18MILtS9fegA8/niQtmRJffkq9YlPwCc/me46W1mrl1AlOX3n0To+QLRSieHnPw/+P/54U7MhkrpW+h12ko4PEFu2xJ933rzs8iH50OgTVaffKKeSe751fIBI0jvpiCOyy4c0V6NPVHm8Ua4VAoo0VscHiCRU2pF2pONaylGAkNxpRElWpWWR6jo+QGg01/xoRElWpWWR+Do+QKgkKSISreMDRF9fOuvZvh0eeCCdgFNtHQpqItIIChApBYgrroDjjoO77kpnfTCwOkTVIyLSSB0fIJKUxiudoAt3N6/I5Ll30ii6OpNWk+Ux2/EBQqO0CjTv6qzTb5RrpHbbHxqLqQHUi0maIY83ykn7BZF6dXyAkPzRfRDSaAqq0To+QOhEkR+6D6Jz6XeYTx0fILKyeXN269aPSdqFAna+dXyASNJIneRg7upKnpc0ty8iUq9MA4SZjTGzRWbWY2YTI6Z/y8yeM7OnzexhM9uvaNo2M3sq/JuWVR6TlMaTBJOFC5PnRZpPV2ciO2T2yFEzGwRMBv4aWA7MMbNp7v5c0WzzgS5332hmXwcuA74cTtvk7gdllb9aNOrkoZNU4+nqTGSgLK8gDgd63H2Ju28BpgLjimdw90fdfWP4dhYwPMP8RMrzyVgnLZHy8vzbjSNp/pvxebMMEMOAZUXvl4dp5ZwJ3F/0/j1m1m1ms8zsC1ELmNlZ4Tzdq1evrj/HIg2mG+WSa/WCU9L8N/PzZlbFlISZnQJ0AZ8uSt7P3VeY2f7AI2b2jLsvLl7O3acAUwC6urpy8XNo1x9lJ8ryu9SNcpKWVh1qYwWwT9H74WFaP2Z2DHAhMNbdewvp7r4i/L8EeAw4OMO81k0/yvQ0+0Y5fZfSClp9qI05wCgzG2lmg4HxQL/eSGZ2MHANQXB4vSh9dzPbOXy9J/BJoLhxO1c2bYL16xu3vXa9StGNciL5klmAcPc+YAIwA3geuM3dF5rZJDMbG852ObAL8MuS7qwHAN1mtgB4FLi0pPdTU5Q7MZ94Itx4Y33r3rABDjwQ5s0rP49ObiLZSqvw1S6FuEzbINx9OjC9JO2iotfHlFnut8CfZ5m3gjROutOnV5+nmieeCO6d+O5361+X1K5dftiSTFqFr3YrxHX8ndQi0H4/bJE0KEDkUKEUq5OWdApdueWTAkQOKUBIp9Axnm8KEAnoYJa06Ua5xtH+SK7jA0QrPnJUB3r9mr0PdaOctIKODxCtpFN++K16B3Ora3bQlPxRgJDc0Mm7ObTfpRwFiJxQ6S0f9D1Iq2nVsZjaTpwvot4vS6W55tB+l7xatiw6vdXHYmoJSU7o27Zllw8RkSizZzdv2x0fIJJoVClT1RwikgcKEDmmag+RxkqrcDZ/fvx5N2+GrVvT2W7aFCBakK4w6peXfagb5fKhnsLYypXBSb7YUUdVX64wyOdBB8ENNyTb5u23w7hx1eerV8cHiFp/RCtXwgsvpJuXanRFUb+87EPdKNc+hg0LhvxP6owzYOlSWLQo+bKXXJJ8mVp0fIBI8mMqDibDhsEBB6SfH1HJV7KRxnE1d250+n331ba+TZtqz0sjdHyA6Otrdg4CGqAvH59dwWmgevbJDTfAkiXZbCPpMsuXJ99GqUoP9MpaM47NTB8Y1AryckJQgJC8SeNYPOMMeN/70t1GYZlt24InMe66a01Za5pW+o13/BVE0sH63nknm3xI/uSl8JBnTz9dfZ5yv5n162HBgv5pY8fCvff2n6ec7dvhRz8qn68336yetwJ919E6PkAk1cwqqZ6e4P/Gjc3LQydopRJeI2zZAmedFX3C/eEPa19vVNXTPffA5z+/4/3BB9e27o9/HI44Iv78d9xR23baXccHiDzeHV2uNFMora1e3bi8SDqmTo1XF19N0jrwadOCgFfPtmfNgmuvhQsvrH0dtXrppdqXXbw4/rxr19a+nWbTWEwZyvOlpUqy7eOkk+KXhh99tPy0pL1efvGL4H+53jdJ5Pm3Itno+ACRNv2IWkMzvqe33oqXjzj1+sV6e6PTdSxKvTINEGY2xswWmVmPmU2MmP4tM3vOzJ42s4fNbL+iaaeb2Yvh3+lZ5fFdKeyBekv6Z58NV15Zfz6kumrfVfFJdds2OOEEePLJxucjiQcfTGfdeenyLfmRWYAws0HAZOA4YDRwkpmNLpltPtDl7h8DbgcuC5cdAlwMHAEcDlxsZrtnlddmu+aagT/yTpa05Ose1JGnbfXqYDiE0zMrnuTLaacl6/kD7XeV0m6fp15ZXkEcDvS4+xJ33wJMBfqNHuLuj7p7oU/OLGB4+PqzwEPuvsbd1wIPAWMyzGtstRxAV18df95Obneo9bPfdVfQyyYPtm4Nhk9oVa++Wn6aTp6dJ8sAMQwoftTF8jCtnDOB+5Msa2ZnmVm3mXWvznHXnm98o9k5aG+V+so32rnnwsiR8MYbta+j0ok4rQLETTfBww+nsy5pX7m4k9rMTgG6gE8nWc7dpwBTALq6ulS+kUR6e+H+++ELX0hvnTNmBP/Xr4c99xw4fcECuPvu9LZXq1NPDf7rqkAqyfIKYgWwT9H74WFaP2Z2DHAhMNbde5MsK62vmSeo88+Hv/kbmDmzcds84gj43veS38Ev0gxZBog5wCgzG2lmg4HxwLTiGczsYOAaguDwetGkGcCxZrZ72Dh9bJiWujyWoKrlKY95rlVXV2O3t2bNjteFm7AaeZNU4cEwjW5ryuMxU7wPas1fHj9XErXmv1HHT2ZVTO7eZ2YTCE7sg4Dr3X2hmU0Cut19GnA5sAvwSws+8SvuPtbd15jZ9wmCDMAkd18TsZm2VnoQtGMDdiNHx1y6FC64IHjd6ieWuNI8ZqL2Wafsx06VaRuEu08HppekXVT0+pgKy14PXJ9d7tJjlu0P5bTTYLfdslt/o3V3939/zDHJenrVakWMSsq0vse468nypJvWetqxYNJOsjz35KKRuplq6XOfpagfY2G4hKhGz1Zwzz1wyCHBQ5Zg4Pg6Dz8MEyfC/vsnW+9TT8HLL+frBBY3L7XmOc5yedof0to01EbOtOOPe+xYOPLI+PPHDcIHH5xODyRVk4hEU4BIWZyTzSGHwKpV2eclT5Ytqz5Po4eIiLu9LAKIgpK0AgWIhFas2FHlE+WFF6p3YZw/H/77v2vPg04u/f3TPzU7BwOV+47yeIXYzEAptSn0hstax7dBJHX88ZVLw1OmwMc+1rj8tIt6Tj5vv51ePuqVRgDI24m40mfKW17r1Sqfp/DwsKzpCiKhOFUlL7+cfT4a6fe/D04Spb2PpPFqDUCtcOL7wQ+anYPW0aiRdxUgcqjwY85LdUThGcG33NLcfBQ//e+GG9Jbb9yT50svwZw51efLk8IxFPUsiryZOOCBANl58cV4hb08aUaQV4BIQS0n8kpfdrMCxJQpjTkIa/1c3/nOjtdnnJFs2XPPhfPO2/HePToflT7//vvD4Ycn227a1iS8XbQQVP/jP9LPSy2uuWbH6zS7+q5bl+xRvB/5COy7b/np27bBv/5r8v0dR14KfnEoQLSod95Jf51f+xr87nfprzeOOIGpnkHufvITmD073ryNbEdIcqPcunXBA4yS5KGwrjR6zaVReDj77PrXEWXoUPjjP05vfffcA//2b0HBIi8ef7zx2+z4AJG3utm4J6eNG6vPU4ukzzzOQrnvJKvPnKZq319hei1BKO6w5lmVUK+/Hp5/fmD6a6/Bli3ZbLOawrGSdp18oZfQ5s3xlyk36OPYsbDrrvXnqRmjAHd8gGglca4aVq2q/NCXemzYkO3lcbV1Z3HVBHDnnfDe96azrsLzoadNqzxfnm3ZAq+/Hj2t9CS1fTt88IM7hg+vVdyC2qZNMHhwfdvKyqfLPKzgnnvy1dMuiVgBwszONbP3W+A6M5tnZsdmnblGaNYVRLntvvpq+RNhoXT/k5+Urxv90Idg771rz9cddwzMW+F9aR1vuwxZfdNNlaf39vZvIC/o6xv4vOpCgPjtb6PX1QrPfT7tNNhrr2S/jdtvzy4/xep5EFMepXH+yfIcFvcK4ivu/hbBsNu7A6cCl2aWqxZTeqKsp5Q9a1b1y9qNG+E3v6l9G5VcfXXQrTVK6ecaNCjd3kQFr70GF17YnHGvoixdCt///sD0Sy4Jnu+QRfffefPKB5kofX0weXI6N1Ddemv965D2EDdAFH5KxwO/cPeFRWkdr/RE1ojnC6RV1RN1FZCklDt1ajr5KN6HU6YEvW7efLP29a1ale7dplEN3E89tWNbaXKHQw+FT34y/jJXXw0TJgRXl0kVl8rz1iYnzRU3QMw1swcJAsQMM9sVaJMKhvRde22zcxDfoEHx53322crTt26F556rvp5ywa3Q0FkIWrUGwXfeCaravv712paPkvaJ84EH4s13443x5lu3rv//JB56qPL0qOq1Zogb8NOsxuv0gBk3QJwJTAQOc/eNwLuBv88sVx0gzoG3cGF923jkkfqWL1Xt9v5zz4WPfrT29f/4x7UvW6zQ2+l//7f8POX2f7n0ctVupQptEGk5/fR011dq8uTq88woepZjteN2y5bs2lkmTIg334YN9W+rle5VyFLcAHEksMjd15nZKcB3gZid7qRWpQ2gxeIEmJNPTicfcUtRjz1WfZ433oD77qsrO01R6U7k4v1TKMHPmBHcrdsoUdt68snKxxDEO+lWqup74YXgf2Ef7Lxz0C6T1OLFA9NKu9QW7uhvFaVdf195pfU6dsQNEFcDG83s48B5wGIg5sWvZCFOb46survW49RT4Wc/G5jejEv5ekuJlZbfuBGuvDL+uirdBGUG//VflbcXNcLw9u2wfHnwuria6M474+Upzncyf/7AtLiPkS3+PFHjl40eXXkZgEmT4Jxz4m2vFoX9V4vS43y//YL8VtqvzzxT+/ayEDdA9Lm7A+OAK919MpDCrR+d64kn6lv+/POTzZ+0bvo//zPZ/HFlMXRBrcqVrpsRrI4+uvL2L7ss/rqiAklxgPjiF+Ovq1Z77FF+WpxHv86dG39bV10Vf96kyt19/8//XH3ZqKq2hx7qf0/ERRf1n17paq1cW1CW1WFxA8QGM7uAoHvrfWb2LoJ2CKnRI48EpcLCJXo95swJDpJKde5DhsBRR8VfZ5y66axddlnQ3TUrUaXD4h9bp9dD19N9t1JBIM4x39VV23aHDOk/3lOpzZujq7OSqrUAtXp1/9EK4l7NQfrtW3HEDRBfBnoJ7od4FRgOXJ5ZrjrEN78JBxxQ/3oKA8hNmlR+Hvf6r1oapXBivuyy/oPM5bX+Nk89XdLMy4IF9a2/XIEky2cZrF0LF1xQfvopp8CHP5xsncU9ySZPru84rLb/Vq0qfxWxfn3jb7SMFSDCoHAz8AEz+xyw2d2rtkGY2RgzW2RmPWY2YDBfMzsqvCu7z8xOLJm2zcyeCv9aeOCC9lZ8wGdd4s7TiRiqf9685TdLUZ+1XIHk7LPTu38mqeIeWZUUf7d33bXj9YIF2Q35AvDoo3DcceWnb9sWryt5WuIOtfEl4Eng74AvAbNLT+gRywwCJgPHAaOBk8ystNnpFeAMIOpJA5vc/aDwb2ycfMoO9Yz9Unriq3SiS6unTl5OpnnJRxKtmOfSHkp5eTbD1VfXt3y1En6ce0qiBkQsuOuu/gELsv3+4z5y9EKCeyBeBzCzocCvgEojsBwO9Lj7knCZqQSN3H+If+6+NJyW08qD1pV1//mCpI3O9VxlNHqk2Wa1QSQZArxVzZ0bdIktSFIXn6Wnnw7+13oXf7V2gpdeSra+0mBx0knBMOSNErcN4l2F4BB6M8ayw4DicsHyMC2u95hZt5nNMrMvRM1gZmeF83SvTvK0kA5QONBrlccG2kY9qD2pRpy8e3uDbpJRGl0vncY4YPfd178DQtyh3Os9LuNuJ8k4WFnp7Y3u6ttIca8gHjCzGcD/hO+/DEzPJkt/sJ+7rzCz/YFHzOwZd+/X/8DdpwBTALq6uppSxsrroxybES9LG++efhpGjUq+njyWlqNKlNVOVj/9aXrbr1TvfXkDu4u4xx8mJIk4V6JpFBDy2tEhSh5G/o0VINz9n83si0Bh+LAp7n5XpWWAFcA+Re+Hh2mxuPuK8P8SM3sMOJjgBr1cGTKk2TlIXy31wY89NrD74sc/DscfH2/54qDQrIfPpC3OCS2N6qRZs5LN36oa9UyFavcMJenimkaX2maK/cAgd7/D3b8V/lULDgBzgFFmNtLMBgPjgVi9kcxsdzPbOXy9J0FgamDbfXxxBzJrxOMC4zz9yqz68w9K72qN8/zsctUO00uuM+OU4PIyOFyzai2T1lMXW7wYLr44vbwksXJltuvfvr0xV5d/X2WUuUrdyYu9/npwr1NSfX1BFdcuu0RPb2T1b8UAYWYbzOytiL8NZlaxcsXd+4AJwAzgeeA2d19oZpPMbGy4/sPMbDlB76hrzKwwPN0BQLeZLQAeBS5191wGiLiK75RNIsmAe3G7v333u9XnyepSPMkjHJst7rAHaZ+0oqoW4n4fU6fCgw8OTG/EifWHP8x2/b29cPPN6a7ze98bOBDj3Xenc1PapElwxRXJl9u8OXhwV1zLlmV3Y2vFKiZ3r2s4DXefTklbhbtfVPR6DkHVU+lyvwX+vJ5tt4uvfCX9dVY7+O++O/vSYKmstvfEE3DkkQPTaxkWu1ShJFdvV984vbOacRdtlErVZo3oZZZ24/EllwR/pZp1n0YtVq0KagWyGJMqbiO1NEmtA+7FfcB9lPvvr33ZWqV580/hRNXXF9zNG3VnbbkRZbdvT34Jn3RcrGJ3352vRvmvfrXy9AcegGOb+LDhVn7Wd1rSGJ4nrthtENIceTp5pKFcQ2Oa7Q6F6pjC/6jnauSlN8t11zU7B/1Va/Cu9KyFPPS6yUIzfoOLFpWfFjVyb1YUIHJs7txs+v43M+iUG6Y8i8/ZKu0dUW0GebV+ffnjp9E3MrazvDwzRVVMOZaXh8dXCihJSuKNrtctdJettf6+UVcZpc+NyPNV48qVrd91U+LTFYTUJUnV0EknZZePYqV3y8YdoK1U1ENsiuXxbvNG0JVC51CAkLps2xaUePN0srwlaujHGuT1LnlprErPWWl3ChBSl97e7Pu/J1Vvl9ClS1PJRtvKcxWYpEsBogOl/QNPY/C2PGnEXe+trNy4UHm6ipR0KEBIVVk+IKWVZXVCnDkzm/WKJKUAIVV9//uVp+et/3s9YxnBjr7+zSoRZ/lIzjSUuwKNesa3tDYFCKlb1I1ozZRkHJtKKt0UBp1bF792bXR6I+/wlcZQgJC6Jentk5c7mNPw8MPNzkFzlBsWpVVuTJT4FCA6UNynamWhndozOrUbbLkrp3JXFtK6FCA6UNrPW9i6VT1YRNqRAoTULS8P+Eminaq6pLNl+ax2BQhJxfz5zc6BSGeqZ2j/ahQgOlAWvW9++cv01yki8WQ1PpYCRAdSbxOR9vLss9msVwFCOlIrtpvkhfZd/mT1nShAiEgiChCdQwFC6tapdxSLtDsFCKlbvcNri0g+ZRogzGyMmS0ysx4zmxgx/Sgzm2dmfWZ2Ysm0083sxfDv9CzzKfVRlYNIe8osQJjZIGAycBwwGjjJzEaXzECRKzMAAAsoSURBVPYKcAZwS8myQ4CLgSOAw4GLzWz3rPIqIiIDZXkFcTjQ4+5L3H0LMBUYVzyDuy9196eB0vtaPws85O5r3H0t8BAwJsO8iohIiSwDxDBgWdH75WFaasua2Vlm1m1m3atXr645oyIiMlBLN1K7+xR373L3rqFDhzY7OyIibSXLALEC2Kfo/fAwLetlRUQkBVkGiDnAKDMbaWaDgfHAtJjLzgCONbPdw8bpY8M0ERFpkMwChLv3ARMITuzPA7e5+0Izm2RmYwHM7DAzWw78HXCNmS0Ml10DfJ8gyMwBJoVpIiLSIOZtchtsV1eXd3d3J15OD7oRkXZQ66nczOa6e1fUtJZupBYRkewoQIiISCQFCBERiaQAISIikRQgREQkkgKEiIhEUoAQEZFIChAiIhJJAUJERCIpQIiISCQFCBERiaQAISIikRQgREQkkgKEiIhEUoAQEZFIChAiIhJJAUJERCIpQIiISCQFCBERiaQAISIikRQgREQkUqYBwszGmNkiM+sxs4kR03c2s1vD6bPNbESYPsLMNpnZU+HfT7PMp4iIDLRTVis2s0HAZOCvgeXAHDOb5u7PFc12JrDW3T9sZuOBHwBfDqctdveDssqfiIhUluUVxOFAj7svcfctwFRgXMk844Abwte3A58xM8swTyIiElOWAWIYsKzo/fIwLXIed+8D1gN7hNNGmtl8M3vczD4VtQEzO8vMus2se/Xq1enmXkSkw+W1kXoVsK+7Hwx8C7jFzN5fOpO7T3H3LnfvGjp0aMMzKSLSzrIMECuAfYreDw/TIucxs52ADwBvunuvu78J4O5zgcXARzLMq4iIlMgyQMwBRpnZSDMbDIwHppXMMw04PXx9IvCIu7uZDQ0buTGz/YFRwJIM8yoiIiUy68Xk7n1mNgGYAQwCrnf3hWY2Ceh292nAdcAvzKwHWEMQRACOAiaZ2VZgO3C2u6/JKq8iIjKQuXuz85CKrq4u7+7uTryc+kyJSDuo9VRuZnPdvStqWl4bqUVEpMkUIEREJJIChIiIRFKAEBGRSAoQIiISSQFCREQiKUCIiEgkBQgREYmkACEiIpEUIEREJJIChIiIRFKAEBGRSAoQIiISSQFCREQiKUCIiEgkBQgREYmkACEiIpEUIEREJJIChIiIRFKAEBGRSAoQIiISSQFCREQiZRogzGyMmS0ysx4zmxgxfWczuzWcPtvMRhRNuyBMX2Rmn80ynyIiMlBmAcLMBgGTgeOA0cBJZja6ZLYzgbXu/mHg/wE/CJcdDYwHPgqMAa4K1yciIg2S5RXE4UCPuy9x9y3AVGBcyTzjgBvC17cDnzEzC9Onunuvu78E9ITrS91OO2WxVhGRxlq3Lv11ZhkghgHLit4vD9Mi53H3PmA9sEfMZTGzs8ys28y6V69eXVMm33mnpsVERHJl8OD019nS5Wd3nwJMAejq6vJa1jF4MHhNS4qItLcsryBWAPsUvR8epkXOY2Y7AR8A3oy5rIiIZCjLADEHGGVmI81sMEGj87SSeaYBp4evTwQecXcP08eHvZxGAqOAJzPMq4iIlMisisnd+8xsAjADGARc7+4LzWwS0O3u04DrgF+YWQ+whiCIEM53G/Ac0Aec4+7bssqriIgMZN4mFfBdXV3e3d3d7GyIiLQUM5vr7l1R03QntYiIRFKAEBGRSAoQIiISSQFCREQitU0jtZmtBl6uYxV7Am+klJ12o31TnvZNedo35eVp3+zn7kOjJrRNgKiXmXWXa8nvdNo35WnflKd9U16r7BtVMYmISCQFCBERiaQAscOUZmcgx7RvytO+KU/7pryW2DdqgxARkUi6ghARkUgKECIiEqnjA4SZjTGzRWbWY2YTm52frJjZ9Wb2upk9W5Q2xMweMrMXw/+7h+lmZj8J98nTZnZI0TKnh/O/aGanF6UfambPhMv8JHx0bEsws33M7FEze87MFprZuWF6x+8fM3uPmT1pZgvCfXNJmD7SzGaHn+fWcEh/wiH6bw3TZ5vZiKJ1XRCmLzKzzxalt+xv0MwGmdl8M7s3fN9e+8XdO/aPYBjyxcD+wGBgATC62fnK6LMeBRwCPFuUdhkwMXw9EfhB+Pp44H7AgE8As8P0IcCS8P/u4evdw2lPhvNauOxxzf7MCfbN3sAh4etdgd8Do7V/nDC/u4Sv3w3MDj/HbcD4MP2nwNfD198Afhq+Hg/cGr4eHf6+dgZGhr+7Qa3+GwS+BdwC3Bu+b6v90ulXEIcDPe6+xN23AFOBcU3OUybcfSbBMzeKjQNuCF/fAHyhKP1GD8wCdjOzvYHPAg+5+xp3Xws8BIwJp73f3Wd5cNTfWLSu3HP3Ve4+L3y9AXie4BnoHb9/ws/4dvj23eGfA38F3B6ml+6bwj67HfhMeLU0Dpjq7r3u/hLQQ/D7a9nfoJkNB04A/jt8b7TZfun0ADEMWFb0fnmY1in2cvdV4etXgb3C1+X2S6X05RHpLSe89D+YoKSs/cMfqlGeAl4nCHqLgXXu3hfOUvx5/rAPwunrgT1Ivs9awY+BfwG2h+/3oM32S6cHCAmFJduO7vNsZrsAdwD/193fKp7WyfvH3be5+0EEz4Y/HPizJmep6czsc8Dr7j632XnJUqcHiBXAPkXvh4dpneK1sPqD8P/rYXq5/VIpfXhEessws3cTBIeb3f3OMFn7p4i7rwMeBY4kqFYrPLK4+PP8YR+E0z8AvEnyfZZ3nwTGmtlSguqfvwKuoN32S7MbeZr5R/BM7iUEjUOFhqCPNjtfGX7eEfRvpL6c/o2wl4WvT6B/I+yTYfoQ4CWCBtjdw9dDwmmljbDHN/vzJtgvRtAu8OOS9I7fP8BQYLfw9XuBJ4DPAb+kf2PsN8LX59C/Mfa28PVH6d8Yu4SgIbblf4PA0exopG6r/dL0ndvsP4IeKb8nqFe9sNn5yfBz/g+wCthKUJ95JkEd6MPAi8Cvik5mBkwO98kzQFfRer5C0JDWA/x9UXoX8Gy4zJWEd+m3wh/wlwTVR08DT4V/x2v/OMDHgPnhvnkWuChM358g6PWEJ8Wdw/T3hO97wun7F63rwvDzL6KoF1er/wZLAkRb7RcNtSEiIpE6vQ1CRETKUIAQEZFIChAiIhJJAUJERCIpQIiISCQFCJEIZvbb8P8IM/s/Ka/7O1HbEskbdXMVqcDMjga+7e6fS7DMTr5jPJ6o6W+7+y5p5E8kS7qCEIlgZoURTC8FPmVmT5nZP4UD111uZnPCZ0F8LZz/aDN7wsymAc+FaXeb2dzwOQpnhWmXAu8N13dz8bbC50xcbmbPhs+O+HLRuh8zs9vN7AUzu7lVnichrW2n6rOIdLSJFF1BhCf69e5+mJntDPzGzB4M5z0EONCDYZsBvuLua8zsvcAcM7vD3Sea2QQPBr8r9bfAQcDHgT3DZWaG0w4mGJZhJfAbgrGAfp3+xxXZQVcQIskcC5wWDn89m2A4jlHhtCeLggPAN81sATCLYOC1UVT2l8D/eDB66mvA48BhRete7u7bCYYCGZHKpxGpQFcQIskY8I/uPqNfYtBW8U7J+2OAI919o5k9RjAeT616i15vQ79daQBdQYhUtoHgMaQFM4Cvh8ODY2YfMbP3RSz3AWBtGBz+jGAk14KtheVLPAF8OWznGErwmNgnU/kUIjVQKUSksqeBbWFV0c8JxvwfAcwLG4pXE/340AeAs83seYJROmcVTZsCPG1m89z95KL0uwietbCAYHTZf3H3V8MAI9Jw6uYqIiKRVMUkIiKRFCBERCSSAoSIiERSgBARkUgKECIiEkkBQkREIilAiIhIpP8P6Ix+jEWeNTIAAAAASUVORK5CYII=\n" + }, + "metadata": { + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "code", + "source": [ + "accuracy(model,loader(X_test, y_test, batch_size))" + ], + "metadata": { + "id": "XbsNfKQmWD_x", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "cae25494-a62f-472a-af18-e766ea3fa1a3" + }, + "execution_count": 34, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "0.98" + ] + }, + "metadata": {}, + "execution_count": 34 + } + ] + }, + { + "cell_type": "code", + "source": [ + "accuracy(model, X_train, y_train)" + ], + "metadata": { + "id": "yEW7IzD4bnha" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "" + ], + "metadata": { + "id": "B__rJyUIQ_zP" + }, + "execution_count": null, + "outputs": [] + } + ] +} \ No newline at end of file