From e653c0e52dd436dd590aedfad84bdf5e714fc688 Mon Sep 17 00:00:00 2001 From: Karina Date: Tue, 5 Nov 2024 17:53:16 +0300 Subject: [PATCH] 2 more weeks and hws --- homeworks/hw1_embeddings/README.md | 5 + .../hw1_embeddings/embedding_based_MT.ipynb | 870 ++ homeworks/hw1_embeddings/en-fr.test.txt | 2943 +++++ homeworks/hw1_embeddings/en-fr.train.txt | 10872 ++++++++++++++++ homeworks/hw2_seq2seq/README.md | 5 + .../hw2_seq2seq/lab01_nmt_24s_advanced.ipynb | 1055 ++ homeworks/hw2_seq2seq/my_network.py | 182 + homeworks/hw2_seq2seq/utils.py | 33 + .../.ipynb_checkpoints/README-checkpoint.md | 0 .../transformer-checkpoint.ipynb | 1852 +++ week05_transformer/README.md | 2 + week05_transformer/transformer.ipynb | 1852 +++ week07_LLM_v1/final_llama_practice.ipynb | 585 + 13 files changed, 20256 insertions(+) create mode 100644 homeworks/hw1_embeddings/README.md create mode 100644 homeworks/hw1_embeddings/embedding_based_MT.ipynb create mode 100644 homeworks/hw1_embeddings/en-fr.test.txt create mode 100644 homeworks/hw1_embeddings/en-fr.train.txt create mode 100644 homeworks/hw2_seq2seq/README.md create mode 100644 homeworks/hw2_seq2seq/lab01_nmt_24s_advanced.ipynb create mode 100644 homeworks/hw2_seq2seq/my_network.py create mode 100644 homeworks/hw2_seq2seq/utils.py create mode 100644 week05_transformer/.ipynb_checkpoints/README-checkpoint.md create mode 100644 week05_transformer/.ipynb_checkpoints/transformer-checkpoint.ipynb create mode 100644 week05_transformer/README.md create mode 100644 week05_transformer/transformer.ipynb create mode 100644 week07_LLM_v1/final_llama_practice.ipynb diff --git a/homeworks/hw1_embeddings/README.md b/homeworks/hw1_embeddings/README.md new file mode 100644 index 0000000..36e65f4 --- /dev/null +++ b/homeworks/hw1_embeddings/README.md @@ -0,0 +1,5 @@ +**Lab1: unsupervised MT via orthogonal embeddings projection** + +*Deadline: Sun 3.03.2024 23:59 AOE* + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/girafe-ai/ml-course/blob/24s_advanced/assignments/lab01_umt/embedding_based_MT.ipynb) diff --git a/homeworks/hw1_embeddings/embedding_based_MT.ipynb b/homeworks/hw1_embeddings/embedding_based_MT.ipynb new file mode 100644 index 0000000..7c2860f --- /dev/null +++ b/homeworks/hw1_embeddings/embedding_based_MT.ipynb @@ -0,0 +1,870 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "eulvfJWl7ueY" + }, + "source": [ + "# Lab 1\n", + "\n", + "\n", + "## Part 1: Bilingual dictionary induction and unsupervised embedding-based MT (30%)\n", + "*Note: this homework is based on materials from yandexdataschool [NLP course](https://github.com/yandexdataschool/nlp_course/). Feel free to check this awesome course if you wish to dig deeper.*\n", + "\n", + "*Refined by [Nikolay Karpachev](https://www.linkedin.com/in/nikolay-karpachev-b0146a104/), [Valery Marchenkov](https://www.linkedin.com/in/vmarchenkoff/)*" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fV4rIjxa7uei" + }, + "source": [ + "**In this homework** **YOU** will make machine translation system without using parallel corpora, alignment, attention, 100500 depth super-cool recurrent neural network and all that kind superstuff.\n", + "\n", + "But even without parallel corpora this system can be good enough (hopefully), in particular for similar languages." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "idSYq2GU7uew" + }, + "source": [ + "### Frament of the Swadesh list for some slavic languages\n", + "\n", + "The Swadesh list is a lexicostatistical stuff. It's named after American linguist Morris Swadesh and contains basic lexis. This list are used to define subgroupings of languages, its relatedness.\n", + "\n", + "So we can see some kind of word invariance for different Slavic languages.\n", + "\n", + "\n", + "| Russian | Belorussian | Ukrainian | Polish | Czech | Bulgarian |\n", + "|-----------------|--------------------------|-------------------------|--------------------|-------------------------------|-----------------------|\n", + "| женщина | жанчына, кабета, баба | жінка | kobieta | žena | жена |\n", + "| мужчина | мужчына | чоловік, мужчина | mężczyzna | muž | мъж |\n", + "| человек | чалавек | людина, чоловік | człowiek | člověk | човек |\n", + "| ребёнок, дитя | дзіця, дзіцёнак, немаўля | дитина, дитя | dziecko | dítě | дете |\n", + "| жена | жонка | дружина, жінка | żona | žena, manželka, choť | съпруга, жена |\n", + "| муж | муж, гаспадар | чоловiк, муж | mąż | muž, manžel, choť | съпруг, мъж |\n", + "| мать, мама | маці, матка | мати, матір, неня, мама | matka | matka, máma, 'стар.' mateř | майка |\n", + "| отец, тятя | бацька, тата | батько, тато, татусь | ojciec | otec | баща, татко |\n", + "| много | шмат, багата | багато | wiele | mnoho, hodně | много |\n", + "| несколько | некалькі, колькі | декілька, кілька | kilka | několik, pár, trocha | няколко |\n", + "| другой, иной | іншы | інший | inny | druhý, jiný | друг |\n", + "| зверь, животное | жывёла, звер, істота | тварина, звір | zwierzę | zvíře | животно |\n", + "| рыба | рыба | риба | ryba | ryba | риба |\n", + "| птица | птушка | птах, птиця | ptak | pták | птица |\n", + "| собака, пёс | сабака | собака, пес | pies | pes | куче, пес |\n", + "| вошь | вош | воша | wesz | veš | въшка |\n", + "| змея, гад | змяя | змія, гад | wąż | had | змия |\n", + "| червь, червяк | чарвяк | хробак, черв'як | robak | červ | червей |\n", + "| дерево | дрэва | дерево | drzewo | strom, dřevo | дърво |\n", + "| лес | лес | ліс | las | les | гора, лес |\n", + "| палка | кій, палка | палиця | patyk, pręt, pałka | hůl, klacek, prut, kůl, pálka | палка, пръчка, бастун |" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cNM3_fjr7ue2" + }, + "source": [ + "But the context distribution of these languages demonstrates even more invariance. And we can use this fact for our for our purposes." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YLppwa527ue6" + }, + "source": [ + "## Data" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MwGoVhRA7ufP" + }, + "source": [ + "In this notebook we're going to use pretrained word vectors - FastText (original paper - https://arxiv.org/abs/1607.04606).\n", + "\n", + "You can download them from the official [website](https://fasttext.cc/docs/en/crawl-vectors.html). We're going to need embeddings for English and French languages." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "KV2-MpR-ugq-", + "outputId": "36bb718d-65c0-4b32-b49d-68e73c15b7cd" + }, + "outputs": [], + "source": [ + "!wget -nc https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.vec.gz\n", + "!gzip -d cc.en.300.vec.gz\n", + "\n", + "!wget -nc https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.fr.300.vec.gz\n", + "!gzip -d cc.fr.300.vec.gz" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Kwg26PKLv88U" + }, + "source": [ + "After downloading and extracting the vectors, we should be able to load them using the [gensim](https://radimrehurek.com/gensim/) library:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "u1JjQv_97ufT" + }, + "outputs": [], + "source": [ + "from gensim.models import KeyedVectors\n", + "import numpy as np\n", + "\n", + "\n", + "en_emb = KeyedVectors.load_word2vec_format(\"cc.en.300.vec\")\n", + "fr_emb = KeyedVectors.load_word2vec_format(\"cc.fr.300.vec\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Sqb_XJhkMyHM" + }, + "source": [ + "Once you've loaded the vectors, you can use the `KeyedVectors` interface to get word embeddings and/or query most similar words by embedding:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "nTkXfT0W7ufk", + "outputId": "6b8ed7a3-f23e-4598-e494-2d5800e62280" + }, + "outputs": [], + "source": [ + "august_embedding = en_emb[\"august\"]\n", + "august_embedding.shape, august_embedding[:5]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "oQ2kCq-7NQPn", + "outputId": "0622f613-479b-4b61-bc1d-a1e2f8fe3b70" + }, + "outputs": [], + "source": [ + "en_emb.most_similar([august_embedding])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "t5EcMMI6pxzL" + }, + "source": [ + "The latter function also allows you to vary the amount of closest words via the `topn` argument:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "bi6AF3z0p9Oo", + "outputId": "420dde14-d208-4bdc-ab4b-9cab0847790c" + }, + "outputs": [], + "source": [ + "en_emb.most_similar([august_embedding], topn=3)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xw345NRXov4p" + }, + "source": [ + "Another feature of `KeyedVectors` is that it allows to compute embeddings for multiple words simultaneously:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "86OuYeLYow0C", + "outputId": "d46d5166-7817-49f8-da47-0ffc2f6cd6c5" + }, + "outputs": [], + "source": [ + "en_emb[[\"august\", \"september\"]].shape" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3uGx5zHXQtfo" + }, + "source": [ + "Everything above is true for the embeddings for French language." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "vdBA8lcg7ufs", + "outputId": "b523b412-214f-4dbe-9bc4-7a34f6771225" + }, + "outputs": [], + "source": [ + "fr_emb.most_similar([fr_emb[\"aout\"]])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "F1Dkka5uQ37-" + }, + "source": [ + "However, french and english embeddings were trained independently of each other. This means, that there is no obvious connection between values in embeddings for similar words in French and English:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "_yJvcKXO7uf0", + "outputId": "562c2733-0564-4080-f916-fec2295df753" + }, + "outputs": [], + "source": [ + "fr_emb.most_similar([en_emb[\"august\"]])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Lia_h7W2qL8C" + }, + "source": [ + "## Translation" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pNdYAR1q7uf6" + }, + "source": [ + "We'll build a simple translator, which will try to predict the french embedding from the english one. For this we'll need a dataset of word pairs." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "CXbH86oQRprk" + }, + "outputs": [], + "source": [ + "def load_word_pairs(filename):\n", + " en_fr_pairs = []\n", + " en_vectors = []\n", + " fr_vectors = []\n", + " with open(filename, \"r\") as inpf:\n", + " for line in inpf:\n", + " en, fr = line.rstrip().split(\" \")\n", + " if en not in en_emb or fr not in fr_emb:\n", + " continue\n", + " en_fr_pairs.append((en, fr))\n", + " en_vectors.append(en_emb[en])\n", + " fr_vectors.append(fr_emb[fr])\n", + " return en_fr_pairs, np.array(en_vectors), np.array(fr_vectors)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wwjYGFE7Ui0N" + }, + "source": [ + "We will train our model to predict embedding for the french word from embedding of its english counterpart. For this reason we split our train and test data into english and french words and compute corresponding embeddings to obtain `X` (english embeddings) and `y` (french embeddings)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "yPvHHq7Cc_Oa", + "outputId": "59827e18-22a5-4917-a905-db94b1a6c9a8" + }, + "outputs": [], + "source": [ + "!wget -O en-fr.train.txt https://raw.githubusercontent.com/girafe-ai/ml-course/23s_nes/homeworks/hw04_umt/en-fr.train.txt\n", + "!wget -O en-fr.test.txt https://raw.githubusercontent.com/girafe-ai/ml-course/23s_nes/homeworks/hw04_umt/en-fr.test.txt" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "K05ari5nSEcn" + }, + "outputs": [], + "source": [ + "en_fr_train, X_train, Y_train = load_word_pairs(\"en-fr.train.txt\")\n", + "en_fr_test, X_test, Y_test = load_word_pairs(\"en-fr.test.txt\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "ithG80uDTYWr", + "outputId": "5ea5c89b-7159-4392-9b90-07d0ab838c1e" + }, + "outputs": [], + "source": [ + "en_fr_train[33:44]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-ZBBNvpz7ugQ" + }, + "source": [ + "## Embedding space mapping (0.3 pts)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "x_Dhk5gL7ugS" + }, + "source": [ + "Let $x_i \\in \\mathrm{R}^d$ be the distributed representation of word $i$ in the source language, and $y_i \\in \\mathrm{R}^d$ is the vector representation of its translation. Our purpose is to learn such linear transform $W$ that minimizes euclidian distance between $Wx_i$ and $y_i$ for some subset of word embeddings. Thus we can formulate so-called [Procrustes problem](https://en.wikipedia.org/wiki/Orthogonal_Procrustes_problem):\n", + "\n", + "$$W^*= \\arg\\min_W \\sum_{i=1}^n\\|Wx_i - y_i\\|_2$$\n", + "\n", + "or\n", + "\n", + "$$W^*= \\arg\\min_W \\|XW^T - Y\\|_F$$\n", + "\n", + "where $\\|\\cdot\\|_F$ denotes Frobenius norm.\n", + "\n", + "> **Note:** in second formula, $W$ and $x$ seem to have switched places. This happens because the $X$ matrix is composed of objects $x_i$ in *rows* not *columns*, i.e. it is kind of composed of $x_i^T$. This means that $X \\in \\mathbb{R}^{N \\times D}$, where $N$ is the number of items and $D$ is the embedding dimensionality. The same is true for the $Y$." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "acOjDdtL7ugY" + }, + "source": [ + "$W^*= \\arg\\min_W \\sum_{i=1}^n\\|Wx_i - y_i\\|_2$ looks like simple multiple linear regression without bias. The `sklearn` allows you to turn off the bias in `LinearRegression` via the `fit_intercept` argument (in fact they simply call bias the intercept). So let's code." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Lb-KN1be7uga" + }, + "outputs": [], + "source": [ + "from sklearn.linear_model import LinearRegression\n", + "\n", + "\n", + "# YOUR CODE HERE\n", + "# mapping = ...\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "X7tqJwoY7ugf" + }, + "source": [ + "Let's take a look at neigbours of the vector of word _\"august\"_ (_\"aout\"_ in French) after linear transform." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "31SrFSbn7ugi", + "outputId": "7cc31f62-521e-4d10-b4db-e6b7c76aeee5" + }, + "outputs": [], + "source": [ + "august = mapping.predict(en_emb[\"august\"].reshape(1, -1))\n", + "fr_emb.most_similar(august)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "o2uY6Y9B7ugt" + }, + "source": [ + "As quality measure we will use precision top-1, top-5 and top-10 (for each transformed english embedding we count how many right target pairs are found in top N nearest neighbours in french embedding space)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "zptuho8LAfIE" + }, + "outputs": [], + "source": [ + "def precision(pairs, mapped_vectors, topn=1):\n", + " \"\"\"\n", + " :args:\n", + " pairs = list of right word pairs [(en_word_0, fr_word_0), ...]\n", + " mapped_vectors = list of embeddings after mapping from source embedding space to destination embedding space\n", + " topn = the number of nearest neighbours in destination embedding space to choose from\n", + " :returns:\n", + " precision_val, float number, total number of words for those we can find right translation at top K.\n", + " \"\"\"\n", + " assert len(pairs) == len(mapped_vectors)\n", + " total = len(pairs)\n", + " correct = 0\n", + " for i in range(total):\n", + " pair = pairs[i]\n", + " predicted_vector = mapped_vectors[i]\n", + "\n", + " # YOUR CODE HERE\n", + "\n", + " return correct / total" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "duhj9hpv7ugy" + }, + "outputs": [], + "source": [ + "assert precision([(\"august\", \"aout\")], august, topn=5) == 1.0\n", + "assert precision([(\"august\", \"aout\")], august, topn=9) == 1.0\n", + "assert precision([(\"august\", \"aout\")], august, topn=10) == 1.0" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "z5A9tWtnuFx3" + }, + "source": [ + "Note that our `precision` function accepts lists of pairs of words, whereas we have dataframes. However, it is not a problem: we can get a list (actually, numpy array) of pairs via the `values` property." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0-iyd5gP7ug5" + }, + "outputs": [], + "source": [ + "assert precision(en_fr_test[:100], X_test[:100]) == 0.0\n", + "assert precision(en_fr_test[:100], Y_test[:100]) == 1.0" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7DVV5lqrua_O" + }, + "source": [ + "Let's see how well our model is doing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "U-ssEJ3x7uhA" + }, + "outputs": [], + "source": [ + "precision_top1 = precision(en_fr_test[:100], mapping.predict(X_test[:100]), 1)\n", + "precision_top5 = precision(en_fr_test[:100], mapping.predict(X_test[:100]), 5)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "JOXKaYj1VHGC", + "outputId": "6056f077-29b4-44b2-9359-9decbe938f53" + }, + "outputs": [], + "source": [ + "print(precision_top1)\n", + "print(precision_top5)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hf6Ou8bx7uhH" + }, + "source": [ + "## Making it better (orthogonal Procrustean problem) (0.3 pts)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4oLs-drN7uhK" + }, + "source": [ + "It can be shown that a self-consistent linear mapping between semantic spaces should be orthogonal. \n", + "We can restrict transform $W$ to be orthogonal. Then we will solve next problem:\n", + "\n", + "$$(W^T)^*= \\arg\\min_{W^T} \\|XW^T - Y\\|_F \\text{, where: } W^TW = I$$\n", + "\n", + "$$I \\text{- identity matrix}$$\n", + "\n", + "Instead of making yet another regression problem we can find optimal orthogonal transformation using singular value decomposition. It turns out that optimal transformation $W^*$ can be expressed via SVD components:\n", + "$$X^TY=U\\Sigma V^T\\text{, singular value decompostion}$$\n", + "$$(W^T)^*=UV^T$$" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "DdFQ7qti7uhL" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "\n", + "# YOUR CODE HERE\n", + "# Compute the orthogonal mapping (W^T)^* as defined in formula above.\n", + "# mapping_svd = ..." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sehLFmlBysc-" + }, + "source": [ + "Now our `mapping` is just a numpy array, meaning that it has no `predict` method. However, from the formulae above we know, that prediction is done using the matrix multiplication:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "OVOFYYa37uhX", + "outputId": "0afda429-5c00-4b7c-9ec7-4bc348db2b88" + }, + "outputs": [], + "source": [ + "fr_emb.most_similar([np.matmul(en_emb['august'], mapping_svd)])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "h4qKCmq7zJDK" + }, + "source": [ + "Now let's compute our precision values and see, whether our trick did improve the results." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "r297sYP37uhb", + "outputId": "03635012-c0f1-4773-fc0e-0e7663e5a7c2" + }, + "outputs": [], + "source": [ + "print(precision(en_fr_test[:100], np.matmul(X_test[:100], mapping_svd)))\n", + "print(precision(en_fr_test[:100], np.matmul(X_test[:100], mapping_svd), 5))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hvUZ72U5AfJg" + }, + "source": [ + "## Unsupervised embedding-based MT (0.4 pts)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LLyuVfHBLrJn" + }, + "source": [ + "Now, let's build our word embeddings-based translator!" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "oa3dAZHv1wjY" + }, + "source": [ + "Now let's translate these sentences word-by-word. Before that, however, don't forget to tokenize your sentences. For that you may (or may not) find the `nltk.tokenize.WordPunctTokenizer` to be very useful." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "FGksC7l_NMi9" + }, + "outputs": [], + "source": [ + "def translate(sentence):\n", + " \"\"\"\n", + " :args:\n", + " sentence - sentence in English (str)\n", + " :returns:\n", + " translation - sentence in French (str)\n", + "\n", + " * find english embedding for each word in sentence\n", + " * transform english embedding vector\n", + " * find nearest french word and replace\n", + " \"\"\"\n", + " translated = []\n", + "\n", + " # YOUR CODE HERE\n", + "\n", + " return \" \".join(translated)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4hbbMy-tNxlf" + }, + "outputs": [], + "source": [ + "assert translate(\".\") == \".\"\n", + "assert translate(\"I walk around Paris\") == \"je marcher autour Paris\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ia6I2ce7O_HI" + }, + "source": [ + "Now you can play with your model and try to get as accurate translations as possible. **Note**: one big issue is out-of-vocabulary words. Try to think of various ways of handling it (you can start with translating each of them to a special **UNK** token and then move to more sophisticated approaches). Good luck!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "17Azt44TW9s3", + "outputId": "d230d2e5-4c2a-4e18-90cc-5227e3abfade" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd\n", + "import nltk\n", + "from nltk.corpus import stopwords\n", + "from nltk.stem import PorterStemmer\n", + "from nltk.tokenize import TweetTokenizer\n", + "from nltk.corpus import stopwords, twitter_samples\n", + "import re\n", + "import string\n", + "\n", + "nltk.download('twitter_samples')\n", + "nltk.download('stopwords')\n", + "\n", + "def process_tweet(tweet):\n", + " '''\n", + " Input:\n", + " tweet: a string containing a tweet\n", + " Output:\n", + " tweets_clean: a list of words containing the processed tweet\n", + "\n", + " '''\n", + " stemmer = PorterStemmer()\n", + " stopwords_english = stopwords.words('english')\n", + " # remove stock market tickers like $GE\n", + " tweet = re.sub(r'\\$\\w*', '', tweet)\n", + " # remove old style retweet text \"RT\"\n", + " tweet = re.sub(r'^RT[\\s]+', '', tweet)\n", + " # remove hyperlinks\n", + " tweet = re.sub(r'https?:\\/\\/.*[\\r\\n]*', '', tweet)\n", + " # remove hashtags\n", + " # only removing the hash # sign from the word\n", + " tweet = re.sub(r'#', '', tweet)\n", + " # tokenize tweets\n", + " tokenizer = TweetTokenizer(preserve_case=False, strip_handles=True,\n", + " reduce_len=True)\n", + " tweet_tokens = tokenizer.tokenize(tweet)\n", + "\n", + " tweets_clean = []\n", + " for word in tweet_tokens:\n", + " # if (word not in stopwords_english and # remove stopwords\n", + " # word not in string.punctuation): # remove punctuation\n", + " if word not in string.punctuation:\n", + " tweets_clean.append(word)\n", + " # stem_word = stemmer.stem(word) # stemming word\n", + " # tweets_clean.append(stem_word)\n", + "\n", + " return \" \".join(tweets_clean)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "nawoCF7kXLyE", + "outputId": "ec0bff98-a916-4e23-d096-ffd0d94913e7" + }, + "outputs": [], + "source": [ + "twitter_samples.strings('positive_tweets.json')[10:15]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "6XW5avSmX1CD", + "outputId": "0eea70b4-9726-46f8-dbef-fc948a2d0b7f" + }, + "outputs": [], + "source": [ + "for i in twitter_samples.strings('positive_tweets.json')[10:15]:\n", + " print(i, process_tweet(i), sep='\\n\\n', end='\\n-----------------\\n')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "x4zEK62iaxzc" + }, + "source": [ + "Your translation:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "9-lFLSclXDip" + }, + "outputs": [], + "source": [ + "for i in twitter_samples.strings('positive_tweets.json')[:10]:\n", + " print(i, process_tweet(i), translate(process_tweet(i)), sep='\\n\\n', end='\\n-----------------\\n')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PXMxWUtipDD8" + }, + "source": [ + "Great! " + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "colab": { + "machine_shape": "hm", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.16" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/homeworks/hw1_embeddings/en-fr.test.txt b/homeworks/hw1_embeddings/en-fr.test.txt new file mode 100644 index 0000000..76b919e --- /dev/null +++ b/homeworks/hw1_embeddings/en-fr.test.txt @@ -0,0 +1,2943 @@ +torpedo torpille +torpedo torpilles +giovanni giovanni +chat discuter +chat discussion +chat causerie +chat bavardage +chat chat +catholics catholiques +herald herald +chuck chuck +pit pit +pit fosse +supplied approvisionné +supplied fournis +supplied fournies +supplied fourni +supplied fournie +optional optionnelles +optional facultatif +optional facultative +optional optionnel +optional facultatives +garrison garrison +garrison garnison +sprint sprint +exile exilé +exile exil +exile exilés +surprised surprise +surprised surpris +surprised étonné +surprised étonnée +achievements réussites +achievements accomplissements +achievements réalisations +biblical biblique +biblical bibliques +rebels rebelles +denis denis +geographical géographique +sit asseoir +sit assis +sit sit +alpine alpin +alpine alpins +alpine alpine +bills factures +glacier glacier +glacier glaciers +binding liant +binding contraignant +binding reliure +indicating indiquant +estonia estonie +eating manger +saving sauver +saving épargne +saving économiser +chi chi +developer développeur +developer promoteur +developer développeurs +indie indie +difficulties difficultés +doctrine doctrine +worn usées +worn portés +worn portées +worn usé +worn porté +fork fourchette +fork fourche +fork fourches +simpson simpson +maintaining maintenir +theological théologique +upcoming prochains +upcoming prochaines +temporarily temporairement +temporarily provisoirement +temporarily momentanément +hotels hôtels +edmonton edmonton +developments développements +literacy littératie +literacy alphabétisation +currency monnaie +currency devise +currency monnaies +currency devises +missionary missionnaire +missionary missionnaires +arrives arrive +hammer marteaux +hammer hammer +hammer marteau +hammer maillet +dollar dollars +dollar dollar +ambassadors ambassadeurs +twitter twitter +centres centres +solomon salomon +solomon solomon +recommend recommander +recommend recommande +descendants descendance +descendants descendants +ruth ruth +handling maniabilité +handling manipulation +handling manutention +customs coutumes +customs douanes +customs douane +customs douanières +customs douanier +collect recueillir +collect ramasser +collect rassembler +collect collecter +collect collectionner +grid grid +grid quadrillage +grid grille +secured sécurisée +secured sécurisés +secured sécurisées +secured sécurisé +certificate certificats +certificate attestation +certificate certificat +destination destination +albania albanie +euro euros +euro euro +consumption consommation +feat exploit +feat prouesse +pushing pousser +pushing poussant +constantly constamment +survivors survivants +survivors rescapés +survivors survivant +mansion manoir +cardiff cardiff +temples temples +blake blake +sheet feuille +sheet drap +sheet feuillet +lift soulever +lift ascenseur +lift élévateur +confidence confidence +confidence confiance +cuisine cuisine +cuisine gastronomie +frankfurt frankfurt +frankfurt francfort +galaxy galaxie +galaxy galaxy +ecuador équateur +ecuador equateur +breeding élevage +breeding reproducteurs +breeding elevage +breeding reproduction +outbreak épidémie +outbreak éclosion +legendary légendaire +legendary légendaires +legendary mythique +handball handball +georgian géorgien +georgian géorgie +georgian géorgiens +georgian géorgienne +copenhagen copenhague +trek trek +ignored ignorée +ignored ignorées +ignored ignoré +ignored ignorés +arch voûte +arch arche +arch arch +keys clés +keys clefs +proceedings procédures +enjoy enjoy +enjoy profite +enjoy profiter +enjoy profitez +quartet quartet +quartet quatuor +aims buts +aims objectifs +aims finalités +propaganda propagande +disk disque +disk disquette +realized réalisé +neat soigné +funny drole +funny drôle +funny rigolo +funny marrant +funny amusant +punishment punition +punishment châtiment +punishment punitions +punishment sanction +accuracy exactitude +accuracy précision +accuracy justesse +meter compteur +meter mètre +theoretical théoriques +theoretical théorique +suspension suspension +suspension sursis +suspension suspendre +suspension suspendu +suspension suspensions +seeds semences +seeds graines +lighting éclairage +lighting eclairage +lighting luminaires +jennifer jennifer +smooth lisser +smooth lisses +smooth lisse +customer client +armstrong armstrong +involve impliquer +philosophical philosophique +philosophical philosophiques +escaped échappée +escaped échappé +escaped évadé +escaped échappés +powell powell +kills tue +taste goûter +taste saveur +taste gouter +taste goût +allmusic allmusic +requiring nécessitant +bros bros +assertion affirmation +assertion assertion +boulevard boulevard +brooks ruisseaux +brooks brooks +sending envoyant +sending envoi +atomic atomiques +atomic atomique +antarctica antarctique +strikes frappes +strikes grèves +reconstruction reconstitution +reconstruction reconstruction +chronicle chroniques +chronicle chronique +traveling voyager +traveling voyageant +leslie leslie +ellis ellis +devon devon +ghana ghana +gen gen +rebel rebelles +rebel rebel +rebel rebelle +duncan duncan +pianist pianiste +canon canonique +canon canon +reformed réformé +reformed réformée +pack pack +iceland islande +solve résoudre +solve résout +cyclists cyclistes +payment payement +payment paiements +payment paiement +suburbs faubourgs +suburbs banlieue +suburbs banlieues +militia miliciens +militia milice +militia milices +pronounced prononcée +pronounced prononcé +pronounced prononcés +exhibit exposer +exhibit exposition +mph mph +glen glen +glen vallon +eugene eugène +eugene eugene +compromise compromis +compromise compromettre +tactical tactique +tactical tactiques +discovers découvre +switched basculé +uganda ouganda +jail prison +yeah ouais +yeah oui +yeah yeah +yeah ouai +withdraw retirer +withdraw retirez +holmes holmes +promise promesses +promise promettre +promise promis +promise promesse +promise promets +convert convertir +convert convertissent +convert convertit +dos dos +noting notant +noting constatant +recall rappeler +recall rappel +arrive arriver +arrive arrivez +warrior guerrière +warrior warrior +warrior guerrier +mammals mammifères +mammals mammifère +dimensions dimensions +dimensions dimension +surrey surrey +gaming jeux +gaming jeu +lutheran luthérienne +ports ports +amy amy +survival survie +survival survivre +responses réponses +collegiate collégiale +collegiate collégial +scandal scandales +scandal scandale +widow veuves +widow veuve +widow veuf +swing balançoire +swing swing +nights nuitées +nights nuits +nights soirs +polo polo +linda linda +adr adr +probability probabilités +probability probabilité +probability vraisemblance +farms fermes +farms exploitations +conferences conférences +conferences colloques +zhang zhang +crazy dingue +crazy fou +crazy fous +crazy cinglé +crazy folle +witness témoins +witness témoin +nephew neveu +sensitive sensible +sensitive délicat +sensitive sensibles +mutual mutuelle +mutual réciproque +mutual mutuel +mutual mutuelles +diet diète +diet diététique +diet régime +clients clients +fringe frange +fringe fringe +fringe marginale +fringe franges +passion passions +passion passion +rings bagues +rings anneaux +millions millions +dialect dialecte +orlando orlando +relay relayer +relay relais +wet humide +wet mouillé +wet mouillés +wet mouillée +wet humides +cruise cruise +cruise croisière +cruise croisières +henri henri +publish publier +publish publie +publish publiez +joy joy +joy joie +julia julia +kitchen cuisine +abstract abstraite +abstract abstrait +abstract abstraits +abstract résumé +snake serpents +snake vipère +snake snake +snake serpent +comedian comique +comedian comédien +comedian humoriste +motorcycle motocyclette +motorcycle motocycle +motorcycle motocyclettes +motorcycle moto +nadu nadu +arsenal arsenal +millennium millénaire +millennium millenium +millennium millennium +assists assiste +bow arc +bow bow +andré andré +serie serie +dimensional dimensionnelle +travelled parcourus +travelled voyagé +travelled parcouru +eurovision eurovision +suite suite +doug doug +gravity gravitation +gravity pesanteur +gravity gravité +stored stocké +stored stockés +stored entreposé +stored stockées +departed défunts +optical optique +optical optiques +frontier frontière +frontier frontier +evaluation évaluation +evaluation evaluation +graph graphique +graph graphe +hybrid hybrides +hybrid hybride +oslo oslo +earn gagnez +earn gagner +earn mériter +metre mètre +keyboard clavier +keyboard claviers +jamie jamie +decorated décorées +decorated décorée +decorated décorés +decorated décoré +complicated compliqué +complicated compliquées +complicated complique +complicated compliquée +nathan nathan +slavery esclavage +circular circulaire +circular circulaires +operators exploitants +operators opérateurs +armor blindage +armor armures +armor armure +mechanics mécano +mechanics mécaniciens +mechanics mécanique +mechanics mécanicien +mechanics mécaniques +bradford bradford +leon léon +leon león +leon leon +rachel rachel +strings chaînes +strings cordes +header entête +hood hood +hood capot +hood cagoule +hood capuche +hood capuchon +inspector inspectrice +inspector inspecteur +warnings avertissements +plains plains +plains plaine +plains plaines +defended défendues +defended défendu +defended défendus +defended défendue +wheels roulettes +wheels roues +criterion critère +ace as +ace eca +ace ace +arrangements arrangements +penn penn +approached approchée +approached approchés +approached approché +joke blague +joke plaisanter +joke plaisanterie +sailed navigué +religions religions +religions cultes +grants bourses +grants subventions +andrews andrews +moderate modéré +moderate modérée +moderate modérées +moderate modérés +stolen volée +stolen volées +stolen volé +stolen volés +stolen dérobée +tributary affluent +pin broche +pin épingle +pin pin +pin épingler +carol carol +carol carole +owns possède +prototype prototype +prototype prototypes +copied copié +copied copiés +copied copiée +canterbury canterbury +midnight minuit +midnight midnight +quarterback quarterback +quarterback quaterback +duchy duché +bailey bailey +arbitrators arbitres +performers interprètes +handled manipulée +handled manipulé +handled géré +handled manipulés +handled manipulées +exploration prospection +exploration exploration +diversity diversité +sixteen sixteen +sixteen seize +findings constatations +findings conclusions +repeat répéter +repeat répète +repeat répétition +repeat répétez +brussels bruxelles +planets planete +planets planètes +theatrical théâtrale +theatrical théâtral +reconnaissance reconnaissances +reconnaissance reconnaissance +shots coups +complaint réclamation +complaint plainte +complaint plaintes +batman batman +exhibited exposées +espn espn +investigate enquêter +verify vérifie +verify vérifier +verify vérifiez +discontinued arrêté +discontinued interrompu +absent absents +absent absent +absent absentes +absent absente +girlfriend copine +resignation résignation +resignation démission +fossil fossile +fossil fossiles +explaining expliquant +explaining expliquer +tang tang +inches pouces +inches centimètres +proven prouvés +proven prouvées +proven prouvée +proven éprouvé +proven démontré +proven prouvé +franco franco +dying mourante +dying mourir +dying mourant +tribal tribaux +tribal tribales +tribal tribal +tribal tribale +tyler tyler +surrender capitulation +surrender reddition +glenn glenn +substance substance +focusing focalisation +luxembourg luxembourg +colored coloré +colored colorée +scholarly érudit +administered administrée +administered administrés +administered administré +explosion explosion +explosion explosions +pushed poussé +pushed poussés +generations générations +duck canards +duck canard +duck duck +porter porteur +porter portier +porter porter +permanently définitivement +memphis memphis +salvador salvador +emma emma +mit mit +zoo zoos +zoo zoologique +zoo zoo +gibson gibson +wording libellé +emerging naissants +emerging émergents +emerging émergent +emerging émergente +portions portions +macedonia macédoine +ethics déontologie +ethics éthique +depot dépôt +depot depot +curtis curtis +rescued secouru +rescued sauvés +rescued secourus +rescued sauvée +rescued sauvé +gaelic gaélique +slovakia slovaquie +elevated élevées +elevated élevée +elevated élevés +elevated élevé +jeremy jérémie +jeremy jeremy +jeremy jérémy +listen ecouter +listen ecoutez +listen écoutez +listen ecoute +listen écouter +listen écoute +impressive impressionnante +impressive impressionnantes +impressive impressionnant +impressive impressionnants +bradley bradley +surely sûrement +surely surement +egg ovule +egg œuf +egg œufs +egg oeuf +egg oeufs +conquest conquête +conquest conquêtes +rod baguette +rod rod +rod canne +rod tige +cdp cdp +algorithm algorithme +burn brûlure +burn graver +burn brûler +burn brûlent +thesis thèses +thesis thèse +lover amoureux +lover amant +capitol capitole +capitol capitol +ferdinand ferdinand +marshal marshal +marshal maréchal +judaism judaïsme +balls boules +balls couilles +balls ballons +balls balles +balls billes +nacional nacional +wrestlers lutteurs +ahmed ahmed +sin pêché +sin pécher +sin péché +sin sin +sin péchés +holocaust holocauste +edgar edgar +saxophone saxophone +retain conserver +retain retenir +wishes voeux +wishes désirs +wishes souhaits +wishes souhaite +wishes vœux +prepare prépare +prepare préparer +prepare préparez +ruins ruines +ruins ruine +ibm ibm +rochester rochester +nigerian nigérien +nigerian nigériane +nigerian nigérian +jesse jesse +malaysian malaisien +malaysian malaisienne +atlas atlas +telegraph telegraph +telegraph télégraphe +telegraph télégraphique +performer interprète +cannon cannon +cannon canon +cannon canons +encounter rencontre +emily émilie +emily emilie +emily emily +dissolved dissoute +dissolved dissout +dissolved dissoudre +dissolved dissous +catalogue catalogue +discrimination discrimination +discrimination discriminations +myspace myspace +reveal dévoiler +reveal révéler +reveal révèlent +wizard assistant +wizard sorcier +wizard magicien +teen teen +teen adolescent +teen ado +spots spots +spots taches +bomber bombardier +bomber poseur +foods nourritures +foods aliments +quest quest +quest quête +connor conner +connor connor +screenplay scénario +motors moteurs +minimal minime +minimal minimale +minimal minimaliste +minimal minimum +minimal minimes +minimal minimal +muscle musculaire +muscle muscles +muscle muscle +muscle musclé +prestigious prestigieux +prestigious prestigieuse +prestigious prestigieuses +sustainable durable +sustainable durables +sustainable soutenable +chelsea chelsea +strict rigoureuse +strict strict +strict rigoureux +strict strictes +strict stricte +kingston kingston +sheep moutons +sheep mouton +sheep brebis +sheep ovins +andrea andrea +complaints plainte +complaints plaintes +complaints réclamations +connects relie +connects connecte +nursing infirmier +nursing infirmières +defenders défenseurs +richardson richardson +triangle triangulaire +triangle triangle +triangle triangles +nato otan +teeth dentition +teeth dents +occasional occasionnels +occasional occasionnellement +occasional occasionnelles +occasional occasionnelle +occasional occasionnel +strictly rigoureusement +strictly strictement +harper harper +fluid fluides +fluid fluide +fed fed +fed nourri +fed nourris +newfoundland neuve +disbanded dissoute +disbanded dissous +disbanded démantelé +comparable comparables +comparable comparable +documentation documentation +brien brien +compounds composés +pointing pointage +pointing pointant +edmund edmund +edmund edmond +naturally naturellement +forcing forçant +forcing forcer +ussr urss +laser laser +laser lasers +lat lat +lat lats +sculptor sculpteur +guild guilde +observer observateurs +observer observateur +observer observatrice +worlds mondes +imprisoned incarcérés +imprisoned emprisonnés +imprisoned emprisonné +wrestler lutteur +wrestler catcheur +praise louanges +praise éloge +praise louange +praise éloges +parishes paroisses +bones bones +bones os +css css +cox cox +contracts contrats +consequences conséquences +provisions dispositions +provisions provisions +circulation circulation +butterfly butterfly +butterfly papillons +butterfly papillon +hugo hugo +abolished abolis +abolished abolie +abolished aboli +algeria algérie +edu edu +sufficiently suffisamment +armies armées +separation séparation +spy espion +spy espionne +spy espionner +spy espions +spy espionnage +cliff falaises +cliff falaise +cliff cliff +technically techniquement +reactions réactions +lithuanian lituanie +lithuanian lituaniens +lithuanian lituanien +lithuanian lituanienne +trick trick +trick astuce +curve courbe +accidents accidents +horizontal horizontales +horizontal horizontaux +horizontal horizontal +horizontal horizontalement +horizontal horizontale +uploader uploader +legends légendes +enzyme enzyme +enzyme enzymatique +freight marchandises +freight fret +hydrogen hydrogène +broadcasts émissions +broadcasts diffusions +viii viii +caroline caroline +pull tirer +pull tirez +plymouth plymouth +twentieth vingtième +twentieth twentieth +twentieth xxe +cuts découpes +cuts coupures +cuts coupes +mediation médiation +airfield aérodrome +catalog catalogue +dale dale +synthesis synthèses +synthesis synthèse +rape viols +rape viol +rape colza +rape violer +seoul séoul +engagement engagement +engagement fiançailles +coin pièce +lucy lucie +lucy lucy +platinum platine +platinum platinum +twins jumelles +twins jumeaux +memories mémoires +memories souvenirs +robertson robertson +verified vérifiés +verified vérifiée +verified vérifiées +verified vérifié +anthology anthologie +milton milton +geological géologiques +geological géologique +defining définition +defining définir +defining définissant +dinner dîner +dinner souper +dinner diner +hosting hébergeur +hosting hébergement +thriller suspense +thriller thriller +retreat retraite +albany albany +abdul abdul +abdul abdel +ignore ignore +ignore ignorez +ignore ignorer +migration émigration +migration migration +migration migrations +carefully prudemment +carefully soigneusement +carefully attentivement +magnitude ampleur +magnitude magnitude +magnitude grandeur +sudan soudan +manages gère +duration duree +duration durée +henderson henderson +explorer explorateurs +explorer explorer +explorer explorateur +marco marco +fusion fusion +aids sida +aids aides +gathered rassemblés +gathered recueillies +reflected reflété +reflected réfléchie +afraid peur +afraid effrayée +afraid effrayé +presbyterian presbytérienne +automobile automobile +fault faute +fault faille +pound dièse +pound livre +allegedly prétendument +delay retarder +delay retard +delay délai +developers promoteurs +developers développeurs +belfast belfast +arctic arctique +kurt kurt +mayors maires +windsor windsor +assumption assomption +assumption supposition +assumption hypothèse +plates assiettes +plates plaques +fourteen fourteen +fourteen quatorze +nominee nominée +nominee nominé +disruption perturbations +disruption perturbation +monroe monroe +hearts coeurs +hearts cœur +hearts cœurs +hearts coeur +belgrade belgrade +victories victoires +extending étendre +pale pâles +pale pale +pale pâle +pursuit poursuite +glory glorieux +glory gloire +glory glory +destroyer destructeur +destroyer destroyer +deeply profondément +lectures conférences +affiliate affilié +affiliate affiliés +preston preston +deceased décédée +deceased décédés +deceased défunt +deceased décédé +speaks parle +gathering rassemblement +gathering cueillette +angry énervé +angry colère +angry furieux +angry furieuse +angry fâché +incomplete incomplets +incomplete inachevée +incomplete incomplet +incomplete incomplète +incomplete incomplètes +enrolled inscrits +enrolled inscrit +configuration configuration +brad brad +skill compétence +skill habileté +skill compétences +intense intense +intense intenses +tasmania tasmanie +commitment engagement +loved adoré +loved aimée +loved aimées +loved aimé +loved aimés +reforms réformes +rulers souverains +rulers dirigeants +uruguay uruguay +sustained soutenue +sustained soutenu +napoleon napoléon +confirm confirmez +confirm confirmation +confirm confirmer +breed race +auxiliary auxiliaire +auxiliary auxiliaires +enabled activées +enabled activés +enabled activée +enabled activé +discography discographie +licence permis +licence licence +refugees réfugiés +adrian adrien +adrian adrian +pipe pipe +pipe tuyau +karen karen +altered altéré +altered altérée +budapest budapest +designers concepteurs +designers designers +designers créateurs +designers dessinateurs +heir héritier +heir héritière +advisor conseiller +advisor conseillère +illustrate illustrer +illustrate illustrent +authorized autorisées +authorized autorisé +authorized autorisés +authorized autorisée +hide masquer +hide cache +hide cacher +announcement annonce +compact compact +compact compresser +compact compactes +compact pacte +compact compacts +compact compacte +particles particules +refuses refuse +receiver récepteur +receiver destinataire +receiver receveur +civilians civils +civilians civiles +marsh marécage +marsh marais +marsh marsh +vinyl vinyles +vinyl vinyle +vinyl vinyl +delayed retardée +delayed retardé +delayed retard +delayed retardés +delayed différé +encountered rencontrée +encountered rencontrées +encountered rencontré +encountered rencontrés +wednesday mercredi +chilean chilienne +chilean chilien +chilean chiliens +hey salut +hey hé +hey hey +chambers chambers +demo démonstration +demo demo +demo démo +demo démos +ahmad ahmad +santos santos +paying payant +paying payer +interpreted interprétés +interpreted interprété +interpreted interprétée +submit soumettre +desired désirée +desired désiré +desired souhaités +desired souhaitée +desired souhaité +followers adeptes +followers abonnés +observatory observatoires +observatory observatoire +problematic problématique +problematic problématiques +springfield springfield +kit kits +kit trousse +kit kit +remarks remarques +remarks observations +burton burton +coached entraînée +monarch monarque +observations observations +beetle coccinelle +beetle scarabée +promised promise +promised promises +promised promis +palomar palomar +cream crèmes +cream crème +cream cream +presenter présentateur +potter potter +copyvio copyvio +favourite préféré +favourite préférée +favourite favori +favourite favoris +favourite préférés +transformation transformation +mcdonald mcdonald +bavaria bavière +kumar kumar +nineteenth xixe +nineteenth nineteenth +severely gravement +severely sévèrement +mixture mélange +browser navigateur +endangered menacées +endangered menacée +endangered menacés +endangered menacé +mate mate +lyon lyonnais +lyon lyon +illustration illustration +kyle kyle +afl afl +brook brook +brook ruisseau +geometry géométrique +geometry géométrie +ping ping +extends étend +aggregate agrégat +aggregate agrégats +variants variantes +baroque baroque +baroque baroques +iso iso +collapsed effondré +collapsed écroulé +collapsed effondrée +integral intégral +integral intégrale +integral intégrales +jake jake +hopes espoirs +cornell cornell +modes modes +servant serviteur +servant servante +kenny kenny +hurt blessé +hurt blessée +hurt blesser +inline inline +carlo carlo +lynn lynn +stability stabilité +hoping espérant +hoping espérer +imposed imposés +imposed imposée +imposed imposées +imposed infligée +imposed imposé +confusing confusion +confusing confus +confusing déroutant +summaries synthèses +summaries résumés +summaries sommaires +beetles coléoptères +joel joël +joel joel +jets jets +logos logos +vital vitales +vital vitale +vital vital +vital vitaux +malcolm malcolm +winnipeg winnipeg +kilometers kilomètres +kilometers kilomètre +kilometers kilométrage +songwriters compositeurs +buddhism bouddhisme +nose nez +respected respectés +respected respecté +respected respectée +respected respectées +pace pace +pace rythme +thunder thunder +thunder tonnerre +centered centrée +centered centré +centered centrés +centered centrer +centered centrées +physicians médecins +bolivia bolivie +forget oubli +forget oublions +forget oublier +forget oublie +forget oubliez +implies implique +crops cultures +crops récoltes +halifax halifax +toll péage +toll péages +monk monk +monk moine +extraordinary extraordinaire +extraordinary extraordinaires +lessons leçons +lessons enseignements +pub pub +paralympics paralympiques +monte monte +maría maría +segments segments +deer cerf +deer cerfs +deer daim +deer biche +deer chevreuil +wireless wireless +commenced commencée +mysterious mystérieuses +mysterious mystérieux +mysterious mystérieuse +consultant consultant +consultant conseiller +consultant conseillère +consultant consultante +fraser fraser +formats formats +jam jam +jam confiture +jam confitures +chicken poulet +chicken poule +chicken volaille +chicken poulets +enable activer +idol idol +idol idole +reid reid +births naissances +births accouchements +amazing incroyable +amazing extraordinaire +amazing amazing +amazing étonnant +pet pet +pet animal +upset bouleversé +upset bouleversée +upset contrariée +upset contrarié +loves aime +loves amours +loves adore +stretch tronçon +stretch stretch +stretch étirer +stretch étirement +nominate désigner +striking frappante +striking frappant +striker buteur +striker striker +striker attaquant +accidentally accidentellement +louisville louisville +hopkins hopkins +eds eds +goddess déesses +goddess déesse +resumed repris +resumed reprise +satisfy satisfont +satisfy satisfaire +notion notion +voltage tension +voltage voltage +betty betty +marion marion +geology géologie +cyclone cyclone +export exportations +export exporter +export exportation +lightning lightning +lightning éclairs +lightning éclair +lightning foudre +impressed impressionné +impressed impressionnés +impressed impressionnée +maintains maintient +maintains entretient +logical logique +logical logiques +aggressive agressif +aggressive agressifs +aggressive agressive +aggressive agressives +aggressive agressivité +jin jin +julie julie +fbi fbi +yankees yankees +ludwig ludwig +pond étang +suburban banlieue +enlisted enrôlé +enlisted enrôlés +enlisted engagé +moments moments +moments instants +conjunction conjonction +interim provisoire +interim intérimaire +interim intérimaires +interim intérim +lucky chanceux +lucky chanceuse +lucky lucky +lucky veinard +targeted ciblée +targeted ciblés +targeted ciblé +targeted ciblées +lon lon +speedway speedway +regiments régiments +picks pics +prevented empêchée +prevented empêché +toy toy +toy jouet +toy jouets +bicycle bicyclettes +bicycle vélo +bicycle bicyclette +purely purement +interactions interactions +fraud fraude +fraud imposture +fraud escroquerie +fraud fraudes +fraud imposteur +lang lang +arcade arcades +arcade arcade +lecture conférence +sanctuary sanctuaires +sanctuary sanctuaire +dragons dragons +copa copa +careful prudent +careful prudents +nurse infirmier +nurse infirmière +rivals rivaux +module module +supplement suppléments +supplement complément +supplement supplément +lens lens +lens objectif +lens lentille +lens lentilles +patron patron +patron mécène +patron patronne +commands commandements +commands commandes +trend trend +trend tendance +superintendent surintendant +gerald gerald +gerald gérald +rap rappeur +rap rappeurs +rap rap +geneva geneve +geneva genève +ash frêne +ash cendres +ash ash +ash cendre +blade blade +blade lame +disappeared disparues +disappeared disparue +disappeared disparu +disappeared disparus +patrolling patrouiller +patrolling patrouilles +predominantly principalement +predominantly majoritairement +committees commissions +committees comités +boom boum +boom boom +sailors marins +sailors matelots +beaten vaincu +beaten battues +beaten battue +beaten battu +beaten battus +smoke fumer +smoke fumée +smoke fumées +assassination assassinats +assassination assassiner +assassination assassinat +lancaster lancaster +reynolds reynolds +divorce divorcer +divorce divorce +divorce divorces +dust dust +dust poussière +dust poussières +saxon saxons +saxon saxon +saxon saxonne +separately séparément +grain grain +grain grains +executives cadres +executives exécutifs +executives dirigeants +translations traductions +translations traduction +zimbabwe zimbabwe +thrown jetés +thrown jeté +cohen cohen +diving plongeon +diving plongeur +diving plongée +neighbouring voisines +neighbouring voisine +neighbouring voisin +carroll carroll +accounting comptables +accounting comptable +accounting comptabilité +mesa mesa +mesa missa +prussia prusse +intelligent intelligente +intelligent intelligents +intelligent intelligent +intelligent intelligentes +cherry cherry +cherry cerisier +cherry cerise +cherry cerises +tobacco tabac +tobacco tabagisme +cleaned nettoyées +cleaned nettoyé +cleaned nettoyés +varieties cépages +varieties variétés +bench banquette +bench banc +directions itinéraire +directions directions +ellen hélène +ellen ellen +padding rembourrage +measurement mesures +measurement mesurage +measurement mesurer +measurement mesure +paradise paradis +paradise paradisiaque +paradise paradise +alexandria alexandrie +complement compléter +complement complément +witch sorcière +witch sorcières +attraction attraits +attraction attraction +attraction attirance +attraction attirer +attraction attrait +diana diana +personalities personnalités +colleagues collègues +busy occupé +busy occupés +busy occupée +cia cia +screenwriter scénariste +rankings palmarès +rankings classements +aboriginal aborigène +aboriginal autochtones +aboriginal aborigènes +aboriginal autochtone +commanders commandants +salem salem +wagner wagner +sanctions sanctions +americas amérique +americas amériques +endings fins +instructor moniteur +instructor instructeur +nobility noblesse +divorced divorcées +divorced divorce +divorced divorcée +divorced divorcé +divorced divorcés +varies varie +varies varient +tomorrow demain +tomorrow tomorrow +manuscripts manuscrits +unified unifiée +unified unifiées +unified unifié +unified unifiés +clarify clarifier +scouts éclaireurs +scouts scoutisme +scouts recruteurs +scouts scouts +investigations enquêtes +silva silva +derek derek +agenda agenda +provision provision +provision disposition +humanity humanité +admit admettre +admit admets +admit admet +terror terreur +terror terrorisme +contestants concurrents +trinidad trinité +trinidad trinidad +distant distante +distant distants +distant distant +distant lointaine +distant lointain +burke burke +circles cercles +assignment affectation +assignment cession +assignment assignation +recalled rappelées +recalled rappelé +recalled rappelés +shrine sanctuaire +shrine autel +sail voiles +sail voilier +sail voile +sail naviguer +willie willie +karnataka karnataka +celebrate fêter +celebrate célébrer +ranch ranch +collaborated collaboré +vampire vampire +vampire vampires +playwright dramaturge +sick malades +sick malade +associates associés +heinrich heinrich +ethiopia éthiopie +ethiopia ethiopie +flags pavillons +flags drapeaux +tel tél +tel tel +drove conduisait +learns apprend +shorts short +shorts shorts +accomplished accompli +accomplished accomplis +accomplished accomplie +autobiography autobiographie +recruited recrutée +recruited recruté +recruited recrutés +uprising révolte +uprising insurrection +uprising soulèvement +edwin edwin +velocity vitesse +terminology terminologie +raiders pillards +raiders raiders +coordinates coordonnées +coordinates coordonnés +brighton brighton +viola alto +viola viola +para para +morrison morrison +propulsion propulsion +boxer boxer +boxer boxeur +finale finale +shoulder épaules +shoulder épaule +disabled handicapé +disabled handicapés +disabled désactivés +disabled désactivée +disabled désactivé +joins rejoint +div div +tactics tactique +tactics tactiques +ernst ernst +innocent innocents +innocent innocentes +innocent innocente +innocent innocent +rapper rappeur +privacy intimité +privacy confidentialité +boeing boeing +cites cites +emmy emmy +indo indo +distinguish distinguer +rosa rosa +thermal thermale +thermal thermique +thermal thermiques +flute flûte +flute flûtes +marines marines +feminist féministe +feminist féministes +trustees administrateurs +trustees fiduciaires +trustees mandataires +sculptures sculptures +bacteria bactériennes +bacteria bactérienne +bacteria bactéries +bacteria bactérie +introduce introduire +landmarks repères +disorders affections +disorders troubles +rivalry rivalité +rivalry rivalités +prevention prévention +honored honorées +honored honorée +honored honorés +honored honoré +healthy sain +healthy saine +circus cirque +circus cirques +speculation spéculations +speculation spéculation +burma birmanie +sec sec +quiet silencieuse +quiet tranquille +quiet calme +quiet silencieux +knee genoux +knee genou +deliver livrer +hypothesis hypothèses +hypothesis hypothèse +referendum referendum +referendum référendum +travelling voyager +estonian estonienne +estonian estonien +estonian estoniens +pastor pasteur +sofia sofia +tribune tribune +permit permis +priority priorité +priority priorités +priority prioritaire +priority prioritaires +cent cent +consequence conséquence +rica rica +furniture ameublement +furniture meubles +furniture mobilier +furniture meuble +macdonald macdonald +honest honnête +honest honnêtes +innovative novatrices +innovative innovante +innovative innovantes +innovative innovatrice +innovative innovant +innovative innovateur +estimate estimation +estimate devis +estimate estimations +atp atp +rotation rotation +syracuse syracuse +lecturer conférencier +automated automatisée +automated automatisé +obscure obscures +obscure obscure +obscure obscurs +obscure obscur +kosovo kosovo +classics classiques +julius julius +julius jules +appreciated appréciés +appreciated apprécié +appreciated appréciée +appreciated appréciées +naples naples +sebastian sebastian +sebastian sébastien +sebastian sebastien +activated activées +activated activés +activated activée +activated activé +varied variée +varied varié +varied variées +varied variés +offense offense +advised conseillé +barnes barnes +acknowledged reconnu +acknowledged reconnue +acknowledged reconnus +exceptions exceptions +exceptions exception +exceptions dérogations +martha martha +martha marthe +quarters quarts +quarters trimestres +drawings dessins +refuge refuge +maharashtra maharashtra +conventions conventions +elliott elliott +elliott elliot +diplomat diplomates +diplomat diplomate +unused inutilisées +unused inutilisés +unused inutilisée +unused inutilisé +searches recherches +brigadier brigadier +particle particule +particle particules +malayalam malayalam +thursday jeudi +icon icônes +icon icône +ulster ulster +genes gènes +infinite infinie +infinite infinies +infinite infini +considerably considérablement +vale vale +portraits portraits +paste coller +paste colle +paste pâte +randy randy +saxony saxe +convoy convois +convoy convoi +annie annie +excessive excessif +excessive excessifs +excessive excessive +believing croyant +believing croire +rhine rhin +mineral minéral +mineral minéraux +mineral minérales +implement implémenter +surgeon chirurgien +badge insigne +badge badge +charleston charleston +clause clause +infection infections +infection infection +electron électrons +electron électron +walt walt +cnn cnn +likewise pareillement +tonight tonight +confederation confédération +casino casino +doctorate doctorat +guatemala guatemala +guatemala guatémaltèque +settings paramètres +settings réglages +settings configuration +settings paramétrage +mask masques +mask masquer +mask masque +shelter abris +shelter abri +dorothy dorothée +dorothy dorothy +ethnicity ethnie +ethnicity ethnicité +ethnicity ethnique +hopefully espérons +elimination elimination +elimination élimination +heath bruyère +heath lande +heath heath +pregnant enceinte +pregnant enceintes +richards richards +theodore theodore +theodore théodore +delegates délégués +blair blair +fac fac +phrases phrases +phrases expressions +crashed accidenté +crashed crashé +crashed écrasé +preference préférence +preference préférences +janeiro janeiro +concerto concerto +bits bits +construct construire +tune tune +unofficial officieux +unofficial officieuse +bulk vrac +lighthouse phare +stan stan +highland highland +highland highlands +mascot mascotte +squadrons escadrons +acceptance acceptation +tight serrée +tight serré +tight serrés +tight étroit +considers considère +hub hub +mess mess +mess désordre +mess bordel +mess pagaille +wilderness sauvage +routine routine +dubbed surnommé +dozens douzaine +dozens dizaines +dozens douzaines +spotted repérés +spotted repéré +spotted repérée +harmony harmonie +harmony harmony +entrepreneur entrepreneur +entrepreneur entrepreneurs +wwe wwe +apollo apollo +apollo apollon +runway piste +naked nues +naked nue +naked nu +naked nus +anton anton +moses moïse +moses moses +legally juridiquement +legally légalement +fake faux +fake imposteur +fake simuler +fake fausse +fake fake +biased partiale +biased partial +revolt révolte +revolt révoltes +equity égalité +equity équité +varying variant +providence providence +investors investisseurs +reliability fiabilité +tenor ténor +fights combats +pocket pocket +pocket poches +pocket poche +sad tristes +sad tristesse +sad triste +sad sad +troy troie +troy troy +treasure trésors +treasure trésor +ion ion +ion ionique +ion ions +rendered rendue +rendered rendu +rendered rendues +rendered rendus +transformed transformée +transformed transformés +transformed transformées +transformed transformé +roberto roberto +adoption adoption +decrease diminution +decrease diminuer +decrease réduire +decrease baisse +reserved réservés +reserved réservée +reserved réservé +forgotten oubliés +forgotten oubliées +forgotten oublié +forgotten oubli +forgotten oubliée +lok lok +crop récolte +licensing licences +advocacy sensibilisation +advocacy plaidoyer +collecting collecter +collecting collectionner +collecting collecte +treasury trésorerie +treasury trésor +trumpet trompette +johnston johnston +uncertain incertain +uncertain incertaine +uncertain indécis +uncertain incertitude +norton norton +collector collecteur +collector collectionneur +collector percepteur +cluster cluster +cluster amas +dear chère +dear cher +dear chers +georges georges +roller rouleaux +roller roller +roller rouleau +clothes vêtements +clothes vêtement +sovereign souveraines +sovereign souverains +sovereign souveraine +sovereign souverain +enhanced améliorés +enhanced renforcée +enhanced améliorée +enhanced amélioré +compensation dédommagement +compensation indemnité +compensation indemnisation +compensation rémunération +consent consentement +outline esquisse +holdings exploitations +holdings participations +holdings dotations +jorge jorge +darkness noirceur +darkness pénombre +darkness obscurité +darkness ténèbres +penalties pénalités +penalties peines +penalties pénalité +penalties sanctions +bombers bombardiers +holes trous +blow souffler +blow souffle +blow coup +cooking cuisiner +cooking cuisine +cooking cuisson +aftermath contrecoup +aftermath séquelles +trainer dresseur +trainer formateur +trainer entraîneur +trainer entraineur +measuring mesurage +measuring mesurer +measuring mesure +lawsuit poursuites +lawsuit procès +chip chip +chip jeton +chip puce +consciousness conscience +archaeology archéologie +latvia lettonie +telugu telugu +blogs blogue +blogs blogs +protecting protéger +protecting protégeant +hardy hardy +nicknamed surnommé +nicknamed surnommée +scorer marqueur +stamp cachet +stamp timbre +stamp timbres +stamp poinçon +stamp tampon +nat nat +fur fourrure +fur fourrures +fur pelage +redirected redirigé +estimates estimations +lit allumé +lit éclairé +lit allumée +lit allumés +ritual rituels +ritual rituel +locality localité +trace tracé +trace traces +trace tracer +trace trace +marble bille +marble marbres +marble marbre +foundations fondements +foundations fondations +politically politiquement +nottingham nottingham +derivative dérivés +derivative dérivé +derivative dérivée +boxers caleçon +boxers boxeurs +dimension dimension +touchdowns touchdowns +crawford crawford +bats chauves +bats battes +yugoslav yougoslaves +yugoslav yougoslave +tanzania tanzanie +succeed réussir +motto devise +concentrated concentré +concentrated concentrés +concentrated concentrée +concentrated concentrées +dirty cochonne +dirty sale +dirty sales +hayes hayes +xbox xbox +identifying identifier +likes aime +genres genres +galleries galeries +forbes forbes +adequate adéquate +adequate adéquates +adequate adéquat +brass cuivre +brass cuivres +brass laiton +bach bach +alias alias +alias pseudonyme +alias pseudonymes +inland intérieure +wore portaient +wore portait +tough dur +tough coriace +advertisement publicité +advertisement annonce +protagonist protagoniste +trails sentiers +demanded exigé +claire claire +mistakes fautes +mistakes erreurs +bruno bruno +dylan dylan +bag sac +bag sachet +bag sacoche +churchill churchill +tan tan +tan bronzage +tan bronzé +climb grimper +witnesses témoins +thames tamise +kazakhstan kazakhstan +presenting présentant +highlights souligne +jumping sauter +jumping sauts +jumping sautant +jumping saut +prof prof +slovak slovaque +slovak slovaques +skull crâne +skull crânes +missionaries missionnaires +ordained ordonnés +ordained ordonné +hoped espéraient +hoped espéré +hoped espérait +myth mythes +myth mythe +mandatory obligatoire +mandatory obligatoires +mandatory obligatoirement +stern poupe +stern sévère +stern stern +fees frais +fees honoraires +bet bet +bet pari +bet parie +bet parier +monks moines +dancers danseuses +dancers danseurs +quantity quantités +quantity quantité +inventor inventeur +cairo caire +graves tombes +graves tombeaux +graves graves +graves fosses +proximity proximité +sue sue +armament armements +armament armement +barrier barrière +creatures créatures +logan logan +erik erik +leicester leicester +silence silences +silence silence +jessica jessica +plateau plateau +finite finie +finite finies +precedent précédent +stationed stationnés +stationed stationné +walsh walsh +zones zones +intensity intensités +intensity intensité +exterior extérieurs +exterior extérieur +exterior extérieures +exterior exterieur +exterior extérieure +murders assassinats +murders meurtres +paragraphs paragraphes +costume déguisements +costume costume +costume déguisement +bike bicyclette +bike vélo +neighborhoods quartiers +imprisonment incarcération +imprisonment emprisonnement +suffolk suffolk +forwards transmet +remarkable remarquables +remarkable remarquable +undelete reprennent +differ diffèrent +tin tin +tin étain +garcia garcia +madagascar madagascar +cameras appareils +cameras caméras +ammunition munition +ammunition munitions +fires feux +fires incendies +explore découvrir +explore explorer +builder constructeur +minneapolis minneapolis +bullet balle +bullet bullet +kerry kerry +subway métro +arrow flèches +arrow arrow +arrow flèche +economist économiste +economist economist +bread pain +lou lou +strategies stratégies +rubber caoutchouc +precise précises +precise précis +precise précise +rifles fusils +rifles carabines +cognitive cognitifs +cognitive cognitif +cognitive cognitive +cognitive cognitives +governorate gouvernorat +nest nids +nest nest +nest nid +slam slam +ancestry ascendance +portsmouth portsmouth +convince convaincre +convince convainc +audiences auditoires +audiences audiences +boarding embarquement +boarding abordage +bonds obligations +joshua joshua +inhabited habité +inhabited habitées +inhabited habités +inhabited peuplé +inhabited habitée +casey casey +attract attirent +attract attirer +attract attire +nonetheless néanmoins +kilometres kilomètres +kilometres kilométrage +pump pompe +pump pomper +feeding nourrir +prey proies +prey proie +ain aïn +ain ain +mathematician mathématicien +diary journal +diary agenda +vulnerable vulnérables +vulnerable vulnérable +inscription inscription +dubai dubai +dubai dubaï +michelle michelle +michelle michèle +lebanese libanais +lebanese libanaises +lebanese libanaise +productive productives +productive productive +productive productif +productive productifs +guided guidé +guided guidées +guided guidés +guided guidée +researcher chercheuse +researcher chercheur +della délia +della della +baden bade +baden baden +upgraded améliorés +upgraded modernisé +upgraded améliorée +upgraded amélioré +demonstration démonstration +demonstration démonstrations +demonstration manifestation +equality égalité +equality egalité +philosophers philosophes +spacecraft vaisseau +trap piège +clara clara +invitation invitation +marking marquages +marking marquage +marking balisage +expertise compétences +expertise expertise +expertise expertises +admission admission +sacramento sacramento +certification accréditation +certification certification +precisely justement +precisely précisément +casting fonderie +casting casting +casting coulée +casting moulage +reassessed réévalué +prohibited interdites +prohibited interdite +prohibited interdit +prohibited interdits +supposedly supposé +supposedly prétendument +governance gouvernance +frog grenouille +vague vague +mhz mhz +secular séculaire +secular laïque +secular laïcité +secular laïques +secular laïcs +tracking repérage +tracking suivi +spa spa +spa thermes +publicity publicité +armoured blindé +armoured blindée +armoured blindés +cleared innocenté +watts watt +watts watts +gibraltar gibraltar +renewed renouvelée +renewed renouvelé +renewed renouvelés +reflects reflète +fever fièvre +melody mélodie +melody mélodies +melody melody +supporter partisan +supporter supporter +elaborate élaboré +elaborate élaborer +jeffrey jeffrey +discusses discute +useless inutile +useless inutilisable +useless inutilité +useless inutiles +swift swift +tuesday mardi +silly idiot +silly idiote +empress impératrice +capabilities capacités +newman newman +scales gammes +scales écailles +scales échelles +scales balances +beatles beatles +clergy clergé +jacksonville jacksonville +sara sara +bee abeille +bee bee +holders titulaires +holders détenteurs +baltic baltique +baltic baltes +czechoslovakia tchécoslovaquie +brandon brandon +loaded chargées +loaded chargée +loaded chargés +loaded chargé +maya maya +maya mayas +evangelical évangélique +enterprises entreprises +imo omi +mature maturité +mature mûr +mature mûres +mature mûre +mature mature +physically physiquement +sequences séquences +breast mammaire +breast poitrine +breast seins +breast sein +beast bête +raja raja +für für +educator éducateur +educator éducateurs +bang bang +griffin griffin +griffin griffon +rhodes rhodes +preparing préparer +proportion proportions +proportion proportion +itv itv +ana ana +ceiling plafond +ceiling plafonds +rainbow rainbow +rainbow arc +demon démon +demon demon +prussian prussienne +prussian prusse +equations équations +answered répondu +ist tsi +ist ist +perception perception +perception perceptions +distributor distributeur +distributor distributeurs +entities entités +jackie jackie +dynamics dynamiques +dynamics dynamique +fiji fidji +fiji fidjien +insufficient insuffisante +insufficient insuffisant +insufficient insuffisance +insufficient insuffisantes +insufficient insuffisants +algebra algèbre +homer homer +larvae larve +larvae larves +limestone calcaires +limestone calcaire +johns johns +bce bce +chaos anarchie +chaos chaos +chang chang +layers pondeuses +layers couches +layers calques +crater cratère +crater cratères +chad tchadien +chad tchad +chad chad +seized saisi +seized saisis +webster webster +excess excès +excess excédentaires +excess excédent +bombardment bombardement +bombardment bombardements +hurling hurling +ashley ashley +dot dot +gif gif +translator traducteurs +translator traducteur +translator traductrice +cowboys cowboys +counted comptés +counted comptabilisé +counted compté +counted comptées +hanging suspendus +hanging pendu +hanging pendaison +soprano soprano +interviewed interviewé +interviewed interrogés +interviewed interrogé +workshops ateliers +terrain terrain +belarus belarus +belarus biélorussie +belarus bélarus +liked apprécié +liked aimés +liked aimées +liked aimée +liked aimé +console pupitre +console consoles +console console +nascar nascar +vandal vandales +vandal vandale +graduates diplômés +jungle jungle +ballot scrutin +placement placement +fairy féerique +fairy fées +fairy fée +tourists touristes +reasonably raisonnablement +performs accomplit +performs effectue +quarterly trimestriellement +quarterly trimestriel +quarterly trimestrielle +quarterly trimestrielles +quarterly trimestriels +shifted décalé +romans romans +romans romains +rpm rpm +diploma diplôme +environments environnements +collaborative collaboratif +swan cygne +swan swan +swan cygnes +carpenter charpentier +carpenter carpenter +carpenter menuisier +petition pétition +boris boris +berry baie +berry berry +berry baies +invention invention +southampton southampton +prairie prairie +prairie prairies +bend plier +bend virage +app application +app app +app appli +finalist finaliste +finalist finalistes +questioned interrogée +questioned interrogés +questioned questionné +questioned interrogé +explicit explicite +explicit explicitement +explicit explicites +draws dessine +draws tirages +governed régis +governed gouvernés +governed gouverné +slight légère +drag drag +drag traîner +drag traînée +drag glisser +maxwell maxwell +planes avions +everyday quotidien +everyday quotidienne +oriental orientale +oriental orientales +oriental oriental +oriental orientaux +manufacture fabrication +airing aérer +acclaimed acclamé +coordinator coordonnateur +coordinator coordinatrice +coordinator coordonnatrice +coordinator coordinateur +bombs bombes +mohammad mohammad +mohammad mahomet +bassist bassiste +superman superman +colombian colombien +colombian colombienne +colombian colombiens +philippe philippe +felix felix +felix félix +bengali bengali +greene greene +voluntary volontaire +voluntary volontaires +floating flottant +floating flottants +floating flotter +floating flottante +montenegro monténégro +sketch croquis +sketch esquisse +sketch esquisser +mann mann +flooding inonder +flooding inondations +flooding inondation +escort accompagnateur +escort escorte +escort escort +dressed habillé +dressed habillée +dressed habillées +dressed vêtue +dressed habillés +astronomy astronomie +sudden soudaine +sudden subite +sudden soudain +variables variables +arbitrary arbitraires +arbitrary arbitraire +skiing ski +timothy timothy +cello violoncelle +rainfall précipitations +rainfall pluies +rainfall pluviométrie +rafael rafael +rafael raphaël +sphere sphère +rewrite réécriture +rewrite réécrire +georg georg +cinematography cinématographie +canvas canevas +canvas canvas +canvas toile +canvas toiles +chest poitrine +chest thoracique +chest torse +chest coffre +krishna krishna +provider prestataire +provider fournisseur +frances frances +frances françoise +crowned couronnés +crowned couronnée +crowned couronnées +crowned couronné +wanting vouloir +wanting voulant +carved taillé +carved gravé +carved taillée +carved sculpté +carved sculptés +poles pôles +poles poteaux +poles mâts +poles polonais +cabin cabine +cabin cabane +civilization civilisation +civilization civilisations +avoided évitées +avoided évitée +avoided évités +avoided évité +lisbon lisbonne +eliminate élimine +eliminate éliminer +panels panneaux +darwin darwin +cheese fromages +cheese fromage +easter pâque +easter pâques +rat rat +rat rats +papua papouasie +insert insert +insert insérer +insert insertion +insert insérez +descriptions descriptions +debates débat +debates débats +informal officieux +informal informel +informal informelles +informal informels +informal informelle +castles châteaux +cry pleurer +cry cry +cry pleure +cry pleurs +loyal fidèle +loyal loyal +loyal loyaux +loyal fidèles +loyal loyale +surfaces surfaces +nicolas nicolas +institutes instituts +humor humour +madonna madone +madonna madonna +worcester worcester +cooperative coopératifs +cooperative coopérative +cooperative coopératives +cooperative coopératif +substantially substantiellement +substantially sensiblement +winston winston diff --git a/homeworks/hw1_embeddings/en-fr.train.txt b/homeworks/hw1_embeddings/en-fr.train.txt new file mode 100644 index 0000000..855f568 --- /dev/null +++ b/homeworks/hw1_embeddings/en-fr.train.txt @@ -0,0 +1,10872 @@ +the le +the les +the la +and et +was fut +was etait +was était +for pour +that que +that cela +with avec +from du +from de +from depuis +this ceci +this cet +this cette +this cela +this ce +utc utc +utc tuc +his sa +his his +his ses +his son +not not +not non +not pas +are sont +talk parler +talk parle +talk talk +talk parlez +which lesquels +which laquelle +which lequel +also également +also aussi +also egalement +were étaient +but mais +have avoir +have ont +one un +one une +one one +new nouveau +new nouvelle +new nouvelles +new nouveaux +new nouveautés +first première +first premières +first premier +first premiers +page pages +page page +you vous +you tu +you toi +you you +they elles +they ils +they eux +had avait +had avaient +had avais +article article +who qui +who oms +who who +all toute +all toutes +all tous +all tout +all all +their leur +their leurs +there là +made fait +made fabriqués +made faite +made fabriqué +its sa +its ses +its son +people gens +people personnes +may mai +may may +may peut +after apres +after after +after après +after aprés +other autre +other autres +should devrait +should devraient +should devrais +two two +two deux +score marquer +score score +score partition +her her +can peuvent +can pouvez +can peux +can peut +would serait +would aurait +would ferait +more davantage +more plus +she she +she elle +when lorsque +when quand +time temps +time heure +team équipe +team equipe +american américaine +american américain +american américaines +american américains +such tels +such tel +such telle +such telles +discussion discussions +discussion discussion +discussion débat +links liens +only seul +only uniquement +only seulement +only seule +some certains +some certaines +some quelques +see consultez +see voyez +see voir +see vois +united unie +united unis +united unies +years années +years ans +school ecole +school école +world monde +world mondial +world mondiale +university université +university universitaire +during durant +during pendant +during lors +out sortir +out sortie +out dehors +out out +state etat +state état +states etats +states états +national nationaux +national national +national nationale +national nationales +wikipedia wikipédia +wikipedia wikipedia +year annee +year an +year année +most most +city ville +city villes +used utilisées +used utilisé +used utilisés +used utilisée +then ensuite +then alors +then puis +county comté +external externe +external extérieurs +external extérieur +external extérieure +external externes +where oú +where où +will will +will volonté +will fera +will sera +what quoi +what quel +what quelle +delete supprimer +delete supprimez +delete effacement +delete supprime +delete effacer +these ces +january janvier +march marche +march mars +august aout +august août +july juillet +being etre +being être +film film +him lui +many nombreux +many beaucoup +many nombreuses +many plusieurs +south méridional +south sud +september septembre +like comme +like aime +like aiment +like genre +like aimez +between entre +october octobre +three trois +three three +june juin +well bien +well puits +well bah +use utilisation +use utiliser +use utilise +use utilisez +war guerre +war war +under sous +under under +them eux +april avril +born naître +born né +born naissance +born née +born born +december décembre +december decembre +link lien +later ultérieur +part partie +november novembre +players joueurs +list liste +list listes +please veuillez +please please +please svp +following suivantes +following suivant +february fevrier +february février +known connues +known connue +known connus +known connu +second second +second deuxième +second deuxieme +second seconde +name nommer +name nom +name dénomination +name noms +group groupe +history histoire +history historique +series série +series séries +just just +just simplement +just juste +north nord +work travaux +work travail +work travailler +before auparavant +before avant +since depuis +since puisque +season saison +season saisons +both both +high élevée +high hautes +high haut +high haute +high élevé +through via +district district +now désormais +now now +now maintenant +comments commentaires +comments remarques +comments observations +because parce +because car +because parceque +football football +music musiques +music musique +however toutefois +however cependant +diff différence +diff diff +century siecle +century siècle +century century +league league +league ligue +edits modifications +debate débat +title titre +articles articles +john jean +john john +same pareil +same meme +same identique +same mêmes +same même +including incluant +including comprenant +could pourrais +could pouvait +could pourrait +could pourraient +english anglaise +english francais +english anglais +album album +number nombre +number numéro +against contre +against against +family familiale +family famille +family familles +user utilisateur +user usager +based basée +based basés +based basé +area région +area zone +area domaine +became devenus +became devenu +became devenue +became devint +york york +life vie +british britanniques +british britannique +international international +international internationaux +international internationale +game gibier +game match +game jeu +club club +your votre +your vos +early early +early précoce +early tôt +best meilleure +best meilleures +best meilleur +best best +best meilleurs +west occidentale +west occident +west ouest +west west +house maison +company compagnie +company entreprise +company société +general généraux +general généralités +general générale +general général +left gauche +very très +very tres +very trés +here ici +here voilà +here voici +don don +living vivante +living vivants +living vivant +living vivre +day journée +day jour +day journee +several plusieurs +place place +place endroit +place lieu +party partie +party parti +party fete +party soirée +party fête +college collège +college université +result résultat +keep maintenir +keep garder +keep conserver +appropriate approprié +appropriate appropriées +appropriate appropriés +appropriate appropriée +four quatre +even meme +even même +class classe +government gouvernement +government gouvernements +how comment +called appelée +called appelé +called appelés +did did +each chacune +each chaque +each chacun +found trouvée +found trouvé +found trouvées +found retrouvé +found trouvés +center center +center centrer +center centre +per per +style style +com ocm +com com +long longues +long longs +long longue +long longtemps +long long +country pays +back dos +back arrière +back retour +back revenir +way façon +way manière +way way +www www +modify modifiez +modify modifier +end end +end fin +make faites +make faire +public publics +public publiques +public public +public publique +played joués +played jouées +played jouée +played joué +won gagné +won gagnée +won gagnées +won won +won gagnés +another another +released publié +released relâché +released libérés +released libéré +added ajoutés +added ajouté +added ajout +added ajoutée +support soutien +support assistance +support appui +games jeux +former ancien +former ancienne +those ceux +films films +church église +church eglise +east orientale +east est +east orient +line ligne +line line +major major +major commandant +major majeur +members membres +members adhérents +good bons +good bien +good bonne +good bon +good bonnes +much much +much beaucoup +image image +show afficher +show montre +show montrer +show spectacle +still still +still encore +still toujours +think pense +think pensez +think réfléchis +think penser +think réfléchir +below dessous +town ville +last dernier +last dernière +last derniers +last dernières +system système +right droite +right droit +song chant +song chanson +notable notable +notable notables +notable remarquable +notable remarquables +section section +single célibataire +single single +single unique +single célibataires +included incluses +included inclus +included incluse +included compris +align aligner +align alignement +home maison +home domicile +home accueil +women femmes +women féminin +women femme +television téléviseur +television télévision +television télé +seed graine +seed semences +seed seed +seed graines +seed semence +member membres +member membre +goals buts +goals objectifs +sources sources +book livre +book réserver +station station +station gare +order ordre +order commande +order commander +old ancienne +old anciens +old ancien +old vieux +old vieille +information informations +information infos +information renseignements +information information +set définir +own propres +own propre +own posséder +text textes +text texte +band groupe +band orchestre +band bande +point point +local locale +local local +local locales +local locaux +around autour +around alentour +river fleuve +river rivière +top top +top haut +main principal +main principales +main main +main principale +main principaux +language langue +language langage +language langues +french francais +french française +french français +french françaises +https https +named nommée +named nommé +named nommés +off off +off hors +note remarque +note note +note notez +career carrière +original originale +original original +original originaux +age ère +age age +age âge +age âges +service service +established établies +established établie +established établi +established établis +located situés +located située +located localisé +located situé +said dit +said disait +website site +population populations +population population +air aérien +air air +german allemand +german allemands +german allemande +law loi +law droit +military armée +military militaire +military militaires +great super +great grande +great génial +great grand +clubs discothèques +clubs clubs +published publiée +published publiées +published publiés +published publié +president présidente +president président +park parc +official fonctionnaire +official officielles +official officiels +official officielle +official officiel +case cas +case étui +case affaire +london londres +times fois +although quoique +small petit +small petits +small petites +small petite +third troisieme +third troisième +third tiers +third third +third troisièmement +different différentes +different différente +different différents +different différent +due dus +due due +due dû +get obtenez +get obtenir +village village +closed fermée +closed fermé +closed fermés +closed clos +art art +art artistique +player joueurs +player joueur +player lecteur +final définitif +final finale +final final +final définitive +community communaute +community communautaire +community communauté +community collectivité +held tenu +again again +again encore +began commencé +army armée +award récompense +without sans +death décès +death death +death mort +built construite +built construit +built bâti +built construits +men masculins +men hommes +men homme +large grands +large grande +large grandes +large grand +site site +using utiliser +using utilisant +deletion suppressions +deletion effacement +deletion suppression +white blanche +white blancs +white blanches +white blanc +five five +five cinq +central central +central centrale +road routier +road route +road chemin +children enfants +children enfant +free gratuit +free gratuits +free free +free gratuitement +free libre +took pris +took prit +england angleterre +include incluent +include inclure +include comprennent +include inclut +association association +down down +down descendre +given donné +given donnés +source source +source sources +california californie +california californienne +man man +man mec +man homme +version version +written ecrit +written écrite +written écrit +written écrites +written écrits +created créée +created créé +created crée +created crées +created créés +media média +media médias +black noire +black black +black noires +black noir +black noirs +php php +report signaler +report reportage +report rapport +building batiment +building immeuble +building bâtiment +take prend +take prendre +take prenez +take prends +division division +division circonscription +comment commentaire +comment commente +comment commenter +having ayant +having avoir +king roi +king king +edit modifier +edit édition +edit éditer +stadium stade +stadium stadium +died morte +died décédé +died mort +ship navire +ship vaisseau +research recherches +research recherche +record enregistre +record record +record enregistrer +archive archivage +archive archiver +archive archive +archive archives +places endroits +places lieux +undo annuler +undo défaire +cup gobelet +cup cup +cup tasse +cup coupe +records enregistrements +records records +often souvent +few peu +received reçu +received reçues +received reçus +received reçue +side latéraux +side côté +side latérale +side latéral +power puissance +power pouvoir +education enseignement +education éducation +know connaitre +know connaître +know sais +know sachez +know savoir +category catégorie +category catégories +water eaux +water eau +species espèces +species espèce +field champ +field domaine +near proche +near près +australia australie +video vidéo +video vidéos +video video +need nécessité +need besoins +need besoin +island île +form forme +form formulaire +find trouver +find trouvez +served servi +served servis +served servie +served servies +served desservis +play jouer +play jouez +project projet +radio radiophonique +radio radio +works travaux +works œuvres +works oeuvres +proposed proposée +proposed proposé +proposed proposées +proposed proposés +every chaque +development developpement +development développement +example exemple +live live +live vivant +live vivre +union union +union syndicat +india inde +india indes +next prochain +next suivante +next suivant +next prochaine +next next +special spéciales +special spécial +special spéciaux +special spéciale +court tribunal +court cour +region région +little petit +little peu +little petite +short courte +short court +short courts +short courtes +william guillaume +william william +province province +western occidentale +western occidental +western ouest +western western +son fils +son fiston +france france +council conseil +others autres +royal royale +royal royal +royal royales +royal royaux +current actuel +current courant +current actuels +current actuelle +street rues +street street +street rue +full complète +full pleine +full plein +full complet +red rouges +red roux +red rouge +too trop +too too +department département +san san +help aidez +help aider +help aide +among parmis +among parmi +preserved préservée +preserved préservé +preserved conservés +preserved préservés +preserved conservées +preserved préservées +james james +open ouverte +open ouvrez +open ouvert +open ouverture +open ouvrir +force force +force forcer +position position +head head +head tête +head têtes +director directeur +director directrice +director réalisateur +father pére +father pere +father père +track piste +track morceau +http http +canada canada +never jamais +never never +australian australienne +australian australiens +australian australien +australian australie +george georges +george george +jpg jpg +level niveau +late tardive +late tardif +late retard +late tard +summer eté +summer été +society société +moved déplacé +moved ému +moved déménagé +moved déplacée +office bureaux +office bureau +period période +championship championnat +round arrondi +round ronde +round rond +round rondes +round ronds +story récit +songs chanson +songs chansons +various divers +various diverses +file dossier +file fichier +days jours +days journées +land terre +land terrain +land terres +business affaires +business entreprise +business entreprises +reason raisons +reason raison +america amerique +america amérique +million million +million millions +european européens +european européennes +european européenne +european européen +term terme +six six +post publier +post poste +post publication +why pourquoi +why why +produced produite +produced produites +subject objet +subject sujet +young jeunes +young jeune +total totale +total total +total totaux +david alain +david laurent +david sylvie +david david +science science +science scientifique +science sciences +related connexe +related connexes +related lié +related liés +rock rocheux +rock rock +archived archivées +archived archivé +archived archivés +railway ferroviaire +become devenez +become deviennent +become deviens +become devenir +led led +students étudiants +students etudiants +students élèves +started commencé +started démarré +started commencée +news nouveautés +news actualité +news nouvelles +news actualités +described décrites +described décrits +described décrit +described décrite +role rôle +election élection +election election +election élections +albums albums +present actuel +present présent +present présents +present présente +present présenter +indian indienne +indian indiens +indian indiennes +indian indien +kingdom royaume +books ouvrages +books livres +important importante +important important +important importantes +important importants +northern septentrionale +northern nordique +northern septentrional +northern northern +northern nord +love aime +love amour +love aimer +love love +run courir +run exécuter +canadian canadiens +canadian canadienne +canadian canadien +press appuyez +press presse +rather plutôt +type taper +type type +type tapez +act agir +act loi +act acte +act act +editor rédacteur +editor éditeur +editor rédactrice +editor editeur +came vint +schools école +schools ecoles +schools écoles +program programme +once jadis +once once +social sociale +social sociaux +social social +germany allemagne +production production +male mâle +male masculin +male homme +might pourrait +awards distinctions +awards trophées +awards récompenses +points points +similar similaires +similar similaire +similar semblables +similar analogue +similar semblable +professional professionnels +professional professionnelle +professional professionnel +professional professionnelles +say dites +say say +say dire +say dis +background fond +background contexte +enough suffisamment +enough assez +lead plomb +either soit +common commune +common fréquent +common fréquents +common commun +overlap chevaucher +overlap chevauchement +overlap chevauchements +data données +color couleur +color coloris +color couleurs +better meilleures +better meilleure +better mieux +better meilleurs +better meilleur +person personne +services services +bgcolor bgcolor +museum musées +museum musée +battle bataille +battle combat +went allé +sports sport +sports sports +already déja +already déjà +already deja +already dejà +currently actuellement +currently présentement +hall hall +buildings bâtiments +buildings immeubles +buildings édifices +historic historique +historic historiques +date date +deleted supprimé +deleted supprimée +deleted effacé +deleted supprimés +deleted supprimées +considered considérée +considered considéré +considered considérées +considered considérés +change modifier +change changements +change changer +change changement +location localisation +location emplacement +location localité +location lieu +seems semble +must doit +must moût +must doivent +must devez +yes oui +yes yes +our notre +our nos +southern méridional +southern méridionale +lost perdu +lost perdues +lost perdue +lost lost +lost perdus +something quelquechose +review revue +review examen +review révision +together ensemble +together together +robert robert +robert thierry +less moins +japanese japonais +japanese japonaise +japanese japonaises +groups groupes +content contenu +content contenus +involved impliqué +involved impliqués +involved impliquée +isbn isbn +board planche +japan japon +control contrôler +control contrôle +policy politiques +policy politique +modern modernité +modern moderne +modern modernes +human humaine +human humains +human humain +half moitié +half demi +design dessin +design design +design conception +event évènement +event événement +event evénement +events évènements +events evénements +events evenements +events événements +available dispo +available disponible +available disponibles +done fait +done faite +washington washington +real réel +real réelle +real véritable +real vrai +real vraie +start démarrer +start commencer +start début +personal personnel +personal personnelle +personal personnelles +personal personnels +action action +space espace +areas domaines +areas zones +star étoile +star vedette +star etoiles +star star +star étoiles +really vraiment +really réellement +china chine +possible possible +possible possibles +paul paul +working travaillant +working travailler +taken prises +taken taken +taken pris +taken prise +far far +far loin +going aller +minister ministre +lake lac +reported signalé +reported rapporté +reported signalés +reported signalée +popular populaires +popular populaire +married mariées +married mariés +married mariée +married marié +founded fondé +founded fondée +europe europe +author auteure +author auteur +away loin +independent indépendant +independent indépendante +independent indépendantes +independent indépendants +process processus +process procédé +teams équipes +teams equipes +character caractère +character personnage +low basses +low basse +low faible +low faibles +low bas +michael alain +michael michael +michael michel +pages pages +light lumiere +light lumière +light légère +light léger +big gros +big grosse +big grande +big big +big grand +seen vu +seen vus +release release +release libération +release libèrent +want veulent +want envie +want vouloir +want veux +want voulez +episode épisode +episode episode +wrote écrivit +wrote écrit +republic république +thomas thomas +companies compagnies +companies entreprises +companies sociétés +via via +russian russes +russian russe +thanks merci +thanks remerciement +thanks cordialement +thanks remerciements +put mis +put mettre +race course +race race +worked fonctionné +worked travaillé +route itinéraire +route route +route trajet +route parcours +recorded enregistrées +recorded enregistrée +recorded enregistrés +recorded enregistré +someone someone +civil civil +civil civile +civil civiles +police policière +police policiers +police police +police policier +charles charles +listed listés +listed listé +listed énumérés +listed répertoriés +users usagers +users utilisateurs +template gabarit +template modèle +eastern orientale +eastern oriental +body organisme +body carrosserie +body organe +body corps +question question +italian italiennes +italian italienne +italian italien +italian italiens +featured vedette +featured recommandés +featured vedettes +week semaine +week semaines +editors rédacteurs +editors éditeurs +texas texas +chief chef +close fermer +close proche +match correspondance +match allumette +match match +roman romain +roman romaine +roman roman +roman romaines +roman romains +come viens +come venez +come come +come venir +opened ouvert +tour visite +tour tour +tour tournée +sea mer +cross croisée +cross croix +cross traverser +cross croisé +playing jouer +playing jouant +health sante +health santé +institute institute +institute institut +caps casquettes +caps chapeaux +caps caps +caps bouchons +forces forces +green verte +green vert +green vertes +rights droits +evidence preuve +evidence preuves +originally initialement +aircraft aéronefs +aircraft avion +aircraft avions +arts arts +range gamme +range portée +probably surement +probably probablement +probably sûrement +consensus consensus +bar barreau +bar bar +bar barre +problem problème +problem problématique +look regardez +look regarde +look regardes +issues problèmes +alumni anciens +average moyen +average moyennes +average moyenne +network reseau +network réseaux +network réseau +win victoire +win gagnez +win gagner +win gagnant +shows spectacles +wife épouse +wife epouse +wife femme +returned retourné +returned retournés +returned retournée +night soirée +night nuit +night soir +magazine revue +magazine magazine +magazine magasine +centre centre +joined rejoints +joined rejoint +usually habituellement +usually généralement +middle middle +middle milieu +completed complété +completed terminée +completed achevés +completed achevé +completed terminées +completed terminé +elected élue +elected élu +elected élues +elected élus +significant significative +significant significatif +significant significatifs +african africains +african africain +african africaine +african africaines +able capable +google google +stage étape +stage stade +stage scène +addition addition +addition ajout +ireland irlande +today hui +today aujourdhui +academy académie +academy academy +saint sainte +saint saint +self self +itself soi +continued continué +stations gares +stations stations +mother mère +mother mere +mother maman +appeared apparu +appeared apparut +appeared paru +appeared parut +appeared semblait +africa afrique +culture culture +spanish espagnol +spanish espagnole +spanish espagnols +grand grandiose +grand grand +committee comité +things choses +fire feu +fire incendie +fire incendies +changed changé +changed changée +gold doré +gold or +gold gold +female femelle +female femelles +female féminin +female femmes +female femme +course cours +directed dirigé +directed réalisé +directed orienté +months mois +chinese chinois +chinese chinoise +previous précédentes +previous précédente +previous précédent +previous précédents +developed développées +developed développés +developed développée +developed développé +size taille +size pointure +size tailles +mentioned mentionnée +mentioned mentionné +mentioned mentionnés +add ajout +add ajouter +add ajoute +add ajoutez +festival festival +festival fête +peter peter +peter pierre +basketball basket +basketball basketball +move déménagement +move bouger +move déménager +move déplacer +performance performance +performance performances +performance rendement +standard norme +standard standard +means moyens +means signifie +give donner +give donne +give donnez +training formation +training entraînement +artist artiste +artist artistes +word mot +blue bleu +blue bleus +blue bleue +blue bleues +primary primaire +primary primaires +announced annoncé +announced annoncés +announced annoncées +announced annoncée +value valeur +christian chrétiens +christian christianisme +christian chrétienne +christian christian +christian chrétien +private privée +private privé +private privés +catholic catholique +catholic catholiques +artists artistes +includes inclut +includes comprend +view afficher +view vue +view voir +view affichage +view visualiser +thus ainsi +almost presque +almost quasi +almost pratiquement +almost quasiment +baseball baseball +seven seven +seven sept +appears semble +appears paraît +appears apparaît +appears apparait +ever jamais +ever ever +provide fournir +provide fournissent +technology technologique +technology technologies +technology technologie +olympics olympiades +olympics olympiques +future future +future futures +future futur +future futurs +future avenir +formed formés +formed formé +formed formées +formed formée +census recensements +census recensement +images images +los los +results résultats +return retourner +return retour +return revenir +quality qualite +quality qualité +construction construction +zealand zélande +front front +front avant +front devant +cover couvercle +cover couverture +cover couvrir +cover housse +model modele +model modèle +model mannequin +model maquette +despite malgré +read lecture +read lisez +read lire +read lis +material matériel +material matériaux +material matériau +strong forte +strong forts +strong strong +strong fort +coach coach +coach entraineur +coach entraîneur +henry henri +henry henry +footballers footballeurs +mark marquer +mark marque +mark mark +rev rév +rev rev +organization organisation +studies etudes +studies études +federal fédéral +federal fédérale +federal fédéraux +federal fédérales +richard richard +html html +virginia virginie +virginia virginia +car voitures +car voiture +attack attaquez +attack attaque +attack attentat +attack attack +attack attaquer +conference conférences +conference conférence +outside extérieur +outside dehors +outside exterieur +study etude +study étudier +study étude +brother frangin +brother frère +brother frere +names noms +writer rédacteur +writer auteur +writer écrivain +writer scénariste +characters personnages +characters caractères +musical musicale +musical musical +nothing rien +border frontière +border frontalier +border frontières +border border +border bordure +medical médicaux +medical médicale +medical médical +countries pays +past passées +past passée +past passé +past passés +writing ecrire +writing rédaction +writing écriture +writing écrire +makes rend +interest intérêts +interest intérêt +provided fournies +provided fourni +provided fournie +provided fournis +killed tué +killed tuée +killed tués +medal médaille +medal médailles +signed signées +signed signée +signed signé +signed signés +label libellé +label label +label étiquette +label étiquettes +fair foire +fair justes +fair équitable +fair équitables +search chercher +search rechercher +search recherche +search recherches +search recherchez +bay baie +bay bay +reference référence +especially particulièrement +especially surtout +especially spécialement +removed retirée +removed enlevé +removed retiré +removed supprimée +removed supprimés +removed supprimé +library bibliothèque +library librairie +eventually éventuellement +eventually finalement +management management +management gestion +references références +features caractéristiques +features fonctions +features fonctionnalités +navy navy +navy marine +guitar guitare +guitar guitares +hill hill +hill colline +sure sure +sure sûre +sure sûr +historical historique +historical historiques +lower abaisser +lower inférieur +lower inférieure +daughter fille +appointed désignés +appointed désigné +appointed nommé +appointed nommés +reading lecture +reading reading +reading lire +yet pourtant +systems systèmes +debut débuts +movement mouvements +movement movement +movement mouvement +specific spécifiques +specific spécifique +always always +always tjrs +always toujour +always toujours +actor comédien +actor acteur +natural naturel +natural naturelles +natural naturels +natural naturelle +clear limpide +clear clair +clear effacer +coast côte +let let +got got +chicago chicago +championships championnats +pennsylvania pennsylvanie +ten rte +ten dix +ten ten +performed effectué +individual individuelle +individual individuels +individual individuel +designed conçu +designed conçues +designed conçue +designed conçus +rule règle +etc etc +lists listes +paris paris +thought réfléchi +thought pensais +thought pensé +thought pensée +brown brown +brown marron +brown bruns +brown brun +brown brunes +brown brune +hand main +hand hand +needs besoins +reliable fiable +reliable fiables +smith smith +smith forgeron +generally généralement +base base +sometimes quelquefois +sometimes parfois +florida floride +capital majuscule +capital capitaux +capital capitale +capital capital +valley valley +valley vallée +bank banques +bank banque +ground moulu +reached atteints +reached atteint +italy italie +energy énergie +energy énergies +energy energie +believe croyez +believe croire +believe croient +believe crois +leader meneur +leader leader +active active +active actif +active actives +active actifs +online online +block bloquer +block block +block bloc +block blocage +bridge bridge +bridge pont +bridge passerelle +families familles +changes modifications +changes changements +followed suivis +followed suivit +followed suivie +followed suivies +industry industrie +collection ramassage +collection recouvrement +collection recueil +collection collection +collection collecte +request demander +request demande +request requête +request demandez +soon bientôt +soon prochainement +soon bientot +olympic olympic +olympic olympique +olympic olympiques +sold vendues +sold vendus +sold vendue +sold vendu +writers scénaristes +writers auteurs +writers écrivains +professor professeur +professor professeure +studio studio +mexico mexique +competition compétition +competition concurrence +competition concours +campaign campagne +org org +theatre théatre +theatre théâtre +theatre théâtres +particular particulière +particular particulier +empire empire +length longueur +length longueurs +islands îles +islands iles +singer chanteur +singer chanteuse +create créer +create créez +create crée +create créent +redirect rediriger +redirect redirection +redirect réorienter +additional supplémentaire +additional supplémentaires +additional additionnel +soviet soviétique +soviet soviet +soviet soviétiques +market marché +words mots +producer producteur +producer productrice +producer producteurs +notes remarques +notes notes +hockey hockey +code code +referee arbitres +referee arbitrer +referee arbitre +fourth quatrième +fourth fourth +fourth quatrièmement +sport sportif +sport sportive +sport sport +van camionnette +van fourgonnette +van fourgon +van van +mary marie +mary mary +mary myriam +airport aéroports +airport aéroport +sound sound +sound sonore +sound son +status état +status statut +irish irlandaise +irish irlandais +placed placés +placed placées +placed placée +placed placé +child enfant +idea idée +foreign étranger +foreign etranger +foreign étrangère +foreign étrangères +municipality municipalité +register enregistrer +register enregistrez +register registre +eight huit +eight eight +problems problèmes +native indigènes +native natif +native indigène +native autochtone +coverage couverture +channel canal +channel chaîne +channel chenal +channel channel +parliament parlement +username username +username identifiant +username pseudo +edition edition +edition édition +minor mineures +minor mineure +minor mineur +says dit +foundation fondement +foundation fondation +foundation fondations +units unités +movie film +ice verglas +ice ice +ice glace +simply simplement +limited limitée +limited limitées +limited limité +limited limités +unit unité +unit unite +unit unit +student étudiant +student étudiante +student étudiants +student etudiant +previously auparavant +previously précédemment +stated déclaré +governor gouverneure +governor gouverneur +complete complète +complete complets +complete complètes +complete complet +test essai +test test +test épreuve +test tester +nominated nominée +nominated nominé +nominated désignés +bill bill +bill facture +bill facturer +parts pièces +parts parties +vocals chant +vocals voix +theory théories +theory théorie +regional régionales +regional régionaux +regional régional +account compte +vote vote +vote votez +vote voter +computer ordinateurs +computer ordinateur +none aucun +none néant +none aucune +carolina caroline +carolina carolina +tournament tournois +tournament tournoi +poland pologne +behind derrière +wales galles +winning gagnante +winning gagner +winning gagnant +lot lot +hospital hopital +hospital hôpital +hospital hospitalisation +hospital hôpitaux +mid mid +taking prendre +taking prenant +mountain montagne +mountain montagnes +higher supérieur +cases cas +angeles angeles +editing montage +editing éditer +editing édition +replaced remplacé +replaced remplacée +replaced remplacées +replaced remplacés +food alimentaire +food nourriture +food alimentation +multiple multiple +multiple multiples +multiple plusieurs +likely probable +likely probablement +terms termes +sir messire +sir monsieur +thing truc +thing chose +square carré +square carrés +square square +square carrées +try essaie +try essayez +try essayer +try essaye +topic thème +topic sujet +woman femme +officer officier +categories catégories +greek grecs +greek grecque +greek grecques +greek grec +recent récents +recent récent +recent récentes +recent récente +sent envoyés +sent envoyées +sent envoyée +sent envoyé +copyright copyright +speed vitesses +speed rapidité +speed vitesse +templates modèles +templates gabarits +money monnaie +money argent +saw saw +saw scie +senior seniors +senior aîné +senior senior +selected sélectionnés +selected sélectionnées +selected sélectionnée +selected sélectionné +introduced introduit +introduced introduits +introduced introduites +politician politicien +true vrais +true true +true vraie +true vrai +true véritable +required exigée +required exigé +required requise +required nécessaire +required obligatoire +required requis +regular ordinaire +regular régulière +regular régulières +regular régulier +awarded décerné +awarded attribué +awarded récompensé +awarded décernés +commercial commercial +commercial commerciaux +commercial commerciales +commercial commerciale +cities villes +contains contient +trade commerce +trade échange +trade échanges +degree diplôme +degree degré +anti anti +birth naissance +sun dim +sun soleil +finished fini +finished terminée +finished finis +finished terminé +rugby rugby +earth earth +earth terre +access accès +access accéder +prior prieur +seasons saisons +journal revue +journal journal +beginning commencement +beginning début +software logiciel +software logiciels +famous fameux +famous célèbre +famous célèbres +religious religieux +religious religieuses +religious religieuse +appear apparaître +appear apparaissent +martin martín +martin martine +martin martin +god god +god dieu +bit bit +bit bits +hours horaires +hours heures +running courir +brought apporté +brought amené +brought amenés +missing manquants +missing manquante +missing manque +missing manquant +missing disparu +economic économique +economic economiques +economic economique +structure structure +rural rurales +rural ruraux +rural rurale +rural rural +remained resta +remained resté +remained restait +decision décision +certain certain +certain certaine +certain certaines +hit touché +hit frappé +hit frapper +minutes minutes +spain espagne +plays joue +whole entiers +whole entier +joseph joseph +lord lord +lord seigneur +web web +web enchaînement +decided décidée +decided décidés +decided décidé +operations opérations +function fonctions +function fonction +louis louis +assembly assemblée +assembly assemblage +queen reine +queen queen +security securité +security sûreté +security sécurité +uses usages +uses utilise +uses utilisations +ohio ohio +owned possédée +owned appartenant +owned possédé +owned possédés +jan jan +jan yann +operation fonctionnement +operation opération +call appeler +call appelez +call appel +call appelle +successful réussi +successful réussie +legal juridique +legal légal +legal légales +legal légale +russia russie +prince prince +jewish juives +jewish juif +jewish juifs +jewish juive +staff personnel +establishments établissements +goal objectif +goal but +towards vers +agree convenir +agree acceptez +bad méchant +bad bad +bad mal +bad mauvaise +bad mauvais +attendance fréquentation +attendance présence +attendance assiduité +attendance participation +populated peuplée +populated peuplés +populated peuplé +populated peuplées +nature nature +allowed autorisé +allowed autorisés +allowed autorisées +captain captain +captain capitaine +mount monter +mount mont +mount monture +calculated calculées +calculated calculé +calculated calculés +calculated calculée +structures structures +hard difficiles +hard dur +hard hard +hard difficile +hard dure +saying disant +saying dicton +manager gestionnaire +manager directeur +manager manager +manager directrice +manager gérant +elections élections +elections elections +meet rencontrez +meet rencontre +meet rencontrer +box coffret +box boîte +box boite +lines lignes +democratic démocratique +democratic démocrate +democratic démocratiques +success succes +success succès +success success +success réussite +associated associée +associated associées +singles célibataire +singles simple +singles simples +singles célibataires +traditional traditionnelles +traditional traditionnel +traditional traditionnelle +traditional traditionnels +rest repos +rest repose +highway route +highway autoroute +highway autoroutes +particularly particulièrement +wide large +wide vaste +month mois +care soins +care soin +admin admin +admin administrateur +cultural culturels +cultural culturelle +cultural culturel +commission commission +plan planifier +plan plan +practice pratiquer +practice pratique +practice pratiques +command commandement +command commande +nomination nomination +jersey maillot +jersey jersey +parties parties +michigan michigan +anyone quiconque +overlaps chevauchements +approximately approximativement +approximately environ +master maitre +master master +master maître +noted notée +noted remarqué +noted noté +usa usa +stop arrêt +stop stop +stop arrêter +stop arrête +stop arrêtez +feature fonctionnalité +feature caractéristique +engine moteur +response réponse +response réaction +needed nécessaires +needed nécessaire +needed besoin +needed requis +illinois illinois +afd afd +experience expériences +experience expérience +engineering ingénierie +engineering génie +silver silver +silver argent +silver argenterie +silver argenté +separate distinct +separate séparés +separate séparé +separate séparées +separate séparer +takes prend +secretary secrétaire +dutch hollandaise +dutch néerlandaise +dutch néerlandais +dutch hollandais +lee lee +recording enregistrement +prime prime +rules règles +rules regles +uploaded téléchargés +uploaded téléchargé +uploaded téléchargée +trying essayant +trying essayer +youth jeunesse +youth jeunes +scotland écosse +scotland ecosse +iii iii +houses maisons +heart heart +heart cœur +heart coeur +room salle +room chambre +room chambres +stone stone +stone pierre +stone pierres +shown montré +deal aubaine +drama drame +drama dramatique +scores partitions +scores scores +dead morts +dead morte +dead dead +dead mort +key touche +key clef +key clé +key clefs +key clés +shot coup +shot shot +shot abattus +shot tir +shot abattu +turn tourne +turn turn +turn tourner +turn tournez +occupation profession +occupation occupation +scottish ecossais +scottish écossaise +scottish écossais +executive exécutif +plant végétal +plant usine +plant plante +promoted promue +promoted promus +promoted promu +promoted promues +villages bourgs +villages villages +languages langues +internet internet +leave congés +leave quitter +leave laisser +leave partir +leave congé +feel sentir +feel ressentir +covered couvert +covered couvertes +covered couverte +covered recouvert +covered couverts +merge fusionner +merge fusionne +merge fusion +mostly principalement +mostly essentiellement +mostly surtout +numerous nombreux +numerous nombreuses +ancient antiquité +ancient anciens +ancient ancien +ancient ancienne +attempt tenter +attempt tentative +property biens +property propriété +programs programmes +picture photo +picture image +finally enfin +finally finalement +ships navires +ships vaisseaux +fiction fictions +fiction fiction +looking regardant +secondary secondaires +secondary secondaire +nations nations +majority majorité +majority majoritaire +majority majoritaires +edward edward +edward édouard +annual annuelle +annual annuels +annual annuel +digital numérique +digital numériques +digital digital +mission mission +lived vécue +lived vécu +claim réclamation +claim réclamer +claim revendication +claim revendiquer +seat banquette +seat siège +seat seat +bbc bbc +profile profil +profile profils +dance danse +dance danser +doing faisant +doing faire +georgia géorgie +georgia georgie +port port +pacific pacifique +castle château +castle chateau +pass passer +pass passe +pass passent +pass pass +transport transports +transport transport +organizations organismes +organizations organisations +ratio ratio +recently récent +recently récemment +fall tomber +fall automne +fall chute +fall fall +global global +global mondiale +global mondial +global mondiaux +era eer +era era +era époque +wing escadre +wing aile +wing ailier +wing wing +wing ailes +opinion opinion +opinion avis +commander commandeur +commander commandant +fort fort +effect effets +effect effet +opening vernissage +opening ouverture +fine fine +fine amende +purpose objet +purpose finalité +purpose but +purpose objectif +winter hiver +winter hivers +genus genre +congress congrès +overall globale +overall globalement +activities activités +met rencontrée +met rencontrées +met rencontré +met rencontrés +income revenu +income revenus +massachusetts massachusetts +comes vient +older âgées +older aîné +peak apogée +peak pic +peak crête +lack manque +bass contrebasse +bass basse +bass bassiste +bass bass +super super +complex complexe +complex complexes +academic académiques +academic universitaire +academic universitaires +academic académique +stars etoiles +stars vedettes +stars étoiles +accounts comptes +appearance apparence +appearance apparition +appearance aspect +asian asiatiques +asian asiatique +asked demandé +friends amies +friends amis +kind aimable +financial financière +financial financier +entry entrée +asia asie +asia asiatique +sense sens +meaning signification +meaning sens +meaning signifiant +actress comédienne +actress actrice +map cartes +map plan +map carte +intended destiné +bishop évêques +bishop évêque +bishop bishop +boston boston +rate taux +rate tarif +literature littérature +forest forestier +forest forêts +forest forêt +forest foret +voice voix +jack valet +jack vérin +jack jack +pre pre +pre pré +justice justice +champion championne +champion champion +double doubler +double double +double doubles +polish polonaise +polish polonaises +polish polonais +numbers numéros +numbers chiffres +numbers nombres +columbia colombie +columbia columbia +temple temple +temple tempe +defeated vaincu +defeated défaite +defeated battu +defeated vaincus +defeated vaincue +administration administration +claims réclamations +claims créances +claims allégations +claims revendications +jones jones +parish paroisse +israel israël +israel israel +actors acteurs +actors comédiens +sister sister +sister soeur +sister sœur +nine neuf +nine nine +scored marqué +table table +table tableau +attended participé +pop pop +newspaper journal +friend ami +friend amie +friend amis +unknown inconnue +unknown inconnu +unknown inconnus +unknown inconnues +winner vainqueur +winner gagnants +winner gagnante +winner gagnant +winner lauréat +chart graphique +chart diagramme +initially initialement +loss perte +loss pertes +sites sites +starting démarrer +starting démarrage +architecture architecture +relations relations +upper supérieure +upper supérieur +supported supportés +supported appuyé +supported soutenue +supported soutenu +supported supporté +tracks morceaux +tracks chenilles +tracks pistes +contract contrat +face visage +face face +directly directement +spent dépensé +spent dépensés +girl fille +girl fillette +clearly manifestement +clearly clairement +junior juniors +junior junior +francisco francisco +politics politiques +politics politique +presented présentés +presented présentées +presented présentée +presented présenté +mar mar +cause cause +volume volume +caused causée +caused causés +caused causé +caused causées +tom tom +flight vol +candidate candidats +candidate candidate +candidate candidat +matches matches +matches correspondances +matches matchs +matches allumettes +claimed réclamé +claimed prétendu +claimed revendiqué +except sauf +except excepté +oil huiles +oil huile +oil pétrole +assistant assistant +assistant adjointe +assistant adjoint +assistant assistante +surface surface +victory victoire +victory victoires +regiment régiment +stories histoires +represented représentées +represented représenté +represented représentée +represented représentés +gets obtient +speedy prompt +speedy speedy +weeks semaines +allow permettent +allow autoriser +allow permettre +branch succursale +branch branche +branch embranchement +retired retraités +retired retraite +retired retraité +communities communautés +communities collectivités +train train +paper papier +paper papiers +adding ajout +adding ajoutant +provides fournit +remains restes +remains demeure +victoria victoria +metal métal +metal métaux +metal metal +metal métallique +metal métalliques +wrong faux +wrong tort +wrong mal +wrong erroné +direct directe +direct directement +direct direct +direct directs +frank franck +frank frank +miles milles +miles kilomètres +miles miles +blocked bloquées +blocked bloquée +blocked bloqué +blocked bloqués +launched lancée +launched lancé +launched lancés +mass masse +mass messe +chairman président +comedy comédie +comedy comique +relationship relation +knowledge connaissances +knowledge savoirs +knowledge savoir +knowledge connaissance +format formater +format format +creek crique +creek ruisseau +creek creek +meeting rencontre +meeting réunion +failed échec +failed raté +failed échoué +officers officiers +draft brouillons +draft draft +draft brouillon +goes goes +fight lutte +fight combattre +fight combat +fight bagarre +figure chiffre +figure figure +faculty professeurs +faculty faculté +camp camp +camp campement +camp camps +ran ran +ran couru +variety variété +owner propriétaire +statistics statistiques +statistics statistique +raised élevé +raised soulevées +heavy heavy +heavy lourde +heavy lourd +heavy lourds +alexander aleksandr +alexander alexandre +alexander alexander +alone seul +alone seuls +alone seule +understand comprenez +understand comprendre +understand comprends +episodes épisodes +educational éducatif +daily quotidien +daily quotidiens +daily journalier +daily quotidienne +williams williams +latin latine +latin latin +latin latines +completely complètement +completely complétement +completely entièrement +completely totalement +completely completement +products produits +dark noir +dark foncé +dark sombre +attention attention +religion religion +religion religieux +von von +mind esprit +mind mind +oppose opposer +corps corps +administrative administratif +administrative administratives +administrative administratifs +cut cut +cut couper +cut coupé +cut coupez +cut coupe +scott scott +becoming devenant +becoming devenir +footballer footballeur +jean jean +mayor bourgmestre +mayor maire +mayor maires +pro pro +beach plage +beach plages +descent ascendance +descent descente +nearly presque +leaving quitter +leaving quittant +highly hautement +cast jeté +cast coulée +cast coulé +cast plâtre +territory territoire +write écrivez +write ecrire +write écrire +write rédiger +towns bourgs +forms formes +forms formulaires +joe joe +inside interieur +inside inside +inside intérieur +wanted recherchés +wanted recherché +wanted recherchée +wanted voulu +wanted voulait +solid solides +solid massif +solid solide +individuals particuliers +individuals individus +individuals individuels +authority autorité +mention mention +mention mentionner +projects projets +del del +continue continuer +continue continue +continue continuons +continue continuez +continue poursuivre +cost coût +cost coûts +vice mœurs +vice vice +drive lecteur +drive conduire +notice remarque +notice préavis +notice avis +johnson johnson +forced forcé +forced obligé +forced contraint +forced forcée +basis base +basis fondement +looks looks +reasons raisons +photo photo +hope espère +hope espoirs +hope espérer +hope espoir +hope espérance +log bûche +parents parents +entered entré +entered entrés +mike mike +basic basiques +basic élémentaire +basic basique +basic basic +scientific scientifiques +scientific scientifique +amount montant +spring spring +spring printemps +spring ressort +oxford oxford +kong kong +opera opéra +tried essayé +tried tenté +critical critique +critical critiques +simple simple +simple simples +founder fondateur +founder fondatrice +hong hong +told racontée +told raconté +husband époux +husband mari +useful utile +useful utiles +technical technique +technical techniques +necessary nécessaires +necessary nécessaire +believed croyait +operated fonctionné +operated exploitée +operated opéré +mountains montagnes +mountains monts +mountains montagne +importance importance +musicians musiciens +hotel hôtel +girls filles +crew équipage +crew équipages +feb fév +feb février +boy garçon +boy boy +ontario ontario +nation nation +defense défense +wiki wiki +champions champions +golden dorée +golden golden +golden doré +districts districts +districts arrondissements +districts quartiers +faith faith +faith foi +racing course +racing racing +racing courses +mainly principalement +mainly essentiellement +auto auto +auto automatique +lives vies +swedish suédoise +swedish suédoises +swedish suédois +hot chaud +hot hot +hot chaude +hot sexy +entertainment divertissements +entertainment divertissement +turned tourné +turned tournés +net net +net filets +net filet +soccer football +soccer foot +soccer soccer +creation création +product produit +product produits +tower tour +tower tower +increased augmentées +increased augmenté +increased augmentation +increased augmentée +votes vote +votes votes +votes voix +squadron escadrille +squadron escadre +squadron escadron +squadron escadrons +contemporary contemporains +contemporary contemporain +contemporary contemporaine +focus focus +focus concentrer +marriage mariage +questions questions +naval navale +naval navales +details détails +details détail +forward forward +memorial commémoration +memorial memorial +memorial mémorial +peace paix +kept gardé +iran iranienne +iran iran +korea corée +korea coréenne +analysis analyses +analysis analyse +winners lauréats +winners gagnants +winners vainqueurs +poor pauvre +poor médiocre +poor pauvres +grade grade +cricket criquets +cricket cricket +cricket criquet +judge magistrat +judge juger +judge juge +electric électrique +electric electrique +exist existent +exist existe +exist exister +corporation corporation +hold tenir +hold tiens +hold tenez +campus campus +brazil brésil +chris chris +chris christophe +beyond beyond +fifth cinquième +increase augmenter +increase accroître +increase augmentation +increase accroissement +summary sommaire +summary récapitulatif +summary résumé +remaining restantes +remaining restant +remaining restants +statement énoncé +statement déclaration +broadcast diffusion +broadcast radiodiffusion +broadcast diffuser +getting obtenir +piano piano +novels romans +serving servant +serving portion +serving servir +hour heure +moving déménagement +moving bouger +moving déménager +resolution résolution +concept concept +alternative alternative +alternative alternatif +brothers freres +brothers frères +brothers brothers +attacks attaques +attacks agressions +attacks attentats +encyclopedia encyclopédie +republican républicaine +republican républicain +republican républicains +representatives représentants +politicians politiciens +difficult difficiles +difficult difficile +ability capacité +ability aptitude +studied étudié +studied étudiée +studied étudiées +studied étudiés +host hôte +wall paroi +wall muraille +wall mur +immediately immédiatement +immediately aussitôt +urban urbain +urban urbaines +urban urbains +pakistan pakistan +becomes devient +marine marin +marine marins +marine marine +physical physique +physical physiques +dec dec +dec déc +troops troupes +interview entrevue +interview entretien +interview entrevues +interview interview +coming venir +semi semi +suggest proposer +suggest suggérez +suggest suggère +suggest suggérer +emperor empereur +emperor empereurs +letter lettre +couple couple +duke duke +duke duc +gallery galerie +gallery gallerie +gallery gallery +gallery galeries +follow suivre +follow suivez +follow suivi +windows windows +windows vitres +windows fenêtres +tree sapin +tree arbre +tree arborescence +tree arbres +hits frappe +hits hits +jazz jazz +protection protection +relevant pertinente +relevant pertinent +relevant pertinentes +relevant pertinents +count compter +count compte +count comte +situation situation +reviews examens +reviews critiques +containing contenant +classical classique +classical classiques +offered offerts +offered offertes +offered offerte +offered offert +lady madame +lady dame +netherlands hollande +reports rapports +reports reportages +influence influence +influence influencer +address allocution +address adresse +linear linéaires +linear linéaire +consider envisager +consider considérer +consider considère +consider considérez +consider considérons +machine machine +domain domaine +elements éléments +elements eléments +minnesota minnesota +types types +nov novembre +nov nov +serve servir +serve sers +serve servez +sydney sydney +ministry ministère +blood sang +blood sanguin +blood blood +distance distance +distance éloignement +distance distances +bottom fond +giving donnant +boys garçons +potential potentialités +potential potentiels +potential potentielle +potential potentiel +toronto toronto +toronto montréal +edited modifié +edited édité +infantry infanterie +jun juin +jun jun +formerly anciennement +formerly jadis +formerly auparavant +formerly autrefois +oct octobre +oct oct +conflict conflits +conflict conflit +workers ouvriers +workers travailleurs +steve steve +philadelphia philadelphie +philadelphia philadelphia +helped aidé +helped aidés +der der +nationality nationalité +nationality nationalités +dispute contestation +dispute litige +dispute conflit +dispute dispute +scene scène +method méthode +titles titres +berlin berlin +conditions conditions +arms bras +arms armes +races courses +discovered découvert +discovered découverts +discovered découvertes +iron fer +extended prolongée +extended étendu +extended prolongé +extended prolongés +extended étendue +churches églises +churches eglises +otherwise autrement +otherwise sinon +positive positive +positive positives +positive positif +positive positifs +santa santa +imperial impérial +imperial impériale +imperial impérialiste +imperial imperial +composed composée +composed composées +composed composé +ball boule +ball ball +ball balle +ball ballon +width largeur +width largeurs +quickly rapidement +quickly vite +correct exact +correct corriger +correct corrigez +correct correct +correct correcte +responsible responsable +responsible responsables +possibly éventuellement +possibly possiblement +indiana indiana +soldiers soldats +examples exemples +korean coréens +korean coréen +korean coréenne +genre genre +genre genres +fish poissons +fish pêcher +fish poisson +senate sénat +effects effets +gun fusil +gun pistolet +gun arme +gun revolver +check cocher +check chèque +check vérifiez +check vérifier +appearances apparitions +appearances apparences +plans plans +renamed rebaptisée +renamed rebaptisé +renamed renommé +renamed renommés +renamed renommée +sign signer +sign signe +sign panneau +sign signez +reporting rapports +reporting reportages +reporting reportage +reporting signalement +reporting signaler +sweden suède +consists consiste +heritage héritage +heritage patrimoine +tag étiquette +tag tag +tag balise +primarily principalement +doctor médecin +doctor docteur +leaders meneurs +leaders leaders +leaders dirigeants +lies lies +lies mensonge +lies mensonges +inc inc +rivers fleuves +rivers rivières +crime délit +crime délinquance +crime criminalité +crime crime +liberal libéral +liberal libérale +liberal libéraux +liberal libérales +stand stand +bob bob +existing existantes +existing existant +existing existants +publishing publier +publishing édition +publishing publication +industrial industrielle +industrial industriels +industrial industrielles +industrial industriel +answer réponse +answer répondez +answer répondre +answer reponse +split diviser +split scinder +split split +split scission +apr apr +apr avril +apr avr +sex sexe +mixed mélangé +mixed mixtes +mixed mélangés +mixed mixte +acting agissant +acting agir +personnel personnel +rail ferroviaire +rail rail +die die +die crève +die mourir +die meurs +premier premier +approach approche +approach démarche +wisconsin wisconsin +sentence phrase +sentence sentence +sentence peine +root root +root racines +root racine +standards normes +comics comics +comics bd +earned méritée +earned gagnés +earned mérité +earned gagné +earned gagnées +miss mademoiselle +miss miss +miss melle +miss mlle +specifically spécifiquement +specifically précisément +specifically spécialement +horse chevaux +horse cheval +horse jument +actual réelle +contributions contributions +contributions cotisations +lieutenant lieutenant +wood bois +plants plantes +plants végétaux +initial initiale +initial initial +initial initiales +origin origine +origin origines +environment environnement +pretty joli +pretty jolie +pretty jolies +rank grade +rank rang +rank classer +bus bus +bus autobus +gas gaz +direction direction +guide guide +resources ressources +accepted acceptées +accepted acceptés +accepted acceptée +accepted accepté +animals animaux +nor nor +activity activité +levels niveaux +laws lois +jim jim +creating créer +cambridge cambridge +composer compositeur +composer compositrice +remove supprimer +remove enlever +remove retirer +remove enlèvent +remove supprimez +agency agence +reserve réserve +reserve réserver +atlantic atlantique +supreme supreme +supreme suprême +supreme suprêmes +weight poids +ask demander +ask demandez +ask demande +fighting combats +fighting combat +fighting bagarre +jackson jackson +widely largement +rose rose +rose rosé +treatment traitement +linked liée +linked lié +linked relié +linked liés +andrew andrew +andrew andré +trial essai +trial procès +expanded élargi +expanded élargie +expanded étendu +expanded agrandi +daniel daniel +certainly assurément +certainly certainement +info info +info infos +sciences sciences +fame célébrité +fame renommée +fame gloire +fame fame +everything tout +avenue avenue +travel voyage +travel voyages +scale echelle +scale échelle +break break +break pause +break rupture +break briser +oregon oregon +produce produire +produce produisent +capacity capacité +capacity capacités +#efefef #efefef +fictional fictive +fictional fictif +exchange échange +exchange echange +exchange échanger +actions actions +cited citées +cited cités +cited citée +typically typiquement +typically habituellement +typically généralement +agreement accord +translation traduction +translation traductions +males masculins +males mâles +males hommes +kansas kansas +managed gérée +managed gérés +managed géré +bring amener +bring apporte +bring amenez +bring apportez +bring apporter +charge charge +fails échoue +dedicated dédiée +dedicated dévoué +dedicated dédié +dedicated dédiés +dedicated dévouée +nearby proximité +nearby proche +residents résidants +residents habitants +residents résidents +piece piece +piece morceau +growth croissance +trust trust +trust fiducie +trust confiance +applied appliquée +applied appliqués +applied appliqué +drums batterie +drums tambours +drums fûts +issued émis +issued émises +issued délivrés +issued émise +issued délivré +murder assassiner +murder meurtre +murder assassinat +murder meurtres +normal normale +normal normaux +normal normales +normal normal +twenty twenty +twenty vingt +avoid éviter +avoid évite +avoid eviter +avoid évitez +tony tony +norwegian norvégiens +norwegian norvégien +norwegian norvégienne +criteria critère +criteria critères +context contexte +suggested suggérée +suggested suggérés +suggested suggéré +suggested suggérées +suggested proposé +revolution revolution +revolution révolution +fully complètement +fully pleinement +fully entièrement +fully totalement +wars guerres +aug aout +aug août +aug aug +leaves feuilles +advanced avancés +advanced avancées +advanced avancé +advanced perfectionné +advanced avancée +distribution répartition +distribution distribution +medicine médecine +medicine médicament +garden jardins +garden jardin +reach portée +reach atteindre +turkey turquie +turkey dinde +females femmes +females femelles +publications publications +impact impact +impact incidence +households ménages +survey enquête +survey sondage +height taille +height hauteur +morning matin +morning morning +morning matinée +morning matinale +honor honneur +honor honorer +deep profond +deep profonds +deep deep +deep profonde +argument argumentation +argument dispute +argument argument +argument arguments +publication publication +arthur arthur +elizabeth elizabeth +elizabeth élisabeth +disambiguation homonymie +worth worth +colorado colorado +median médiane +median médian +maryland maryland +falls chutes +falls tombe +falls falls +zone zone +solo solo +solo soliste +learning apprendre +learning apprentissage +pay paye +pay payez +pay payer +pay paie +resolves décide +resolves résout +choice choix +flag pavillon +flag flag +flag drapeau +flag drapeaux +engineer ingénieur +cars voitures +farm ferme +farm fermes +wilson wilson +principal commettant +principal principale +principal principal +acquired acquise +acquired acquises +acquired acquis +constructed construite +constructed construit +constructed construits +secret secrets +secret secret +secret secrète +poet poète +poet poètes +build bâtir +build construire +remain demeurer +remain rester +orchestra orchestre +versions versions +follows suit +fixed corrigé +fixed fixe +fixed fixes +fixed fixé +fixed réparé +efforts efforts +documentary documentaire +documentary documentaires +equipment équipements +equipment équipement +equipment equipements +equipment matériel +equipment equipement +ray raie +ray ray +yellow jaunes +yellow jaune +guard gardes +guard garde +guard gardien +pressure pression +pressure pressions +grant subvention +grant grant +prison prison +freedom liberté +norway norvège +store stocker +store boutique +store magasin +taylor taylor +quarter quart +quarter trimestre +designated désignée +designated désignés +designated désigné +designated désignées +independence indépendance +platform plate +platform quai +platform plateforme +platform plateformes +rome rome +teacher professeur +teacher enseignante +teacher enseignant +teacher institutrice +copy copiez +copy exemplaire +copy copier +copy copie +effort effort +nuclear nucléaires +nuclear nucléaire +pictures images +pictures photos +models maquettes +models modèles +sep sep +easily facilement +easily aisément +thank remercier +description descriptif +description description +agreed accepté +agreed convenu +institutions institutions +covers couvertures +covers couvercles +covers couvre +covers housses +facilities équipements +facilities installations +target cibles +target objectif +target cible +target cibler +stack empiler +stack pile +stack stack +rationale raisonnement +rationale justification +stat stat +combined combiné +combined combinée +combined combinées +combined combinés +bronze bronze +sort tri +sort sorte +sort trier +hosted hébergé +hosted accueillis +hosted accueilli +hosted hébergés +programming programmation +sri sri +railroad railroad +railroad ferroviaire +unique uniques +unique unique +defined définie +defined définis +defined définies +defined défini +ocean océan +cell cell +cell cellules +cell cellule +missouri missouri +concert concert +improve améliorez +improve améliore +improve améliorer +biography biographie +biography biographiques +loan emprunt +loan prêt +contact contacter +contact contactez +contact contact +contact contacte +holy saintes +holy sainte +holy sacré +holy saint +tennessee tennessee +sub sub +safety sûreté +safety sécurité +safety securite +stephen stéphane +stephen stephen +stephen étienne +policies politiques +painting peinture +painting peintures +price prix +price tarif +entirely entièrement +mexican mexicaines +mexican mexicaine +mexican mexicains +mexican mexicain +leadership leadership +flying volants +flying battant +flying volant +flying voler +message message +municipal municipales +municipal municipale +municipal municipaux +serious sérieux +serious serieux +serious graves +serious grave +serious sérieuse +headquarters siège +officially officiellement +cemetery cimetière +memory souvenir +memory souvenirs +memory mémoire +fields domaines +fields champs +generation génération +generation générations +join joindre +join rejoignez +join adhérer +join rejoindre +copies exemplaires +copies copies +copies copie +finals finale +finals finals +finals finales +fox renard +fox fox +fox renards +continues poursuit +continues continue +representative représentant +representative représentante +destroyed détruite +destroyed détruits +destroyed détruit +destroyed détruites +feet pieds +guy mec +guy guy +guy gars +philippines philippines +philippines philippins +revealed révélés +revealed révélé +revealed dévoilé +revealed révélées +revealed révélée +organized organisé +organized organisés +organized organisée +organized organisées +serves sert +conservative conservatisme +conservative conservateur +conservative conservatrice +conservative conservateurs +share partage +share share +share part +share partager +share partagez +maria maría +maria marie +maria maria +disease maladie +disease maladies +sections sections +philosophy philosophique +philosophy philosophie +ways façons +arrived arrivés +arrived arrivé +divided divisé +divided divisée +divided divisées +divided divisés +floor etage +floor étage +floor sol +floor plancher +logo logo +logo logos +cancer cancer +offer offrir +offer offre +tax impôts +tax taxe +tax taxes +tax fiscalité +tax fiscale +tax impôt +expected attendus +expected escompté +expected attendue +expected attendu +traffic trafic +traffic circulation +concerns inquiétudes +concerns préoccupations +graduated diplômée +graduated diplômé +guest invité +guest invitée +guest invités +jews juifs +meant signifiait +economy économie +economy economique +economy economie +storm orage +storm tempête +storm storm +tells raconte +mile mile +protected protégées +protected protégés +protected protégée +protected protégé +bowl gamelle +bowl cuvette +bowl bol +letters lettres +providing fournissant +begins débute +begins commence +classic classic +classic classique +classic classiques +damage dégats +damage dommages +damage dégâts +damage dommage +harry harry +offers offres +offers offre +davis davis +challenge gageure +challenge challenge +challenge contestation +challenge défi +views affichages +views vues +marked marquées +marked marquée +marked marqué +marked marqués +allows permet +density densités +density densité +literary littéraire +literary littéraires +htm htm +ben ben +transportation transports +transportation transport +kentucky kentucky +sales soldes +sales ventes +sales vente +fleet flotte +supporting soutenir +captured capturées +captured capturée +captured capturé +captured capturés +extra supplémentaires +extra supplémentaire +extra extra +recognized reconnu +recognized reconnues +recognized reconnue +recognized reconnus +arizona arizona +compared comparés +compared comparé +theme thème +francis francis +francis françois +moscow moscou +interested intéressée +interested intéressés +interested intéressées +interested intéressé +heard entendu +heard entendue +heard entendues +heard entendus +behavior comportement +transferred transférées +transferred transférés +transferred transférée +transferred transféré +environmental environnemental +blank vide +blank vierge +blank blanc +musician musicienne +musician musicien +assigned attribué +assigned assignée +assigned assignés +assigned affecté +assigned assignées +assigned assigné +seats sièges +tennis tennis +percent pourcent +percent pourcentage +logs grumes +display afficher +display affichage +display affiche +convention convention +ring bague +ring anneau +joint joint +brian brian +deputy adjoint +deputy député +deputy adjointe +planned prévu +planned planifié +planned planifiés +planned planifiée +planned prévue +universities universités +yards verges +yards yards +communist communisme +communist communistes +communist communiste +agent mandataire +agent agent +difference différence +animal animale +animal animal +czech tchèque +czech tchèques +positions positions +exactly exactement +stay reste +stay séjour +stay rester +titled intitulée +titled intitulé +titled titré +combat combattre +combat combat +palace palais +palace palace +ordered commandés +ordered ordonné +ordered commandé +ordered commandée +ordered ordonnée +opposition opposition +attempts tentatives +understanding compréhension +understanding compréhensif +understanding comprendre +understanding compréhensive +wrestling catch +wrestling lutte +wrestling wrestling +wrestling lutter +critics critiques +growing grandir +growing croissant +growing grandissant +growing croissante +establish établir +hands mains +participated participé +poetry poésie +materials matières +materials matériaux +materials materiaux +turkish turques +turkish turque +turkish turc +turkish turcs +paid rémunéré +paid payées +paid payé +paid payée +paid payés +promotion promotion +apparently apparemment +apparently apparement +battalion bataillon +mobile mobile +mobile portable +additions ajouts +additions additions +row rangées +row ligne +row rang +row rangée +merged fusionnées +merged fusionnée +merged fusionnés +merged fusionné +metropolitan métropolitain +metropolitan métropolitaine +metropolitan metropolitan +figures chiffres +existence existence +eye oculaire +eye yeux +eye oeil +eye œil +louisiana louisiane +lewis lewis +melbourne melbourne +austria autriche +brigade brigade +screen screen +screen ecran +screen écran +risk risques +risk risque +conducted mené +lats lat +lats lats +ban ban +ban bannir +ban bannissement +ban interdiction +ban interdire +legislative législative +legislative législatives +legislative législatif +definition définition +definition définitions +indeed effectivement +draw tirage +draw dessine +draw dessiner +application application +application candidature +steel acier +presence présence +expansion agrandissement +expansion extension +expansion expansion +earl earl +earl comte +max max +max maxi +max maxime +max maximum +wild wild +wild sauvage +wild sauvages +planning planifier +planning planification +comic comic +comic comique +adopted adoptées +adopted adoptée +adopted adopté +adopted adoptés +easy facile +easy faciles +easy easy +easy facilité +plus plus +happy heureuses +happy joyeux +happy heureux +happy heureuse +happy joyeuse +acts actes +classes classes +iowa iowa +save économiser +save enregistrer +save sauvegarder +save sauver +wins gagne +wins victoires +theater théatre +theater théâtre +exists existe +roles rôles +chance hasard +chance chance +prevent empêcher +prevent prévenir +linecolor linecolor +candidates candidats +object objet +felt sentis +felt feutre +felt senti +felt ressenti +powers pouvoirs +powers powers +birds oiseaux +spread propagation +spread répandre +defeat défaite +defeat défaites +defeat vaincre +cape cape +cape cap +identified identifiée +identified identifié +identified identifiées +identified identifiés +regions régions +mine miens +mine mine +mine mienne +mine mien +sides côtés +jul jul +showing montrant +teaching enseignement +guidelines directives +simon simon +depth profondeur +depth profondeurs +lyrics paroles +lyrics lyrique +christmas noël +christmas noel +declined décliné +declined refusée +declined refusé +greece grece +greece grèce +express exprès +express exprimer +express express +express expresse +federation fédération +journalist journaliste +intelligence renseignement +intelligence intelligence +connection branchement +connection raccordement +connection connexion +displayed affichée +displayed affiché +displayed affichées +displayed affichés +portuguese portugaise +portuguese portugais +declared déclarées +declared déclarée +declared déclaré +declared déclarés +constitution constitution +presidential présidentielle +presidential présidentiel +standing standing +sons fils +sons sons +plot complot +plot parcelle +dates dates +ends ends +ends extrémités +pilot pilot +pilot pilote +pilot pilotes +relatively relativement +receive reçois +receive reçoivent +receive recevez +receive recevoir +educated éduquée +educated instruit +educated éduqués +educated éduqué +opposed opposés +manchester manchester +queensland queensland +americans américains +introduction introduction +directors administrateurs +directors réalisateurs +directors directeurs +vehicle véhicule +stock stock +vehicles véhicules +israeli israélien +israeli israélienne +israeli israéliens +israeli israéliennes +frequently fréquent +frequently souvent +frequently fréquemment +hills collines +performing performante +performing exécutant +northwest northwest +drug drogue +drug médicament +visit visite +visit visitez +visit visiter +portion portion +residence résidence +walter walter +pov pov +interesting intéressante +interesting intéressant +interesting intéressantes +interesting intéressants +moon lunaire +moon lune +moon moon +limit limite +limit limites +limit limiter +minute minute +bell cloche +bell clochette +bell bell +athletics athlétisme +reduced réduit +reduced réduite +reduced réduites +wind vent +wind éolien +wind éolienne +wind vents +oklahoma oklahoma +architect architecte +architect architectes +ideas idées +electronic électroniques +electronic electronique +electronic électronique +crown couronnes +crown crown +crown couronner +crown couronne +anderson anderson +step step +step étape +weapons armements +weapons armes +weapons armement +unable incapable +neutral neutre +neutral neutres +neutral neutralité +connected connectée +connected connectés +connected raccordée +connected connecté +switzerland suisse +expatriate expatrié +expatriate expatriés +armed armés +armed armées +armed armé +weekly hebdo +weekly hebdomadaire +weekly hebdomadaires +rating notation +rating cotation +rating cote +programme programme +squad escouade +squad équipe +squad escadron +squad brigade +multi multi +dynasty dynastie +cold rhume +cold froide +cold froid +cold froids +granted accordées +granted accordé +granted octroyé +granted accordée +socorro socorro +alliance alliance +alliance alliances +methods méthodes +sam sam +alabama alabama +albert albert +tropical tropiques +tropical tropicale +tropical tropical +tropical tropicaux +tropical tropicales +vietnam vietnam +vietnam viêtnam +dvd dvd +heat chaleur +heat thermique +heat chauffer +fans fans +fans ventilateurs +fans adeptes +surrounding entourant +credit crédit +commons commons +boat bateau +boat canot +boxes boîtes +boxes cartons +boxes boites +boxes coffrets +ethnic ethnique +ethnic ethniques +speaking parlant +fell tombé +fell tombée +fell tomba +arena arena +arena aréna +arena arène +roads routes +roads chemins +core core +core cœur +core coeur +core noyau +dog chiens +dog chien +dog chienne +kill kill +kill tuez +kill tuer +kill tue +athletic athlétique +athletic athlétisme +oldest aîné +negative négative +negative négatif +negative négatives +negative négatifs +confirmed confirmé +confirmed confirmées +confirmed confirmés +confirmed confirmée +sixth sixth +sixth sixième +edge bord +edge bordure +edge arête +edge edge +jesus jésus +jesus jesus +tools outillage +tools outils +colonel colonel +weak faiblesse +weak faible +weak faibles +chosen choisi +chosen choisis +chosen choisie +brand marque +resulting résultante +resulting résultant +nfl nfl +rise rise +supply approvisionnement +tradition tradition +tradition traditions +elementary primaire +elementary élémentaire +elementary élémentaires +household ménages +household ménage +spirit esprit +spirit spirit +task tâche +slightly légèrement +howard howard +incident incident +incident incidents +develop développez +develop développer +sunday sunday +sunday dimanches +sunday dimanche +discuss discuter +discuss discutez +stats stats +stats statistiques +climate climat +topics sujets +topics thèmes +purchased achetée +purchased acheté +purchased achetées +purchased achetés +communications communications +chapter chapitre +broken brisé +broken broken +broken cassé +broken brisée +broken cassée +singapore singapour +situated située +situated situé +license permis +license licences +license licence +haven haven +deaths morts +deaths décès +passing passer +passing passant +citizens citoyens +guns fusils +guns armes +guns pistolets +guns guns +guns canons +trees arbres +gone partis +gone parti +gone gone +improved améliorés +improved améliorées +improved améliorée +improved amélioré +visual visuelle +visual visuels +visual visuel +visual visuelles +pope pape +pope papes +officials fonctionnaires +officials officiels +sat assise +sat assis +sat sat +glass vitre +glass verre +glass verres +miller miller +miller meunier +posted publiée +posted affiché +posted posté +posted publié +estimated estimées +estimated estimée +estimated estimations +estimated estimé +estimated estimation +contain contiennent +contain contenir +brazilian brésilien +brazilian brésilienne +brazilian brésiliens +brazilian brésiliennes +sexual sexuel +sexual sexuelles +sexual sexuelle +defence défense +respectively respectivement +concerning concernant +rich riches +rich rich +rich riche +fast rapide +fast rapidement +fast vite +fast rapidité +fast rapides +properties propriétés +taught enseignée +taught enseigné +taught appris +extensive extensive +exhibition expositions +exhibition exposition +speech discours +speech allocution +proposal proposition +straight hétéro +internal internes +internal interne +effective efficace +effective efficaces +solution solution +fashion mode +foot foot +foot pied +foot pieds +orange orange +orange orangé +orange oranges +argentina argentine +brief bref +brief brèves +brief brève +performances performances +performances spectacles +performances représentations +adult adulte +adult adultes +newly nouvellement +identity identité +singers chanteurs +singers chanteuses +inspired inspiré +inspired inspirées +inspired inspirés +inspired inspirée +discussed discuté +require requiert +require exige +require exigent +require nécessite +require exiger +require requièrent +facility facilité +transfer virement +transfer transfer +transfer transférer +transfer transfert +egypt égypte +egypt egypte +cells cellules +patrick patrick +quebec québec +quebec québécois +quebec quebec +connecticut connecticut +scoring marquer +scoring pointage +scoring notation +anthony anthony +anthony antoine +permanent permanente +permanent permanentes +permanent permanent +permanent permanents +phase phase +audience audience +audience auditoires +audience auditoire +motion motion +motion mouvement +blues blues +blues cafard +blues bleus +blues bleues +hungarian hongroise +hungarian hongroises +hungarian hongrois +arab arabe +arab arabes +trains trains +sets ensembles +ranked classés +ranked classées +ranked classé +unlike contrairement +begin commencez +begin commencer +setting paramètre +setting réglage +eyes yeux +studios studios +gmina gmina +criminal criminels +criminal pénale +criminal criminel +criminal criminelle +commonwealth commonwealth +finish finis +finish finir +finish terminer +finish termine +communication communication +scope portée +accused accusés +accused inculpé +accused accusé +accused accusée +divisions divisions +accept accepte +accept accepter +accept acceptent +accept acceptez +warning alerte +warning avertissement +warning avertissements +alan alain +alan alan +objects objets +diego diego +contest concours +fighter chasseur +fighter fighter +fighter combattant +fighter boxeur +finds trouve +finds trouvailles +coaches autocars +coaches entraîneurs +coaches coachs +beat beat +beat battez +beat battu +beat battre +beat battement +extremely extrêmement +ford gué +ford ford +swiss suisses +swiss suisse +sorry désolé +sorry désolée +sorry désolés +sorry pardon +houston houston +worldwide mondiale +worldwide mondial +showed montré +holds détient +holds tient +holds cales +cathedral cathédrale +cathedral cathédrales +losing perdant +losing perdre +advance advance +advance avancée +advance avance +reality réalité +reality réalités +broadcasting radiodiffusion +broadcasting diffusion +adam adam +vandalism vandalisme +enemy ennemies +enemy ennemis +enemy ennemie +enemy ennemi +youtube youtube +assessed évaluées +assessed évalué +assessed évaluée +assessed évalués +billion billion +billion milliard +billion milliards +buried enfoui +buried inhumé +buried enterré +buried enterrés +buried enterrée +belgium belgique +respect respect +respect respecter +respect respecte +rare rareté +rare rare +rare saignant +rare rares +detroit détroit +detroit detroit +graduate diplômés +graduate diplômée +graduate diplômé +colleges collèges +explain expliquez +explain expliquer +explain explique +authorities autorités +killing tuerie +killing tuer +maximum maximale +maximum maximal +maximum maximales +maximum maximum +neither ni +fan ventilateur +fan fan +fan éventail +fan ventilateurs +fan adepte +notify aviser +notify informer +notify avertir +notify notifier +painter peintre +hamilton hamilton +returning retournant +returning revenant +attempted essayé +attempted tentatives +attempted tentée +attempted tenté +attempted tentative +universe universe +universe univers +passes passe +passes passes +obvious évident +obvious évidence +obvious évidente +suffered subie +suffered souffert +pieces morceaux +apply postuler +apply appliquer +apply appliquez +actresses actrices +competitions concours +competitions compétitions +aid aides +aid aide +driver pilote +driver conducteur +folk folk +dan dan +khan khan +baby bébé +baby baby +denmark danemark +tokyo tokyo +billboard panneau +calling appeler +calling appelant +anne anne +danish danoises +danish danois +danish danoise +wants veut +formula formule +formula formules +interior interieur +interior intérieur +interior intérieurs +kevin kévin +kevin kevin +weather temps +weather météorologie +weather météo +weather climat +weather intempéries +powerful puissant +powerful puissants +powerful puissante +powerful puissantes +muslim musulmans +muslim musulmane +muslim musulman +registered enregistrée +registered inscrits +registered enregistrés +registered enregistré +registered inscrit +publisher éditeur +publisher editeur +preceding précédant +sounds sons +sounds bruits +eric éric +eric eric +approved approuvé +approved approuvée +approved approuvés +approved agréée +approved approuvées +approved agréé +achieved atteint +douglas douglas +provincial provincial +provincial provinciales +provincial provinciaux +provincial provinciale +fund fonds +portugal portugal +athletes athlètes +athletes sportifs +bird oiseau +bird oiseaux +bird bird +bands bandes +bands groupes +audio acoustique +audio audio +cat chats +cat félin +cat chatte +cat chat +cat cat +centuries siècles +valid valables +valid valide +valid valable +valid valides +chemical chimique +chemical chimiques +lane lane +holding holding +counties comtés +update actualisation +update actualiser +ncaa ncaa +speak parlez +speak parler +speak parle +finding trouver +domestic domestiques +domestic domestique +ali ali +false faux +false fausse +false false +false fausses +equivalent équivalentes +equivalent équivalente +equivalent équivalents +equivalent équivalent +caught attrapée +caught pris +caught attrapé +caught capturé +caught capturés +christ christ +ending finissant +puerto puerto +perform effectuer +partner associé +partner partenaire +partner associée +partner partenaires +romania roumanie +aviation aviation +aviation aéronautique +failure échec +failure défaillance +ward ward +ward pupille +strength résistance +strength solidité +strength force +knight knight +knight chevalier +knight chevaliers +nominations nominations +nominations candidatures +hungary hongrie +concern préoccupation +concern préoccupations +concern inquiétudes +concern inquiétude +recordings enregistrements +juan juan +functions fonctions +mississippi mississippi +calls appels +criticism critique +criticism critiques +involving impliquant +magic magie +magic magic +magic magique +gordon gordon +treaty traité +antonio antonio +selection choix +selection sélections +selection sélection +rear arrière +rear arriere +rear arrières +colonial colonial +colonial coloniales +colonial coloniaux +colonial coloniale +motor moteur +obtained obtenues +obtained obtenue +obtained obtenus +obtained obtenu +circuit circuit +wish souhait +wish voeu +wish souhaite +wish souhaiter +compilation compilation +compilation recueil +harvard harvard +islamic islamiste +islamic musulman +islamic islamique +determined déterminées +determined déterminée +determined déterminés +determined déterminé +geography géographie +arkansas arkansas +fuel combustible +fuel carburant +artillery artillerie +medieval médiévale +medieval médiévales +medieval médiéval +medieval médiévaux +locations localisations +locations lieux +inclusion inclusion +recognition reconnaissance +moment moment +moment instant +grounds motifs +succeeded réussi +historian historien +condition état +condition condition +physics physique +newspapers journaux +newspapers quotidiens +represent représenter +represent représentent +allen allen +watch regardez +watch montre +watch montres +watch regarder +watch regarde +kitt kitt +protect protège +protect protégez +protect protéger +protect protègent +grey gris +grey grise +launch lancement +launch lancements +launch lancer +launch lancez +dave dave +philip philippe +philip philip +iraq irak +iraq iraq +changing changeant +changing changer +ukraine ukraine +municipalities communes +municipalities municipalités +mix mix +mix mélanger +mix mélange +mix mixage +tamil tamoul +tamil tamil +tamil tamouls +shift maj +shared partagées +shared partagée +shared partagé +austrian autrichiennes +austrian autrichien +austrian autrichienne +door porte +investigation enquête +institution institution +princess princesse +princess princess +princess princesses +trail sentier +trail piste +parks parcs +applications applications +applications demandes +hundred hundred +hundred centaine +hundred cent +requirements exigences +requirements prescriptions +talking parler +talking parlant +kim kim +ltd ltée +ltd ltd +metres mètres +gray gray +gray gris +gray grise +sector secteur +sector sectoriel +dean doyen +dean dean +dean doyenne +agricultural agricole +agricultural agricoles +incorporated incorporés +incorporated incorporé +incorporated incorporées +incorporated incorporée +escape échapper +escape fuite +escape évasion +escape escape +orders ordonnances +orders commandes +orders ordres +corner corner +corner coin +commissioned commandé +commissioned commandée +commissioned commandées +founding fondatrice +mill moulin +mill mill +mill laminoir +mrs mme +mrs mrs +subjects sujets +temperature température +temperature températures +settled réglés +settled réglé +spacewatch spacewatch +remember mémoriser +remember souviens +miami miami +promote promouvoir +values valeurs +spot spot +progress progrès +progress progression +progress progresser +progress avancement +learn apprenez +learn apprendre +learn apprends +planet planete +planet planète +occupied occupé +occupied occupés +occupied occupées +occupied occupée +usage utilisation +usage usage +refused refusées +refused refusés +refused refusé +refused refusée +borough arrondissement +borough borough +truth verite +truth vérité +clark clark +sufficient suffisantes +sufficient suffisants +sufficient suffisamment +sufficient suffisant +sufficient suffisante +equal égales +equal égaux +equal égal +equal égale +administrator administratrice +administrator administrateur +persons personnes +factory fabrique +factory usine +fought combattu +derived dérivé +derived dérivée +derived dérivées +outstanding remarquables +outstanding remarquable +magazines revues +magazines magazines +flow flux +flow débit +flow flow +flow écoulement +peer pair +peer peer +peer pairs +attacked attaqués +attacked agressé +attacked agressée +attacked attaqué +attacked attaquée +generate génère +generate générer +shape shape +shape forme +creator créatrice +creator créateur +requires requiert +requires exige +requires nécessite +option option +lincoln lincoln +starts débute +starts démarre +starts commence +stands stands +stands gradins +establishment etablissement +establishment établissement +selling vendre +causes cause +causes causes +budget budgétaire +budget budget +battles batailles +sky sky +sky ciels +sky ciel +legend légendes +legend légende +arrested arrêtés +arrested arrêté +arrested arrêtée +forum forum +metro métro +metro metro +broke brisé +broke fauché +broke cassé +broke rompu +broke cassée +strike strike +strike frappe +strike grève +strike frapper +injury préjudice +injury blessure +injury lésion +injury lésions +injury blessures +ryan ryan +zero zero +zero zéro +converted convertis +converted convertie +converted converti +violence violences +violence violence +significantly significativement +significantly sensiblement +statements déclarations +controlled contrôlées +controlled contrôlés +controlled contrôlée +controlled contrôlé +welsh galloise +welsh gallois +welsh welsh +dropped chuté +roger reçu +roger roger +pdf pdf +distinguished distinguées +distinguished distingués +distinguished distinguée +distinguished distingué +samuel samuel +translated traduites +translated traduits +translated traduite +translated traduit +papers papiers +detail détails +detail détail +chapel chapelle +chapel chapel +frederick frédéric +frederick frederick +thousands milliers +banks banques +offensive offensif +offensive offensive +offensive offensant +kings rois +kings kings +factor facteur +factor factor +rename renommer +replace remplacez +replace remplacer +replace remplacement +replace remplace +museums musées +resistance résistance +resistance résistant +resistance resistance +resistance résistants +resistance résistances +junction junction +junction jonction +tim tim +engines moteurs +contributed contribué +medium moyen +medium moyennes +medium médium +medium moyenne +medium milieu +device appareil +device dispositif +device périphérique +profit profit +profit bénéfice +profit profits +profit bénéfices +dream reve +dream rêves +dream dream +dream rêver +dream rêve +enter entrer +enter entrez +enter saisir +enter saisissez +twelve twelve +twelve douze +universal universal +universal universels +universal universelle +universal universel +typical typiques +typical typique +skills aptitudes +skills compétences +bought achetées +bought achetée +bought achetés +bought acheté +passenger passager +passenger passagère +passenger passagers +passenger voyageur +cleveland cleveland +funding financement +agriculture agriculture +parent parents +parent parent +decades décennies +receiving recevant +receiving recevoir +signal signaux +signal clignotant +signal signal +reform reforme +reform réformer +reform réforme +reform réformes +organisation organisation +column colonne +column chronique +column colonnes +defunct défunte +defunct défunt +utah utah +managers directeurs +managers gestionnaires +managers dirigeants +qualified qualifiés +qualified qualifiées +qualified qualifiée +qualified qualifié +indicate indiquez +indicate indiquer +ukrainian ukrainiennes +ukrainian ukrainien +ukrainian ukrainienne +ukrainian ukrainiens +gay homosexualité +gay gay +gay homosexuel +amateur amateurs +amateur amateur +obviously manifestement +obviously evidemment +obviously visiblement +obviously évidemment +flora flora +flora flore +gene gènes +gene gène +gene gene +soul âmes +soul ame +soul soul +soul âme +alt alt +alt alat +discussions discussions +montreal montréal +montreal montreal +turns virages +walker walker +walker rôdeur +entrance entrée +path sentier +path chemin +path sillon +nice sympa +nice gentil +nice joli +nice jolie +nice nice +string chaîne +string ficelle +string string +string cordes +influenced influencé +influenced influencées +influenced influencée +influenced influencés +occur survenir +developing développer +abandoned abandonnés +abandoned abandonné +abandoned abandonnées +abandoned abandonnée +humans humains +pair pair +pair paire +pair paires +flat plats +flat plat +sample échantillon +sample échantillons +contained contenues +contained contenaient +banned interdits +banned interdite +banned interdit +banned bannis +banned bannie +moore moore +strongly fortement +visited visités +visited visitée +visited visité +increasing croissante +attorney avocat +attorney avocate +arm bras +arm arm +mathematics mathématique +mathematics mathématiques +canal canal +charts graphiques +charts diagrammes +thinking réfléchir +thinking pensant +thinking pensée +thinking penser +dublin dublin +suggests suggère +surname nom +surname patronyme +brain cerveaux +brain cérébrale +brain cerveau +brain cervelle +pittsburgh pittsburgh +blog blog +blog blogue +economics économie +economics economie +seventh septième +seventh seventh +alex alex +heavily fortement +heavily lourdement +authors auteurs +paintings peintures +paintings toiles +paintings tableaux +concerned préoccupé +concerned préoccupée +concerned concerné +concerned concernés +recipients bénéficiaires +recipients lauréats +recipients destinataires +recipients récipiendaires +controversial controverse +controversial controversée +controversial polémique +controversial controversé +controversy controverse +controversy polémique +controversy controverses +expressed exprimée +expressed exprimées +expressed exprimé +expressed exprimés +josé jose +josé josé +bodies organes +bodies carrosseries +conservation préservation +conservation conservation +maps cartes +marie marie +arguments argumentation +arguments arguments +chain enchaîner +chain chaîne +chain chaine +focused concentré +focused concentrée +readers lecteurs +carl carl +violation violation +violation infraction +offices bureaux +wave vague +wave onde +circle cercle +invasion envahir +invasion invasion +invasion invasions +jimmy jimmy +opportunity opportunités +opportunity opportunité +determine déterminer +determine détermine +colspan colspan +orthodox orthodoxe +orthodox orthodoxie +orthodox orthodoxes +voted votés +voted votées +voted votée +voted voté +formal formels +formal formel +formal formelle +describes décrit +seconds secondes +seconds seconds +cycle cycle +doubt doute +doubt doutes +doubt douter +doubt doutez +golf golf +walls murs +productions productions +constituency circonscription +closely étroitement +occurs survient +huge énormes +huge immense +huge énorme +huge enorme +andy andy +representing représenter +representing représentant +indonesia indonésie +sell vends +sell vendre +mon mon +mon lun +drawn dessiné +diocese diocèse +tank réservoir +tank tank +tank citerne +tank cuve +advice conseil +advice conseils +senator sénateur +senator sénatrice +generated générés +generated générées +generated générée +generated généré +malaysia malaisien +malaysia malaisie +asking demandant +finland finlande +causing causant +leads prospects +lawyer avocat +lawyer juriste +lawyer avocate +seattle seattle +gain gain +index index +index indice +saints saintes +saints saints +runner runner +runner coureur +crisis crise +cinema cinémas +cinema cinéma +cinema ciné +matt matt +matt matthieu +matt mat +hollywood hollywood +reaction réaction +medals médailles +documents documents +reader lecteur +lawrence laurent +lawrence lawrence +pattern motif +pattern schéma +archives archives +atlanta atlanta +voting vote +voting voter +voting votants +reviewed revu +reviewed examiné +bear ours +bear bear +perfect parfaits +perfect parfait +perfect parfaites +perfect parfaite +restored restaurée +restored restaurés +restored rétabli +restored restauré +bruce bruce +baltimore baltimore +baron baron +pan pan +pan casserole +commune commune +fantasy imaginaire +fantasy fantasy +fantasy fantasme +fantasy fantaisie +duty devoir +chair chaise +chair président +chair présidence +chair fauteuil +scenes scènes +broad large +opposite opposé +opposite opposée +opposite contraire +opposite opposés +stuff trucs +aged vieilli +streets rues +nick nick +anna anna +billy billy +extension prolongement +extension prolongation +extension extension +extension vulgarisation +kent kent +parliamentary parlementaire +kelly kelly +shooting fusillade +shooting tirs +shooting tournage +shooting tir +shooting tirer +ready prête +ready pret +ready prêtes +ready prêt +ready prêts +pick pick +songwriter compositeur +songwriter compositrice +aware consciente +aware conscient +aware conscients +jordan jordanie +jordan jourdain +jordan jordanienne +jordan jordan +dictionary dictionnaire +dictionary dictionnaires +composition composition +salt salé +salt sel +bangladesh bangladesh +bot bot +benefit bénéfice +benefit avantage +benefit prestation +lands terres +interests intérêts +scheduled prévu +scheduled planifiée +scheduled programmée +scheduled planifié +scheduled programmé +teachers professeurs +teachers enseignants +teachers enseignant +closing fermer +closing clôture +closing fermeture +advertising publicitaire +advertising publicité +advertising publicités +contribution contribution +contribution cotisation +maine maine +retirement retraite +scientists scientifiques +dam dam +dam barrage +dam digue +blocks blocages +blocks blocs +las las +print imprimé +print impression +print imprimer +techniques techniques +participate participez +participate participer +anniversary anniversaire +requested demandée +requested demandé +requested sollicité +requested demandés +discovery découverte +discovery découvertes +discovery discovery +explained expliqués +explained expliqué +explained expliquée +expedition expédition +expedition expedition +citation citation +und und +meanwhile entretemps +hampshire hampshire +creative créatifs +creative créative +creative créatif +maintain maintenir +pierre pierre +detailed détaillées +detailed détaillée +detailed détaillé +facts faits +frame ossature +frame cadre +finance financement +finance finances +finance finance +socialist socialistes +socialist socialiste +script scénario +script script +camera appareil +camera caméra +returns retours +returns retourne +engaged fiancé +engaged fiancée +engaged engagés +engaged fiancés +engaged engagée +engaged engagé +assistance assistance +experienced expérimentées +experienced expérimentée +experienced expérimenté +experienced expérimentés +underground underground +underground souterraine +underground souterrain +underground souterrains +sale soldes +sale vente +beautiful beau +beautiful superbe +beautiful magnifiques +beautiful magnifique +beautiful belle +jane jane +jane jeanne +abc abc +supposed supposés +supposed supposée +supposed supposé +successor successeur +successor successeurs +classification classement +classification classification +tool outil +mining minier +mining mines +mining minière +cabinet armoire +cabinet cabinet +bytes bytes +bytes octets +ross ross +russell russel +russell russell +citations citations +maintained maintenu +maintained entretenu +maintained entretenue +maintained maintenus +maintained maintenue +evening soirée +evening soir +singing chant +singing chanter +fifa fifa +gender sexe +gender sexes +gender genre +venues lieux +lakes lacs +lakes laques +mail courrier +mail mail +jeff jeff +electoral électorale +emergency urgence +emergency urgences +mode mode +christopher christopher +christopher christophe +heads tête +heads têtes +proved prouvé +proved prouvée +proved prouvées +proved prouvés +priest priest +priest prêtre +priest curé +funds fonds +investment investissement +investment investissements +romanian roumains +romanian roumaine +romanian roumain +session session +session séance +capture capturer +capture capture +capture capter +aspects aspects +reduce réduire +trophy trophée +trophy trophy +trophy trophées +abuse sévices +abuse abus +abuse abuser +abuse maltraitance +prefecture préfecture +walk marche +walk promenade +walk balade +walk marcher +normally normalement +snow neige +snow snow +shop boutique +shop magasin +dakota dakota +bush brousse +bush bush +bush buissons +bush buisson +coal houille +coal charbon +inhabitants habitants +gary gary +employees employés +employees collaborateurs +employees salariés +error error +error erreur +error erreurs +invited invité +invited invitée +invited invitées +cable câbles +cable cable +cable câble +protein protéines +protein protéique +protein protéine +accident accident +decade décennie +measure mesurer +measure mesure +watched surveillés +watched regardée +watched regardé +watched regardées +patients patients +patients patientes +downtown downtown +animated animées +animated animé +animated animés +animated animée +satellite satellite +johnny johnny +combination combinaison +courts tribunaux +courts juridictions +sequence séquences +sequence séquence +hook hook +hook hameçon +hook crochet +clean propre +clean nettoient +clean nettoyer +owners propriétaires +twin jumelle +twin jumeau +twin jumeaux +twin jumelles +distributed distribué +distributed distribués +distributed distribuée +distributed distribuées +describe décrivez +describe décrire +defensive défensives +defensive défensive +defensive défensif +islam islam +photos photos +photos photographies +ottoman ottoman +ottoman ottomane +trained formés +trained entraîné +trained formé +affected affectée +affected affectés +affected touchés +affected touchée +affected affecté +affected touché +routes routes +routes itinéraires +ministers ministres +wine vins +wine vin +elsewhere ailleurs +lanka lanka +carlos carlos +landing atterrissage +landing débarquement +landing landing +landing atterrir +collected recueillis +collected collecté +collected collectés +collected collectées +collected recueilli +revival renouveau +rio rio +rio río +communes communes +saturday samedi +saturday saturday +mps parlementaires +mps députés +guess suppose +guess deviner +guess devinez +guess devine +drop goutte +drop drop +sarah sarah +laid pondu +swimming nager +swimming natation +swimming baignade +membership membres +membership adhésion +edinburgh edimbourg +edinburgh édimbourg +fit fit +fit ajuster +harris harris +dallas dallas +degrees degrés +degrees diplômes +degrees degré +bachelor célibataire +bachelor bachelor +bachelor baccalauréat +personally personnellement +briefly brièvement +files fichiers +files dossiers +extreme extrême +extreme extrêmes +extreme extreme +courses cours +reaching atteignant +reaching atteindre +sought recherchée +sought recherché +vision vision +demand exiger +vertical verticales +vertical verticalement +vertical verticaux +vertical vertical +vertical verticale +updated actualisé +marketing marketing +marketing commercialisation +jason jason +consisted consistait +appeal pourvoi +appeal appel +plane avion +quick rapide +quick rapidement +quick vite +quick rapides +victor viktor +victor victor +dyk dyk +solar solaires +solar solaire +ages âges +neighborhood voisinage +neighborhood quartier +fairly équitablement +wings wings +wings ailes +acid acid +acid acide +acid acidité +acid acides +rfc rfc +constant constante +constant constantes +constant constant +constant constants +hip hip +hip hanches +hip branché +hip hanche +admins administrateurs +nova nova +ceremony cérémonie +chile chilien +chile chili +composers compositeurs +nazi nazie +nazi nazi +nazi nazis +scholar érudit +liverpool liverpool +hero héro +hero héros +hero hero +designer designer +designer styliste +designer dessinateur +designer concepteur +designer créateur +learned appris +learned apprise +learned apprises +instruments instruments +welcome bienvenues +welcome bienvenue +welcome bienvenus +welcome bienvenu +hair cheveu +hair cheveux +hair coiffure +consecutive consécutifs +consecutive consécutif +consecutive consécutives +consecutive consécutive +movies films +movies cinéma +movies ciné +adjacent adjacents +adjacent contiguës +adjacent adjacentes +adjacent adjacente +adjacent adjacent +pool piscine +pool pool +tue mar +tue aut +norman norman +norman normand +norman normands +norman normande +collections collections +belgian belge +belgian belges +austin austin +ensure assurez +driving conduire +driving conduite +phone téléphone +fly vole +fly voler +fly mouches +fly mouche +ian ian +window hublot +window vitre +window fenetre +window fenêtre +window fenêtres +document document +adams adams +collaboration collaboration +collaboration collaboratif +margaret margaret +margaret marguerite +kennedy kennedy +leg patte +leg jambes +leg cuisse +leg jambe +videos vidéos +assume présumer +assume supposer +assume supposez +attached attachée +attached attaché +attached attachées +attached attachés +dry sèche +dry sèches +dry sec +dry secs +dry sécher +expand étendre +expand agrandir +expand développez +expand élargir +bible biblique +bible bible +matthew matthew +matthew mathieu +matthew matthieu +serbian serbe +serbian serbes +instrument instrument +covering couvrant +random random +random hasard +random aléatoire +random aléatoires +represents représente +participants participants +thorough minutieux +thorough minutieuse +mentions mentionne +mentions mentions +portrait portrait +drivers chauffeurs +drivers conducteurs +drivers pilotes +drivers conducteur +airlines airlines +franklin franklin +viewers téléspectateurs +viewers spectateurs +finnish finnoise +finnish finlandais +finnish finnois +finnish finlandaise +differences différences +venue lieu +vocal vocaux +vocal chant +vocal vocale +vocal vocal +element element +element élément +regularly régulièrement +rejected rejetées +rejected refusées +rejected refusé +rejected rejetés +rejected rejetée +rejected rejeté +relative relatif +relative relative +relative parent +illegal illégaux +illegal illégales +illegal illégale +illegal illégal +illegal illicite +stewart stewart +roof toit +roof toiture +leagues ligues +leagues lieues +colour couleur +colour coloris +colour couleurs +morgan morgane +morgan morgan +prisoners prisonnières +prisoners prisonniers +prisoners détenus +facebook facebook +attend assister +nelson nelson +survived survécu +insurance assurances +insurance assurance +expert expert +steam vapeur +steam vapeurs +cards cartes +manufacturing fabrication +testing essai +testing test +testing essais +testing tests +testing tester +coastal côtières +coastal littoral +coastal côtier +coastal côtière +yorkshire yorkshire +rescue secours +rescue sauvetage +rescue sauveteurs +territories territoires +thu thu +thailand thaïlande +thailand thailande +struck frappé +choose choisissez +choose choisir +choose choisis +vienna vienne +journey voyage +journey journey +journey parcours +storage stockage +storage entreposage +storage rangement +costs coûts +singh singh +distinct distinctes +distinct distinct +distinct distincte +distinct distincts +notably notamment +soldier soldat +colony colonie +evolution evolution +evolution évolution +taiwan taiwan +taiwan taïwan +hurricane ouragans +hurricane ouragan +judges magistrats +judges juges +gardens jardins +poems poèmes +driven conduit +responsibility responsabilités +responsibility responsabilité +sentences phrases +birmingham birmingham +engineers ingénieur +engineers ingénieurs +visible visible +visible visibles +substantial substantielle +substantial substantiel +gulf golfe +gulf gulf +installed installées +installed installé +installed installés +installed installée +revolutionary révolutionnaires +revolutionary révolutionnaire +trip voyage +trip trip +restaurant restaurant +restaurant gastronomie +graham graham +stores magasins +rice riz +prove prouvez +prove prouver +prove prouve +reasonable raisonnable +reasonable raisonnablement +reasonable raisonnables +skin peau +committed engagée +committed engagés +committed commis +committed engagé +volleyball volley +volleyball volleyball +chose choisit +chose choisi +factors facteurs +hundreds centaines +hundreds centaine +injured blessé +injured blessée +injured blessées +injured blessés +injured lésée +devices dispositifs +devices appareils +devices périphériques +phrase phrase +stanley yves +stanley stanley +lemmon lemmon +thompson thompson +suicide suicide +suicide suicides +advantage avantage +automatically automatiquement +disc disque +minimum minimales +minimum minimum +minimum minimal +minimum minimale +minimum minimums +goods marchandises +goods marchandise +goods biens +charges charges +alfred alfred +operator exploitant +operator opérateur +finishing finition +finishing parachèvement +finishing finitions +finishing terminer +fred fred +identify identifier +producers producteurs +ann ann +ann anne +campbell campbell +portland portland +latest dernier +latest dernières +latest derniers +latest dernière +releases communiqués +releases rejets +victims victimes +explanation explication +explanation explications +operate opérer +threat menaces +threat menace +crossing croisement +crossing franchissement +crossing traverser +crossing traversée +slow ralentir +slow lentement +slow lenteur +slow lent +slow lente +poets poètes +stopped stoppé +strategy stratégie +wayne wayne +ranking classement +disney disney +wright wright +residential résidentiel +residential résidentielle +residential résidentiels +associate associé +associate associés +associate associée +associate associer +significance signification +significance importance +ruled gouverné +ruled statué +excellent excellents +excellent excellente +excellent excellent +observed observés +observed observée +observed observées +observed observé +threatened menacés +threatened menacée +threatened menacé +threatened menacées +friendly amicale +friendly sympathique +friendly convivial +friendly amical +redirects redirections +temporary provisoire +temporary temporaires +temporary temporaire +masters masters +masters capitaines +masters maîtres +peninsula péninsule +networks réseaux +passengers voyageurs +passengers passagers +assumed supposé +assumed présumé +artistic artistiques +artistic artistique +safe sécuritaire +safe sûr +safe coffre +festivals festivals +festivals fêtes +compete rivaliser +compete concourir +png png +hunter chasseur +hunter hunter +alaska alaska +partnership partenariats +partnership partenariat +maintenance entretien +maintenance maintenance +monitoring surveillance +monitoring suivi +evil diaboliques +evil mal +evil evil +evil diabolique +relief soulagé +relief relief +relief soulagement +relief secours +charlie charlie +poverty pauvreté +hop houblon +hop hop +fri ven +fri fri +suspected présumés +suspected soupçonnée +suspected suspecté +suspected soupçonné +filled remplies +filled remplis +filled remplie +filled rempli +nba nba +decide décider +decide décide +decide décident +decide décidez +breaking brisant +breaking cassant +breaking rompre +breaking rupture +breaking briser +argentine argentins +argentine argentin +argentine argentine +resigned démissionné +resigned démission +oblast oblast +drew drew +hawaii hawaï +hawaii hawaii +brooklyn brooklyn +historians historiens +speaker orateur +speaker intervenant +speaker enceintes +speaker conférencier +moth papillon +permission autorisation +permission permission +wounded blessé +wounded blessée +wounded blessées +wounded blessés +racial raciale +marshall marshall +gate gate +gate porte +springs ressorts +springs springs +roy roy +photography photographie +photography photographique +helping aidant +helping aider +knights knights +knights chevaliers +roll rouler +roll roll +roll rouleau +progressive progressiste +progressive progressifs +progressive progressif +progressive graduel +progressive progressive +contrast contrastes +contrast contraste +continuing continuer +continuing continue +processes processus +processes procédés +terminal terminal +terminal terminaux +executed exécuté +executed exécutés +executed exécutée +svg svg +spouse conjoint +spouse époux +spouse conjoints +spouse epouse +spouse épouse +infrastructure infrastructures +infrastructure infrastructure +principle principe +painters peintres +painted peintes +painted peinte +painted peint +painted peints +properly correctement +frequency répétition +frequency périodicité +frequency fréquence +frequency fréquences +shaped façonné +shaped façonnés +shaped façonnée +joining rejoindre +robinson robinson +waters eaux +waters waters +ridge dorsale +ridge ridge +ridge arête +ridge crête +bridges passerelles +bridges ponts +ceo pdg +monument monument +mental mentale +mental mental +carter carter +karl karl +rowspan rowspan +mac mac +orleans orleans +orleans orléans +portal portal +portal portail +parallel parallèlement +parallel parallèles +parallel parallèle +thirty trente +thirty thirty +giant géants +giant géant +giant giant +giant géante +qualifying qualifications +qualifying qualifiée +qualifying qualification +qualifying qualifier +murray murray +afghanistan afghanistan +assessment évaluation +assessment appréciation +counter compteur +counter comptoir +bears bears +bears ours +bears oursons +purchase achats +purchase acheter +purchase achat +purchase achetez +expression expression +uefa uefa +improvement amélioration +improvement améliorations +madrid madrid +closure bouclage +closure clôture +closure fermeture +wheel roue +wheel roues +ambassador ambassadeur +ambassador ambassadrice +desert désert +desert désertique +bringing apportant +iranian iraniennes +iranian iranienne +iranian iranien +reign règne +reign régner +uncle tonton +uncle oncle +severe sévères +severe graves +severe grave +severe sévère +rain pluies +rain rain +rain pluie +admiral amiral +fishing peche +fishing pêcher +fishing pêche +existed existait +existed existé +existed existaient +raise élever +broadway broadway +principles principes +grow grandir +grow croître +grow pousser +tests essais +tests tests +tests épreuves +roughly grossièrement +tech tech +trouble trouble +trouble perturbation +rico rico +paragraph paragraphe +paragraph alinéa +bat bat +bat batte +prepared préparés +prepared préparées +prepared préparée +prepared préparé +measures mesures +robin robin +hired engagée +hired engagés +hired embauché +hired embauchés +hired engagé +fear peur +fear fear +fear crainte +fear craintes +merit mérites +merit mérite +participation participation +massive massifs +massive massif +massive massive +massive massives +designs dessins +agencies organismes +agencies agences +technique technique +alberta alberta +egyptian égyptiens +egyptian égyptienne +egyptian égyptien +clerk clerc +clerk greffier +clerk commis +knew saviez +knew savait +knew savais +knew savaient +narrow étroite +narrow étroits +narrow étroites +narrow étroit +adapted adaptée +adapted adaptés +adapted adaptées +adapted adapté +commissioner commissaire +rapid rapide +rapid rapid +rapid rapides +credited crédités +credited crédité +dating rencontres +businesses entreprises +bomb bombe +bomb bombes +capable capables +capable capable +poem poème +stages étapes +honorary honoraire +honorary honorifique +honorary honorifiques +dragon dragons +dragon dragon +charged débité +charged inculpé +charged facturés +charged chargé +propose proposer +modified modifié +modified modifiée +modified modifiées +modified modifiés +fired virée +fired viré +fired virés +mlb mlb +send envoie +send envoyez +send envoi +send envoyer +proof justificatif +proof preuve +proof preuves +practices pratiques +arabic arabe +arabic arabes +attractions attraits +attractions attractions +mouth bouche +mouth embouchure +fix fix +fix correctif +fix réparer +licensed licencié +licensed agréé +symbol symboles +symbol symbole +organ orgue +organ organe +damaged endommagé +damaged endommagées +damaged endommagés +damaged abîmés +damaged abîmé +damaged endommagée +warren warren +exception exception +costa costa +unfortunately hélas +unfortunately malheureusement +jerusalem jerusalem +jerusalem jérusalem +replacement remplacements +replacement remplacement +replacement remplaçant +indians indiens +soundtrack soundtrack +virgin virgin +virgin vierges +virgin vierge +virgin puceau +thousand millier +thousand mille +thousand milliers +vancouver vancouver +legislation législation +legislation législations +beauty beauté +credits générique +credits crédits +buy acheter +buy achat +buy achetez +organisations organisations +serbia serbie +christianity chrétienté +christianity christianisme +opinions opinions +cavalry cavalerie +tribe tribu +richmond richmond +chess échecs +chess echec +channels chaînes +channels canaux +claiming revendication +claiming réclamer +claiming affirmant +claiming prétendant +claiming revendiquer +exact exacte +exact exactement +exact exact +exact exactes +baker boulanger +baker baker +allied alliés +allied allié +allied allied +allied alliées +allied alliée +involvement implication +anime animé +anime anime +donald donald +sisters soeurs +sisters sœurs +requests demandes +requests requêtes +unusual inhabituelle +unusual inhabituel +unusual insolites +unusual insolite +impossible impossible +impossible impossibles +colors coloris +colors couleurs +cook cuisiner +cook cuisinier +cook cuire +cook cook +cook cuisinière +drawing dessin +drawing tirage +drawing dessiner +wikimedia wikimedia +jonathan jonathan +removal déménagement +removal enlèvement +removal suppression +indicates indique +admitted admise +admitted avoué +admitted admis +admitted admises +ownership appropriation +ownership propriété +shore rive +shore rivages +shore shore +shore rivage +monitored surveillés +monitored surveillée +monitored surveillé +nebraska nebraska +regulations réglementation +regulations règlements +regulations règlementation +regulations réglementations +regulations règlement +crash plantage +crash accident +crash crash +crash krach +guitarist guitariste +supports supporte +supports soutient +supports soutiens +supports supports +abbey abbaye +deleting effacement +deleting supprimant +deleting suppression +nevada nevada +barry barry +tone tonalité +tone tonus +operates fonctionne +operates opère +indigenous indigènes +indigenous indigène +indigenous autochtone +personality personnalité +reception réception +transit transit +buffalo buffle +buffalo buffles +buffalo buffalo +buffalo bisons +buffalo bison +flowers fleurs +bond cautionnement +bond caution +bond bond +jay jay +adventure aventures +adventure aventure +definitely assurément +definitely certainement +definitely définitivement +guinea guinée +guinea guinéen +guinea guinéenne +horror horreur +rangers rangers +pointed pointé +pointed pointu +apple pommier +apple pommes +apple apple +apple pomme +popularity popularité +occasionally occasionnellement +occasionally parfois +coalition coalition +franchise franchise +franchise franchises +franchise franchisé +franchise franchisés +starred étoilé +critic critique +journals revues +rolling laminage +rolling rouler +rolling roulant +rolling roulement +percentage pourcentages +percentage pourcentage +silent silencieuse +silent muet +silent silencieux +silent silencieuses +laboratory laboratoire +laboratory laboratoires +microsoft microsoft +movements mouvements +charter affrètement +charter charter +charter charte +charter chartes +suitable approprié +suitable convenable +alternate suppléant +alternate alterner +offering offrir +offering offrant +offering offrande +missions missions +experimental expérimental +experimental expérimentale +rooms salles +rooms chambres +concluded conclu +concluded conclue +concluded conclus +reputation renommée +reputation notoriété +reputation réputation +accurate exacte +accurate précises +accurate précis +versus versus +websites sites +interpretation interprétation +tagged étiquetés +tagged identifié +tagged identifiées +tagged étiqueté +tagged marqué +tagged identifiés +endemic endémique +endemic endémiques +chemistry chimie +achieve réaliser +knows sait +knows connait +manga manga +manga mangas +journalists journalistes +forests forêts +forests forêt +cbs cbs +symphony symphonique +symphony symphonie +promotional promotionnel +promotional promotionnelles +promotional promotionnels +electrical électrique +electrical electrique +tags identifications +tags balises +tags étiquettes +tags tags +meters compteurs +meters mètres +jerry jerry +tigers tigers +tigers tigres +commerce commerce +remix remix +addressed adressé +addressed adressée +phil phil +automatic automatiques +automatic automatisé +automatic automatique +gang gang +gang bande +printed imprimé +printed imprimée +printed imprimées +printed imprimés +oak chênes +oak oak +oak chêne +warner avertisseur +warner warner +tend tendent +quote cotation +quote citation +quote devis +quote citer +quote cite +separated séparées +separated séparée +separated séparés +separated séparé +bishops évêques +glasgow glasgow +essentially essentiellement +essentially fondamentalement +wait attendre +wait attends +wait attendez +battery batterie +battery pile +battery accumulateur +favor faveur +benjamin benjamin +apparent apparent +apparent apparente +shopping achats +shopping shopping +patrol patrouilles +patrol patrouiller +patrol patrouille +eagle eagle +eagle aigle +angel ange +angel angélique +angel angel +angel anges +martial martial +martial martiaux +restoration restauration +delhi delhi +hans hans +indicated indiquée +indicated indiqués +indicated indiqué +morris morris +centers centres +mills fraises +mills mills +mills moulins +mills broyeurs +helpful serviable +helpful utile +helpful utiles +delivered livrés +delivered livrée +delivered délivré +delivered livré +components composants +components composantes +victorian victorien +victorian victorienne +legislature législatif +legislature législateur +legislature législature +tourism tourisme +treated traitée +treated soigné +treated traitées +extent étendue +kids enfants +barbara barbara +essay essai +circumstances circonstances +repeated répétés +repeated répétée +repeated répété +repeated répétées +plain plaine +superior supérieurs +superior supérieure +superior supérieur +superior superieure +superior superieur +strategic stratégique +strategic stratégiques +similarly pareillement +duties fonctions +duties devoirs +effectively efficacement +blp blp +considering considérant +arranged arrangée +arranged arrangé +arranged arrangés +ken ken +grammar grammaire +amendment amendement +amendment modification +alleged présumés +alleged prétendument +alleged prétendue +alleged présumé +alleged présumée +alleged prétendu +relation relation +habitat habitats +habitat habitat +spoken parlé +spoken parlés +spoken parlée +shell coquillage +shell coquille +shell shell +mounted montées +mounted montée +mounted montés +mounted monté +entries entrées +conflicts conflits +conflicts conflit +philippine philippin +philippine philippins +philippine philippine +philippine philippines +montana montana +appearing apparaître +appearing apparaissant +triple triples +triple triplé +triple tripler +triple triple +caribbean caraïbes +caribbean antilles +caribbean caraïbe +hosts hôtes +signs signes +signs panneaux +seriously sérieux +seriously sérieusement +seriously serieux +bristol bristol +warring belligérantes +mitchell mitchell +industries industries +colombia colombie +comparison comparatif +comparison comparaison +comparison comparaisons +basin bassine +basin bassin +eleven onze +eleven eleven +ill ill +ill malade +ill malades +pradesh pradesh +charity bienfaisance +charity aumône +charity charité +output sortie +dna adn +carbon carbone +boats bateaux +desc desc +architectural architecturaux +architectural architectural +representation représentation +commentary commentaire +rising hausse +rising rising +rising montante +visitors visiteur +visitors visiteurs +markets marchés +plate assiette +plate plaque +giants géants +giants giants +processing transformation +processing traitement +landscape paysager +landscape paysage +landscape paysages +dick bite +dick dick +hunt hunt +hunt pourchasser +hunt chasser +hunt chasse +summit sommet +psychology psychologie +ride chevaucher +ride balade +greatly grandement +guardian guardian +guardian tuteur +guardian gardienne +guardian gardien +terminus terminus +losses pertes +balance balance +balance solde +balance équilibre +democracy démocratie +nicholas nicolas +nicholas nicholas +usual habituel +usual habituelle +peru pérou +eighth eighth +eighth huitième +instrumental instrumental +instrumental instrumentale +hindu hindou +hindu hindous +hindu hindoue +defender défenseur +riding équitation +arrival arrivée +arrival arriver +arrival arrivées +evans evans +turning tournant +imply impliquer +imply impliquent +imply insinuer +prose prose +cargo cargaison +cargo fret +hidden masquée +hidden cachée +hidden masqué +hidden caché +hidden cachés +hidden masqués +volunteer volontaires +volunteer volontaire +volunteer bénévole +volunteer bénévolat +volunteer bénévoles +bio bio +bio biographie +holder titulaire +holder porteur +holder détenteur +sugar sucre +sugar sucres +daughters filles +wildlife faune +fun fun +fun plaisir +fun marrant +fun amusant +integrated intégrée +integrated intégrées +integrated intégré +integrated intégrés +partners partenaires +rates tarifs +rates taux +grace grâces +grace grâce +grace grace +feed nourrir +feed fil +childhood enfance +accompanied accompagné +accompanied accompagnée +accompanied accompagnées +accompanied accompagnés +milan milan +photographs photographies +honour honneur +soil terre +soil sols +soil sol +server server +server serveur +manual manuel +manual manual +manual manuels +manual manuelle +concrete concret +concrete concrètes +concrete béton +possibility possibilité +ghost ghost +ghost fantôme +ghost fantômes +confused perplexe +confused confus +confused confuse +confused troublé +tunnel tunnel +larry larry +styles styles +elevation altitude +elevation élévation +muhammad muhammad +muhammad mahomet +considerable considérable +considerable considérables +inter inter +lose perdez +lose perdre +lose perd +phoenix phénix +phoenix phoenix +sweet doux +sweet sucré +sweet douce +sweet sweet +waste déchets +waste gaspillages +waste gaspillage +waste déchet +operational opérationnelles +operational opérationnel +operational opérationnels +operational opérationnelle +tall tall +qualify qualifier +constitutional constitutionnelle +constitutional constitutionnel +constitutional constitutionnels +peoples peuples +acceptable acceptables +acceptable acceptable +fruit fruitières +fruit fruits +decisions décisions +depression dépressions +depression dépression +perspective perspective +midfielder milieu +crystal cristal +crystal cristalline +crystal cristaux +crystal crystal +monastery monastère +monastery monastères +resident résidente +resident résidant +resident résident +resident résidents +cincinnati cincinnati +tied liées +surgery chirurgie +steps marches +steps étapes +carrier porteur +carrier transporteur +stream flux +stream ruisseau +alice alice +kick kick +kick botter +strange étrange +strange bizarre +strange strange +strange etrange +predecessor prédécesseur +bernard bernard +nigeria nigeria +nigeria nigéria +pain souffrance +pain douleur +pain peine +pain douleurs +influential influente +influential influentes +influential influents +influential influent +punk punk +punk voyou +suggestion suggestion +interaction interaction +interaction interactions +retained conservés +retained conservées +retained conservé +retained retenue +retained retenu +achievement accomplissement +mechanical mécanique +mechanical mécaniques +drugs drogues +drugs drogue +missed loupé +missed manqué +missed manquée +missed raté +missed manqués +trinity trinity +trinity trinité +classified classées +classified classifié +classified classés +classified classé +minority minoritaires +minority minorité +minority minoritaire +coat manteau +powered alimenté +powered motorisé +powered propulsé +alive vivante +alive vivants +alive vivant +alive vivantes +alive alive +nbc nbc +nhl lnh +keith keith +bobby bobby +harbor harbor +behaviour comportements +behaviour comportement +croatian croates +croatian croate +maritime maritimes +maritime maritime +terry terry +virtual virtuel +virtual virtuels +virtual virtuelle +virtual virtual +virtual virtuelles +indoor intérieure +indoor intérieur +periods périodes +spiritual spirituelle +spiritual spirituel +spiritual spirituelles +croatia croatie +lions lions +archbishop archevêque +luis luis +merchant commerçant +merchant marchand +merchant négociant +azerbaijan azerbaïdjan +lots lots +contested contesté +contested contestées +contested contestée +editorial éditorial +editorial rédaction +initiative initiative +charlotte charlotte +pure pures +pure purs +pure pur +pure pure +borders bordures +borders bordure +borders frontières +persian persique +persian perse +persian perses +persian persan +marks marks +marks marques +armenian arménienne +armenian arménien +armenian arméniens +romantic romantisme +romantic romantiques +romantic romantique +replacing remplacer +replacing remplaçant +talent talents +talent talent +unlikely improbable +unlikely invraisemblable +unlikely improbables +panel panneau +panel panel +jump sauter +jump saute +jump jump +jump saut +animation animation +animation animations +agents agents +agents mandataires +employment emploi +employment emplois +trading négoce +trading trading +trading négociation +parker parker +statue statue +dated datés +dated datée +dated daté +wonder émerveillement +wonder merveille +wonder wonder +filed déposé +filed déposées +filed classé +provinces provinces +friday vendredi +jobs emploi +jobs emplois +cuba cuba +são são +são sao +scientist scientifique +schedule horaires +schedule horaire +schedule calendrier +waiting attente +waiting attendre +waiting attendant +familiar familières +familiar familière +familiar familier +familiar familiers +suspect suspecte +suspect suspects +suspect suspect +suspect suspectes +disagree désaccord +suggestions suggestions +turner turner +forming formant +forming formage +formally officiellement +formally formellement +locomotives locomotives +barcelona barcelone +barcelona barcelona +consistent cohérents +consistent cohérentes +consistent cohérente +consistent cohérent +recommended recommandés +recommended conseillé +recommended recommandée +recommended recommandé +recommended recommandées +desire désir +desire désirs +desire désirer +patient patiente +patient patients +patient patient +bulgaria bulgarie +vincent vincent +hear entendez +hear entendre +texts textes +belief croire +belief croyance +visitor visiteur +vessels navires +vessels vaisseaux +basically essentiellement +basically fondamentalement +continental continentale +continental continental +continental continentaux +hole hole +hole trou +fail échec +fail échouer +passage passage +sees voit +wedding mariage +wedding noces +wedding mariages +archaeological archéologique +archaeological archéologiques +layer couche +layer calque +designation désignation +designation dénomination +designation appellation +clan clan +revenue revenus +revenue revenu +revenue recettes +couples couples +suit costard +suit costume +soft douce +soft doux +soft douceur +soft douces +soft tendre +soft soft +weekend weekend +approval approbation +approval agrément +approval homologation +democrats démocrates +democrats démocrate +crimes crimes +collins collins +expatriates expatriés +horses chevaux +horses cheval +wear usure +supporters partisans +supporters supporters +supporters soutiens +cash argent +cash comptant +cash liquidités +dennis dennis +dennis denis +resource ressource +sculpture sculpture +sculpture sculptures +practical pratique +harrison harrison +pink rose +pink rosé +oliver oliver +oliver olivier +limits limite +limits limites +cooper cooper +illustrated illustrée +illustrated illustrées +illustrated illustré +illustrated illustrés +hell hell +hell enfer +statistical statistique +referenced référencés +referenced référencé +arbcom arbcom +wolf loup +wolf loups +wolf wolf +wolf louve +warriors warriors +warriors guerriers +incidents incidents +fresh frais +fresh fraiche +fresh fraîche +fresh fraîcheur +editions éditions +roots racines +roots racine +signature signature +clinical clinique +clinical cliniques +volumes volumes +worst pire +worst pires +adults adultes +adults adulte +contribute cotiser +contribute contribuer +necessarily forcément +necessarily nécessairement +immediate immédiates +immediate immédiat +immediate immédiats +immediate immédiate +immediate immédiatement +feeling feeling +feeling sensation +feeling sentiment +theories théories +essential essentiel +essential indispensable +essential essentielles +essential essentielle +essential essentiels +completion achèvement +conclusion conclusion +technologies technologies +strip strip +bound lié +praised loué +stayed resté +stayed séjourné +hull hull +hull coque +hull coques +diamond losange +diamond diamond +diamond diamant +diamond diamants +origins origine +origins origines +empty vides +empty vider +empty vide +eliminated éliminée +eliminated éliminés +eliminated éliminées +eliminated éliminé +valuable précieux +valuable précieuse +cite citer +cite cite +doubles doubles +branches branches +branches succursales +honors honneurs +brick brick +brick briques +brick brique +experiences expériences +beijing pékin +beijing beijing +tie cravate +tie nouer +tie cravates +lgbt lgbt +lgbt homosexualité +liberty liberty +liberty liberté +siege siege +siege siège +baptist baptiste +ron ron +hebrew hébreu +hebrew hébreux +hebrew hébraïque +affect affecter +affect affectent +decline refuser +decline déclin +decline décliner +decline baisse +coaching accompagnement +coaching coaching +coaching entraîneurs +alpha alpha +equipped équipée +equipped équipés +equipped équipées +equipped équipé +identical identique +identical identiques +submitted soumise +submitted soumises +submitted présenté +submitted soumis +enterprise enterprise +enterprise entreprise +touch toucher +touch touchez +touch touch +touch touche +transmission transmission +transmission transmissions +platforms plateformes +cave caverne +cave grotte +cave cave +filmed filmés +filmed filmé +filmed tourné +filmed filmées +filmed filmée +inch inch +inch pouce +inch centimètre +cool cool +bulgarian bulgare +bulgarian bulgares +liga liga +manhattan manhattan +destruction destructions +destruction anéantissement +destruction destruction +activist militant +activist militante +activist activiste +weapon arme +clay argileux +clay argile +clay clay +keyboards claviers +dangerous dangereux +dangerous dangereuse +dangerous dangereuses +viewed visionné +email email +email courriel +biology biologie +bold audacieuse +bold téméraire +bold audacieux +bold gras +bowling quilles +bowling bowling +compare comparez +compare compare +compare comparons +compare comparer +compare comparaison +treaties traités +affiliated affiliées +affiliated affilié +affiliated affiliée +affiliated affiliés +sock chaussette +assault agression +assault assaut +assault agressions +monthly mensuel +monthly mensuellement +monthly mensuels +monthly mensuelle +foster foster +cousin cousin +cousin cousine +urls url +hispanic hispaniques +hispanic hispanique +logic logique +logic logiques +craig craig +trivial insignifiant +trivial trivial +pioneer pionnier +pioneer pioneer +pioneer pionniers +pioneer pionnière +muslims musulmans +muslims musulman +lay lay +rated évalué +rated classé +rated notée +rated nominale +rated évalués +rated noté +absence absence +amsterdam amsterdam +publishers éditeurs +tribes tribus +percussion percussions +percussion percussion +runners coureurs +themes thèmes +#the #le +#the #the +#the #la +#the #les +benefits avantages +benefits bénéfices +benefits prestations +guards gardes +guards gardiens +flows flux +attributed attribuées +attributed attribué +attributed attribués +attributed attribuée +athens athènes +herbert herbert +celebrated célébré +celebrated célébrée +celebrated fêté +sponsored parrainé +sponsored sponsorisé +sponsored commandité +sponsored sponsorisée +raf raf +delaware delaware +neil neil +pole pôle +pole poteau +pole pole +pole polonais +ref réf +ref ref +ref arbitre +historically historiquement +tail queue +tail filature +tail tail +tours circuits +tours tours +tours tournées +tours visites +stable stabilité +stable écurie +stable stables +stable stable +decides décide +vessel récipient +vessel navire +vessel vaisseau +identification identification +delta delta +writes écrit +mediterranean méditerranéen +mediterranean méditerranéenne +mediterranean méditerranée +volunteers volontaires +volunteers bénévole +volunteers bénévoles +reply réponse +reply réplique +reply répondez +reply répondre +reply reponse +stuart stuart +marvel émerveillement +marvel merveille +marvel marvel +luke luke +luke luc +grave tombe +grave grave +odd etrange +odd impair +odd étrange +odd bizarre +odd odd +hearing audition +hearing audience +hearing ouïe +hearing entendre +uss uss +mall mall +penalty pénalités +penalty peine +penalty pénalité +penalty sanction +solutions solutions +secure sécurisée +secure sécurisés +secure sécurisé +hugh hugues +hugh hugh +steven stéphane +steven steven +sole sole +sole semelle +architects architecte +architects architectes +characteristics caractéristiques +falling tomber +falling chute +falling tombant +spin vrille +spin spin +spin tourner +spin tourne +clinton clinton +villa villa +select choisissez +select choisir +select sélection +select sélectionnez +select sélectionner +metric métrique +metric métriques +criticized critiquée +criticized critiqué +criticized critiqués +surviving survivant +surviving survivre +roberts roberts +standings classements +biological biologiques +biological biologique +lloyd lloyd +munich munich +belongs appartient +adelaide adélaïde +adelaide adelaide +belong appartiennent +belong appartenir +harold harold +norfolk norfolk +butler butler +butler majordome +coi coi +rival rival +rival rivale +rival rivaux +acoustic acoustiques +acoustic acoustique +posts postes +posts poteaux +posts publications +adaptation adaptation +greg greg +reporter journaliste +reporter reporter +url url +absolutely absolument +nobody personne +scholarship bourse +vast vastes +vast vaste +exit quitter +exit sortir +exit sortie +inquiry enquête +dual double +dual dual +belt courroie +belt ceinture +belt belt +noticed remarquée +noticed remarqua +noticed remarqué +patent brevets +patent brevet +mathematical mathématique +mathematical mathématiques +rarely rarement +submission soumission +demographics démographiques +demographics démographie +crowd foule +rick rick +governments gouvernements +bonus bonus +bonus prime +bonus bonification +tourist touriste +mystery mystérieux +mystery mystère +mystery mystérieuse +mystery mystères +click cliquer +click cliquez +click cliquant +walking marche +walking marcher +nevertheless néanmoins +voters électeurs +voters votants +rifle fusil +rifle carabine +component composante +component composant +civilian civil +civilian civils +civilian civile +civilian civiles +partial partiel +partial partielles +partial partiellement +partial partiels +partial partielle +encouraged encouragées +encouraged encouragée +encouraged encouragés +encouraged encouragé +birthday anniversaire +birthday anniversaires +eddie eddie +eddie eddy +christians chrétiens +denver denver +petersburg pétersbourg +petersburg petersbourg +researchers chercheurs +partly partiellement +photographer photographe +runtime runtime +runtime exécutable +jon jon +obama obama +seemed sembla +seemed semblé +seemed semblaient +seemed semblait +clock horloge +clock horloges +violin violon +highways routes +highways autoroutes +holiday vacances +distinction distinction +distinction distinguer +distinction distinctions +artwork oeuvres +artwork artwork +makeup maquillage +makeup makeup +makeup maquiller +catherine catherine +font polices +font font +font police +font fontes +farmers fermiers +farmers agriculteurs +occasions occasions +photograph photographie +photograph photographier +struggle lutte +timestamp timestamp +yale yale +options options +pen stylo +pen pen +pen plume +pen stylos +procedure procédure +jacob jacob +convicted condamnée +convicted condamné +convicted condamnés +touring tournées +touring tournée +transition transition +anglo anglo +legacy héritage +legacy legacy +legacy legs +denied refusées +denied refusé +denied refusée +relationships relations +ottawa ottawa +derby derby +surrounded entourés +surrounded encerclé +surrounded encerclés +surrounded entourée +surrounded entouré +libraries bibliothèques +competing concurrentes +competing rivaliser +speakers enceintes +speakers orateurs +speakers conférenciers +speakers intervenants +grades grades +hudson hudson +administrators administrateurs +sacred sacrées +sacred sacrée +sacred sacré +sacred sacrés +signing signer +signing signant +signing signature +rob rob +rob cambrioler +rob braquer +citizen citoyenne +citizen citoyen +dogs chiens +dogs chien +argue argumenter +believes croit +annually annuellement +cardinal cardinal +nepal népal +intersection croisement +intersection intersection +intersection carrefour +reveals révèle +disputes contentieux +disputes litiges +disputes conflits +disputes différends +beam poutre +beam rayon +beam faisceau +overseas outremer +perry perry +nickname surnom +nickname pseudo +nickname pseudonyme +syria syrienne +syria syrie +wells puits +wells wells +contributing contribuant +contributing contribuer +ultimate ultimes +ultimate ultimate +ultimate ultime +ranks grades +ranks rangs +danny danny +danny dany +retail détail +favorite préféré +favorite préférée +favorite favori +favorite favoris +favorite préférés +vermont vermont +begun commencé +begun commencée +download télécharge +download téléchargements +download télécharger +download téléchargement +trusted confiance +appointment nomination +ballet ballet +jefferson jefferson +anywhere partout +sand sables +sand sable +angle angle +sessions séances +sessions sessions +recreation loisirs +recreation récréation +wearing portant +kenya kenya +accessible accessible +accessible accessibles +ralph ralph +thread filetage +thread thread +thread fil +disruptive perturbateur +disruptive perturbant +disruptive perturbateurs +spend dépenser +ninth ninth +ninth neuvième +arrest arrestation +arrest arrestations +choir chœur +choir chœurs +choir chorale +choir choeur +mines mines +injuries blessures +injuries blessés +injuries lésions +rounds rondes +rounds ronds +competitive compétitif +competitive concurrentiel +competitive compétition +competitive concurrentielle +competitive compétitifs +opportunities possibilités +opportunities occasions +opportunities opportunités +meetings rencontres +meetings réunions +commented commentées +commented commenté +commented commentés +wang wang +woods woods +exercise exercices +exercise exercice +jacques jacques +objective objectif +demolished démolis +demolished démoli +demolished démolie +preferred préféré +preferred préférée +preferred privilégiées +preferred préférence +preferred préférés +pedro pedro +robot robots +robot robot +robot robotique +venezuela venezuela +segment segment +studying étudier +edwards edwards +aim aim +aim viser +dancing danse +dancing danses +dancing danser +dancing dansant +eagles eagles +eagles aigles +demonstrated démontré +demonstrated démontrée +demonstrated montré +tribute hommage +tribute tribut +tribute hommages +continuous continue +continuous ininterrompu +continuous continu +continuous continus +encourage encourager +spider spider +spider araignée +acted agi +convinced persuadé +convinced convaincue +convinced convaincu +convinced convaincus +heroes héros +heroes heroes +describing décrivant +rocks roches +rocks rochers +rocks cailloux +rocks pierres +bed lit +gap fossé +gap écart +gap gap +reflect reflètent +reflect refléter +mars mars +participating participer +participating participantes +participating participant +cooperation coopération +cooperation coopérations +obtain obtenir +gothic gothique +gothic gothiques +protest protestation +protest protestations +protest contestation +protest protester +protest manifestation +hunting chasser +hunting chasse +rfa rfa +frequent fréquentes +frequent fréquent +frequent fréquents +frequent fréquente +conversion conversion +conversion reconversion +stress contrainte +stress stress +stress stressé +manufacturers constructeurs +manufacturers fabricants +checkuser checkuser +voiced exprimé +traditionally traditionnellement +jose jose +jose josé +adventures aventures +tiger tiger +tiger tigresse +tiger tigres +tiger tigre +totally carrément +totally complètement +totally totalement +concentration concentration +sing chantons +sing chanter +sing chante +rocket roquettes +rocket fusée +rocket fusées +rocket rocket +rocket roquette +electricity electricité +electricity électricité +shadow shadow +shadow ombre +shadow ombres +boxing boxe +senators sénateurs +doc doc +doc toubib +stanford stanford +machines machines +vegas vegas +saved enregistré +saved enregistrée +saved sauvés +saved sauvé +saved sauvegardé +saved enregistrés +jury juré +jury jury +jury jurés +jury jurys +calendar calendriers +calendar agenda +calendar calendrier +noble noble +noble nobles +noble noblesse +tommy tommy +guilty coupable +guilty coupables +leo léon +leo léo +leo leo +handle manipuler +handle poignée +extinct éteinte +extinct éteint +extinct éteints +responded répondu +shares partage +shares parts +shares actions +shares partages +scotia scotia +manufacturer constructeur +manufacturer manufacturier +manufacturer fabricant +tales contes +tales récits +implementation implémentation +truck camionnette +truck fourgon +truck camion +spelling orthographe +item élément +load charger +load chargement +load charge +customers clients +customers clientèle +adds ajouts +adds ajoute +spaces espaces +cap casquette +cap pac +cap cap +cap capuchon +orphaned orphelin +orphaned orpheline +orphaned orphelines +orphaned orphelins +ferry ferry +ferry traversier +ferry traversiers +prefer préfère +prefer préférer +prefer préférez +prefer préfèrent +push push +push poussez +push pousser +push pousse +lie mentir +lie mensonge +lie mensonges +berkeley berkeley +lebanon liban +madison madison +throne trône +attracted attiré +attracted attirés +attracted attirée +attracted attirées +lion lion +lion lionne +retrieved récupérés +retrieved récupéré +retrieved récupérées +retrieved récupérée +manor manor +promoting promouvoir +saudi saoudien +serial serial +serial série +abroad etranger +rogers rogers +lights lumières +lights lumière +lights luminaires +gauge gabarit +gauge jauge +gauge écartement +concerts concerts +elder ancien +elder aîné +renaissance renaissance +uniform uniforme +uniform uniformes +chase pourchasser +chase chase +aka alias +aka aka +computers informatique +computers ordinateurs +brisbane brisbane +susan susan +susan sylvie +raymond raymond +flower fleurs +flower fleur +col col +thai thaï +thai thaïlandaise +thai thaïlandais +disaster sinistre +disaster désastre +disaster catastrophe +disaster catastrophes +survive survivent +survive survivre +clothing habillement +clothing vêtements +clothing vêtement +murphy murphy +sharp tranchant +sharp pointu +sharp sharp +explains explique +yugoslavia yougoslave +yugoslavia yougoslavie +buddhist bouddhisme +buddhist bouddhiste +buddhist bouddhistes +publicly publiquement +meat viande +meat viandes +literally littéralement +spam spam +spam indésirable +spam spams +telephone telephone +telephone téléphone +moral morale +moral moral +moral moraux +moral moralité +sung chanté +sung sung +sung chantée +partially partiellement +lawyers avocats +lawyers juristes +citing citant +interviews entretiens +interviews entrevues +interviews interviews +brunswick brunswick +radar radar +radar radars +spending dépenser +spending dépenses +spending dépense +grove grove +grove bosquet +tea thés +tea tea +tea thé +elite élite +elite élitiste +elite élites +elite elite +bright brillant +bright bright +bright lumineux +improving améliorer +sierra sierra +heaven paradis +heaven heaven +heaven cieux +heaven ciel +athlete athlètes +athlete athlète +aspect aspect +answers réponses +ted ted +consumer consommateurs +consumer consommateur +funded financé +funded financées +funded financée +funded financés +exclusive exclusives +exclusive exclusive +exclusive exclusivité +exclusive exclusifs +exclusive exclusif +ibn ibn +manuel manuel +allies alliés +reviewer réviseur +reviewer examinateur +missile missiles +missile missile +mechanism mécanisme +helen hélène +helen helen +withdrawn retirée +withdrawn retirées +withdrawn retiré +intention intention +mini mini +casualties pertes +casualties victimes +diseases maladies +rhythm rythmique +rhythm rythme +pat pat +catch attrape +catch captures +catch attraper +poll sondages +poll sondage +poll scrutin +deck pont +newcastle newcastle +antarctic antarctique +leeds leeds +lasted duré +ranges gammes +ordinary ordinaire +ordinary ordinaires +insects insectes +insects insecte +suffering souffrance +suffering souffrir +suffering souffrant +suffering souffrances +flash instantané +flash flash +flash éclair +flash flashs +worship adoration +worship culte +boundaries frontières +boundaries limites +blind aveugle +blind blind +blind aveugles +pakistani pakistanaises +pakistani pakistanaise +pakistani pakistanais +assuming supposer +assuming supposant +interstate interstate +arrangement arrangement +globe globe +honours distinctions +honours honneurs +gross dégoûtant +gross gross +gross brut +gross brutes +gross brute +gilbert gilbert +applies applique +gradually progressivement +gradually graduellement +managing gérer +experiment expérimenter +experiment expérience +experiment expérimentation +radical radical +radical radicaux +radical radicale +gov gov +legs cuisses +legs jambes +legs pattes +opponent opposant +opponent adversaire +diameter diamètre +diameter diamètres +supplies approvisionnement +supplies fournitures +pitch lancer +pitch pitch +utility utilitaire +utility utilité +cleanup nettoyage +opponents adversaires +opponents opposants +regime régime +revised révisée +revised révisées +revised révisé +genera genres +diplomatic diplomates +diplomatic diplomatique +diplomatic diplomate +diplomatic diplomatiques +germans allemands +seal phoque +seal sceau +seal phoques +gregory gregory +gregory grégoire +gregory grégory +corresponding correspondante +corresponding correspondant +concepts notions +concepts concepts +sword épée +sword glaive +sword sabre +purple violette +purple pourpre +purple violet +virus virus +populations populations +bull bull +bull taureau +bull taureaux +drummer batteur +presents présente +holland hollande +bias biais +bias préjugé +merger fusion +remote distants +remote reculées +remote lointain +remote distante +sean sean +messages messages +rebellion révolte +rebellion rébellion +premiere première +premiere premiere +physician médecin +physician physicien +victim victime +con con +cloud nuage +cloud cloud +cloud nuages +angels anges +angels angels +noise bruits +noise bruit +heading rubrique +duo duo +beer bières +beer biere +beer bière +palestinian palestinienne +palestinian palestinien +palestinian palestiniens +us$ usd +us$ us$ +us$ $us +copper cuivre +jurisdiction compétence +jurisdiction juridiction +improvements amélioration +improvements améliorations +ski ski +ski skier +peaked culminé +hms hms +loop boucles +loop boucle +renaming renommer +drum batterie +drum tambour +drum fût +dramatic spectaculaire +dramatic dramatique +dramatic dramatiques +saskatchewan saskatchewan +talks pourparlers +earthquake tremblement +earthquake séismes +earthquake séisme +rhode rhode +hat casquette +hat hat +hat chapeau +requirement exigence +den tanière +den den +tanks réservoirs +tanks chars +tanks citernes +tanks cuves +presidents présidents +min mn +min min +defending défendre +alcohol alcools +alcohol alcool +dominated dominée +dominated dominés +dominated dominé +dominated dominées +sang sang +sang chanté +sang chantait +eat manger +eat mangez +eat mange +graphics graphiques +graphics graphique +graphics graphismes +graphics infographie +graphics graphisme +constituencies circonscriptions +asp asp +coffee café +chancellor chancelier +chancellor chancelière +destroy détruisez +destroy anéantir +destroy détruire +tons tonnes +cruz cruz +warsaw varsovie +exclusively exclusivement +connections raccordements +connections connexions +rush ruée +rush rush +heights hauteurs +playstation playstation +outcome aboutissement +apartment appartement +cardinals cardinaux +fill remplissage +fill remplissez +fill remplir +recipient destinataire +recipient receveur +recipient bénéficiaire +correctly correctement +traditions traditions +fundamental fondamentale +fundamental fondamental +fundamental fondamentaux +thin maigre +thin mince +thin minces +chan chan +resolved résolus +resolved résolues +resolved résolu +resolved résolue +mario mario +departments départements +dame dame +shield bouclier +shield shield +fighters boxeurs +fighters combattants +ivan ivan +writings écritures +writings écrits +bosnia bosnie +sentenced condamnée +sentenced condamné +sentenced condamnés +violent violentes +violent violent +violent violents +violent violente +caption légende +harbour port +harbour harbour +margin marge +margin marges +auckland auckland +postal postaux +postal postal +postal postale +pirates pirates +collective collective +collective collectifs +collective collectif +diesel gazole +diesel diesel +liberation liberation +liberation libération +confederate confédérés +confederate confédéré +devil démon +devil devil +devil diable +activists militants +activists activistes +sultan sultan +rider rider +rider motard +rider cavalier +amazon amazones +amazon amazonie +amazon amazon +amazon amazone +florence florence +marc marc +arnold arnold +shah shah +blogspot blogspot +reduction réduction +contents sommaire +contents contenus +contents contenu +genetic génétique +genetic génétiques +somerset somerset +locally localement +milk lait +romance romantisme +romance idylle +romance romance +romance romantique +intellectual intellectuelle +intellectual intellectuel +intellectual intellectuels +latino latino +latino latinos +failing échouer +mason maçon +mason mason +pete pete +advisory consultatif +arbitration arbitrage +arbitration arbitrages +interface interface +hitler hitler +default défaut +accessed accédé +accessed consulté +sheffield sheffield +departure départ +departure depart +departure départs +hindi hindi +anglican anglicane +suggesting suggérer +suggesting suggérant +mistake erreur +residing résidant +embassy ambassade +embassy ambassades +murdered assassiné +murdered assassinés +murdered tué +murdered assassinée +sox sox +sleep sommeil +sleep dormir +sleep dors +suspended suspendu +suspended suspendues +suspended suspendue +suspended suspendus +sum somme +sum sum +mythology mythologie +bengal bengale +confusion confusion +confusion désarroi +oscar oscar +therapy thérapie +therapy thérapeutiques +therapy thérapies +occasion occasion +exposed exposées +exposed exposés +exposed exposé +assisted aidé +assisted assistée +assisted assisté +possession possession +defend défendons +defend défendre +defend défendez +devoted dévouée +devoted dévoués +devoted dévoué +devoted consacré +graphic graphiques +graphic graphique +milwaukee milwaukee +informed informée +informed informé +informed informés +anonymous anonyme +reverse inverse +reverse inverser +reverse inversé +soap savon +soap soap +soap savons +territorial territoriale +territorial territoriales +territorial territorial +lisa lisa +paulo paulo +northwestern northwestern +playoffs playoffs +playoffs éliminatoires +boss boss +boss patron +boss patronne +nasa nasa +quoted cité +quoted citée +quoted cotés +byzantine byzantin +byzantine byzantine +idaho idaho +poster poster +poster affiches +poster affiche +geographic géographique +geographic géographiques +rebounds rebonds +congo congolais +congo congo +venture venture +worse pire +hoax canular +restricted limité +restricted restreint +doors portes +naming nommer +situations situations +instructions consignes +instructions instructions +sullivan sullivan +tables tables +tables tableaux +leaf feuille +leaf leaf +leaf feuilles +shoot shoot +shoot tirer +shoot tire +shoot tirez +substitute remplaçant +substitute suppléant +substitute substituer +substitute remplaçants +restaurants restaurants +restaurants gastronomie +contributor collaborateur +contributor contributeur +contributor contributeurs +errors fautes +errors erreurs +enjoyed apprécié +framework cadre +rocky rocheux +rocky rocky +kerala kerala +shakespeare shakespeare +quantum quantum +quantum quantique +immigration immigration +mirror miroir +mirror miroirs +certified certifiées +certified certifiée +certified certifié +certified certifiés +assets actif +assets actifs +npov npov +potentially potentiellement +presentation présentation +cotton coton +sitting assise +sitting assis +tournaments tournois +syndrome syndrome +checked vérifiés +checked cochée +checked vérifiées +checked vérifié +checked vérifiée +checked coché +forty forty +forty quarante +sourcing approvisionnement +journalism journalisme +journalism journalistique +unsuccessful infructueuses +towers tours +conductor conducteur +hospitals hôpital +hospitals hôpitaux +bone osseuse +bone osseux +bone os +bone bone +essex essex +rebuilt reconstruits +rebuilt reconstruite +rebuilt reconstruit +wellington wellington +ideal idéale +ideal ideal +ideal idéal +raw cru +raw premières +raw brut +raw crue +raw raw +raw crus +sharing partage +sharing partager +labels labels +labels étiquettes +labels libellés +leonard léonard +leonard leonard +watson watson +governors gouverneurs +posting poster +harvey harvey +bases bases +bases fondements +hello bonjour +hello coucou +hello hello +hello salut +hello bjr +rabbi rabbi +rabbi rabbin +rabbi rabin +hardware quincaillerie +hardware matériel +hardware matériels +ensemble ensemble +monster monstrueuse +monster monstrueux +monster monster +monster monstre +pitcher carafe +pitcher lanceur +pitcher cruche +pitcher pichet +emphasis emphase +recovery rétablissement +recovery recouvrement +recovery récupération +recovery guérison +respond répondez +aaron aaron +lesser moindre +qualification qualifications +qualification qualification +organic bio +organic biologiques +organic biologique +organic organiques +organic organique +exposure exposition +palestine palestine +palestine palestinienne +palestine palestinien +thoughts réflexions +thoughts pensées +drafted rédigée +drafted rédigé +maurice maurice +immigrants immigrants +immigrants immigrés +variant variant +variant variante +lap lap +lap giron +legitimate légitimes +legitimate légitime +autonomous autonomes +autonomous autonomie +autonomous autonome +wallace wallace +succession successions +succession succession +throw jeter +monday lundi +reserves réserve +reserves réserves +donated donné +donated donnés +increases hausses +increases augmentations +increases augmente +kid gamin +delivery livraison +delivery livraisons +delivery accouchement +joan joan +fifty cinquante +fifty fifty +slave slave +slave esclave +slave esclaves +feedback commentaires +feedback rétroaction +columbus colomb +columbus columbus +stones pierres +manage gérer +cgi cgi +initiated initiée +initiated initiées +initiated initiés +initiated initié +favour faveur +printing impression +printing imprimer +printing imprimeries +printing imprimerie +variable variable +variable variables +theology théologique +theology théologie +todd todd +parameters paramètres +traveled voyageait +traveled voyagé +canton canton +han han +reed roseaux +reed reed +celtic celtique +celtic celtic +characteristic caractéristique +commanded commandé +commanded commandée +searching recherchant +inappropriate inapproprié +inappropriate inappropriée +inappropriate inconvenant +inappropriate inappropriés +inappropriate inadéquat +switch interrupteur +switch basculer +switch commutateur +ties cravates +tube tube +otto otto +debt endettement +debt dette +debt dettes +outdoor extérieurs +outdoor extérieur +outdoor extérieure +navigation navigation +eligible admissible +eligible éligible +eligible éligibles +eligible admissibles +experts experts +expensive coûteux +expensive chères +tier tier +gospel évangile +gospel évangélique +gospel evangile +newton newton +essays essais +shanghai shanghai +conventional conventionnelle +conventional conventionnel +conventional conventionnels +campaigns campagnes +feelings sentiments +bath bains +bath baignoire +bath bain +bath bath +venice venise +#aaa #aaa +cats chats +variations variations +variations variantes +emerged émergé +socks chaussettes +socks chaussette +connecting connecter +connecting raccordement +connecting reliant +connecting connexion +flood inonder +flood déluge +flood inondation +flood inondations +flood crue +documented documenté +documented documentés +documented documentées +documented documentée +custom personnalisé +custom personnalisée +custom personnalisés +custom coutume +touchdown touchdown +touchdown touché +profession métier +profession profession +layout disposition +layout agencement +academics universitaires +settlers colons +merging fusionner +sony sony +competitors concurrents +competitors compétiteurs +phillips philips +phillips phillips +grass pelouse +grass graminées +grass gazon +grass herbe +reservoir réservoir +artificial artificiels +artificial artificiel +artificial artificielle +artificial artificielles +novelist romancier +tip pourboire +tip pointe +tip astuce +prague prague +abu abu +abu abou +faces faces +faces visages +guitars guitares +aspx aspx +laura laura +laura laure +fellows boursiers +internationally internationalement +attacking attaquant +attacking attaquer +johann johann +dreams reves +dreams rêve +dreams dreams +dreams rêves +hughes hughes +hughes hugues +suburb faubourg +suburb banlieue +understood compris +specialized spécialisé +specialized spécialisée +specialized spécialisées +specialized spécialisés +warned prévenu +warned prévenus +warned averti +pearl perle +pearl nacre +pearl pearl +pearl perles +chorus chœurs +chorus refrain +chorus choeur +chorus chœur +dependent dépendant +dependent dépendantes +dependent dépendante +dependent dépendants +restrictions restrictions +restrictions restriction +killer tueur +killer killer +killer meurtrier +oakland oakland +trio trio +influences influences +blocking bloquer +blocking bloquant +blocking blocage +mtv mtv +cattle bovins +cattle bétail +gear engrenage +gear engins +gabriel gabriel +traded échangés +traded négociés +traded négociées +traded échangé +skating patinage +fifteen fifteen +fifteen quinze +fifteen quinzaine +palm palme +palm palm +palm paume +palm palmier +wikis wikis +tale récit +tale conte +demonstrate démontrer +vary varient +vary varie +liquid liquide +liquid liquides +cycling vélo +cycling cyclisme +princeton princeton +respective respectifs +voices voix +friedrich friedrich +friedrich frédéric +jet jet +horn trompette +horn corne +horn cor +horn avertisseur +horn horn +erected érigée +erected érigés +erected érigées +erected érigé +burning brûlant +burning brûlure +burning gravure +burning brûler +worker travailleur +worker travailleuse +worker ouvrier +atmosphere atmosphère +atmosphere ambiance +characterized caractérisée +characterized caractérisé +characterized caractérisés +syrian syrien +syrian syrienne +syrian syriens +java java +monitor moniteurs +monitor surveiller +monitor moniteur +graduating diplômé +columns poteaux +columns colonnes +repair réparer +repair réparateur +repair réparation +repair réparations +bin bin +bin poubelle +stick stick +stick bâton +dollars dollars +organised organisé +organised organisés +organised organisée +organised organisées +parameter paramètre +truly vraiment +truly réellement +truly véritablement +resolve résoudre +resolve détermination +buenos buenos +parade défilé +parade parade +backed adossés +awareness conscience +awareness notoriété +depends dépend +depends dépendra +define définit +define définir +define définis +spencer spencer +republicans républicains +conspiracy complot +conspiracy complots +conspiracy conspiration +dies meurt +dies décède +dies meurent +clarke clarke +rough rude +rough bruts +engage engager +engage engagez +pine pine +pine pin +equation équation +feels ressent +democrat démocrates +democrat démocrate +cutting couper +cutting découpage +cutting coupage +cutting découpe +cutting coupe +button button +button boutons +button bouton +brands marques +queens reines +queens queens +abraham abraham +neck nuque +neck cou +neck encolure +forever forever +drink boisson +drink bois +drink boire +drink verre +drink boissons +sheriff shériff +sheriff sheriff +sheriff shérif +miguel miguel +aires aires +montgomery montgomery +vanity coiffeuse +vanity vaniteux +vanity vanité +gift cadeau +gift cadeaux +riders coureurs +riders cavaliers +functional fonctionnels +functional fonctionnelle +functional fonctionnel +crossed traversé +diverse varié +diverse diversifié +diverse diverses +numbered numéroté +numbered numérotées +numbered numérotés +quotes cotation +quotes citations +quotes devis +quotes cotes +quotes soumissions +slowly lentement +slowly doucement +attitude attitude +mouse souris +justin justin +protests protestations +gods dieux +amounts montants +variation variations +variation variante +variation variation +smart intelligente +smart malin +smart intelligent +smart smart +smart futé +prices tarifs +prices prix +prayer prières +prayer prière +prayer prier +terrorism terrorisme +beta beta +beta bêta +beta béta +durham durham +counts comtes +iraqi irakiens +iraqi irakien +iraqi irakienne +detective inspectrice +detective inspecteur +detective détective +josh josh +linking reliant +compositions compositions +oval ovale +filming tournages +filming tournage +filming filmer +perfectly parfaitement +indianapolis indianapolis +funeral funérailles +funeral obsèques +funeral funéraire +funeral enterrement +recovered récupérées +recovered récupéré +recovered retrouvé +recovered récupérée +recovered récupérés +farmer agriculteur +farmer paysan +farmer fermier +protestant protestants +protestant protestant +protestant protestante +cameron cameron +unclear imprécis +indonesian indonésiens +indonesian indonésien +indonesian indonésienne +mixing mélange +mixing mixage +mumbai bombay +mumbai mumbai +nashville nashville +danger danger +rally rally +rally rallyes +rally rassemblement +rally rallye +narrative récit +camps camps +camps campements +surprise surprise +surprise surprenant +surprise surpris +surprise surprendre +surprise surprises +manufactured fabriqués +manufactured fabriqué +deployed déployé +deployed déployées +deployed déployés +deployed déployée +kate kate +molecular moléculaire +molecular moléculaires +unnecessary inutile +unnecessary superflu +unnecessary inutiles +isle isle +theorem théorème +colonies colonies +cyprus chypre +wake veillée +wake wake +wake réveiller +wake sillage +brings apporte +winds vents +magnetic magnétiques +magnetic magnétique +magnetic magnétisme +conversation conversation +sussex sussex +gates portails +gates vannes +gates gates +gates portes +ram ram +ram bélier +plastic plastique +plastic plastiques +electronics electronique +electronics électronique +restore restaurer +restore restaure +restore rétablir +stockholm stockholm +inn inn +buses bus +buses autobus +buses autocars +connect connecter +connect connexion +connect connectez +wmflabs wmflabs +guests invités +guests hôtes +radiation rayonnement +radiation radiations +radiation rayonnements +receives reçoit +lancashire lancashire +playoff playoff +playoff playoffs +playoff éliminatoires +cork bouchon +cork liège +cork cork +generals généraux +intermediate intermédiaire +intermediate intermédiaires +verifiable vérifiable +verifiable vérifiables +cheers tchin +cheers santé +cheers acclamations +filipino philippin +filipino philippins +filipino philippine +oriented orienté +oriented orientée +hamburg hambourg +hamburg hamburg +creates crée +orbit orbite +orbit orbites +massacre massacrer +massacre massacre +massacre massacres +dialogue dialogue +dialogue dialogues +dialogue dialoguer +illness maladie +dress robe +codes codes +dawn aube +dawn dawn +dawn aurore +isolated isolées +isolated isolés +isolated isolée +isolated isolé +nancy nancy +violations violation +violations infractions +violations violations +perth perth +tenure titularisation +ladies mesdemoiselles +ladies mesdames +ladies dames +autumn automne +ratings cotes +ratings appréciations +ratings évaluations +incorrect incorrect +incorrect inexacte +incorrect incorrecte +incorrect erroné +scout scout +scout éclaireur +difficulty difficulté +difficulty difficultés +pupils élèves +wealth richesses +wealth richesse +hart hart +allegations allégations +regulation réglementation +regulation règlement +regulation régulation +regulation règlementation +watching regardant +watching regarder +lodge lodge +eggs œufs +eggs oeufs +disputed contesté +disputed disputées +disputed contestées +disputed contestée +citizenship nationalité +citizenship citoyenneté +specialist spécialiste +tasks tâches +intent intention +intent intentions +instruction instruction +ceased cessé +pride pride +pride orgueil +pride fierté +banner bannières +banner banderole +banner banner +banner bannière +friendship amitié +friendship amitiés +panama panama +panama panamá +corruption corruption +sunk coulés +sunk coulé +harm harm +ernest ernest +pilots pilotes +pursue poursuivre +tape ruban +tape cassette +tape bande +emigrants émigrants +emigrants émigrés +cancelled annulée +cancelled annulé +cancelled annulés +revenge revanche +revenge vengeance +revenge venger +revision révisions +revision révision +dominant dominante +dominant dominant +fee honoraires +fee redevance +computing informatique +examination examen +chen chen +matrix matrice +matrix matrix +das das +biographical biographiques +biographical biographique +kiss baiser +kiss kiss +kiss bisou +valign valign +nationalist nationaliste +nationalist nationalistes +luck chance +crosses croix +heavyweight heavyweight +heavyweight lourd +bid bid +bid enchère +appreciate apprécier +appreciate apprécie +enemies ennemies +enemies ennemis +mercury mercury +mercury mercure +interactive interactivité +interactive interactif +interactive interactives +interactive interactive +interactive interactifs +math maths +math mathématique +math mathématiques +preserve conserver +preserve préserver +nobel nobel +grande grande +structural structurelle +structural structurel +structural structurels +marry marier +marry épouser +airports aéroports +veterans vétérans +axis axes +axis axe +execution exécution +cult secte +cult culte +cult cultes +reducing réduire +reducing réducteur +reducing réducteurs +colin colin +chester chester +ticket billet +ticket ticket +belonging appartenance +belonging appartenant +entity entité +judicial judiciaire +explicitly explicitement +explicitly expressément +bombing bombardement +bombing bombardements +bombing attentat +recognised reconnu +recognised reconnues +recognised reconnue +recognised reconnus +applicable applicable +applicable applicables +founders fondateurs +fitted équipée +fitted équipé +wilhelm wilhelm +suddenly subitement +suddenly soudainement +suddenly soudain +parking stationnement +parking parking +absolute absolue +absolute absolu +françois françois +locomotive locomotive +locomotive locomotives +preparation préparation +nintendo nintendo +declaration déclaration +presumably vraisemblablement +burial enfouissement +burial inhumation +burial enterrement +governing régissant +governing gouverner +jamaica jamaïque +knowing sachant +knowing connaissant +vladimir vladimir +beating battant +beating battre +avg avg +methodist méthodiste +utf utf +challenges défis +kenneth kenneth +evolved évolué +celebration célébrations +celebration célébration +discipline discipline +discipline discipliné +bearing portant +bearing roulements +belonged appartenait +belonged appartenaient +belonged appartenu +fauna faune +manuscript manuscrit +manuscript manuscrits +experiments expériences +experiments expérimentations +chiefs chefs +compound composé +tampa tampa +arabia arabie +arabia saoudite +associations associations +targets cibles +alien étranger +alien alien +alien extraterrestre +depicted dépeint +depicted représenté +sergeant sergent +sergeant adjudant +diffs diffs +subsidiary filiale +subsidiary subsidiaires +thirteen treize +thirteen thirteen +thick épaisse +thick épais +thick épaisses +extend étendre +extend prolonger +dismissed congédié +dismissed licencié +dismissed rejeté +neo neo +neo néo +wire fils +wire fil +phd doctorat +phd doctorats +phd doctorants +measured mesurés +measured mesuré +measured mesurée +measured mesurées +fat grasse +fat gros +fat gras +fat graisses +fat graisse +visits visites +linux linux +teach enseigner +flights vols +verse verset +verse couplet +bennett bennett +bennett bennet +warm chaud +warm chaleureux +warm chaleureuses +warm chaude +dynamic dynamisme +dynamic dynamic +dynamic dynamiques +dynamic dynamique +shaw shaw +breaks ruptures +breaks casse +breaks pauses +monuments monuments +lying mentir +lying mensonge +lying menteur +lords lords +lords seigneurs +michel michel +treat traitez +treat traiter +raid raid +raid raids +congregation congrégation +temperatures températures +temperatures température +testament testament +drinking boire +drinking boisson +drinking potable +companion compagnon +companion compagne +manila manille +km² km² +punjab pendjab +imagine imaginez +imagine imagines +imagine imaginons +imagine imaginer +imagine imagine +consideration considération +veteran vétéran +doctors docteurs +doctors médecins +eldest aîné +ruler règle +ruler dirigeant +ruler souverain +wise sage +wise wise +shipping livraison +shipping expédition +shipping envoi +afc afc +worthy dignes +worthy digne +registration inscription +registration enregistrement +registration immatriculation +directory annuaire +directory annuaires +directory répertoire +wyoming wyoming +manitoba manitoba +vietnamese vietnamiens +vietnamese vietnamien +vietnamese vietnamienne +vietnamese vietnamiennes +ronald ronald +cuban cubain +cuban cubaine +cuban cubains +cuban cubaines +burns brûlure +burns burns +burns brûlures +burns brûlés +justify justifier +justify justifie +justify justifient +divine divine +divine divin +divine divins +suppose suppose +suppose supposer +suppose supposez +suppose supposons +fate fatalité +fate destin +rovers rovers +cole cole +oral orale +oral buccale +oral oral +oral oraux +trans trans +boards planches +bryan bryan +santiago santiago +episcopal épiscopale +terrorist terroriste +terrorist terroristes +okay okay +okay ok +waves ondes +waves vagues +invented inventés +invented inventée +invented inventé +landed atterri +landed débarqué +landed débarqués +sandy sable +sandy sandy +acres hectares +acres acres +paint peinture +paint peindre +actively activement +indication indication +stops arrêts +excellence excellence +integration intégration +bibliography bibliographie +nonsense bêtises +nonsense absurde +nonsense absurdité +nonsense sottises +marathon marathon +beliefs croyances +beliefs croyance +redundant redondant +redundant superflu +freestyle freestyle +freestyle acrobatique +aerial antenne +aerial aérien +aerial aérienne +preservation préservation +preservation conservation +altitude altitude +freely librement +simultaneously simultanément +simultaneously simultané +psychological psychologique +fernando fernando +cultures cultures +taxes impôt +taxes impôts +taxes taxes +taxes fiscalité +marcus marcus +stakes pieux +stakes enjeu +stakes enjeux +stakes piquets +dominican dominicaine +dominican dominicain +franz franz +coins pièces +coins monnaies +oxygen oxygène +civic civisme +civic citoyenne +civic civique +civic civiques +isaac isaac +spell sort +spell sortilège +spell orthographe +inspiration inspiration +pairs paires +vector vecteur +arc arc +professionals professionnels +vii vii +contrary contrairement +contrary contraires +contrary contraire +accusations accusations +approaches approches +slaves esclave +slaves esclaves +mad furieux +mad fou +mad mad +mad furieuse +mad folle +spectrum spectre +client client +dozen douzaines +dozen dizaine +dozen dizaines +dozen douzaine +travels voyages +symbols symboles +plaza plaza +banking banques +banking banque +banking bancaire +inherited hérité +inherited héritée +inherited héréditaire +inherited héritées +legion légions +legion légion +symptoms symptômes +symptoms symptôme +mosque mosquée +guys mecs +guys gars +lab laboratoire +lab labo +lab lab +sailing voile +orientation orientation +virtually virtuellement +virtually quasiment +virtually pratiquement +generic génériques +generic générique +reasoning raisonnement +reasoning raisonnements +stroke attaque +stroke caresse +stroke avc +unions syndicats +efficient efficace +efficient efficaces +opens ouvre +impression impression +discover découvrez +discover découverte +discover découvrir +relocated réinstallés +roosevelt roosevelt +dancer danseuse +dancer danseur +dancer dancer +phenomenon phénomène +preliminary préliminaires +preliminary préliminaire +recognize reconnaître +recognize reconnaissent +recognize reconnais +anchor ancrage +anchor ancres +anchor ancre +anchor ancrer +arguing argumenter +abilities aptitudes +procedures procédures +emotional émotif +emotional émotionnelle +emotional émotions +emotional affectif +emotional émotionnel +timber bois +fisher fisher +fisher pêcheur +prod prod +cartoon cartoon +disorder désordre +disorder trouble +fled enfui +fled fuit +fled fui +demands exigences +lithuania lituanie +continent continent +fellowship camaraderie +lock écluse +lock serrure +lock verrouiller +lock verrou +relegated relégué +relegated relégués +warrant mandat +pictured imaginais +recurring récurrents +recurring récurrente +recurring récurrent +overview aperçu +wealthy riches +wealthy riche +acquisition acquisition +eve ève +eve eve +filter filtres +filter filtrer +filter filtre +filter filtrage +filter filtrant +addresses adresses +addresses allocutions +independently indépendamment +slovenia slovénie +observation observation +challenged contesté +challenged défié +challenged contestées +challenged contestée +threats menaces +threats menace +fallen tombés +fallen tombée +fallen déchu +fallen déchus +fallen tombé +protocol protocole +protocol protocol +judgment jugement +grammy mamie +grammy grammy +colours colorants +colours coloris +colours couleurs +colours teintes +distinctive distinctif +opposing opposé +opposing opposée +opposing opposés +landmark repère +package paquetage +package forfait +package paquet +package colis +controls contrôles +controls commandes +completing achèvement +sabha sabha +prisoner prisonnière +prisoner prisonnier +prisoner détenu +signals signaux +owen owen +owen fabien +inaugural inaugurale +intervention intervention +arriving arriver +arriving arrivant +cylinder cylindre +cylinder bouteille +cylinder cylindres +cylinder vérin +tenth dixième +tenth tenth +liu liu +tested testé +tested testés +tested testée +tested testées +renowned renommée +renowned renommés +renowned renommé +shops commerces +shops magasins +shops boutiques +dome dôme +dome coupole +dome dome +philosopher philosophes +philosopher philosophe +epic épique +epic epic +epic épopée +stem tige +stem souches +stem potence +specified précisé +specified spécifiés +specified spécifiée +specified spécifié +davies davies +collapse effondrement +allan allan +albanian albanaise +albanian albanais +canyon canyon +samples échantillons +perceived perçu +perceived perçus +perceived perçue +perceived perçues +celebrity célébrités +celebrity célébrité +priests prêtres +louise louise +workshop atelier +workshop ateliers +claude claude +fortune fortune +bars bars +bars barres +cornwall cornouailles +cornwall cornwall +palmer palmer +presidency présidence +tiny minuscule +tiny minuscules +tiny tiny +appeals pourvois +appeals appels +istanbul istanbul +rookie recrue +rookie rookie +expanding expansion +calgary calgary +shock chocs +shock choc +shock choquer +stevens stevens +employee salarié +employee travailleur +employee employé +yang yang +housed logés +tomb tombes +tomb tombe +tomb tombeau +tomb caveau +earning gagner +innovation innovation +streams ruisseaux +unity unité +unity unity +lucas lucas +grows pousse +grows grandit +grows croît +armenia arménienne +armenia arménie +interchange échangeur +proteins protéines +proposals propositions +swimmers nageurs +mainland continentale +seminary séminaire +hamlet hameau +hamlet hamlet +timeline chronologie +timeline journal +realize réalise +newport newport +negotiations négociations +exhibitions expositions +malta malte +hate déteste +hate détester +hate haine +hate haineux +hate haïr +westminster westminster +installation montage +installation installation +enters pénètre +goalkeeper gardien +julian julian +julian julien +morocco maroc +efficiency efficacité +efficiency efficience +efficiency rendement +chapters chapitres +helicopter hélico +helicopter hélicoptère +helicopter hélicoptères +fortress forteresses +fortress forteresse +ani ani +burned brulé +burned brûlés +burned brûlées +burned brûlée +burned brûlé +displays présentoirs +displays affichages +compiled compilés +compiled compilé +compiled compilées +ips ips +contributors collaborateurs +contributors contributeurs diff --git a/homeworks/hw2_seq2seq/README.md b/homeworks/hw2_seq2seq/README.md new file mode 100644 index 0000000..6bcb612 --- /dev/null +++ b/homeworks/hw2_seq2seq/README.md @@ -0,0 +1,5 @@ +**Lab2: neural machine translation for ru->en language direction** + +*Deadline: Sun 10.03.2024 23:59 AOE* + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/girafe-ai/ml-course/blob/24s_advanced/assignments/lab02_nmt/lab02_nmt_24s_advanced.ipynb) diff --git a/homeworks/hw2_seq2seq/lab01_nmt_24s_advanced.ipynb b/homeworks/hw2_seq2seq/lab01_nmt_24s_advanced.ipynb new file mode 100644 index 0000000..7df3af2 --- /dev/null +++ b/homeworks/hw2_seq2seq/lab01_nmt_24s_advanced.ipynb @@ -0,0 +1,1055 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "BmwNc0Bb5X3p" + }, + "source": [ + "## Lab assignment 02" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ydqh6Nv-5X3q" + }, + "source": [ + "### Neural Machine Translation in the wild\n", + "In the third homework you are supposed to get the best translation you can for the RU-EN translation task.\n", + "\n", + "Basic approach using RNNs as encoder and decoder is implemented for you.\n", + "\n", + "Your ultimate task is to use the techniques we've covered, e.g.\n", + "\n", + "* Optimization enhancements (e.g. learning rate decay)\n", + "\n", + "* Transformer/CNN/ encoder (with or without positional encoding)\n", + "\n", + "* attention/self-attention mechanism (**highly recommended**)\n", + "\n", + "* custom tokenization (BPE units, other subword approaches)\n", + "\n", + "to improve the translation quality.\n", + "\n", + "--------\n", + "\n", + "* __Please use at least three different approaches/models and compare them (translation quality/complexity/training and evaluation time).__\n", + "\n", + "* Write down some summary on your experiments and illustrate it with convergence plots/metrics and your thoughts. Just like you would approach a real problem." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "7_dvIsbm5X3r" + }, + "outputs": [], + "source": [ + "# You might need to install the libraries below. Do it in the desired environment\n", + "# if you are working locally.\n", + "\n", + "# ! pip install subword-nmt\n", + "# ! pip install nltk\n", + "# ! pip install torchtext" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "meRCsofE5X3r" + }, + "outputs": [], + "source": [ + "# Thanks to YSDA NLP course team for the data\n", + "# (who thanks tilda and deephack teams for the data in their turn)\n", + "\n", + "import os\n", + "path_do_data = '../../datasets/Machine_translation_EN_RU/data.txt'\n", + "if not os.path.exists(path_do_data):\n", + " print(\"Dataset not found locally. Downloading from github.\")\n", + " !wget https://raw.githubusercontent.com/neychev/made_nlp_course/master/datasets/Machine_translation_EN_RU/data.txt -nc\n", + " path_do_data = './data.txt'" + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Grading criteria\n", + "\n", + "**100%**\n", + "- implementation of at least 3 model improvements over baseline\n", + "- threshold of 27 BLEU on test corpus\n", + "- experimental results and conclusions in human-readable format :)\n", + "\n", + "**70%**\n", + "- implementation of at least 2 model improvements over baseline\n", + "- threshold of 25 BLEU on test corpus\n", + "- experimental results and conclusions in human-readable format :)\n", + "\n", + "**30%**\n", + "- implementation of at least 1 model improvement over baseline\n", + "- threshold of 21 BLEU on test corpus\n", + "- experimental results and conclusions in human-readable format :)\n", + "\n", + "\n", + "------\n", + "\n", + "#### **Note: Please do not use pretrained machine translation / BERT / LLM checkpoints. All such solutions will be graded at 30% pts.**\n" + ], + "metadata": { + "id": "mSVNrhIm560f" + } + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gTCdJvym5X3r" + }, + "source": [ + "### Warning! The code below is deeeeeeeply deprecated and is is provided only as simple guide.\n", + "We suggest you to stick to most recent pipelines here, e.g. by Huggingface:\n", + "* Example notebook: [link](https://github.com/huggingface/notebooks/blob/main/examples/translation.ipynb)\n", + "* Converting your own dataset to specific format: [link](https://discuss.huggingface.co/t/correct-way-to-create-a-dataset-from-a-csv-file/15686/15)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0QEVmpe95X3s" + }, + "outputs": [], + "source": [ + "# old deprecated code\n", + "import torch\n", + "import torch.nn as nn\n", + "import torch.optim as optim\n", + "\n", + "import torchtext\n", + "from torchtext.datasets import TranslationDataset, Multi30k\n", + "from torchtext.data import Field, BucketIterator\n", + "\n", + "import spacy\n", + "\n", + "import random\n", + "import math\n", + "import time\n", + "\n", + "import matplotlib\n", + "matplotlib.rcParams.update({'figure.figsize': (16, 12), 'font.size': 14})\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "from IPython.display import clear_output\n", + "\n", + "from nltk.tokenize import WordPunctTokenizer\n", + "from subword_nmt.learn_bpe import learn_bpe\n", + "from subword_nmt.apply_bpe import BPE\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "45VrR5c75X3s" + }, + "source": [ + "### Main part\n", + "__Here comes the preprocessing. Do not hesitate to use BPE or more complex preprocessing ;)__" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Y_9BgbGv5X3s" + }, + "outputs": [], + "source": [ + "tokenizer_W = WordPunctTokenizer()\n", + "def tokenize(x, tokenizer=tokenizer_W):\n", + " return tokenizer.tokenize(x.lower())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "A-BlIYlJ5X3s" + }, + "outputs": [], + "source": [ + "SRC = Field(tokenize=tokenize,\n", + " init_token = '',\n", + " eos_token = '',\n", + " lower = True)\n", + "\n", + "TRG = Field(tokenize=tokenize,\n", + " init_token = '',\n", + " eos_token = '',\n", + " lower = True)\n", + "\n", + "dataset = torchtext.data.TabularDataset(\n", + " path=path_do_data,\n", + " format='tsv',\n", + " fields=[('trg', TRG), ('src', SRC)]\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "x_fOj6235X3s" + }, + "outputs": [], + "source": [ + "train_data, valid_data, test_data = dataset.split(split_ratio=[0.8, 0.15, 0.05])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "LVwb_lXu5X3s", + "outputId": "a7b743eb-6074-4fa7-c660-3e95b9e94bc4" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of training examples: 40000\n", + "Number of validation examples: 2500\n", + "Number of testing examples: 7500\n" + ] + } + ], + "source": [ + "print(f\"Number of training examples: {len(train_data.examples)}\")\n", + "print(f\"Number of validation examples: {len(valid_data.examples)}\")\n", + "print(f\"Number of testing examples: {len(test_data.examples)}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "o4T1u_KB5X3t" + }, + "outputs": [], + "source": [ + "SRC.build_vocab(train_data, min_freq = 3)\n", + "TRG.build_vocab(train_data, min_freq = 3)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "L7_STXhR5X3t", + "outputId": "1666267e-1cd9-4f13-f2c9-261338bb7628" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Unique tokens in source (ru) vocabulary: 9267\n", + "Unique tokens in target (en) vocabulary: 6699\n" + ] + } + ], + "source": [ + "print(f\"Unique tokens in source (ru) vocabulary: {len(SRC.vocab)}\")\n", + "print(f\"Unique tokens in target (en) vocabulary: {len(TRG.vocab)}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZYFFDCXJ5X3t" + }, + "source": [ + "Here are tokens from original (RU) corpus:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "v9WNMVnz5X3t", + "outputId": "2068d040-45ce-4560-9d2d-e27db0b9e7b8" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "['',\n", + " '29',\n", + " 'соль',\n", + " 'комо',\n", + " '―',\n", + " 'электрическая',\n", + " 'ming',\n", + " 'утренний',\n", + " 'детском',\n", + " 'таунус']" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "SRC.vocab.itos[::1000]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1ZU4KfiM5X3t" + }, + "source": [ + "And from target (EN) corpus:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "fi7qQJbh5X3t", + "outputId": "426ddeda-f4e2-4330-fd66-da324a599ffa" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "['', 'king', 'buffets', 'catch', 'media', 'schedule', 'maraunenhof']" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "TRG.vocab.itos[::1000]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "i8Ry9Vmg5X3t" + }, + "source": [ + "And here is example from train dataset:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "LyKi-7uf5X3t", + "outputId": "9230c88a-c8f5-4e2d-f9d6-1eb6970b8600" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'trg': ['laundry', 'service', 'is', 'provided', '.'], 'src': ['помимо', 'этого', ',', 'гостям', 'предоставляются', 'услуги', 'прачечной', '.']}\n" + ] + } + ], + "source": [ + "print(vars(train_data.examples[9]))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TQMnac635X3t" + }, + "source": [ + "Let's check the length distributions:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "T1ObGYYM5X3t", + "outputId": "58977d29-617a-407c-d8b8-6a51d71f5083" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Length distribution in Train data\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAAEICAYAAABGRG3WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAer0lEQVR4nO3df7ReVX3n8fdHIr+0kCApxQRNKhlbZC0rzUhcOB3HOPyybVhr1MFxDdGmTVdLW9vpTAvTrmFGZQbXOEVYKpURSrAWpFRLRqw0RRlXp8OPoA7yQ0rKryQFuZIAVqs19jt/nH3hId6be3Pvzb3PPff9WutZ95y99zlnn3Of/XzP2c9+zklVIUmS5r8XzHUFJEnSzDCoS5LUEwZ1SZJ6wqAuSVJPGNQlSeoJg7okST1hUNcBkWRFkkqyaA62/c4kfznb25XmQpKrkrxvGsv/XZIfnck6tfU+nORNM73eSWx3zj57hoFBXfPaQm/AGg5zFcD2V5Jbkvz8YFpVvbiqHpyrOk3XfDn2s8WgrudJctBc10HqG086NVsM6vNIkt9OsjPJN5Pcn2RtSz8kyQeT/G17fTDJIS3vB7qi25Xt8W36qiSXJflskm8B/yLJcUk+lWQkyZNJPjSw7M8luS/J7iQ3JXn5JOt+ZJIrkjzW9uF9oycQo3VM8oG23oeSnDGw7MokX2z7/RdJPpzkD1v2F9vfp1o34usGlhtzfdJMSvJx4GXA/2rvwd8a6EHakORR4POt7B8neTzJ0+09/aqB9VzV3ts3tvf6bUle0fKS5OIkTyR5JslXk5w4Rl2WJPlMa7u72/Tylnch8M+AD7V6fqilD34eHJnk6rb8I0l+N8kLWt4+2+kEx+gFSc5L8jftM+W6JEe1vNFjtT7Jo0m+keR3BpY9LMmmts372vHdMd6xH9jsO8ZaX+9Vla958AJeCWwHXtrmVwCvaNPvAW4FfhhYCvwV8N6W907gL/daVwHHt+mrgKeBU+hO8l4E/D/g4jZ9KPD6VnYdsA34cWAR8LvAX41T3xVtO4va/KeBj7Z1/jBwO/CLA3X8HvALwEHALwF/C6Tl/1/gA8DBwOuBZ4A/HGs7k1mfL18z/QIeBt40MD/6vry6vecPa+k/B/wQcAjwQeArA8tcBTwJvLa1r08A17a804A7gcVAWhs8dmC597XplwD/Cji8beePgT8d2MYtwM/vVffBz4OrgRvasiuAvwY2tLz9aleDxwR4N91n1PK27x8FrtnrWP1P4DDg1cB3gR9v+RcB/xtY0pa/C9gxiWM/5vr6/przCvia5D8KjgeeAN4EvHCvvL8BzhyYPw14uE2/k4mD+tUDea8DRhgIkgN5fzbawNv8C4BvAy8fo+xow1oEHNMa1WED+W8HvjBQx20DeYe3ZX+E7ix8D3D4QP4fMnFQH3N9c/1/9NXP1z4Cy4/uY5nFrcyRbf4q4GMD+WcCX2vTb6QLsGuAF+y1nqtoQX2MbfwEsHtg/hbGCep0gfofgBMG8n4RuKVN71e74vlB/T5g7UDesXQnCIsGjtXygfzbgbPb9IPAaQN5P8/kgvqY6+v7y+73eaKqtgG/Dvxn4Ikk1yZ5act+KfDIQPFHWtpkbR+YPg54pKr2jFHu5cAlSZ5K8hSwi+6qYdkE63858ELgsYFlP0p3xT7q8dGJqvp2m3xx249dA2l713c8461Pmk3PvleTHJTkotYF/QxdMAI4eqD84wPT36a9Z6vq88CHgA/Ttf/Lkxyx98aSHJ7ko63r/Bm6r6cWZ3JjZY6ma6d7f5YMtu+ptquXA58eaP/3Ad+nO+H/gXUzsO90nwGDbX4y7X9f6+s1g/o8UlV/VFWvp2sgBby/Zf1tSxv1spYG8C26M2oAkvzIWKsemN4OvCxjD+zZTtdlvnjgdVhV/dUEVd9Od6V+9MByR1TVqyZYDuAx4Kgkhw+kHTdO3aW5Mt77cDD939B9hfUm4Ei6K0roTown3kDVpVX1k8AJwD8B/sMYxX6T7qu6k6vqCOCn9trGvtrLN+iunvf+LNk5mfpNYDtwxl6fHYdW1WTW/Rhdt/uo4/bK9zNggEF9nkjyyiRvTDcA7jvA3wP/2LKvAX43ydIkRwP/ia6LGrrvx1+V5CeSHEp3pb8vt9M1oouSvCjJoUlOaXm/D5w/OrinDap560R1r6rHgD8H/keSI9qgmVck+eeTWPYRYCvwn5Mc3AbC/cxAkRG64zDjv7OV9sPXmfg9+EN0J7dP0p1o/9fJrjzJP01ycpIX0p2of4fn2v/e2/h7uoGjRwEXTLaeVfV94DrgwiQ/lG4Q7L/juc+S6fj9tt6Xt/1ZmmTdJJe9ju5zZ0mSZcCv7JU/mWO/YBjU549D6AaMfIOuW+mHgfNb3vvoAt9dwFeBL7U0quqv6QbS/QXwALDPm7K0hv0zdN+xPQrsAP51y/s0Xe/Ata1r725gsqPKz6Eb6HYvsBu4nu57tcl4B913/U+2/fok3YfjaBfghcD/aV17aya5Tmkm/Te6E+unkvz7ccpcTdedvZOuHdy6H+s/gm7g1+62jieB/z5GuQ/SDQ77Rlv/5/bKvwR4SxtJfukYy/8q3UnDg3SfFX8EXLkf9RzPJcBm4M+TfLPV7eRJLvseus+hh+g+x66ntf9mMsd+wRgdXSzNG0k+STeAaO+rEEk9l+SX6Aa9TdjTtxB5pa6h17oeX9G67U+n+17yT+e6XpIOvCTHJjmltf9X0o0b+PRc12tYeZcjzQc/AnyK7je4O4Bfqqovz22VJM2Sg+l+LbMSeAq4FvjInNZoiNn9LklST9j9LklST8zb7vejjz66VqxYMdfVkIbanXfe+Y2qWjrX9dgX27I0OZNpzxMG9SRXAj8NPFFVJ7a0o+h+VrSC7q5Ib6uq3UlC99OFM+nu4PPOqvpSW2Y93b3Cobul4aaW/pN0tzk8DPgs8O6axHcCK1asYOvWrRMVkxa0JI9MXGpu2ZalyZlMe55M9/tVwOl7pZ0H3FxVq4Cb2zx0v1le1V4bgctaRUZvgnAy3cMKLkiypC1zGd0DAkaX23tbkiRpEiYM6lX1Rbp7fA9aB2xq05uAswbSr67OrXT3HD6W7gEjW6pqV1XtBrYAp7e8I6rq1nZ1fvXAuiRJ0n6Y6kC5Y9qtP6G7u9noTfmX8fyb7e9oaftK3zFG+piSbEyyNcnWkZGRKVZdkqR+mvbo93aFPSu/i6uqy6tqdVWtXrp0qMf+SJI066Ya1L/eus5pf59o6Tt5/hN0lre0faUvHyNdkiTtp6kG9c3A+ja9HrhhIP2cdNYAT7du+puAU9tTdpYApwI3tbxnkqxpI+fPGViXJEnaD5P5Sds1wBuAo5PsoBvFfhFwXZINdE8Melsr/lm6n7Nto/tJ27sAqmpXkvcCd7Ry76mq0cF3v8xzP2n7s/aSJEn7acKgXlVvHydr7RhlCzh3nPVcyRiP8KuqrcCJE9VDkiTtm7eJlSSpJ+btbWJn04rzbpywzMMXvXkWaiJpumzP6jOv1CVJ6gmDuiRJPWFQlySpJwzqkiT1hEFdkqSeMKhLktQTBnVJknrCoC5JUk8Y1CVJ6gmDuiRJPWFQlySpJwzqkiT1hEFdkqSeMKhLktQTBnVJknrCoC5JUk8Y1KUFJMmVSZ5IcvdA2lFJtiR5oP1d0tKT5NIk25LcleSkgWXWt/IPJFk/kP6TSb7alrk0SWZ3D6WFzaAuLSxXAafvlXYecHNVrQJubvMAZwCr2msjcBl0JwHABcDJwGuBC0ZPBFqZXxhYbu9tSTqADOrSAlJVXwR27ZW8DtjUpjcBZw2kX12dW4HFSY4FTgO2VNWuqtoNbAFOb3lHVNWtVVXA1QPrkjQLDOqSjqmqx9r048AxbXoZsH2g3I6Wtq/0HWOk/4AkG5NsTbJ1ZGRk+nsgCTCoSxrQrrBrFrZzeVWtrqrVS5cuPdCbkxYMg7qkr7euc9rfJ1r6TuC4gXLLW9q+0pePkS5plhjUJW0GRkewrwduGEg/p42CXwM83brpbwJOTbKkDZA7Fbip5T2TZE0b9X7OwLokzYJFc10BSbMnyTXAG4Cjk+ygG8V+EXBdkg3AI8DbWvHPAmcC24BvA+8CqKpdSd4L3NHKvaeqRgff/TLdCPvDgD9rL0mzxKAuLSBV9fZxstaOUbaAc8dZz5XAlWOkbwVOnE4dJU2d3e+SJPWEQV2SpJ4wqEuS1BMGdUmSesKgLklSTxjUJUnqCYO6JEk9Ma2gnuQ3ktyT5O4k1yQ5NMnKJLe15yl/MsnBrewhbX5by18xsJ7zW/r9SU6b3i5JkrQwTTmoJ1kG/BqwuqpOBA4CzgbeD1xcVccDu4ENbZENwO6WfnErR5IT2nKvonv28keSHDTVekmStFBNt/t9EXBYkkXA4cBjwBuB61v+3s9mHn1m8/XA2nZ/6HXAtVX13ap6iO6WlK+dZr0kSVpwphzUq2on8AHgUbpg/jRwJ/BUVe1pxQafp/zsM5hb/tPASxj/2cw/wGcwS5I0vul0vy+hu8peCbwUeBFd9/kB4zOYJUka33S6398EPFRVI1X1PeBTwCnA4tYdD89/nvKzz2Bu+UcCTzL+s5klSdJ+mE5QfxRYk+Tw9t34WuBe4AvAW1qZvZ/NPPrM5rcAn29PgdoMnN1Gx68EVgG3T6NekiQtSFN+9GpV3ZbkeuBLwB7gy8DlwI3AtUne19KuaItcAXw8yTZgF92Id6rqniTX0Z0Q7AHOrarvT7VekiQtVNN6nnpVXQBcsFfyg4wxer2qvgO8dZz1XAhcOJ26SJK00E0rqEtSH60478YJyzx80ZtnoSbS/jGoz5DJfAiAHwSSpAPHe79LktQTBnVJknrC7ndJvTHZr8GkvvJKXZKknjCoS5LUEwZ1SZJ6wqAuSVJPGNQlSeoJg7okST1hUJckqScM6pIASPIbSe5JcneSa5IcmmRlktuSbEvyySQHt7KHtPltLX/FwHrOb+n3JzltrvZHWogM6pJIsgz4NWB1VZ0IHET3eOT3AxdX1fHAbmBDW2QDsLulX9zKkeSEttyrgNOBjyQ5aDb3RVrIDOqSRi0CDkuyCDgceAx4I3B9y98EnNWm17V5Wv7aJGnp11bVd6vqIWAbYzyKWdKBYVCXRFXtBD4APEoXzJ8G7gSeqqo9rdgOYFmbXgZsb8vuaeVfMpg+xjLPSrIxydYkW0dGRmZ+h6QFyqAuiSRL6K6yVwIvBV5E131+QFTV5VW1uqpWL1269EBtRlpwDOqSAN4EPFRVI1X1PeBTwCnA4tYdD7Ac2NmmdwLHAbT8I4EnB9PHWEbSAWZQlwRdt/uaJIe378bXAvcCXwDe0sqsB25o05vbPC3/81VVLf3sNjp+JbAKuH2W9kFa8Hz0qiSq6rYk1wNfAvYAXwYuB24Erk3yvpZ2RVvkCuDjSbYBu+hGvFNV9yS5ju6EYA9wblV9f1Z3RlrADOqSAKiqC4AL9kp+kDFGr1fVd4C3jrOeC4ELZ7yCkiZk97skST1hUJckqScM6pIk9YRBXZKknjCoS5LUEwZ1SZJ6wqAuSVJPGNQlSeoJg7okST3hHeVm2YrzbpywzMMXvXkWaiJJ6huv1CVJ6gmDuiRJPTGtoJ5kcZLrk3wtyX1JXpfkqCRbkjzQ/i5pZZPk0iTbktyV5KSB9axv5R9Isn78LUqSpPFM90r9EuBzVfVjwKuB+4DzgJurahVwc5sHOIPu2cqrgI3AZQBJjqJ7MtTJdE+DumD0RECSJE3elIN6kiOBn6I9X7mq/qGqngLWAZtasU3AWW16HXB1dW4FFic5FjgN2FJVu6pqN7AFOH2q9ZIkaaGazuj3lcAI8AdJXg3cCbwbOKaqHmtlHgeOadPLgO0Dy+9oaeOlz4rJjEaXJGk+mE73+yLgJOCyqnoN8C2e62oHoKoKqGls43mSbEyyNcnWkZGRmVqtJEm9MJ2gvgPYUVW3tfnr6YL811u3Ou3vEy1/J3DcwPLLW9p46T+gqi6vqtVVtXrp0qXTqLokSf0z5aBeVY8D25O8siWtBe4FNgOjI9jXAze06c3AOW0U/Brg6dZNfxNwapIlbYDcqS1NkiTth+neUe5XgU8kORh4EHgX3YnCdUk2AI8Ab2tlPwucCWwDvt3KUlW7krwXuKOVe09V7ZpmvSRJWnCmFdSr6ivA6jGy1o5RtoBzx1nPlcCV06mLJEkLnXeUkySpJwzqkiT1hEFdkqSeMKhLktQTBnVJknrCoC5JUk8Y1CVJ6gmDuiQAkixOcn2SryW5L8nrkhyVZEuSB9rfJa1sklyaZFuSu5KcNLCe9a38A0nWj79FSTPNoC5p1CXA56rqx4BXA/fRPaTp5qpaBdzMcw9tOgNY1V4bgcsAkhwFXACcDLwWuGD0REDSgWdQl0SSI4GfAq4AqKp/qKqngHXAplZsE3BWm14HXF2dW4HF7QFOpwFbqmpXVe0GtgCnz+KuSAuaQV0SwEpgBPiDJF9O8rEkLwKOaQ9eAngcOKZNLwO2Dyy/o6WNly5pFhjUJUH3HIiTgMuq6jXAt3iuqx149vkNNRMbS7IxydYkW0dGRmZilZIwqEvq7AB2VNVtbf56uiD/9datTvv7RMvfCRw3sPzyljZe+vNU1eVVtbqqVi9dunRGd0RayAzqkqiqx4HtSV7ZktYC9wKbgdER7OuBG9r0ZuCcNgp+DfB066a/CTg1yZI2QO7UliZpFkz3eeqS+uNXgU8kORh4EHgX3Yn/dUk2AI8Ab2tlPwucCWwDvt3KUlW7krwXuKOVe09V7Zq9XZAWNoO6JACq6ivA6jGy1o5RtoBzx1nPlcCVM1s7SZNh97skST1hUJckqSfsfpekKVhx3o0Tlnn4ojfPQk2k53ilLklSTxjUJUnqCYO6JEk9YVCXJKknDOqSJPWEQV2SpJ4wqEuS1BMGdUmSesKgLklSTxjUJUnqCYO6JEk9YVCXJKknDOqSJPWEQV2SpJ7w0atDyEc6SpKmYtpX6kkOSvLlJJ9p8yuT3JZkW5JPJjm4pR/S5re1/BUD6zi/pd+f5LTp1kmSpIVoJrrf3w3cNzD/fuDiqjoe2A1saOkbgN0t/eJWjiQnAGcDrwJOBz6S5KAZqJckSQvKtIJ6kuXAm4GPtfkAbwSub0U2AWe16XVtnpa/tpVfB1xbVd+tqoeAbcBrp1MvSZIWouleqX8Q+C3gH9v8S4CnqmpPm98BLGvTy4DtAC3/6Vb+2fQxlnmeJBuTbE2ydWRkZJpVlySpX6Yc1JP8NPBEVd05g/XZp6q6vKpWV9XqpUuXztZmJUmaF6Yz+v0U4GeTnAkcChwBXAIsTrKoXY0vB3a28juB44AdSRYBRwJPDqSPGlxGkiRN0pSv1Kvq/KpaXlUr6Aa6fb6q3gF8AXhLK7YeuKFNb27ztPzPV1W19LPb6PiVwCrg9qnWS5KkhepA/E79t4Frk7wP+DJwRUu/Avh4km3ALroTAarqniTXAfcCe4Bzq+r7B6BekiT12owE9aq6BbilTT/IGKPXq+o7wFvHWf5C4MKZqIskSQuVt4mVJKknDOqSJPWEQV3Ss7ztszS/GdQlDfK2z9I8ZlCXBHjbZ6kPDOqSRs3abZ+95bN0YBjUJc36bZ+95bN0YByIm89Imn+87bPUA16pS/K2z1JPeKUuaV+87bM0jxjUJT2Pt32W5i+73yVJ6gmDuiRJPWFQlySpJwzqkiT1hEFdkqSeMKhLktQTBnVJknrCoC5JUk8Y1CVJ6gmDuiRJPWFQlySpJwzqkiT1hEFdkqSeMKhLktQTPnp1nlpx3o0Tlnn4ojfPQk0kScPCoC5JB8hkTr7BE3DNHLvfJUnqCYO6JEk9YVCXJKknDOqSJPWEQV2SpJ4wqEuS1BMGdUmSemLKQT3JcUm+kOTeJPckeXdLPyrJliQPtL9LWnqSXJpkW5K7kpw0sK71rfwDSdZPf7ckSVp4pnOlvgf4zao6AVgDnJvkBOA84OaqWgXc3OYBzgBWtddG4DLoTgKAC4CTgdcCF4yeCEiSpMmbclCvqseq6ktt+pvAfcAyYB2wqRXbBJzVptcBV1fnVmBxkmOB04AtVbWrqnYDW4DTp1ovSZIWqhn5Tj3JCuA1wG3AMVX1WMt6HDimTS8Dtg8stqOljZc+1nY2JtmaZOvIyMhMVF2SpN6YdlBP8mLgT4Bfr6pnBvOqqoCa7jYG1nd5Va2uqtVLly6dqdVKktQL0wrqSV5IF9A/UVWfaslfb93qtL9PtPSdwHEDiy9vaeOlS5olDnyV+mE6o98DXAHcV1W/N5C1GRhtyOuBGwbSz2kfBmuAp1s3/U3AqUmWtA+MU1uapNnjwFepB6bz6NVTgH8LfDXJV1rafwQuAq5LsgF4BHhby/sscCawDfg28C6AqtqV5L3AHa3ce6pq1zTqJWk/tRPsx9r0N5MMDnx9Qyu2CbgF+G0GBr4CtyYZHfj6BtrAV4AkowNfr5m1nZEWsCkH9ar6SyDjZK8do3wB546zriuBK6daF0kzZzYGvibZSHeFz8te9rKZq7y0wHlHOUnPmq2Brw56lQ4Mg7okwIGvUh8Y1CU58FXqiekMlJPUHw58lXrAoN5jK867cVLlHr7ozQe4Jhp2DnyV+sHud0mSesKgLklSTxjUJUnqCYO6JEk9YVCXJKknHP0uSXNsMr9U8Vcqmgyv1CVJ6gmDuiRJPWFQlySpJwzqkiT1hEFdkqSeMKhLktQTBnVJknrCoC5JUk948xl54wtJ6gmv1CVJ6gmDuiRJPWFQlySpJwzqkiT1hAPlJGkecECrJsMrdUmSesKgLklST9j9rkmx60+Shp9BXdLQm8xJpSS73yVJ6g2DuiRJPWFQlySpJ3r9nbrfw80uB9NJc8s2KK/UJUnqiaG5Uk9yOnAJcBDwsaq6aI6rpAPAK4n+sy1Lc2cognqSg4APA/8S2AHckWRzVd07tzWTtD9sy8Nvsl9LenI9Pw1FUAdeC2yrqgcBklwLrAP8IFiAZnIshB9Ms8623BMz1Q5tg7NrWIL6MmD7wPwO4OS9CyXZCGxss3+X5P4x1nU08I0Zr+HssO4zLO+fVLGhrPskTKbeL5+NigyYybYMw/u/sV6TlPcPX52a+VivCdvzsAT1Samqy4HL91UmydaqWj1LVZpR1n1uzNe6z9d6w+TaMgzvPlqvyRvGOkF/6zUso993AscNzC9vaZLmF9uyNIeGJajfAaxKsjLJwcDZwOY5rpOk/WdblubQUHS/V9WeJL8C3ET3M5grq+qeKa5uwi69IWbd58Z8rfvQ1XuG2zIM4T421mvyhrFO0NN6papmqiKSJGkODUv3uyRJmiaDuiRJPdGroJ7k9CT3J9mW5Ly5rs++JDkuyReS3JvkniTvbulHJdmS5IH2d8lc13UsSQ5K8uUkn2nzK5Pc1o79J9sgqaGTZHGS65N8Lcl9SV43j475b7T3yt1Jrkly6Hw57lMxDO152NvpMLbDYWxjw9J2klyZ5Ikkdw+kjXls0rm01e+uJCdNZhu9Cep57vaUZwAnAG9PcsLc1mqf9gC/WVUnAGuAc1t9zwNurqpVwM1tfhi9G7hvYP79wMVVdTywG9gwJ7Wa2CXA56rqx4BX0+3D0B/zJMuAXwNWV9WJdIPQzmb+HPf9MkTtedjb6TC2w6FqY0PWdq4CTt8rbbxjcwawqr02ApdNagtV1YsX8DrgpoH584Hz57pe+1H/G+jul30/cGxLOxa4f67rNkZdl7c33xuBzwChuwPSorH+F8PyAo4EHqINEB1Inw/HfPRObUfR/WrlM8Bp8+G4T3F/h7I9D1M7HcZ2OIxtbNjaDrACuHuiYwN8FHj7WOX29erNlTpj355y2RzVZb8kWQG8BrgNOKaqHmtZjwPHzFG19uWDwG8B/9jmXwI8VVV72vywHvuVwAjwB63L8mNJXsQ8OOZVtRP4APAo8BjwNHAn8+O4T8XQtechbKfD2A6Hro3Ng7Yz3rGZUhvoU1Cfl5K8GPgT4Ner6pnBvOpOz4bqN4dJfhp4oqrunOu6TMEi4CTgsqp6DfAt9uoGHMZjDtC+Z1tH96H5UuBF/GA3ng6QYWunQ9wOh66Nzae2MxPHpk9Bfd7dnjLJC+k+KD5RVZ9qyV9PcmzLPxZ4Yq7qN45TgJ9N8jBwLV3X3yXA4iSjNzMa1mO/A9hRVbe1+evpPoCG/ZgDvAl4qKpGqup7wKfo/hfz4bhPxdC05yFtp8PaDoexjQ172xnv2EypDfQpqM+r21MmCXAFcF9V/d5A1mZgfZteT/cd3tCoqvOranlVraA7xp+vqncAXwDe0ooNXb0BqupxYHuSV7aktXSPBB3qY948CqxJcnh774zWfeiP+xQNRXse1nY6rO1wSNvYsLed8Y7NZuCcNgp+DfD0QDf9+GZrsMIsDUA4E/hr4G+A35nr+kxQ19fTdbPcBXylvc6k+17sZuAB4C+Ao+a6rvvYhzcAn2nTPwrcDmwD/hg4ZK7rN06dfwLY2o77nwJL5ssxB/4L8DXgbuDjwCHz5bhPcX/nvD3Ph3Y6bO1wGNvYsLQd4Bq67/W/R9ersWG8Y0M38PHD7f3/VbrR+xNuw9vESpLUE33qfpckaUEzqEuS1BMGdUmSesKgLklSTxjUJUnqCYO6JEk9YVCXJKkn/j90WDhb6Ns32gAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "src_length = map(len, [vars(x)['src'] for x in train_data.examples])\n", + "trg_length = map(len, [vars(x)['trg'] for x in train_data.examples])\n", + "\n", + "print('Length distribution in Train data')\n", + "plt.figure(figsize=[8, 4])\n", + "plt.subplot(1, 2, 1)\n", + "plt.title(\"source length\")\n", + "plt.hist(list(src_length), bins=20);\n", + "\n", + "plt.subplot(1, 2, 2)\n", + "plt.title(\"translation length\")\n", + "plt.hist(list(trg_length), bins=20);" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "lP_SH6Ym5X3t", + "outputId": "296a4302-a92b-4fd2-aa03-ae28ec856227" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Length distribution in Test data\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe0AAAEICAYAAAByPazKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAfXElEQVR4nO3df7RdZX3n8fdHIij+4GdETMCkktqiq1YmBVzYjiVWAa241qDFsWPUdNJatLbaarBdpctKJ06dIi4tNRUKTK1AqdaMUjFFrdNpQYNaFFBJEUjSYCK/bKVq0e/8sZ8Lh8tN7k3uz33O+7XWWWfv53n23s++5z7nu/ezn7N3qgpJkrTwPWq+KyBJkqbGoC1JUk8YtCVJ6gmDtiRJPWHQliSpJwzakiT1hEFb05JkWZJKsmgetv3qJH8/19uV5kOSi5O8YxrL/1uSH5nJOrX13pbk+TO93ilsd96+e+aTQVu9MKoNVAvLfAWovZXkM0l+aTCtqh5fVbfOV52mqy9/+9lm0B5RSfab7zpIw8aDSs02g/YClOStSbYn+dckX0uyqqUfkOTdSf6lvd6d5ICW94iu4nZmekybvjjJBUmuSvId4GeTHJXkw0l2JbkryXsHln1tkpuT3JPk6iRPnWLdD0pyYZIdbR/eMXaAMFbHJO9q6/1GklMHll2e5LNtv/82yfuS/HnL/mx7v7d18z1nYLkJ1yfNpCT/Gzga+D/tf/AtAz1Aa5LcAXyqlf3LJHcmua/9Tz9jYD0Xt//tj7f/9euSPK3lJcl5SXYm+XaSLyd55gR1OSTJx1rbvadNL2155wI/Dby31fO9LX3w++CgJJe25W9P8jtJHtXy9thOJ/kbPSrJuiT/3L5TrkhyaMsb+1utTnJHkm8l+e2BZR+b5JK2zZvb33fb7v72A5t95UTrG1pV5WsBvYCnA1uBp7T5ZcDT2vTbgWuBJwGLgX8Afr/lvRr4+3HrKuCYNn0xcB9wEt3B2uOAfwLOa9OPAZ7byp4ObAF+HFgE/A7wD7up77K2nUVt/iPA+9s6nwR8DvjlgTr+B/Dfgf2A1wH/AqTl/yPwLmB/4LnAt4E/n2g7U1mfL18z/QJuA54/MD/2f3lp+59/bEt/LfAE4ADg3cCXBpa5GLgLOL61rw8Cl7W8FwLXAwcDaW3wyIHl3tGmDwP+C3Bg285fAn89sI3PAL80ru6D3weXAh9tyy4Dvg6saXl71a4G/ybAG+m+o5a2fX8/8KFxf6s/BR4LPAv4HvDjLX898HfAIW35G4BtU/jbT7i+YX3NewV8jftA4BhgJ/B84NHj8v4ZOG1g/oXAbW361UwetC8dyHsOsIuBIDiQ9zdjDbjNPwq4H3jqBGXHGs4i4IjWaB47kP8K4NMDddwykHdgW/bJdEfRDwAHDuT/OZMH7QnXN9+fo6/hfO0hcPzIHpY5uJU5qM1fDHxgIP804Ktt+mS6AHoi8Khx67mYFrQn2MZPAvcMzH+G3QRtukD8feDYgbxfBj7TpveqXfHwoH0zsGog70i6A4BFA3+rpQP5nwPObNO3Ai8cyPslpha0J1zfsL7sHl9gqmoL8OvA7wE7k1yW5Ckt+ynA7QPFb29pU7V1YPoo4PaqemCCck8Fzk9yb5J7gbvpjvqXTLL+pwKPBnYMLPt+ujPuMXeOTVTV/W3y8W0/7h5IG1/f3dnd+qS59OD/apL9kqxvXcTfpgs2AIcPlL9zYPp+2v9sVX0KeC/wPrr2vyHJE8dvLMmBSd7fura/TXf56OBMbazK4XTtdPx3yWD73td29VTgIwPt/2bgB3QH9I9YNwP7TvcdMNjmp9L+97S+oWTQXoCq6i+q6rl0DaCAd7asf2lpY45uaQDfoTsiBiDJkyda9cD0VuDoTDxwZitdl/bBA6/HVtU/TFL1rXRn2ocPLPfEqnrGJMsB7AAOTXLgQNpRu6m7NF929384mP5f6S4xPR84iO6MELoD38k3UPWeqvpPwLHAjwK/NUGxN9NdSjuhqp4I/My4beypvXyL7ux3/HfJ9qnUbxJbgVPHfXc8pqqmsu4ddN3iY44al+93AAbtBSfJ05OcnG6A2XeBfwd+2LI/BPxOksVJDgd+l64LGbrr089I8pNJHkN3pr4nn6NrJOuTPC7JY5Kc1PL+BDh7bPBMG7TyssnqXlU7gE8C/yvJE9uglKcl+c9TWPZ2YDPwe0n2bwPNfn6gyC66v8OM/85U2gvfZPL/wSfQHbzeRXcg/QdTXXmSn0pyQpJH0x2If5eH2v/4bfw73cDMQ4FzplrPqvoBcAVwbpInpBtk+iYe+i6Zjj9p631q25/FSU6f4rJX0H3vHJJkCfD6cflT+dsPPYP2wnMA3YCMb9F1+zwJOLvlvYMusN0AfBn4Qkujqr5ON1Dtb4FbgD3edKQ13J+nu8Z1B7AN+IWW9xG6s/vLWtfbV4Cpjsp+Fd1AspuAe4Ar6a5rTcUr6a6139X263K6L7+xLrpzgf/Xut5OnOI6pZn0P+gOnO9N8pu7KXMpXXfzdrp2cO1erP+JdAOr7mnruAv4wwnKvZtu8NW32vo/MS7/fOCMNhL7PRMs/wa6g4Jb6b4r/gK4aC/quTvnAxuBTyb511a3E6a47Nvpvoe+Qfc9diWt/TdT+dsPvbFRu9KCk+RyugE6488iJA25JK+jG1Q2aU/dKPFMWwtG6xp8WutWP4XuuuBfz3e9JM2+JEcmOam1/6fTXbf/yHzXa6Hx7j1aSJ4MfJjuN6jbgNdV1Rfnt0qS5sj+dL82WQ7cC1wG/PG81mgBsntckqSesHtckqSeWNDd44cffngtW7ZsvqshLXjXX3/9t6pq8XzXY09sz9LU7Kk9L+igvWzZMjZv3jzf1ZAWvCS3T15qftmepanZU3u2e1ySpJ4waEuS1BMGbUmSesKgLUlSTxi0JUnqCYO2JEk9YdCWJKknDNqSJPWEQVuSpJ5Y0HdEm0nL1n18RtZz2/oXzch6JM2uqbR527P6xjNtSZJ6wqAtSVJPGLQlSeoJg7YkST1h0JYkqScM2pIk9YRBW5KknjBoSyMkyUVJdib5yrj0NyT5apIbk/zPgfSzk2xJ8rUkLxxIP6WlbUmybi73QRplI3NzFUkAXAy8F7h0LCHJzwKnA8+qqu8leVJLPxY4E3gG8BTgb5P8aFvsfcDPAduAzyfZWFU3zdleSCPKoC2NkKr6bJJl45JfB6yvqu+1Mjtb+unAZS39G0m2AMe3vC1VdStAkstaWYO2NMvsHpf0o8BPJ7kuyd8l+amWvgTYOlBuW0vbXfojJFmbZHOSzbt27ZqFqkujxaAtaRFwKHAi8FvAFUkyEyuuqg1VtbKqVi5evHgmVimNtEmD9kQDV5L8YRu0ckOSjyQ5eCDPgStSv2wDPlydzwE/BA4HtgNHDZRb2tJ2ly5plk3lTPti4JRxaZuAZ1bVTwBfB86GRwxcOQX44yT7JdmPbuDKqcCxwCtaWUnz76+BnwVoA832B74FbATOTHJAkuXACuBzwOeBFUmWJ9mfrs1vnJeaSyNm0oFoEw1cqapPDsxeC5zRph24Ii1gST4EPA84PMk24BzgIuCi1pv2fWB1VRVwY5Ir6NrpA8BZVfWDtp7XA1cD+wEXVdWNc74z0giaidHjrwUub9NL6IL4mMEBKuMHrpww0cqSrAXWAhx99NEzUD1JY6rqFbvJ+sXdlD8XOHeC9KuAq2awapKmYFoD0ZL8Nt0R+AdnpjoOXJEkaXf2+Uw7yauBFwOrWlca7HmAigNXJEmahn06005yCvAW4CVVdf9AlgNXJEmaJZOeae9m4MrZwAHApvZzzmur6leqyoErkiTNkqmMHp9o4MqFeyg/1ANXlq37+JTK3bb+RbNcE0nSqPGOaJIk9YRBW5KknjBoS5LUEwZtSZJ6wqAtSVJPGLQlSeoJg7YkST1h0JYkqScM2pIk9YRBW5KknjBoSyMkyUVJdib5ygR5b05SSQ5v80nyniRbktyQ5LiBsquT3NJeq+dyH6RRZtCWRsvFwCnjE5McBbwAuGMg+VS6J/WtANYCF7Syh9I9OOgE4HjgnCSHzGqtJQEGbWmkVNVngbsnyDqP7nG7NZB2OnBpda4FDk5yJPBCYFNV3V1V9wCbmOBAQNLMM2hLIy7J6cD2qvqncVlLgK0D89ta2u7SJ1r32iSbk2zetWvXDNZaGk0GbWmEJTkQeBvwu7Ox/qraUFUrq2rl4sWLZ2MT0kgxaEuj7WnAcuCfktwGLAW+kOTJwHbgqIGyS1va7tIlzTKDtjTCqurLVfWkqlpWVcvourqPq6o7gY3Aq9oo8hOB+6pqB3A18IIkh7QBaC9oaZJmmUFbGiFJPgT8I/D0JNuSrNlD8auAW4EtwJ8CvwpQVXcDvw98vr3e3tIkzbJF810BSXOnql4xSf6ygekCztpNuYuAi2a0cpIm5Zm2JEk9YdCWJKknDNqSJPWEQVuSpJ6YNGhP9ICBJIcm2dQeFrBp7L7DPmBAkqTZM5Uz7Yt55H2F1wHXVNUK4Jo2Dz5gQJKkWTNp0N7NAwZOBy5p05cALx1I9wEDkiTNgn29pn1EuzMSwJ3AEW3aBwxIkjRLpj0Qrd2AoSYtOPX1+YABSZImsK9B+5ut25v2vrOl+4ABSZJmyb7exnQjsBpY394/OpD++iSX0Q06u6+qdiS5GviDgcFnLwDO3vdqD4dl6z4+aZnb1r9oDmoiSeqDSYN2e8DA84DDk2yjGwW+HriiPWzgduDlrfhVwGl0Dxi4H3gNdA8YSDL2gAHwAQOS9tFUDnalYTVp0N7DAwZWTVDWBwxIkjRLvCOaJEk9YdCWJKknDNqSJPXEvo4e1yQcLKOFKMlFwIuBnVX1zJb2h8DPA98H/hl4TVXd2/LOBtYAPwB+raqubumnAOcD+wEfqKr1c70v0ijyTFsaLRfzyFsIbwKeWVU/AXyd9nPMJMcCZwLPaMv8cZL9kuwHvI/uWQPHAq9oZSXNMoO2NEImepZAVX2yqh5os9fS3fwIumcJXFZV36uqb9D9lPP49tpSVbdW1feBy1pZSbPMoC1p0GuBv2nTPktAWmC8pi0JgCS/DTwAfHCm1llVG4ANACtXrpyxZxTMJe9cqIXEoC2JJK+mG6C2qt0kCfb8zACfJSDNA7vHpRHXRoK/BXhJVd0/kLURODPJAUmWAyuAz9HdjnhFkuVJ9qcbrLZxrustjSLPtKURsptnCZwNHABsSgJwbVX9SlXdmOQK4Ca6bvOzquoHbT2vB66m+8nXRVV145zvjDSCDNrSCNnNswQu3EP5c4FzJ0i/iu4BQZLmkN3jkiT1hEFbkqSeMGhLktQTBm1JknrCoC1JUk84elzSyPJpfOobz7QlSeoJg7YkST1h0JYkqScM2pIk9YRBW5KknphW0E7yG0luTPKVJB9K8pj25J/rkmxJcnl7ChDtSUGXt/TrkiybiR2QJGlU7HPQTrIE+DVgZVU9k+5pP2cC7wTOq6pjgHuANW2RNcA9Lf28Vk6SJE3RdLvHFwGPTbIIOBDYAZwMXNnyLwFe2qZPb/O0/FVpzwGUJEmT2+egXVXbgXcBd9AF6/uA64F7q+qBVmwbsKRNLwG2tmUfaOUPG7/eJGuTbE6yedeuXftaPUmShs50uscPoTt7Xg48BXgccMp0K1RVG6pqZVWtXLx48XRXJ2lAkouS7EzylYG0Q5NsSnJLez+kpSfJe9o4lBuSHDewzOpW/pYkq+djX6RRNJ3u8ecD36iqXVX1H8CHgZOAg1t3OcBSYHub3g4cBdDyDwLumsb2Je29i3nkwfU64JqqWgFc0+YBTgVWtNda4ALogjxwDnACcDxwzliglzS7phO07wBOTHJguza9CrgJ+DRwRiuzGvhom97Y5mn5n6qqmsb2Je2lqvoscPe45MHxJuPHoVxanWvpDsiPBF4IbKqqu6vqHmATM9DLJmly07mmfR3dgLIvAF9u69oAvBV4U5ItdNesL2yLXAgc1tLfxENH85Lm1xFVtaNN3wkc0aYfHIfSjI1R2V36IzhGRZpZ03rKV1WdQ9dNNuhWui6z8WW/C7xsOtuTNLuqqpLMWA9YVW2gO5hn5cqV9qxJ0+Qd0SR9s3V70953tvQHx6E0Y2NUdpcuaZYZtCUNjjcZPw7lVW0U+YnAfa0b/WrgBUkOaQPQXtDSJM2yaXWPS+qXJB8CngccnmQb3eWt9cAVSdYAtwMvb8WvAk4DtgD3A68BqKq7k/w+8PlW7u1VNX5wm6RZYNCWRkhVvWI3WasmKFvAWbtZz0XARTNYNUlTYPe4JEk9YdCWJKknDNqSJPWEQVuSpJ4waEuS1BMGbUmSesKgLUlSTxi0JUnqCYO2JEk9YdCWJKknDNqSJPWEQVuSpJ4waEuS1BMGbUmSesKgLUlSTxi0JUnqCYO2JACS/EaSG5N8JcmHkjwmyfIk1yXZkuTyJPu3sge0+S0tf9n81l4aDQZtSSRZAvwasLKqngnsB5wJvBM4r6qOAe4B1rRF1gD3tPTzWjlJs2xaQTvJwUmuTPLVJDcneU6SQ5NsSnJLez+klU2S97Qj8xuSHDczuyBphiwCHptkEXAgsAM4Gbiy5V8CvLRNn97mafmrkmQO6yqNpOmeaZ8PfKKqfgx4FnAzsA64pqpWANe0eYBTgRXttRa4YJrbljRDqmo78C7gDrpgfR9wPXBvVT3Qim0DlrTpJcDWtuwDrfxh49ebZG2SzUk279q1a3Z3QhoB+xy0kxwE/AxwIUBVfb+q7uXhR+Djj8wvrc61wMFJjtznmkuaMa1H7HRgOfAU4HHAKdNdb1VtqKqVVbVy8eLF012dNPKmc6a9HNgF/FmSLyb5QJLHAUdU1Y5W5k7giDb94JF5M3jULml+PR/4RlXtqqr/AD4MnER3cL2olVkKbG/T24GjAFr+QcBdc1tlafRMJ2gvAo4DLqiqZwPf4aGucACqqoDam5XanSbNizuAE5Mc2K5NrwJuAj4NnNHKrAY+2qY3tnla/qdae5c0i6YTtLcB26rqujZ/JV0Q/+ZYt3d739nyHzwybwaP2h9kd5o091o7vhL4AvBluu+GDcBbgTcl2UJ3zfrCtsiFwGEt/U2MO2CXNDsWTV5kYlV1Z5KtSZ5eVV/joSPzm+iOwNfzyCPz1ye5DDgBuG+gG13SPKuqc4BzxiXfChw/QdnvAi+bi3pJesg+B+3mDcAH2w0XbgVeQ3eEfkWSNcDtwMtb2auA04AtwP2trCRJmqJpBe2q+hKwcoKsVROULeCs6WxPkqRR5h3RJEnqCYO2JEk9YdCWJKknDNqSJPWEQVuSpJ4waEuS1BMGbUmSesKgLUlSTxi0JUnqCYO2JEk9YdCWJKknDNqSJPWEQVuSpJ4waEsCIMnBSa5M8tUkNyd5TpJDk2xKckt7P6SVTZL3JNmS5IYkx813/aVRMN3naWuWLVv38UnL3Lb+RXNQE42A84FPVNUZSfYHDgTeBlxTVeuTrAPWAW8FTgVWtNcJwAXtXdIs8kxbEkkOAn4GuBCgqr5fVfcCpwOXtGKXAC9t06cDl1bnWuDgJEfOcbWlkeOZtiSA5cAu4M+SPAu4HngjcERV7Whl7gSOaNNLgK0Dy29raTsG0kiyFlgLcPTRR89a5efbVHrEwF4xTZ9n2pKgO4A/Drigqp4NfIeuK/xBVVVA7c1Kq2pDVa2sqpWLFy+escpKo8qgLQm6M+VtVXVdm7+SLoh/c6zbu73vbPnbgaMGll/a0iTNIoO2JKrqTmBrkqe3pFXATcBGYHVLWw18tE1vBF7VRpGfCNw30I0uaZZ4TVvSmDcAH2wjx28FXkN3YH9FkjXA7cDLW9mrgNOALcD9raykWWbQlgRAVX0JWDlB1qoJyhZw1qxXStLD2D0uSVJPTDtoJ9kvyReTfKzNL09yXbtT0uWtq40kB7T5LS1/2XS3LUnSKJmJ7vE3AjcDT2zz7wTOq6rLkvwJsIbubklrgHuq6pgkZ7ZyvzAD25ekXvAOh5quaZ1pJ1kKvAj4QJsPcDLdz0XgkXdQGruz0pXAqlZekiRNwXS7x98NvAX4YZs/DLi3qh5o82N3SYKBOyi1/Pta+YdJsjbJ5iSbd+3aNc3qSZI0PPY5aCd5MbCzqq6fwfp4ByVJknZjOte0TwJekuQ04DF017TPp3twwKJ2Nj14l6SxOyhtS7IIOAi4axrblyRppOzzmXZVnV1VS6tqGXAm8KmqeiXwaeCMVmz8HZTG7qx0Riu/V/cxliRplM3G77TfCrwpyRa6a9YXtvQLgcNa+psY9zACSZK0ZzNyR7Sq+gzwmTZ9K3D8BGW+C7xsJrY33lQfizes/BmJhsWot2VpMt4RTZKknjBoS5LUEwZtSZJ6wqAtSVJPGLQlSeoJg7YkST1h0Jb0IB+1Ky1sBm1Jg8YetTtm7FG7xwD30D1iFwYetQuc18pJmmUGbUmAj9qV+sCgLWmMj9qVFjiDtiQftSv1xIzce1xS7/moXakHPNOW5KN2pZ4waEvaEx+1Ky0gdo9Lepj5ftSupN3zTFuSpJ4waEuS1BMGbUmSesKgLUlSTxi0JUnqCYO2JEk9YdCWJKknDNqSJPXEPgftJEcl+XSSm5LcmOSNLf3QJJuS3NLeD2npSfKeJFuS3JDkuJnaCUmSRsF0zrQfAN5cVccCJwJnJTmW7naG11TVCuAaHrq94anAivZaC1wwjW1LkjRy9jloV9WOqvpCm/5X4Ga6Z+yeDlzSil0CvLRNnw5cWp1r6Z4edOQ+11ySpBEzI9e0kywDng1cBxxRVTta1p3AEW16CbB1YLFtLW38utYm2Zxk865du2aiepIkDYVpPzAkyeOBvwJ+vaq+neTBvKqqJHv1uL6q2gBsAFi5cqWP+pshy9Z9fErlblv/olmuiSRpX03rTDvJo+kC9ger6sMt+Ztj3d7tfWdL3w4cNbD40pYmSZKmYDqjx0P3TN2bq+qPBrI2Aqvb9GrgowPpr2qjyE8E7hvoRpckSZOYzpn2ScB/A05O8qX2Og1YD/xckluA57d5gKuAW4EtwJ8CvzqNbUuaQf6EU+qHfb6mXVV/D2Q32asmKF/AWfu6PUmzauwnnF9I8gTg+iSbgFfT/YRzfZJ1dD/hfCsP/wnnCXQ/4TxhXmo+ZKYy/sSxJ6PLO6JJ8iecUk8YtCU9jD/hlBYug7akB43/CedgXrvEtdc/4ayqlVW1cvHixTNYU2k0Tft32pKGw55+wllVO/wJ58Lhde/R5Zm2JH/CKfWEZ9raax7lD6Wxn3B+OcmXWtrb6H6yeUWSNcDtwMtb3lXAaXQ/4bwfeM3cVlcaTQZtPcxUb3eq4eJPOKV+sHtckqSeMGhLktQTBm1JknrCoC1JUk8YtCVJ6gmDtiRJPWHQliSpJwzakiT1hEFbkqSe8I5okjSEvN3wcDJoS9KImuptiw3uC4dBW7PCo3xJmnle05YkqScM2pIk9YTd45o3Xk+TpL1j0NaC5/VxaX7ZBheOOe8eT3JKkq8l2ZJk3VxvX9LMsC1Lc29Oz7ST7Ae8D/g5YBvw+SQbq+qmuayHho9nAnPLtqzxpnq5azK20z2b6+7x44EtVXUrQJLLgNMBG7pm3UIM7AuxTlNkW9ascKzLns110F4CbB2Y3wacMFggyVpgbZv9tyR3Ad+am+rNq8MZ/v1c8PuYd87IamZ0P6dYp6fO1PamaNK2DBO256/NQd3myoL/f94HvdmnvWyrvdmvZrftecENRKuqDcCGsfkkm6tq5TxWaU6Mwn6Owj7C6OznVIxvz8NkGD/nYdwnGK79muuBaNuBowbml7Y0Sf1iW5bmwVwH7c8DK5IsT7I/cCawcY7rIGn6bMvSPJjT7vGqeiDJ64Grgf2Ai6rqxkkWG8qutQmMwn6Owj7CCOznPrblYTOMn/Mw7hMM0X6lqua7DpIkaQq897gkST1h0JYkqScWdNAextskJjkqyaeT3JTkxiRvbOmHJtmU5Jb2fsh813UmJNkvyReTfKzNL09yXftML2+DmHorycFJrkzy1SQ3J3nOsH6Wo2rY2+ywtdFhb5MLNmgP3CbxVOBY4BVJjp3fWs2IB4A3V9WxwInAWW2/1gHXVNUK4Jo2PwzeCNw8MP9O4LyqOga4B1gzL7WaOecDn6iqHwOeRbevw/pZjqphb7PD1kaHu01W1YJ8Ac8Brh6YPxs4e77rNQv7+VG6+zd/DTiypR0JfG2+6zYD+7aUroGcDHwMCN1diRZN9Bn37QUcBHyDNqBzIH3oPktfD/t8h6bNDlsbHYU2uWDPtJn4NolL5qkusyLJMuDZwHXAEVW1o2XdCRwxT9WaSe8G3gL8sM0fBtxbVQ+0+b5/psuBXcCfte7FDyR5HMP5WYqhbLPD1kaHvk0u5KA91JI8Hvgr4Ner6tuDedUdDvb6t3hJXgzsrKrr57sus2gRcBxwQVU9G/gO47rdhuGzVGfY2uyQttGhb5MLOWgP7W0SkzyarvF/sKo+3JK/meTIln8ksHO+6jdDTgJekuQ24DK67rfzgYOTjN3Up++f6TZgW1Vd1+avpPvCGLbPcuQNaZsdxjY69G1yIQftobxNYpIAFwI3V9UfDWRtBFa36dV01816q6rOrqqlVbWM7rP7VFW9Evg0cEYr1uv9rKo7ga1Jnt6SVtE9mnKoPstRN6xtdhjb6Ci0yQV9R7Qkp9Fdcxm7TeK581ylaUvyXOD/Al/moetIb6O7RnYFcDRwO/Dyqrp7Xio5w5I8D/jNqnpxkh+hO6o/FPgi8ItV9b35rN90JPlJ4APA/sCtwGvoDoaH8rMcRaPQZoepjQ57m1zQQVuSJD1kIXePS5KkAQZtSZJ6wqAtSVJPGLQlSeoJg7YkST1h0JYkqScM2pIk9cT/B/odrf8G1COhAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "src_length = map(len, [vars(x)['src'] for x in test_data.examples])\n", + "trg_length = map(len, [vars(x)['trg'] for x in test_data.examples])\n", + "\n", + "print('Length distribution in Test data')\n", + "plt.figure(figsize=[8, 4])\n", + "plt.subplot(1, 2, 1)\n", + "plt.title(\"source length\")\n", + "plt.hist(list(src_length), bins=20);\n", + "\n", + "plt.subplot(1, 2, 2)\n", + "plt.title(\"translation length\")\n", + "plt.hist(list(trg_length), bins=20);" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sQ2_5JVX5X3t" + }, + "source": [ + "### Model side\n", + "__Here comes simple pipeline of NMT model learning. It almost copies the week02 practice__" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "UehB1dvv5X3t" + }, + "outputs": [], + "source": [ + "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6a_ftiAc5X3t", + "outputId": "807578fd-007f-4237-caf4-7f5a0feeb802" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "device(type='cuda', index=1)" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "device" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_Ypx-EQB5X3t" + }, + "outputs": [], + "source": [ + "def _len_sort_key(x):\n", + " return len(x.src)\n", + "\n", + "BATCH_SIZE = 128\n", + "\n", + "train_iterator, valid_iterator, test_iterator = BucketIterator.splits(\n", + " (train_data, valid_data, test_data),\n", + " batch_size = BATCH_SIZE,\n", + " device = device,\n", + " sort_key=_len_sort_key\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4qIa4dX35X3u", + "outputId": "f376ebd6-efcb-4c70-c102-80720815be70" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "[torchtext.data.batch.Batch of size 128]\n", + "\t[.trg]:[torch.cuda.LongTensor of size 55x128 (GPU 1)]\n", + "\t[.src]:[torch.cuda.LongTensor of size 59x128 (GPU 1)]\n", + "torch.Size([59, 128]) torch.Size([55, 128])\n" + ] + } + ], + "source": [ + "for x in train_iterator:\n", + " break\n", + "print(x)\n", + "print(x.src.shape, x.trg.shape)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "arXnRMtD5X3u" + }, + "outputs": [], + "source": [ + "import my_network\n", + "Encoder = my_network.Encoder\n", + "Decoder = my_network.Decoder\n", + "Seq2Seq = my_network.Seq2Seq" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "pLmSYcCY5X3u" + }, + "outputs": [], + "source": [ + "INPUT_DIM = len(SRC.vocab)\n", + "OUTPUT_DIM = len(TRG.vocab)\n", + "ENC_EMB_DIM = 256\n", + "DEC_EMB_DIM = 256\n", + "HID_DIM = 512\n", + "N_LAYERS = 2\n", + "ENC_DROPOUT = 0.5\n", + "DEC_DROPOUT = 0.5\n", + "\n", + "enc = Encoder(INPUT_DIM, ENC_EMB_DIM, HID_DIM, N_LAYERS, ENC_DROPOUT)\n", + "dec = Decoder(OUTPUT_DIM, DEC_EMB_DIM, HID_DIM, N_LAYERS, DEC_DROPOUT)\n", + "\n", + "# dont forget to put the model to the right device\n", + "model = Seq2Seq(enc, dec, device).to(device)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "mx28_DhC5X3u", + "outputId": "71dbccbb-260b-487d-9c21-aa0ecc8aa0fc" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "Seq2Seq(\n", + " (encoder): Encoder(\n", + " (embedding): Embedding(9267, 256)\n", + " (rnn): LSTM(256, 512, num_layers=2, dropout=0.5)\n", + " (dropout): Dropout(p=0.5, inplace=False)\n", + " )\n", + " (decoder): Decoder(\n", + " (embedding): Embedding(6699, 256)\n", + " (rnn): LSTM(256, 512, num_layers=2, dropout=0.5)\n", + " (out): Linear(in_features=512, out_features=6699, bias=True)\n", + " (dropout): Dropout(p=0.5, inplace=False)\n", + " )\n", + ")" + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def init_weights(m):\n", + " # \n", + " for name, param in m.named_parameters():\n", + " nn.init.uniform_(param, -0.08, 0.08)\n", + "\n", + "model.apply(init_weights)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "TOFx0cFT5X3u", + "outputId": "6e87f8b5-1532-423d-8cf2-7a2de6597e87" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The model has 14,880,299 trainable parameters\n" + ] + } + ], + "source": [ + "def count_parameters(model):\n", + " return sum(p.numel() for p in model.parameters() if p.requires_grad)\n", + "\n", + "print(f'The model has {count_parameters(model):,} trainable parameters')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "QAMgluTT5X3u" + }, + "outputs": [], + "source": [ + "PAD_IDX = TRG.vocab.stoi['']\n", + "optimizer = optim.Adam(model.parameters())\n", + "criterion = nn.CrossEntropyLoss(ignore_index = PAD_IDX)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "eN4l2xZl5X3u" + }, + "outputs": [], + "source": [ + "def train(model, iterator, optimizer, criterion, clip, train_history=None, valid_history=None):\n", + " model.train()\n", + "\n", + " epoch_loss = 0\n", + " history = []\n", + " for i, batch in enumerate(iterator):\n", + "\n", + " src = batch.src\n", + " trg = batch.trg\n", + "\n", + " optimizer.zero_grad()\n", + "\n", + " output = model(src, trg)\n", + "\n", + " #trg = [trg sent len, batch size]\n", + " #output = [trg sent len, batch size, output dim]\n", + "\n", + " output = output[1:].view(-1, output.shape[-1])\n", + " trg = trg[1:].view(-1)\n", + "\n", + " #trg = [(trg sent len - 1) * batch size]\n", + " #output = [(trg sent len - 1) * batch size, output dim]\n", + "\n", + " loss = criterion(output, trg)\n", + "\n", + " loss.backward()\n", + "\n", + " # Let's clip the gradient\n", + " torch.nn.utils.clip_grad_norm_(model.parameters(), clip)\n", + "\n", + " optimizer.step()\n", + "\n", + " epoch_loss += loss.item()\n", + "\n", + " history.append(loss.cpu().data.numpy())\n", + " if (i+1)%10==0:\n", + " fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12, 8))\n", + "\n", + " clear_output(True)\n", + " ax[0].plot(history, label='train loss')\n", + " ax[0].set_xlabel('Batch')\n", + " ax[0].set_title('Train loss')\n", + " if train_history is not None:\n", + " ax[1].plot(train_history, label='general train history')\n", + " ax[1].set_xlabel('Epoch')\n", + " if valid_history is not None:\n", + " ax[1].plot(valid_history, label='general valid history')\n", + " plt.legend()\n", + "\n", + " plt.show()\n", + "\n", + "\n", + " return epoch_loss / len(iterator)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "h3vj5LiB5X3u" + }, + "outputs": [], + "source": [ + "def evaluate(model, iterator, criterion):\n", + "\n", + " model.eval()\n", + "\n", + " epoch_loss = 0\n", + "\n", + " history = []\n", + "\n", + " with torch.no_grad():\n", + "\n", + " for i, batch in enumerate(iterator):\n", + "\n", + " src = batch.src\n", + " trg = batch.trg\n", + "\n", + " output = model(src, trg, 0) #turn off teacher forcing\n", + "\n", + " #trg = [trg sent len, batch size]\n", + " #output = [trg sent len, batch size, output dim]\n", + "\n", + " output = output[1:].view(-1, output.shape[-1])\n", + " trg = trg[1:].view(-1)\n", + "\n", + " #trg = [(trg sent len - 1) * batch size]\n", + " #output = [(trg sent len - 1) * batch size, output dim]\n", + "\n", + " loss = criterion(output, trg)\n", + "\n", + " epoch_loss += loss.item()\n", + "\n", + " return epoch_loss / len(iterator)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cI8vToS55X3x" + }, + "outputs": [], + "source": [ + "def epoch_time(start_time, end_time):\n", + " elapsed_time = end_time - start_time\n", + " elapsed_mins = int(elapsed_time / 60)\n", + " elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n", + " return elapsed_mins, elapsed_secs" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "fJniU2JM5X3y" + }, + "outputs": [], + "source": [ + "train_history = []\n", + "valid_history = []\n", + "\n", + "N_EPOCHS = 10\n", + "CLIP = 1\n", + "\n", + "best_valid_loss = float('inf')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "HUppdQGT5X3y", + "outputId": "1afa82e3-444a-4ccc-ea72-de374c23ef1d" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAskAAAHwCAYAAABUqPIVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nOzdeXxU1f3/8dfJAmFPAggikOCGInuCG7hiFTfcW6y20tri1tp+269V237VWv3Vqm1ta9WqdWndt7buxVYUFyqbgKIoLkEWhbAJAQJJ5vz+uDOTmclMZu5kMnfuzPvpAzOZucuZJclnPvM5n2OstYiIiIiISKsirwcgIiIiIpJrFCSLiIiIiMRQkCwiIiIiEkNBsoiIiIhIDAXJIiIiIiIxFCSLiIiIiMRQkCx5xRjzgjHmvDT3rTPGHJPpMYmIiIj/lHg9ABFjTEPEt92BnUBL8PsLrLUPpnosa+3xmRybiIiIFCYFyeI5a23P0GVjTB3wHWvtv2O3M8aUWGubszk2ERERKUwqt5CcZYw50hizyhhzuTHmC+BeY0yFMeZZY0y9MWZT8PLgiH1eMcZ8J3h5ujHmdWPMzcFtPzXGpJRpNsZ0NcbcYoxZE/x3izGma/C2fsHzbjbGbDTGvGaMKQredrkxZrUxZqsx5gNjzOROeGhERESkkylIllw3EKgEqoAZOK/Ze4PfDwV2ALe2s/9BwAdAP+BG4C/GGJPCeX8GHAyMBcYABwI/D972Y2AV0B8YAPwUsMaY4cD3gAnW2l7AcUBdivdTREREcoiCZMl1AeBqa+1Oa+0Oa+0Ga+2T1trt1tqtwPXAEe3sv8Jae5e1tgW4H9gdJ7BN5hzgWmvtOmttPfAL4BvB25qCx6my1jZZa1+z1lqcOuquwAhjTKm1ts5a+3Fa91pEREQ8pSBZcl29tbYx9I0xprsx5s/GmBXGmC3AbKDcGFOcYP8vQhestduDF3sm2DbSIGBFxPcrgtcB3AR8BMw0xnxijLkiePyPgB8C1wDrjDGPGGMGISIiIr6jIFlynY35/sfAcOAga21v4PDg9amUULixBqekI2Ro8DqstVuttT+21u4JTAV+FKo9ttY+ZK2dFNzXAr/O8LhEREQkCxQki9/0wqlD3myMqQSu7qTzPAz83BjT3xjTD7gKeADAGHOSMWbvYG3zlzhlFgFjzHBjzNHBCX6NwXEGOml8IiIi0okUJIvf3AJ0A9YD/wVe7KTzXAfMB5YA7wALg9cB7AP8G2gA5gC3WWtn4dQj3xAc2xfAbsCVnTQ+ERER6UTGmW8kIiIiIiIhyiSLiIiIiMRQkCwiIiIiEkNBsoiIiIhIDAXJIiIiIiIxFCSLiIiIiMQo8XoA8fTr189WV1d7PQwREdcWLFiw3lrb3+txZJN+Z4uIX7X3Ozsng+Tq6mrmz5/v9TBERFwzxqxIvlV+0e9sEfGr9n5nq9xCRERERCSGgmQRERERkRg5WW4hIiKdwxhTB2wFWoBma21tzO1HAv8EPg1e9ZS19tpsjlFEJBcoSBYRKTxHWWvXt3P7a9bak7I2GpEOaGpqYtWqVTQ2Nno9FMlhZWVlDB48mNLS0pT3UZAsIiIivrVq1Sp69epFdXU1xhivhyM5yFrLhg0bWLVqFcOGDUt5P9Uki4gUFgvMNMYsMMbMSLDNIcaYxcaYF4wxB2RzcCJuNTY20rdvXwXIkpAxhr59+7r+tEGZZBGRwjLJWrvaGLMb8JIxZpm1dnbE7QuBKmttgzHmBOAfwD6xBwkG2DMAhg4dmo1xiySkAFmSSec1okyyiEgBsdauDn5dB/wdODDm9i3W2obg5eeBUmNMvzjHudNaW2utre3fv6DWThHJaUceeWTcvuW33HIL27dvd328q666in//+98pb3/ffffxve99L+5tJ5xwAps3b064b7pj7CwKkkVECoQxpocxplfoMnAs8G7MNgNNMOVijDkQ5+/EhmyPVUTis9YSCARc79deANrS0pJwv2uvvZZjjjnG9fnief755ykvL094ezpBcntj7ygFySIihWMA8LoxZjEwF3jOWvuiMeZCY8yFwW3OBN4NbvMHYJq11no0XhFf+OUvf8nw4cOZNGkSZ599NjfffDMAH3/8MVOmTKGmpobDDjuMZcuWATB9+nQuvfRSDj30UPbcc0+eeOKJ8LFuuukmJkyYwOjRo7n66qsBqKurY/jw4Xzzm99k5MiRrFy5kosuuoja2loOOOCA8HaJ/OEPf2DNmjUcddRRHHXUUQD07NmTH//4x4wZM4Y5c+Zw7bXXMmHCBEaOHMmMGTMI/dhPnz49PL7q6mquvvpqxo8fz6hRo8L3J9aaNWuYMmUK++yzDz/5yU/C11dXV7N+/Xq2bdvGiSeeyJgxYxg5ciSPPvpo3DE+/PDDjBo1ipEjR3L55ZeHjxM59uuvv55TTz01fNtLL73EaaedlsKzlpxqkkVECoS19hNgTJzr74i4fCtwazbHJZIpv3hmKe+t2ZLRY44Y1JurT048f3XevHk8+eSTLF68mKamJsaPH09NTQ0AM2bM4I477mCfffbhrbfe4uKLL+bll18G4PPPP+f1119n2bJlTJ06lTPPPJOZM2eyfPly5s6di7WWqVOnMnv2bIYOHcry5cu5//77OfjggwG4/vrrqayspKWlhcmTJ7NkyRJGjx4dd4yXXnopv/3tb5k1axb9+jnVU9u2beOggw7iN7/5jXM/R4zgqquuAuAb3/gGzz77LCeffHKbY/Xr14+FCxdy2223cfPNN3P33Xe32WbRokW8/fbbdO3aleHDh/P973+fIUOGhG9/8cUXGTRoEM899xwAX375JX369Ika45o1a7j88stZsGABFRUVHHvssfzjH//g1FNPjRq7tZb999+f+vp6+vfvz7333su3v/3t9p/UFCmTLCIiIpKmN954g1NOOYWysjJ69eoVDiwbGhp48803Oeussxg7diwXXHABn3/+eXi/U089laKiIkaMGMHatWsBmDlzJjNnzmTcuHGMHz+eZcuWsXz5cgCqqqrCATLAY489xvjx4xk3bhxLly7lvffeczXu4uJizjjjjPD3s2bN4qCDDmLUqFG8/PLLLF26NO5+p59+OgA1NTXU1dXF3Wby5Mn06dOHsrIyRowYwYoVK6JuHzVqFC+99BKXX345r732Gn369GlzjHnz5nHkkUfSv39/SkpKOOecc5g9e3absRtj+MY3vsEDDzzA5s2bmTNnDscff7yrxyIRZZJFREQkL7SX8c22QCBAeXk5ixYtint7165dw5dDpQ3WWq688kouuOCCqG3r6uro0aNH+PtPP/2Um2++mXnz5lFRUcH06dNdtzcrKyujuLgYcNroXXzxxcyfP58hQ4ZwzTXXJDxeaNzFxcU0NzcnvW/xttt3331ZuHAhzz//PD//+c+ZPHlyOIvtduwA3/rWtzj55JMpKyvjrLPOoqQkM+GtMskiIiIiaZo4cSLPPPMMjY2NNDQ08OyzzwLQu3dvhg0bxuOPPw44AfDixYvbPdZxxx3HPffcQ0NDAwCrV69m3bp1bbbbsmULPXr0oE+fPqxdu5YXXngh6Th79erF1q1b494WCoj79etHQ0NDVI10Z1izZg3du3fn3HPP5bLLLmPhwoVtxnjggQfy6quvsn79elpaWnj44Yc54ogj4h5v0KBBDBo0iOuuu45vfetbGRunMskiIiIiaZowYQJTp05l9OjRDBgwgFGjRoXLBx588EEuuugirrvuOpqampg2bRpjxrSZFhB27LHH8v7773PIIYcAzgS1Bx54ICprCjBmzBjGjRvHfvvtx5AhQ5g4cWLScc6YMYMpU6YwaNAgZs2aFXVbeXk53/3udxk5ciQDBw5kwoQJbh8GV9555x0uu+wyioqKKC0t5fbbb487xhtuuIGjjjoKay0nnngip5xySsJjnnPOOdTX17P//vtnbJwmFyct19bW2ng9/kREcp0xZoG1ttbrcWSTfmeLl95///2MBkbpaGhooGfPnmzfvp3DDz+cO++8k/Hjx3s6pkLzve99j3HjxnH++ecn3Cbea6W939nKJIuISPZt+Bj67uX1KEQyYsaMGbz33ns0NjZy3nnnKUDOspqaGnr06BHu1JEpCpJFJKf86LFF9Opawi9OGen1UKSz1H8Id0yCkafD8TdCWW+vRyTSIQ899JDXQyhoCxYs6JTjauKeiOSUj+u38cn6bV4PQzpT5TCY9ENY8qgTLH/2ltcjEhFpQ0GyiIhkV3EpHPVT+NYLgIV7p8CsX0FL/HZSIiJeUJAsIrnFWnJwPrF0hqEHw4VvwOivwas3OMHyxk+8HpWICKAgWURyjAUsipILRllvOO0OOPOeYK3yYfD2g+idkoh4TUGyiOQUaxUfFaSRZ8BFb8DuY+GfF8Pj02H7Rq9HJeI7Rx55JJloyRh5nBNOOIHNmze32eaaa67h5ptvbnP99OnT4y5IsmbNGs4888yE59y8eTO33XZbB0adWQqSRSSnWFRuUbDKh8B5T8Pkq2HZs3D7RPh0ttej6nyBACx+FP50MLx5q9ejkRxnrSUQCGT1nM8//zzl5eUdPs6gQYPaXc0vnSA50dLYmaAgWURyigLkAldUDIf9CL7zb+jSHe6fCjP/D5p3eT2yzLMWPvwX/Pkw+PsM2FQHr/xKGXQf+uUvf8nw4cOZNGkSZ599dji7+vHHHzNlyhRqamo47LDDWLZsGeBkWi+99FIOPfRQ9txzz6jA8aabbmLChAmMHj2aq6++GoC6ujqGDx/ON7/5TUaOHMnKlSu56KKLqK2t5YADDghvl8iLL77IWWedFf7+lVde4aSTTgJI6TjV1dWsX78egOuvv559992XSZMm8cEHHyQ85+zZs9vcv7q6OkaOdNp7Ll26lAMPPJCxY8cyevRoli9fzhVXXMHHH3/M2LFjueyyy7DWctlllzFy5EhGjRrFo48+Gh7/YYcdxtSpUxkxYgRXXXUVt9xyS/jcP/vZz/j973/f7mOSCvVJFpGco5pkYdA4uGA2/Otn8OYf4JNX4Iy7of9wr0eWGZ/9F/59DXw2Byr3dGqy++8Ptx/q3N9jrvF4gD71whXwxTuZPebAUXD8DQlvnjdvHk8++SSLFy+mqamJ8ePHU1NTAziLjNxxxx3ss88+vPXWW1x88cW8/PLLAHz++ee8/vrrLFu2jKlTp3LmmWcyc+ZMli9fzty5c7HWMnXqVGbPns3QoUNZvnw5999/PwcffDDgBKuVlZW0tLQwefJklixZwujRo+OO8ZhjjmHGjBls27aNHj168OijjzJt2jTXx1mwYAGPPPIIixYtorm5Oeq+xop3/yLdcccd/OAHP+Ccc85h165dtLS0cMMNN/Duu++yaNEiAJ588kkWLVrE4sWLWb9+PRMmTODwww8HYOHChbz77rsMGzaMuro6Tj/9dH74wx8SCAR45JFHmDt3bsLnLFUKkkUkp6gmWcK69ICTb4F9vgJPfx/+fAQcdx3Ung/GeD269KxdCv/5JXz4AvQcACf+FsZ/02mLB05t9lt3wsGXQM/+3o5VUvLGG29wyimnUFZWRllZGSeffDLgLFX95ptvRmVwd+7cGb586qmnUlRUxIgRI1i7di0AM2fOZObMmYwbNy58jOXLlzN06FCqqqrCATLAY489xp133klzczOff/457733XsLgtqSkhClTpvDMM89w5pln8txzz3HjjTe6Ps5rr73GaaedRvfu3QGYOnVqwscl3v2LdMghh3D99dezatUqTj/9dPbZZ58227z++uucffbZFBcXM2DAAI444gjmzZtH7969OfDAAxk2bBjgZLr79u3L22+/zdq1axk3bhx9+/ZNOLZUKUgWkZxig/9EwvY7EfaogX9cDM/9GJa/BFNv9VcQuWmFU0qx+BHo2hsmXwUHXei8EYh0xOWw9Cl48/dw7HXejNXP2sn4ZlsgEKC8vDycFY3VtWvX8GUbzAxYa7nyyiu54IILoratq6ujR4/W18qnn37KzTffzLx586ioqGD69Ok0Nja2O55p06Zx6623UllZSW1tLb169UrrOKmKd/8iff3rX+eggw7iueee44QTTuDPf/4ze+65Z8rHj3w8AL7zne9w33338cUXX/Dtb387/YFHUE2yiOQUa62iZGmr10A45wmY8mv4eBbcfogTLOe6hnp44XL4Yw0s/TtMvBR+sAgO+3HbABmg/74w6qsw927Y2jb7Jrln4sSJPPPMMzQ2NtLQ0MCzzz4LQO/evRk2bBiPP/444PxuW7x4cbvHOu6447jnnntoaGgAYPXq1axbt67Ndlu2bKFHjx706dOHtWvX8sILLyQd5xFHHMHChQu56667wqUWbo9z+OGH849//IMdO3awdetWnnnmmaTnTeSTTz5hzz335NJLL+WUU05hyZIl9OrVi61bt4a3Oeyww3j00UdpaWmhvr6e2bNnc+CBB8Y93mmnncaLL77IvHnzOO6449IeVyRlkkVExB+KiuDgC2HYYfDkd+HBM+HAGfCVa6G0m9eji9a4Beb8CebcCk07YNy5Tpa4zx7J9z3iJ/DO4/DGLTDlV50/VumQCRMmMHXqVEaPHs2AAQMYNWoUffr0AeDBBx/koosu4rrrrqOpqYlp06YxZsyYhMc69thjef/99znkkEMA6NmzJw888ADFxcVR240ZM4Zx48ax3377MWTIECZOnJh0nMXFxZx00kncd9993H///WkdZ/z48Xzta19jzJgx7LbbbkyYMCHpeRN57LHH+Nvf/kZpaSkDBw7kpz/9KZWVlUycOJGRI0dy/PHHc+ONNzJnzhzGjBmDMYYbb7yRgQMHhidARurSpQtHHXUU5eXlbR6vdJl4KXCv1dbW2kz0+BMR/znud7Pp3a2Exy881OuhpMUYs8BaW+v1OLLJk9/ZTY3wn1/Af2+D/vs5k/oGjsruGOJp3gnz74HZN8H2DTDiFDj6/6Bf23rLdv3jEidQ/sFi6L1754w1T7z//vvsv//+no6hoaGBnj17sn37dg4//HDuvPNOxo8f7+mYCk0gEGD8+PE8/vjjceubIf5rpb3f2Sq3EJGck4Pv3SXXlJY5WdZzn4Idm+Cuo50ew1nuHxsWaIFFDzllFS9eAQNGwndfhq/+1X2ADHD4/4Jtgdd/m/mxSsbNmDGDsWPHMn78eM444wwFyFn23nvvsffeezN58uSEAXI6VG4hIjnFqgGcuLH3ZLhojtP9YubP4KOX4NTbofeg7JzfWvjgBfjPtVD/vrNi4NQ/wl5Hdey4lcNg7Dmw4D6Y+APoMzgjw5XO8dBDD3k9hII2YsQIPvnkk4wfV5lkEckpTgs4hcniQo++MO1BOPn3sHKu02v4vX92/nlXvAn3HAePnA0tu+Cs++C7szoeIIccfpnzA/HabzJzPBFxRUGyiOQUhceSFmOgZjpc8BqUV8Fj34R/XgI7GzJ/ri/egQfPgnuPh82fOcH5JW/BAac5kwszpXwI1JwHC//mtJCThPTGWpJJ5zWiIFlEcoq1KreQDui3N5z/Ekz6Ebz9oLPk86oFmTn2xk+drhp3HAYr34JjfgHfX+gE56HFQDJt0o/AFDkTASWusrIyNmzYoEBZErLWsmHDBsrKylztp5pkEckpFk3ckw4q6QLHXA17HwN/vwD+8hU48ko47EdQlEZrqIZ18OqNTn1wUQlM+qFTJ9ytIuNDb6PPHlD7LZh7lzP+ytQXWygUgwcPZtWqVdTX13s9FMlhZWVlDB7srrZfQbKI5BzFyPmtsamFOZ9sYPiAXgwq78T+xtUT4cLXnVX6Zl0HH/0bTr8TKqpSHOiX8OYfYc5t0NzolD4c/pPst2Sb9D9OgP7qTXDa7dk9tw+UlpaGlycWySSVW4hIblEqOe9t2LaLb907j5lLv+j8k3UrhzP/AqffBevegzsmwZLH2t+nqdEJjn8/xilz2Pc4+N48OOl33vQs7jUQJnwHljwC6z/K/vlFCpSCZBHJKQqP898e5d0Y1KeM+Ss2Ze+ko7/qZJV3GwFPfReeOB92bI7epqXZmST3x/Ew8+cwaBzMeAXOuhf67pW9scYz8QdQUgav/trbcYgUEAXJIpJTNHGvMNRUVzK/blN2J1tVVMH05+Con8PSvztZ5bo3nE8u3n/GaR339PeczO03n4Zv/N0JlHNBz93gwO86q/DVf+D1aEQKgoJkEckpqrYoDBOqK/hiSyOrN+/I7omLS+CIy+D8mc4kvPtOhNsOhkfPBRuAr/4NvvMf2POI7I4rFYf+ALr0gFdu8HokIgVBQbKI5BRr0Zp7BaCmyukMMb8uiyUXkQbXOuUX478BzTudVfIu/i+MmOr0XM5FPfrCQRc4WfC1S70ejUjeU5AsIjlHmeT8t9/A3vTsWsL8FRu9G0TXnk5w/INFMP6bTpY51x3yPejaS9lkkSxQkCwiOcViFSQXgOIiw7ih5d5lkv2qeyUcfDG8/zR8vsTr0YjkNQXJIpJTFCAXjtqqSj5Yu5UvdzR5PRR/OfgiKOujbLJIJ0s5SDbGFBtj3jbGPBvntq7GmEeNMR8ZY94yxlRH3HZl8PoPjDHHZWbYIpKvnJpkKQQTqiuwFt7+TNlkV7qVwyHfhw+egzVvez0akbzlJpP8A+D9BLedD2yy1u4N/A74NYAxZgQwDTgAmALcZoxJY01QESkkWW0LJp4ZO7Sc4iKjkot0HHSBsyz2rP/n9UhE8lZKQbIxZjBwInB3gk1OAe4PXn4CmGyMMcHrH7HW7rTWfgp8BBzYsSGLSD5TgFw4uncp4YBBvb2dvOdXZb3h0Eth+UxYOc/r0YjkpVQzybcAPwECCW7fA1gJYK1tBr4E+kZeH7QqeJ0ErdvSyPqGnV4PQySnKE4uHDVVFSxauZmmlkR/XiShA2dA977wirLJIp0haZBsjDkJWGetXdCZAzHGzDDGzDfGzK+vr+/MU+WUHz++mKv++a7XwxDJGYqPC0ttVSWNTQGWrtni9VD8p2tPmPhD+PhlWDHH69GI5J1UMskTganGmDrgEeBoY8wDMdusBoYAGGNKgD7AhsjrgwYHr2vDWnuntbbWWlvbv39/V3fCz7Y0NrO1sdnrYYjkDC0mUlhqq0OLiqjkIi0TvgM9dlM2WaQTJA2SrbVXWmsHW2urcSbhvWytPTdms6eB84KXzwxuY4PXTwt2vxgG7APMzdjo84FVT1iRSOqTXFgG9C5jSGU3Td5LV5fuMOl/4NPZ8OlrXo9GJK+k3SfZGHOtMWZq8Nu/AH2NMR8BPwKuALDWLgUeA94DXgQusda2dGzI+cUCAUUEImFqAVd4JlRVMn/FJk3aTFftt6DnQHjlVyroF8kgV0GytfYVa+1JwctXWWufDl5utNaeZa3d21p7oLX2k4h9rrfW7mWtHW6tfSGzw/c/a/U7TSSSRR0uCk1NdQXrG3ayYsN2r4fiT6Xd4LAfw4o34NNXvR6NSN7Qinses8H/REQK1YTqSgDmr1DJRdrGfxN67+H0TdabTJGMUJDsMWWSRaKp3KLw7N2/J73LSjR5ryNKy+Dw/4WVb8HH//F6NCJ5QUFyDlBAIBJJUXKhKSoy1FRVKJPcUWPPhT5DlU0WyRAFyR6zFgUEIhGUSS5MtdWVfLSugU3bdnk9FP8q6QJHXAarFzgr8YlIhyhI9pi6W4hE08S9wlRb5fRLXqBscseMORsqqmHW9comi3SQgmSPWatpeyKRFCAXpjFDyiktNiq56KjiUjjicvh8MSx7zuvRiPiaguQcoKBAJJp+IgpPWWkxI/foo8l7mTDqq1C5l9M3ORDwejQivqUg2WOqvxSJ5pRbeD0K8cKE6kqWrPqSxiatOdUhxSVw5BWw9l14/2mvRyPiWwqSPaYleEWiOW8c9UNRiGqqKtjVEuDd1V96PRT/G3kG9Ns3mE3Wmw6RdChI9pjTJ1kBgUiItXrjWKhqgpP3VJecAUXFTja5fhks/bvXoxHxJQXJHlMHOJFo+nkoXP16dmXPfj2YX6cgOSNGnAb994dXblA2WSQNCpI9pqyZSAytQlnQaqoqWLBioz5hy4SiIjjqStiwHN55wuvRiPiOgmSPOZlk/TEQEQFn8t6m7U18XL/N66Hkh/1OhgGj4NUboKXZ69G4Z606dIhnFCTnACVMRFppMZHCVlMdrEtWK7jMKCqCo34KGz+BJY96PRp3Vs6DPx0E950Au/SmSbJPQbLX9NGySBQtsFPY9uzXg8oeXTR5L5OGHw+7j4VXfw0tTV6PJrmmRnjpKrjnWNi5BVa+BY+d54+xS15RkOwxLUstEk0/DYXNGENNVYUyyZlkDBz1M9i8AhY95PVo2rdqAfz5cHjj9zDuG3DJXDjpd/DRS/D0pcoqSVYpSPaYPlYWiWb16UrBq62qoG7Dduq37vR6KPljn6/AHrUw+yZo3uX1aNpq3gn/vgb+coxTWnHuUzD1D1DWG2qmw5E/hcUPOduIZImCZI9pdTGRaDb4nxSu2upKABao5CJzjHFqk79cCW//1evRRFsdzB6//jsYew5c/CbsPTl6myN+ArXfhjdugf/e7s04peAoSPaYVhcTaUtvHAvbyD1606WkSCUXmbbX0TDkYJj9G6fu12vNO+E/18LdX4HGLXDOE3DKrVDWp+22xsAJN8P+J8OLV6ilnWSFgmSPaVlqyWefbdjuuqTIeeMohaxrSTFjB5dr8l6mhbLJW9fAwvu9Hcuat+HOI+G138CYs+HiOU5JSHuKiuH0u6FqEvz9Qvj45awMVQqXgmSPWauJe5KfPqlv4PCbZrHws82u9tNPg4DTCu7d1V+yY5dWisuoYYc7QeZrv4GmHdk/f/MuePk6uGsy7NgEX38cTv0TdCtPbf/SMpj2IPTbFx79hhNsi3QSBckeU9ZM8tXmHU67pi93uJwkpIl7gjN5rzlgWbzK3ZssScIYZxW+hrUw/57snnvNIid7PPsmGP1VJ3u877Huj9OtHM59ErpVwoNnOT2gRTqBguRcoIBA8lC6ga4NrkMpha2myllURJP3OkH1JBh2hDNRLhuLdDTvgln/D+6eDNs3wNmPwGl3QLeK9I/Ze3f4xlMQaIG/nQYN6zI3XtFd3JIAACAASURBVJEgBcke08IJkr+cV7bbYFkt4ASgvHsX9tmtJ/M0ea9zHPVT2FYP8+7u3PN88Q7cdbSzkMnIM5zs8fDjM3PsfvvAOY87AfIDZziT/0QySEGyx7QEr+S7dF7e+okQcFrBLVixiUBAr4iMG3ow7DUZXr8Fdm7N/PFbmuCVG5zyioa1MO0hOP1O6F6Z2fMMroWv/hXWvQePnut0zBDJEAXJOUC//yUfhYJjty9vvXGUkNqqCrY2NvPhuk4I4sRZhW/HRph7Z2aP+8W7Tvb4lV/BAafBJW/Bfidm9hyR9vkKTL0VPn3V6XoRCHTeuaSgKEj2mPokS74Kvardt4DTz4M4JgQXFZlfp7rkTjG4BvadAm/8ITOlCi1N8OpNTvZ46+fwtQfgjLsznz2OZ+zZcMwvYOlT8K+fqmZLMkJBssfUJ1nyVfoT91RuIY4hld3o36urFhXpTEdeCY2b4a07Onacte85E/NmXQcjpsLFbzkLf2TTxB/AwZfAW7c7K/OJdFCJ1wModJqkJPkqlBF2XW6hnwkJMsZQW1WhRUU606CxsN9J8OatcOCM1PsVh7Q0OwHpKzc4K+V99a8w4pTOGWsyxsCx18G2dfDva6DHbjDuHG/GInlBmWSPKRaQfNVabpHGvoqSJai2upJVm3bwxZc5sIxyvjryCtj5Jcz5k7v91i2DvxwDL//SqTm+5C3vAuSQoiI45TbY8yh4+vvw4b+8HY/4moJkjzlZMwUEhWLjtl1c+dQ7NDbl/ypirS9r969v/URISG2wX/L8FSq56DQDRznB7X9vh+0pPM4tzU6P5T8fBps/gzPvha/eDz36df5YU1HSBb72N+d+PXYerJzn9YjEpxQke86qu0UB+fULy3h47mc8vXiN10PJSXrD2PmMMXXGmHeMMYuMMfPj3G6MMX8wxnxkjFlijBnvxThDRgzqTbfSYk3e62xHXAG7GuDNP7a/Xf0HcM+xTjnDvlOc2uORp2dliK507QXnPAG9BsJDZ0H9h16PSHxIQbLH1N2isARCdboFEAzaNBYTCW+b/w+P146y1o611tbGue14YJ/gvxnA7VkdWYzS4iLGDilXJrmzDRjhBLtv/Rm2rW97e6AF3vg93HGYswz0GX9x6o979s/+WFPVs7+zKl9RKTxwOmxRckLcUZDsMacnrNejkGwxxvlaEM95Gn2SFSPnhFOAv1rHf4FyY8zuXg5oQnUF763ZQsPOZi+Hkf+OuAKadzjBcKT1y+Ge4+Clq5yexJfMhVFntv5Cy2WVe8K5T8COzc6qfDv0iYSkTkGyx7QsdWExOH9UCuE5T2fini2gTLuHLDDTGLPAGDMjzu17ACsjvl8VvM4zNdWVBCws+myzl8PIf/33hVFnwdy7nKWeAy1O+cUdk5xA+fS7nd7HPXfzeqTu7D4Gpj3o3IeHvw5NO7wekfiEgmSPKZNcWAopk9yR5agL4OHx0iRr7XicsopLjDGHp3MQY8wMY8x8Y8z8+vr6zI4wxrih5RijyXtZccTl0LILXrwS7j0eZv4c9jra6Vwx+ix/ZI/j2fMIOP3P8NkcePI7zhsAkSQUJHtM3S0KSzhILoAwMFyTXAD31U+stauDX9cBfwcOjNlkNTAk4vvBwetij3OntbbWWlvbv3/n1qX2Litlv4G9WaB+yZ2v714wZhq8+wTUL4PT7oRpDzkT4Pxu5Bkw5QZY9iw89+PCyFZIh2gxkRygH9NCEiy3KIAnPXQf05m4VwiPjxeMMT2AImvt1uDlY4FrYzZ7GvieMeYR4CDgS2vt51keahu1VRU8tXAVzS0BSoqV3+lUx1wDvfeA2m9Db0/L0TPv4AuhYS28/lsn8D/yCq9HJDlMv2k8Zq1VJrmAtGaSC4e7iXvKPneyAcDrxpjFwFzgOWvti8aYC40xFwa3eR74BPgIuAu42JuhRqutrmDbrhaWfbHV66Hkv567wdE/y78AOWTyVTD2XHjlVzDvL16PRnKYMskesxRWwFTowtV8BfDGqHXiXur3VZnkzmWt/QQYE+f6OyIuW+CSbI4rFbXVlQDMr9vIyD36eDwa8TVj4OTfw7Z6eP5/nTcF+5/s9agkBymT7DWrgKCQFFImuSOfkBTC4yPu7FHejd37lDFfdcmSCcUlcNZ9sEcNPHE+1L3h9YgkBylI9pjT3UIhQaEwhVSTnM4+BfC4SPpqqyuZX7dJvzMlM7p0h68/BhVV8PDZsHap1yOSHKMg2WNOTbLXo5BsaW0BVwBPekdKJwrg4RH3aqsq+GJLI6s3q8+tZEj3Sjj3KejSw1lsZPNnXo9IcoiCZI+pJrmwhGqSC+E5T2cSnibuSXtqqysA1ApOMqt8CJz7JDRth7+dDts2eD0iyREKkj2mPsmFxZgCKrdIZzERTdyTduw3sDc9u5Ywr06LikiGDRgBZz/iZJIf+irs2ub1iCQHKEj2mFXOrCAV0nPuqk9yzFeRSMVFhnFDy5lfp0yydIKqQ+HMe2DNQnh8OrQ0eT0i8ZiCZI9ZdbcoKIVUk5zeYiL5/7hIx9RWVfLB2q18uUMBjHSC/U+CE38Ly2fC05fqD3SBU5DsMQsE9ENYMExrp+S8l05WOJ3eylJYaqsrsBbe/kzZZOkktd+CI38Kix+Cf1/j9WjEQwqSvWb10XIhac0kezuObFCfZOkMY4eUU1xkNHlPOtcRP3GW5X7jFvjv7V6PRjyiFfdygSKCgtHa3SL/n3StuCedoUfXEkbs3luT96RzGQMn3OysyvfiFdCjP4w60+tRSZYpk+wxq6l7BaWwMsnBr6526oyRSL6pra5g0crNNLUEvB6K5LOiYjj9bqiaCH+/ED6e5fWIJMuSBsnGmDJjzFxjzGJjzFJjzC/ibPM7Y8yi4L8PjTGbI25ribjt6UzfAb/TxL3CEm4B5/E4ssN9lKw3jJKK2qpKGpsCLF2zxeuhSL4rLYNpD0G/feHRc2HJYxDQm7NCkUomeSdwtLV2DDAWmGKMOThyA2vt/1hrx1prxwJ/BJ6KuHlH6DZr7dSMjTxPaOJeYQmXWxTQU+5qMREbebmAHiRxJbSoyHyVXEg2dCt3Fhvpuzc89V246yj4dLbXo5IsSBokW0dD8NvS4L/2/nqdDTycgbEVBGuVOysooXKLAnjW01pMpIP7S2EY0LuMIZXd1C9Zsqf37vDdWXDanbBtPdx/Mjz0Naj/wOuRSSdKqSbZGFNsjFkErANesta+lWC7KmAY8HLE1WXGmPnGmP8aY07t8IjzjEXBQCEJtYArhOe8deJex/YXiae2qpL5KzbpEwfJnqIiGPM1+P58OOYaWPEm3HYIPPND2LrW69FJJ0gpSLbWtgRLKQYDBxpjRibYdBrwhLW2JeK6KmttLfB14BZjzF7xdjTGzAgG0/Pr6+td3AV/0+/3wmIKp01yWhP3IgMeBT/SntrqCtY37OSzjdu9HooUmtJuMOl/4NJFMOE78Pbf4A/j4NUbtZx1nnHV3cJauxmYBUxJsMk0YkotrLWrg18/AV4BxiU49p3W2lprbW3//v3dDCsvKCDwr+ornuOap5emtG1rTXL+P9+hkpJ0lqUWSaa2qhKAeSq5EK/06Asn3AiXzIW9j4ZZ18Mfa2DhXyHQknx/yXmpdLfob4wpD17uBnwFWBZnu/2ACmBOxHUVxpiuwcv9gInAe5kZuv9FZ808HIh02H1v1qW0XSG2gEt3nwJ4iKQD9tmtJ73LSliwQpP3xGN994KvPQDf/hf0GQxPfx/uOAw++rfXI5MOSiWTvDswyxizBJiHU5P8rDHmWmNMZLeKacAjNjpFtj8w3xizGCcDfYO1VkFyUOQjpQ4XhSFck+zxOLKhdVlqF90t0BtHSU1RkaGmqkKZZMkdQw+G81+Cs+6Dpm3wwBnw11Phi3e8HpmkKemKe9baJcQpkbDWXhXz/TVxtnkTGNWB8eU1m+Cy5K9CyiSHuLqvUZnkAnqQJC211ZXM+uADNm3bRUWPLl4PR8T5JX/AaTD8RJj/F3j1105WeezX4aifQZ89vB6huKAV9zykcovCU1DLUgdf1OneU/1MSDK1VU6/5AUrlE2WHFPSBQ6+CC59Gw79PrzzuFOv/J9fQqMWwfELBck5ohCCJiGcSlYAGJ8eFnFjzJBySosN8xUkS67qVgHH/hK+Nx/2Pwleu9nphDH3Lmhp8np0koSCZA9p4YTC05pJzn/h17SLF7d+DsSNstJiRu7RR5P3JPdVVMEZdzsLkvTfD57/X6fH8rLn9IsvhylI9lD0ErzejUM8UABPeLgFXBr7QEE8RJIBtVUVLF71JTub1XJLfGCP8TD9WTj7EeeTxUe+DvedCKsXeD0yiUNBsoeiAoKCyC3mH7f9jkPbBwrg6Q4vJuKmT7Im7olLtdWV7GoO8O7qL70eikhqjIHhx8NFc+DE38L6D+Guo+GJb8OmOq9HJxEUJHtImWT/cxvshjZvKYAnvDVIdtMCru3+Iu2pCU7eUys48Z3iEphwvjO57/DLYNnzcOsE+NfPYIdez7lAQXKOUDzgT277WwfCmWQ948noEZJU9OvZlWH9ejBfQbL4VddecPTP4dKFMOqrMOdP8PuxztfmnV6PrqApSPZQdCZZIYEfuQ12Q5sHCqDewsZ8TWkf/RxIGmqrKliwYqNeP+JvvQfBqX+CC1+HPWrgXz91MsvvPqWP1jyiINlD0TXJ4kduf2+FYuMCiJFb+ySnW5OsPwqSotrqCjZtb+Lj+m1eD0Wk4waOhG88Bec+5WSZn/gW3H0MrJjj9cgKjoJkD0UFBAHvxiHpcxvHhd4YtRRAlJxOJjne/iLJ1FZXAqgVnOSXvSfDBbPhlNtgy2q4dwo8cg6s/8jrkRUMBckeil6WWiGBH6VbblEQWdI07qIms0o69uzXg8oeXTR5T/JPUTGMOwe+v9CpW/7kFbjtIPj7RbDkcdiyxusR5rUSrwdQyLQstf+5D5KDmeQCeMLDfZJddbeI6gEnkhJjDDVVFVqeWvJXl+5OB4zx58ErNzjLXC9+yLmtYhhUTYTqiVB1KJRXhVd3lY5RkOwhm+Cy+Ifb562wapK9HoEUktqqCl56by31W3fSv1dXr4cj0jl67gYn/RZOuAm+eAdWvAEr3oQPnoNFDzjb9B7sBMtVh0L1JOi7t4LmNClIzhEF8fF7HnJbS15I3S1CtJiIZENttdMvecGKTUwZOdDj0Yh0sqJiGDTW+XfIJRAIQP2yYND8hlOW8c5jzrY9dgsGzcFsc//9oUjVtqlQkOyhyICggGKmvKI+yYmlcw+1mIika+QefehSUsSCFRsVJEvhKSqCASOcfwd+1/kFuuFjWPG6k2muewPe+4ezbbcKGBrKNE+EAaOchU2kDT0qXlLWzPfSDXZbCqCbSXiSoovXdlSdfqYHJHmta0kxYwb30eQ9EXDKK/rt7fyrme5ct2mFEzCHAucPnnOu79ILhh7cWp6x+1go6eLZ0HOJgmQPaZKS/7mvSXY/mc2vWifuudkn4nIBPEaSWbXVldz92ifs2NVCty7FXg9HJLdUVDn/xp7tfL9lTTBoDtY1/+cXzvWl3WHwhNbyjD1qobTMu3F7SEGyh6xiZN9Le8W9AggAWzPJ7vcRSUdtVQW3v2JZvGozB+/Z1+vhiOS23oNg1JnOP4Bt6yOC5jfglV8BFoq7OIFyqDxj8IHQtaenQ8+WvAmSv9zRxOKVmzl83/5eDyVlqr/0P/cr7oVawHXCYHJMeDGRNO9rATxEkmE1Va2T9xQki7jUox+MmOr8A9ixCT57qzVofv138NrNUFTilGSEyjMGjYee/om93MibIPn7D7/N7A/rmf/zY+jX0x/tfyI/Ti6EzGI+cp1JTnM/X0rrPqp3uKSvvHsX9tmtJ/PqtPKeSId1q4DhU5x/ADsbYOVbrdnmt+6AN//g3NZzAAwcBQNGtn7tu7fvJwT6e/QRPl3fAMD2nS3gk08B1CfZ/1wvSx3qblFA7UzcTdxLbz+RkNrqCp5b8jmBgKWoSL1hRTKma09nqey9JzvfN+2A1Qvg88Xwxbuw9h345FUINDm3l5TBbvtHB84DR0JZH+/ug0t5EySXBnv+7fJR24DoJXgVEPiRapITS1ZuYa3l8fmrOGnM7nTvUhK1T9tvRFJTW1XJw3NXsnxdA8MH9vJ6OCL5q7SbU25RPan1uuZdsP5DWPuus9jJ2nfhg+fh7b+1blM+1Gk7N3BkawBdXpWTvZvzJ0gudh7c5oCPgmR9tOx7adck++dlmrZkj03dhu385MkldO9azEmjB6W0j0gyoUVF5tVtVJAskm0lXZzgd+BIGDPNuc5a2PpFa+AcCp4/fKF1Ra4uvWDAARGB82gnC92lu3f3hXwKkkucj9Wamn30V9ZHQ5X40s0kF8InB8nuY3PwnUJzxCzGqDeOnTMsyXNDK7vTv1dXFqzYxLkHV3k9HBExBnrv7vzb5yut1+/aDvXvBwPnd53AefGjsOvu4H5FULlXdOA8cCT02j1ry2znT5Bc7MNyi8jLigh8yX0m2fnaUgBPeGu5Rfz7GkhSelIAD5F0AmMMtVUVmrwnkuu6dIc9apx/IYEAbF4RzDoHA+fVC2Dp31u36VYZzFaPbq1z7je8UxZAybsguclHQXKkQqhRzUfuu1uElqXujNF0nuaWAC3W0rUk9QUaWrPm8W9vLT2JX3akiXuSrpqqCl549wu++LKRgX0KcxEEEV8qKoLKYc6//U9uvb7xS1i71Amcv1jiBM/z7obmxuB+pdB/OHz9MeizR8aGk0dBcrDcwkdBci4tJrJy43YAhlR6W//jN26D3fDEPZ9FyTf+6wMWfbaZxy48JOV9bMzXWK2rD0bsk+CyiBsTqisBmL9iY7jeXUR8rKyP05e56tDW61qaYePHrXXO696DHpnt15xHQXJw4p6PVmmInrjn7bgPu3EWAHU3nOjpOPzG7fMWbgHnswjwiy8b+WJLo6t9kj028Tp9KHssmTBiUG+6lRYzv26TgmSRfFVc4mSP+w9vXTUww3Kv30aafFmTnEOZZEmP2+ctXJPss0yyJf0ANlm5RSBRJjmts4k4fw/GDilnwYpNXg9FRHwsb4LkLj6sSdbEPf9Ld8U9vz3fAWs7sLx0uhP3fPYgSU6pra7gvc+3sG1ns9dDERGfypsg2Z81yblTbiHpCbXlTrUbTXiymt+eb5vO6oLRX2O11iSru4VkXm11JS0By6KVm70eioj4VB4FycFyi2Y/BckRl70bhnRAKNBLuWOjT1fcczLJ6XXySLSXTVJuIdIR44aWYwxqBSciacubILnEh0FyJAUHhSFch+u3mmTr/o1cstd0vHILTdyTTOldVsrwAb1UlywiacubILlLsNxip4+CZPWE9b9wJjnFeovWjg6dNaLOYXFfkxzePNFiIoEkE/d89hhJ7plQXcnCFZvCqzuKiLiRN0FyiR+7W0S1gPNwIJK2UICXarlFvAU0/CBg3b+RC9ckt3NMiM6qR01m1RtH6aDa6gq27Wph2RdbvR6KiPhQHgXJTpjip3ILt1mzloDlhheWsW6ru3610nlaM8mpbR96mv1Wk2zTmLgXuW/869v2jI6ezJre+URCaoOLiqjkQkTSkTdBcpHxYZAccTmVoGnlxu3c8erHvPbh+s4blLjivuODPxcTAet+dcEkmeDWmuRE+4t0zB7l3di9T5km74lIWvImSA7FHH4Kkt0Kt8zyeBzSKhT0mhQLLvxak+yMN91yi0R9kuNkkqP299mDJDmptrqS+XWb9HoSEdfyJ0gO/nn1VU2yy4+Wky2+INkXDnZd9kn2X3eLDiwm4qJPsl7akmm1VRV8saWR1Zt3eD0UEfGZ/AmSfZhJdjtJycYJKsRbbvsk+7YmmXRawCXrk+x8jX6/ED+rLJKumqoKQHXJIuJeHgXJfswkx7+cSLIaTsk+t7Fu6Lnz24p7Aev+zVnyPsnxJu6lvr9IKvYb2IueXUuYX6cgWUTcyaMg2fnqp0xyZK4slcxieAUzBQ85I1yTnGp3i3C5RWeNqHNY674hW2j7xOUW0V8j94n3nUg6SoqLGDe0XJP3RMS1/AmSg1/9FCS7XZY6FFj57aP6fNbaJ9ntxD1/PYfWpl9HnXTiXoLj+uwhkhxWW1XJB2u3sqWxyeuhiIiP5E+QHMok+6ncIvJySuUWqknONe77JPuzBZxNY2mP8F101SfZ/dhEkqmtrsBaWKi6ZBFxIX+C5FB3C59mklPJJfu1fVg+c/tUhD4N8NF7OSD42kuzT3Ki3eKWWyRoByfSEWOHlFNcZDR5T0RcyZ8g2Yc1yW6Xpc50FnLHrhYWrdyckWMVKvfdLfz5aUAgnZrkFCfuJQqMffYQSQ7r0bWEEbv3Vl2yiLiSR0Gyv7tbpJIdznR3ix89tohT//QGG7ftyswBC1DrxL3UwmS/drew6XS3CO+bqCY59DVBdwvlkiWDaqoqWLRyM00++hshIt7KnyA5+NVXmeQEHzMnkuma5Lc/c7LIO5tbMnK8QhQqn0g1kxx6ofpuMRHSeHMWfr0mujn0yUjkeVSfLJ1jQnUljU0B3luzxeuhiIhP5E+Q7MNyi0ipxAM2SdDhViibWZTqrDNpI/xUuF1xz2cBoNMCLs1McsJjOl/9NolR/Km22llURCUXIpKq/AmSg3+Kd/ooSHabNcv0stShbKaC5PQVzIp7NvOZ3UC8N31Rn65k9nxS2Ab0LmNIZTdN3hORlOVPkOzHFnBJ6i/nfLyBv82pa7N9prKQbtuXSVvua5Kd7Vt8lkpOo7lF+PWabDGRyMciOl7212Mkua+2qpJ5dZt8N3FWRLyRNEg2xpQZY+YaYxYbY5YaY34RZ5vpxph6Y8yi4L/vRNx2njFmefDfeZm+AyGhv7N+LbeIFw+cfdd/+b9/Lg1/H28Z344IJAliJLnwYiIpr7gX/dUvAtZ2oAVcool77fdJ9ttjJLmvpqqC9Q07+Wzjdq+HIiI+UJLCNjuBo621DcaYUuB1Y8wL1tr/xmz3qLX2e5FXGGMqgauBWpw/sQuMMU9bazvh8y7nL6qfZi67726R2Yl7oXILZezS5/YNS+i582V3C7c1yUneECSbuCeSaROqKwGYV7eJqr49PB6NiOS6pJlk62gIflsa/JfqX7LjgJestRuDgfFLwJS0RppE6A9xs48+xo6qSXaxmEim4qtwgOefhyznhB7C/K9Jtq7LfJJt3vpJhjpaSHbss1tPepeVsGCFJu+JSHIp1SQbY4qNMYuAdThB71txNjvDGLPEGPOEMWZI8Lo9gJUR26wKXhfvHDOMMfONMfPr6+td3AVH6I+rn2o93X60nOnOCC0ZPl4haq3rdleTnE7fYS9Z0uiTnGTzZOVDoat3Nrfw8NzPfPV4SW4qKjLUVFUwv06T90QkuZSCZGtti7V2LDAYONAYMzJmk2eAamvtaJxs8f1uB2KtvdNaW2utre3fv7/b3cOZ2JaA9c0f0wST+hNvn/HuFqFz++PxykWuM8kuS2xyRRolyUlXF4y7LHWc/d/4aD1XPvUO732u/raZEEx6vG2MeTbObQnnl+SL2upKlq9rYPN2LaIkIu1z1d3CWrsZmEVMyYS1doO1dmfw27uBmuDl1cCQiE0HB6/LuMg/tH4puYj+mDn5mDNek6xMcoe57RAS+Vj76VOPgLVpl0Ik2s3GySTHK71oavFnR5Ac9gPg/XZuf9RaOzb47+5sDSpbaqucfslqBSciyaTS3aK/MaY8eLkb8BVgWcw2u0d8O5XWX8D/Ao41xlQYYyqAY4PXZZz1YfARlTVLYciZbgEXLrfwyePVWV75YB1bGpvS2tdt4BgZBPqpLrm1Ht7FmJPU0IcnjibMJEfvX+Av04wwxgwGTsRJZhSkMUPKKS02zFPJhYgkkUomeXdgljFmCTAPpyb5WWPMtcaYqcFtLg22h1sMXApMB7DWbgR+GdxvHnBt8LqMiywZ8E8mOeJyCh9mhzPJGSqP8GMdd6at29LI9Hvn8f2H3k5r/9ZAN3Eq+f3Pt7BuSyMQW27hn8c9NmB1s08icRfHibOTzfAnKAXuFuAnQHttgOLNL2mjo/NIvFJWWswBg/po8p6IJJW0BZy1dgkwLs71V0VcvhK4MsH+9wD3dGCMqYnMJLf45Y+pu1n9nZVR88ubis7Q2OTECp+sb0iyZXzhmuR2yi2O//1rdC0p4oPrjo96g+Onhz0cqKa1T6Ka5LZlFNGrUEaXA/no4cpJxpiTgHXW2gXGmCMTbPYM8LC1dqcx5gKc+SVHx9vQWnsncCdAbW2tr56eCdUV3D9nBTubW+haUuz1cEQkR+XPinsRl5sD/umVHOKuu0Vm/x4VciY5FNym+5Cmuix1aLl0v9Ykp1NukbxPcpLbw1/blmVIWiYCU40xdcAjwNHGmAciN2hnfkleqamqZFdzgHdXf+n1UEQkh+VPkBzxF9QvmdEknzK30Vkr5PkpWOssmZ6Ulvg87iZr5opAOpnkmK+Jjplsxb3Wl6d/Hq9cZK290lo72FpbDUwDXrbWnhu5TTvzS/JKbbUzeU+t4ESkPXkTJPuyu0Xk5RQCpvDHzxm+f36qjc20VLtSJOK2u4UfJ5hCmjXJSbaNV5McvU90LXIBv0w7VSrzS/JNv55dGdavhybviUi7UlmW2hci/376pSbZ/bLUqW/rhl/eVHSmdLO6oYfOpNgpOfIsTT55nULy+uL2941/fbwWhPE21cKQmWetfQV4JXg5pfkl+aa2qoJ/v78Wa23KiwGJSGHJm0xydLmFP2qSowMz285toS0y290ixE8ZzUwL/XFM9xGwLjPJkVnTxqaWNM+afeksiW6TFFzE61oRr09y6DiF3qpQMqu2uoJN25v4uH6b10MRkRyVP0FyxGW/BH3R5RbRt+1qaRvou6lJ3rGrhVc/TK0tk18er84Qim1TDf5e/bCew2+cxc7mFlf7WprMDwAAIABJREFUhVgLXUucH7vGZh8FyaGvaZRbJM4kR3+NPE/k5daVIUUyp6aqEkCt4EQkobwJkiP/gvrlY+z2Ju7tam4bJMdboSyR59/5nPPumUv91p1Jt/VL5r0zpZqdv/aZpXy2cTsrN24HUu9uEXmebl2cllOh9nN+0JFyi0SSdWuJXYZdNcmSSXv170FF91JN3hORhPImSI784+2XzGh0T9jo2+IFyW5awIWylPEy0m2O659YLePcvlJKi50fmV3N0fW0qdY0BgLQvdQJknfs8k8mOZ3OKskm3IWOGdUnOaq7RXRHjUyXGUlhM8ZQU1XJfC1PLSIJ5E2QHBno+SYzGjVxLzoAiFtuEQhtm/zQrUFN8o1bPE7RvbPqS6bfO5emFAL6TIu3NHJ7ugRLJUJjTfb4xrs9nEn2VblFdMCa2j7R+7a5PW4QHacWXzP3pJNMqK7g0/XbWN+Q/BM3ESk8eRMk+zOTHP8yJCi3CH1N4e65aZvV4vGbiv99fDGvfFDPx/XprXqXCam+ZEKZ5NYguf3tY1+LAWvp3sVpKrPThxP33LQLTF6T3PaTkXglSIqRpbOoX7KItCd/guSIv6B+aWkW76PlkJ0RQXJsLXJqPZWJ2qc9HiRwo3j5MbrbJHppsVNWsSu8gp5zgKIEP0mxWXproVupH2uSo7+62jfB9XH7JMc5Z7ztRDJh5B596FJSpMl7IhJX/gTJEZebszxx78O1W9m2s9n1fu0Fh7uiguTQ19Rrkt1MdorMJHu5ClyqvYYzqfWxTO1+h2uSW6KXmU409tgkfcC2Ttzb4atMsvt0brI3P3H7JEdlkmPfHKZ+7vbMq9vIA/9dkZmDia91LSlmzOA+WlREROLKnyA5KpOcvQxdS8By7O9mc/GDCzt0nNgAICqTHPzqZjERN9m35gQTp7LFywShmxIWgC7BIDmUBU624l6bTDLQPdzdwj9BcrjG3UWUnCz7HPvmr+0GUV8y9nnDkwtW8fv/LM/Q0cTvDhxWyburv2TdlkavhyIiOSZvgmQ8qknesqMJgIWfuc9ERK+4FzNxLyJIjq3dTCXwje0M0J6o7gIpbJ9poXN6seiVm8cJWifu7Uxx0l3s6o/W+rQFXHiJaDf7RO8bKzRpMrpPctvXYrxFRzoiYK3ayUnYWTVDaLGWv+nTBRGJkTdBcsBCUTDIymZN8pfBILl3WanrfePVX4ZEBmGxGblU/sC7qV+ODJK9rPsMxciPzP2Ml5etzco53XQBgdZyi1D7tlCglyi+j1eTXFbqx3KL4Nc09km0U9ya5DjbZnrinnNeRcniqO7Xg2P2H8AD/13hq093RKTz5U2QbK2lJBjAZDOTHAqSe5WVuN43agnemNviZZLdZNTclGYk6lObLbH35y+vf8pj81Zl6+wR/0+uNFxuEQySQzXJCdLg8bpbFBtD15IiX3W3SG/yXIo1yQk+yWgz+TRDr82AtSl3M5HCcP6kYWza3sRTC1d7PRQRySH5EyQDpcFUcjb77XYoSI68nEqfZFc1yW4m7uVIJtm0jiFbHS/cdm3oUuIMckdsTXKC7WMfT4vziUdZabHPslZplFskiW1bJ6K2vc7ZL/r2TL0mrPV2gqrknoOGVXLAoN785fVPot60iUhhy58g2UJpiZeZZPflFpF/81PJJLurSY7etz1et8yLPbu1qfct7ii35RYlRdGZ5GSTytpkkgMWYwxlpUUdrkm21mbtD7qNCVS372pm+r1z+WzD9hT2TVCTnOQ12qarS4be+yqTLLGMMXznsGF8XL+NV5fXez0cEckR+RMk0xrAeFGTnF4mOXGUvDNuC7jor+0eO412cakeO+PC53Tysc7EqiwFfy7LLUKPVThITlICExskW5yMeVlpcYdrkh+dt5LDbpzVoWOkKrbkYeXGHbzyQT2LV21OuE+yTHK8FnBR+6d4HLcCyiRLHCeOGsRuvbpyz+ufej0UEckR+RMkW0uX4EIP/qlJbr2c6e4WrRnS5OPIvXKL7AXrbs8TGySHy1qSbB95PoOhWwbKLVZv3sHqzTuyEvDFZsxTeXORvE9ycLtkE/fCpR6ZuZ/qbiHxdCkp4rxDq3lt+XqWfbHF6+GISA7IoyCZ8MS9bGaStzQ6QXKX4mLX+0YvnBBtV5w+yel1t0i+ba60gAtxPg7PzkjcTgoLLVSzI2biXqLhtskkW0uRga6lxTTGWXrcjUwvstGe2PKdUOlDe4FrstdrvE87olrAxWSaM3U3rfVyjUfJZeccNJSy0iJlk0UEyKcgGUtJMJPcnMWJe6E+yekEdZF7xO4eOXHPBi+mlUlOIRzImUxy8Gs2a5LdfpQfaukWqieOrdWNFft4Bmyw3KKkqMOZ5FSCx4adzSnVDSc/V3RAnkrf5Njsc6JjJl5xL/65OyoQ0BLXEl959y6cWTOYf7y9hvqtO70ejoh4LH+CZNu6GpoX5RbpiDeTPyR6xb3oYCKVuxevc0Ai3q+4FxtIZi+TnGgMiYQmyu1oU5Mcf/vY92sWS5ExdOvS8XKLVCZnfu3Pczj8plkdOo9zsqgvKZ07duJdrKR9kttkolVuIZ3vWxOHsasloKXLRSS/guRwJtmDIDnTmeRAVHY3tH3qtZnJss6Rx4/uk5z96CFeuUW2JKspjhV6bbWpSU6x3CJgAQNlJZkIkpNnWJeuyUxtZdte3UR9jTu+JHXL8R676Bg5+pyZ+rEOWGWSJbG9+vdk8n67aXEREcmjIBkbXujBi0xyR//oxsumxt6WSvaudf/QvvFvj1wJzuvFRMLnDn7NZhDjps4b4k3cS2371hNCUYZawLkpv+moRBnkjpw7tGuyn1e3z1Hy86omWdp3/qRhbNi2i38u0uIiIoUsb4LkgIXSUAu4Fi+CZPf7tjdxL/L78MfSgdQzask+6k5Uh+xFhi02CHL6/2bp3OGvqd3vlnAmOaYmOcHjFvupRsBaDJlZTCT2dbBk1WbWbW3s0DETiRPrR32Nv1P7G8ULtON1usj0YiIBa72ZoSq+cchefdl/99785fVP1S5QpIDlTZCMhaIiZ1JUc7YiLCKDpXR+kSbO4Mab8e9mln+yj6gjg7eomuQUjt15Wsec7e4WqZ4uVGO8I8UWcPH6JDuZ5I73SY4NMqfe+gbH/m52h46Z6rliyy/iSfYGJN6nHVHlFjHBccYm7qncQpIwxnD+pGF8uLaB15av93o4IuKRvAmSLRaDoaTIZLUmOZzdTSMujwoOEqXqoM1H3G5qkhNmkiOy7YEEWeVsiQ2CAln8ONx1d4vgE912MZH427ftbmHDi4ns7PCKe6Fjtl63eXv6E0nbPVfMOVOqSY55bN74aH3UZKi4JRtxLma6BVw2X1/iXyeP2Z1+PbvyF7WDEylY+RMkWyeLXFJUlNWa5JZ4f+hTZBNcjj1ebCDnpiY5cSY5EHG5nbqPLArfv0D2Vtxze4dD7y1CnyAkKwVo2yfZyVKVFJmouvB0JFvtL6NiXoOpTKaLvemcu9/i5/94t/X2eOUWcfokZ/p+WmWSJQVdS4o575AqXv2wnuVrt3o9HBHxQP4EyYSCZJPVmuRQEJRWsUWcbHG872OzwqlkrVvLAJIHb9H1ycmPnWnxspPZGkf4PCmeLxCuSQ5mkpOUAkRm6UPPn8F5rXY0UHPTErCj2nS3CF7fXk42WbY5EO7/3XafeMfJFLWAk1Sdc3AVXUuKuOcNZZNFClH+BMnW6T9bXGzCH4ln57zO1/QyyYnLHOIld910FEgWoDQnCJKTfRBdt34bm7fvSnp+N2JrT7PZJ9nt8xd6rHY2x6y4l2j7OJ8IFBmDMabDgZrb1QI7Iva9RCqTSJMF0slKgmJf94/MXcnhN85KccSJuSlbksJW2aMLp48fzJMLV7OhQYuLiBSavAmSQ3+sS4oMTdkstwhlktM4Zbv1nJEBdExAksq5kgXUUZnkOIFcItPvncsf/vNR8gGkIbLGNlvxi9tAKfS4NbU4JSHJFxNp+0bIGCgy6Z0/UmwQ2ZnarrgXGkTqb9jaXB8+dtvr4p3rs43b+WxjJlYPbH9cIpHOn1TNruYAD771mddDEZEsy5sg2Sm3MBQXmahJaZ0tYzXJ7ZRbxF6XyrSjZF0bojLJLW0DuUQadjazpbGTJodFZHWzleVzOykstr90sg8touttHUXGySZHnj8dsaUPqWyb/rmijxNvSWm35wzdnrhPd+y5MpQB7sCnP1J49t6tF0cO789f56wIf4IkIoUhb4JkrNN/tqSoyJvuFmllkhOXOUTe1qYmOYVzJSsjiCxJcZNJDlhojl1rOUMi63uz9RS6WcUQogO65oCNeN4SPc6tl1szyQYTc106Wmt608/mpsLGCfRbl4xOfOD2H5nImur4PwexJUNu2/Ul4naVRZHzJw1jfcNOnl60xuuhiEgW5U2Q7GSSnaWps1mTnKlVz2J3jzeZyc25kq64F4i87CZItp1WzhJ5P7OW5XOZSY4cV1NLIIXHue1jawwUBestOnI3I4PGQJLnpCOPpo37Woz+2t5JE07cixP0/n/23jzejqO6Fl7Vfc69V7ItyYM84HnAgDFgg/GADWFyAiExCYRA+GwMmCFfSML3SMiD5D14EJKQkADJBwmTAQdihzkhgAFjTIzxIAvbeMajLHmQLVuzdIdzuuv90b2rdlXt6j7n3itd6aiWf/6doburqvv0Va9atfbe8goKKc5Os7PGzqxUmDAaOOu4A/CUg/ZJxUUSEvYwjA5J1lXWgHwn5knWWjOSNAu7BTvEH7JUBW+YbAZ2KV7eOZYCrs3KUZZ63u0sUtntnfUcGtafyjOn9AvdqkpKqfwUFGq3xdyUZHbvtbUzp/LRznvtvTYd16wly0py2K+/31xJSvIkJwwLKi5y59otuPreJxZ6OAkJCTsJo0OSUWW32Jkp4NpSVw2DwG4hvJ9NMZGYqO4ElA2RAk7rHVfRkFcW3GnZLYbUJR0luSwDr64PKXMI9yTPyQbBiGpbzuV5s1t4E7VGu0ULGZXyJDcdP5d0i1K/iSQnDINzTnoSDth7LBUXSUjYgzAyJLksq2XsfCd6kqXMBcNA8l/azzp430Y6ONoC0qIp4AZQJHvzrSSbvnc+gRm2H8eTXLDsFpH9eUYLOlQpzIsneRjlfS415vifU5AvuaHZQaw7fvv8IJ4SUHp9x8U34Fs3Ptg6/rBft/2EhEEw0c1x7ulH4sd3PoZ7Htu60MNJSEjYCRgZklxTEXR3oifZtUQMf/ygBMNXKwfzJDfv6weg2ePa2t2RSrK8BN+GstS4ZpZLoMOSVJ8kty3d06pGppT5/TKlrJI85Hg5OMlsqzI5H4o1b8f3yTcdF51AkMc4Zrfw73sv3eJ3b34E/+Mrv2g/Ab/fpCQnzBLnnn4kxjoZvpCKiyQk7BEYHZKsda0k7zxPsus3nYWS7BBh93iJgA/nSW4eVz+a9q1dSZ5vOwsf62wyD1x73xP4vc9eizvXbh6+7yH3L+r7DKjsFm0WGLJBKLi/2872JM/NbiH1PfjvFLs2kpIsBu6Z/QfobADMZiKWkAAAB+w9jt8+6VB844YHsWHb/BZVSkhI2PUwMiQZQJ0CTrWqavOFuZZz5of4xzsE2lt2HoSQD1PkYjglWc/7JIQrjrPJPLB1ug8A2D4zixyms7BbjHeqP5vKbhFvZs367fjl2i0Aag9yvZOjJM9BlOe/8TD5mofvJ3w/kJLccG349pjdx5Lj6t1ccpK7/Q4/EUtIILz5rKMx1Stx8YpUXCQhYdQxMiRZa1gleWcF7gk5cIeBFBBl2wv7GUYBa1OduWWC21PafaQ7ME+yHs53bcfUPHnYPtPHn3/rFrEIyrC/W1lqjHdyAFUKOGNFEJp5/t9dYYN8lOtPng8lmV+rViV51r14dotgwjb7PqWVGNdu4fYxX351M/adlykyYYTwlIP3wfOffAAuunoVZvrpJkpIGGWMDkk22S2yHeaZ9TFnT7LzXnvbQmICQ1AGH9tAZamHCEDcIYF7pjk9KyWZlO3Ydbno6gdw8XWr8dkr72voe/C+JrqZeV8OmLs3s0LyDvAk6wGyW8zd1lG1U78O0G7bhGfQvx/axrNbzMf5pMC9hNnigrOOxmNbpvGdm1NxkYSEUcbIkORS82IiO/bhd8PqDfjSNau8SnVzY8n+4dIStxToFG26ZanbsViwOUVzMGFlL5jL9b30lkdwxS8fi7Q/uxy2NJ7YuKZ6lQ1DkXzLMLSSrF0leVALjIJylORsXjzJljS2FROZj/LXTp8DqbrNtgY5awbv1+2fe7Dncg8OUlI7IaEJv3L8cjz5wL3xuZ+m4iIJCaOMkSHJWmso1HmSd/DT71X/fDX+93/e5hCT2XFkrha7kIpQDBOV30be+Ni58t6krlFTvTko9Z+68j583sszynXy2SjJRdl8TK+2h3SzkCQP+7NxT7JTca/lOKXs9VNKGcI+H4F7VYXClp3nQpKF9/aebDiOkVrpPhTJd0M7dl+If+Nr1m/H534arhY0jSshYTZQSuHNZx2N2x/ZjGvvW7/Qw0lISNhBGB2SDAC1J7lNZdo63cemydCfOiwK4SE/DJxDvOP5x7DiXntfbYSaj527J5qapjbn4vkuyjL4ffiyvC7d7wYBEabYMbS92wlv92F/tqLUGO/awD2jlra0w1PAKVhP8mx42oZtM3jvN28xCrnWAxQTmQNL1sJKwyCWBb5FsuiINg4ncC8y+YmsZlx66yP40HfvwBbBe+72m5TkhLnjt08+FPvtlYqLJCSMMkaGJENTdovMKIcxnPKhy/CsD/xwzl1Ky8XDgB/RVJbaKq2DP9zbCDVvY9DAPTpmLoF7RRm3RfBUZsPmSW46hoJrurlEkof73YpSY6K2W/TLciCyCFT3Ju3heJJnQdR+/sAGXLJiNe54ZEvdht6xdgtBSx7mXgTk3NpS6XWn34hKryFnWCm8ANcYrGqdWHLC7DHRzXHuaUfg8jsfxf2Pb1vo4SQkJOwAjAxJ1qiWwLJMtT4kp3rzE9g3TDlnCY7/sqEsta8KD0IgfR9nbDvQlDPZhVGS5+IHLSU/qV1qn02hh7bAPSJoY/k82C20xkSXPMl64PEqL7vFXDzJJiVafV6DlPGei7VA9MdrYWOkT62BXr9ZSZa86LH7PVY8xZZib5kwzGK1IiFBwrlnHIluloqLJCSMKkaHJGuNTAG5mh3xmA2GKefchjBwL2x7GALZ5ruU7BxAM2mk3eZEknU8E4PWw1lKeJtNxxBBk5Xkgbup+ijh5Eke3JOsTF+Zwpw8yaX3O2i0e5LnoiRLWSjaJmHVuOhVY0ZYfZA8yeLxwt+GpExTe4Nm+kgkOWGuOHCfCZxz0pPwtZUPYuP2VFwkIWHUMDIkudTVknamdmIxEYE8DIOmwL0m9W4QQt5GqHnfg6aAo21tdpYmFDpUku2yuh6IfPkgJTymINJ4OwJJHrostWae5LK0v0VLMxlXkqFAmvZsiJohg6SgN1hYzDFz8SQLffPfLD5O+yrdMw75ZoQ/7Mu7XyCfr7VbtCjJs5iIJSTE8OYzj8Zkr8AlK9Ys9FASEhLmGa0kWSk1oZRaoZT6hVLqNqXUB4R93qWUul0pdbNS6nKl1JFsW6GUuqn+/9vzfQIEDW3sFvzZd//j23DUe76LH93+6Pz32aKEtR8vv/fb8x/q8+JJZpzFVcSb2qw2ziVwT7JbaPZmmImAP67YdZFUzKDvAUCpxyY61m4xCFm0x1evld1i9p5kOoQrya3Xaw580LUF0evgSjLg3jPSRKgp9Z/k15fuQWu3iI+J75cocsJ84IQnLcGZx+2Pi65eNScBISEhYdfDIEryNIAXa62fBeAkAC9TSp3u7XMjgFO01s8E8HUAf8e2TWqtT6r/P2deRi1AGyXZJYY3rt4AADsk6XsxYH7hGCSFTmrPVyuHym4xQN/9MiQwYpv1+c5FqZeUZD6m2RAY60luVpIlpXk4Ml692uwWPE9y+7GWJCtkGX0//LW0k5XS9N1mMZivPMlmVWMAXy9XnflERZrASb+79l75Buke8m1JMcxmIpaQ0IQLzjoaazdP4Xu3PLLQQ0lISJhHtJJkXWFr/bFb/6+9fa7QWm+vP14L4LB5HeUA0BqACu0W9DYTCknMFXP2JDd4gSWVbRglORb0ZLeHS93SONwxVVvnkidZsgZwMjWXPMmx34BUR4lIDvOzUT+mmEipHXK3YdsM3nHxDdHy16S+ZkxJnosnmauvbT/JvNktvAIhAxW2gXYUNskTLE42IvewhuyLp9+31XqSPMkJ84wXHn8gjlm+Fy68KhUXSUgYJQzkSVZK5UqpmwA8BuAyrfV1DbtfAOBS9nlCKbVSKXWtUuq35jDW9nEizG5hctPuAJI8r2Wpg39YOXF1ieMwnuR4dgv73q2+V71f9fg2nPKhH+HhjZNBmzqi5A2CslFJ1kypHJ4kt9ktRPVxCPJI528D90onQPJTV96L7978CL587QPhwdqOj9+Ks8uK4h5UFRNpI4bD98Pbtw253w3SrO9JlpRkybZi3nudxO6/Qb3Gs8mgkpDQhCxTePOZR+PmBzfh+lUbFno4CQkJ84SBSLLWutBan4RKIT5VKXWitJ9S6lwApwD4CPv6SK31KQBeD+DjSqljI8e+rSbTK9etWzfUSQDVgy+rU2tJy8NCsbU5oy3PaxuaPcnhtkEzKTjjiSnJkMdO7y65fjUe3zqN/7jpIXFMs/XeFWWY3cKqkrMjMAPbLeaoJFM/43UKuD7zJAMw4Xiyr9b6hnme5Nk4Y/32NdonLXMJUnMVX9Ypmicz2rlf4pO+2PhiqyGllrNbDLrSkgL3EnYEXv3sw7BscRcXXtVe9TEhIWH3wFDZLbTWGwFcAeBl/jal1EsB/AWAc7TW0+yYh+rX+wD8BMDJkbY/o7U+RWt9yvLly4cZVn28DYiSKuHtSLtFN7fq9fptM/jaysGinLVATgmiyjaEFcFfko9tB+TAPbIUTPfCZXL/mGEQC7oCyJNs3w/cZtm8zE4ErUl9HATWblGXpS6ZJxmuQuyDnxvA7RaD90+QSOOOVJK18969B5ua5dYM125RvXKe2xy4F06qZE9y9Tqw3aJxr4SE4bBoLMf/c9oR+OHtj+KBJ1JxkYSEUcAg2S2WK6WW1e8XATgbwJ3ePicD+DQqgvwY+35fpdR4/f4AAGcCuH3+hm+hoSu7hVJikY8dwJHNw7uTZebB+45/uwHv/vrNA/0j6ZAPYUnZ72coK4JHrIPN7GspcI+IoBRwBcw+w0VRhoSOTwJm40luK0vda7JbDNFP6ZHkvldMpOkWq/bjSjL/fjj4p6EHIMlzQSlMouibpn7NJi2T5Jgv3uznt2M+24p7fIWozZvu95+U5IT5xhvOOAqdTOELP1u10ENJSEiYBwyiJB8C4Aql1M0ArkflSf6OUuqDSinKVvERAHsD+JqX6u1pAFYqpX6BSoH+sNZ6x5BkpiRLBLPNk7xuyzTe+81bMN0vBu6TFMpOpkw/j26ecra1jdkfp/TZJySDPNtbPcmRPMn0joggV5L5mGYbvFeUYUlhruwNo5b744rbLeLbh6FJNO6xDs9uEe4nkbTKSlK9V8pO2mZzGaVMKO0Wg3CHlavW4yX/8BNMzgx+zw9zL3KfsVTVsdTV347bXkjIw0mBvWf5CpGpRJg8yQkLhIOWTOA3n/kkfHXlGmyaDAN4ExISdi902nbQWt8MwSKhtX4fe//SyLFXA3jGXAY4KGi5O1PuQ5LetXmS3/eft+LSW9fihU9Zjl97+sED9UkP6pzZLai/QZTrtme0Uq5PdxgvZRtxjNkt/OA0Pmngbc1WSS51gxKu+TkO3qYpJtKqJAtdzoKM55lCN1folW7YH/3mUpNac3+8MpO22WSdkPzrs7Fb/OV3bse967bhzrWbcfIR+w50bFBMpElJZufGVyQ40e7kCv2SVS7U0vHhpIB+c06SJRuHhJQCLmFH4s1nHY1v3vgQvnL9arztBWIITkJCwm6Ckam4VynJCjlTdavvw4ephK3TfQDARB2UNQhMJTdmtxjGA23HFj6wS62RG9Zlv2MfG9FKMiP+Yvqa1NKZfonP/fQ+vOZTV3sWjdkpyaUWlGTzymwDQ/CXdiV5fgL3zKRIKXSyrM5uMZg6zX3DVBly2P55W05/A9gtpO2ksEvlumPH+uS4qVe7r5/dgivJmfOdFo4PlGTYDClKsFsMnN2ica+EhNnhxEOX4vRj9sMXf7bK5DJPSEjYPTFCJFlDoSLKTo5hwbsoYXu95Lx4bHCS3DckQzmEAGj2p/rIlAoe2FpXaYWAZitGDG22BUdJ5t5QoyTXgXv9Eveu24Z7122bFyW5KBtSwGmrAg7nSY4XCwGAXr8pBdzgMCQ5U+jkCr1CJqeiksy+V8qWpZ5VVhTvc6nb1VOpF34+gx7r50ke6F5E3JNMfTcWsQnuf/ubi3aL1sC9wceekDAbXHDWMXh40xQuvXXtQg8lISFhDhgdkgxrt5AD95qJAJFkKjk8CEgR7eSKqVOhwtUG30cNuEqy/1AfxIrQZluQvJ+AJT+dvOp7pl8apdIJ3JuDkhyQGDbBmEsxkXie5Dh5mk0/ld0iQ78sRXIqWSgcJVmBVdwbuPvomB0FPgJpM/2GnTaSLPmEW4Ilq3HZ43tiWWrNPMnhOO3x4XgKYfKb8iQn7Cp4yVMPxFH7L8bnUnGRhITdGqNDknWV3cK3W3Bi0oTJmcpuMYxHlB7UnSxrTGEFVHYOX+m0yqK0hG4VPkMcTSng9jG2EYE28kzHTfeLiuB5CvAggYkSGpVk8AnBcG1Wx8gHka96znYL5knuZKrKkyzsR21y8sn7cTzJs3iAioFsrSQ53E6TvLYJpJSzWwvbwk7ti1xMJLzHnSsamTBpNnbXkzzYBGs2E7GEhGGQZQpvPuto/GLNRtywOhUXSUjYXTF4X9o5AAAgAElEQVQ6JBmUNaCyW/ikoM0jTEryUAFjTInzSSkngttn+jjx/T/A337fyZzHyhRH7BauJXkoJbltSTnWhH/cTFGiKBFcU35+26b7g6W8q9XoGKHz8/0OSiBtCrgISe4NZrdo669kxKybZ+gVWjyGvvFtDJIneVbppoUJlZhCjV9LoRmyzLRf57Ad2138WE54yfLC+3OUZDMBlHr12tVo8SRHh+RsTxw5YUfid55zGJYu6uLCq+5f6KEkJCTMEqNDkjWRD/sZ4IF0zcdPzsTVxhhMCrg8CwKa+IN682SlUvPqdXyMmQof2KXg1xzGS9m+/B4jqu7YYnYLrgye//kV+JWP/KR1THwCIRE4brfg+7ehbCBHRamby1I7qw7N/RC57xhPcilf5/q7MS8gju6XLLOe9flQkv3fRtpP6mZYUlm14xLrJteNZvdSrwivs9ZVZhg6ByAWuOdPClieZPaHTe02eZJjNqOEhPnG4rEOfu/UI/D9W9dizfrtCz2chISEWWCESLKuslsoz6JgiGgzS95GdouhPKoVQ+g6nmQ7HrMfLdN7Y6A9qrHpYFteG1ftErdLlpvge0dj28PvrcoHVIF7hSHJ9iCeoWLlA4MtJ7oBgmLvbtDlkEqytD9PYddUpS223enHkNzablFG8iTXr+TrJsz06X7J5r3injR2KbUfhwl4bJ1Q8b7c75rsSZq98rzavGqfn91CujF9Iq5h//ZEu0XDRXUIPzT+8ju34we3peCqhB2D8593JDKl8MWrVy30UBISEmaBESLJFBBVPTQLj+y1pZuYzRIskbM8U4HqJWbY8ORsM7SokuyOTVL0YhimmIh7nPs606+IoJ+LtzeL1EactPHAP66UD6Ps2v100D6BF0ORbB5uf4Nd0zxit/AP972+PGWgXfGYu5KsI+20qfJ0/7Zmg/DyW/Dvmg7VjPj2+uFYpMA96Xj/XtWa50mu/P7v/ebN2DzVax2Tf02+ccOD+O+71sUPSEiYAw5ZugiveOYh+Mr1a7BlKhUXSUjY3TA6JBkwZakBd6kXGCxvMdBOzDgZ6QsV97ilwOxX2v3cMVuSLQbueQUnyiEIpBgI5WyPHRkqyaWuyGCb6joocQdcdZArjr7SNwistzbcNsWLoYjZLVh/Ld0F2S2KUjwmZhOwSrIN3JuNkhxW3NPiBKBNSS4arhuH81t5k6hBOb6UAk4O3GN9we7njEfzPMkKn/vpfbhkxRr86I7HADQHMfpBvUWhzXVISNgRuOCso7F1uo+vXL9moYeSkJAwJEaHJGttUsABPLuDVZwGQZua6GZ4sMqgIQ1CO2ZpOKIkSyngtNYsTzKc10HG6ZOZ2PbY93TdpnsFylIHSrKUJ7nV0xtRknnfs/EkN2W3mOm3KMns/cBKcu1J7peR7BaQFVpzv+QZq843PEEL75WIH7ul7d6gdgshcM+/T+Tj7KuUi1trbSwpvu/eeS+cr81uAUz23LLaTefjtI+qHb+4TULCfOKZhy3DqUfthy+k4iIJCbsdRockw80a4AclDa4kNz8w+QPVpIDLVfCQ583M1EvNvieZoKTsFoAQuNesDErnEQ/Qa1aYaTMpyU2Be4S2ZXvOiyV10s/3O6gn2Vprwm08YKytLHXrb18wJTnLooF7MXvtjJlUsRUPbyxf/Nn9rcuyUnENMbuFcI056PdqTx8Xvh+kah2/Ds7kh407N55k2ibt5/bCK+5lSjmWGqDNk8wnYVU7s835nZAwKN581tF4aOMkfnj7ows9lISEhCEwOiRZu57k0nuYt+WC5e00oe+oodX7bp6xh29IPIhQ+inBuMotBWP5xURElS2C2aa58r2mM/0qBZzWsoXEH3PzmNqV5GHOkdCkJPN+5DRpbHwtXKkwv1etJBdaHGOM3E3XqvZYJwtWPADgmnufwP/5r9vx/v+8rXEcwYRKy8VEJPWWoy11nm3fvvcnX42qrXnVzqBL9nuRBakwYwnbkaxIPE/ydN/94Zoma769pl+WSUlO2OE4+4SDcMR+i1M6uISE3QyjQ5KhkSkVBETR86+JIruFDtrUxJK9t8qiT0qlIDdfzaZdeOAf30aEX/IktxFI6Ri3/WYl2XiSi1IkRLMhyZy08fe81PGslOQGssdtIbLdYvD+iERXdosMvbK5LLVP1sj60clkTzLl6t44ObyS3Jbd4qY1G3HzgxvFcbb68BvtFu6+L/v4lTj3c9cFG6U0blq3eJIjanU1YbN/hzMeSR40cK+obUTJk5ywo5FnCm868yj8/IENuDEVF0lI2G0wMiS51ABUaLegB22TkLxtuu+20wBXSQ5TwEk2B1pmzyJXO1Nh4F6prT2D+MAwnmRTnCG2PbLBt3bM1CngAHcyIXnr2u0WLkGxfdq+3XNsbM6OxWRpiG/z+/f7rvprmSCVdkWgm6l44F4kVR9PAacEJZneta15hH3KijZv+6+/dwf+7vu/FNtrsidU7fC+3cmnf83uXLsFV93zeD0qO17/OtNnyiU9qPpbtcvzJLtp/qQxOceye4TaSEpyws7Aa045HPtMdJKanJCwG2FkSDJ0nd0iYrcwqpnwQNzqkOQ2JZkrxKQkZ4YR0FZO2Mx+kTzJXIk227RmSrL9btBxti2H09c0JBtIRu3bfek9P3cxcG9AuwIQJ0WzqbjHl++DPtmgJCV5GFIeBO7FFMgIgbSBe9aT7FZhbp/QAeE9XGr53Pg1nilKM1kL9hvwXnK+Y+p//Dj76tsc6NpQ4J5kt7AOJl+FtuqvZLcotcbF163G11aG2QT4b2KLzCRPcsKOx97jVXGRS29di4c2Ti70cBISEgbAyJBkDTe7BT3YiReYzwIJ2zZd8IYawT2uppiIkALOsVv027JbyL5Lmyc5JIHthM7tI9zuEvdOg7WjNEotOyeBWAyTGcRRkulVu4RsQI7c6K11A/fiZA8YZPzVa67IbhEJ3KP9tcbZJxyE804/EgC3W8ieZPuumSX7Pbop0dzvzTFavj60bdD+/EnUIBMZjfA60ye/mIhr7XBVa348/eYKED3JX/v5GnzzBrfCJe8HsBO9pCQn7Cyc/7yjAAAXpeIiCQm7BUaHJGsvu4UhyZTmCvXnkGhStb3qu8GVZF5MxFeuJU9yqCTXalimAubDA/eMSi2ojjG0ZbegbzNDkmuyIlg7aDlbyuzBMUyWhBhhdTJgDKoka5lM+f00+YcH6c/Jk5w1BO5pm1f6hEOW4NjlewGw90GXKcm+wgoMoCQLEyqjQvP9vHnMIOq9BLeEuHszNh1qbSfuddKsz65RktnGlvFpMOUZVZpCd7zVxK5t5YB+j+iKQELCPOPQZYvw8hMPxiXXrXZWMBMSEnZNjA5JRkUuco98+CWLJVLEH5JDeZKpmAjPbkFkmREU60mOBO4JnmQNnqkjJN5t/NGSdnk75ZUmVtVpsHaQ0s6XpXtinuTZKcmcd/EW+qXGRy+7C09snW5sl34HqX/uo24rgNL223OSnCkFXf/nnYbTVqasBWi6sJ5ku5+j0wJo9yRLyiovrmHG612P2Pm1VtzjfzPeJKopMDQ2qePbOp4nWVKt/S60dv+uJSW50HKpbj6OnrFbJJKcsPNwwVlHY8t0X7QDJSQk7FoYHZKsq+wWJiCKHqLmYVp9L6XFalMbObjdggfueRzZU5JrctXoSfYJDUsBZ87R3d6EQTzJjCMjz+OEnIo1uBMEwW4xjCdZUCe5VxUAfnr34/iny+/G+1pSovk5sTn6rXmS7fu2ADb6vTu5AlR1vtIh/DwyZYkr5fPlnmSXSFavrdkKGxRSfqhP/tr86dHuhPex4ETCZK/w7lf3vVGSvYmg1K+UGYMmbFoLdgutUZSx1QqLnrFbJE9yws7DyUfsi+ccuS8+/7P70wQtIWEXx8iQ5FK7dgujEnt+1VhWBftdC1ESgteqintuP6Ldwi/7R0vkKgzcg25WkgcNMoudjk2Z53qSbdCZ3XdyhpRkdu6zsFuUZXjt3DG51397bYOJBZz5/TblApYmItUx/H3z+IlUdbMMCiHJJWjWVpYpQ1yN3SLLTKYT18pQQbVoyZKSbPOBx88ndn7tmVLCvw9ppYKfy5apvus154q7dgvxAG42Gj+I1IcG/7sGZvq+3UJXdgvhHnU9yXumkqyUypVSNyqlviNsG1dKfUUpdY9S6jql1FE7f4SjjwvOOhpr1k/islRcJCFhl8bIkGQAgFIs72r1lV9ogj/wjZI8BPmMVdzzfbH8Ad+L2S0AE2zok3OxmAg/tk1J9vZ7fOs0Nm23+XdLXfVNhMTPVysV/mjPbtE8plhxCz5ZcbzQtfI6ljffpuY3FvqnsY93sgEC9xq7MeplpQTTOYTnVGrrreYTkZl+iawueDMbT/KL/v4nuGTFatGTLE8A3O9ic41Wu4Xw3jbNJ4OcJPdsOkH4qf0sgR3reIF72k50Y7m+OcmW7RaI2i0cT3L9QbIOjTjeCeCOyLYLAGzQWh8H4GMA/nanjWoPwq+ecBAO23cRPp/SwSUk7NIYCZLMg5b8/LPWu1h9XwjkbCi7BWMa9JDt8GIiVHGPp9/qU+CeP26rfvvPc42wLPVQSrLn8TzlQz/Csz90mdu3spolBe6FJMiSCFdJnlt2C0mJ5goswMo4+xcu0m5T4N5YJxPHN4yFpVfY31spIqfseOE+yjN7T/aK0nhwlbcfYO8diST3ihL3P74N9z++TVCSLQHmKnRgt4gG7olf23EJLNlXlAE3X/HmKTcoybeVmN8lzwHYSZeGzVAj9l/33eRJLpuUZPYdZZ3Zk5RkpdRhAF4B4HORXV4J4KL6/dcBvEQNWq40YWB08gxvfN5RWLFqfVDkJyEhYdfBiJDk6lWxYiJ+2rImJXkY8in5cjuMzFJToifZqyZSpa1TYgq4Uod5kofJ/CCNw7WaaMeeYgm53W7HX3Xcaw3caxxS1P/LRUmXdNmUaU1oKktN44wqyQNYbW55cBO2Tvft751nNnCP2QNsukHuSbZVIGeK0nhwlXJ/WzoOCO0WazdN4YEntgOoMjlouON0ylJzu4U3j5HS+jWdt7Tdz4/Mt3GyuoWRZK39AEmbeYKUZPpMkzd674+7+mzzJJc6zG5B7cdKnxPMCsme5Un+OIA/AxA76UMBrAEArXUfwCYA+++coe1ZeO1zD8fe46m4SELCrozRIMn1q0LcbmFUY4EQ82XooTzJxu+amT7sg90eYz3J3rhrJRmSkqyt8mxUYcG/GkNTWjQ6XspuIR3Xl5TkWVTca1eStUjOuwMqydJvRxaJsajdgu8btl2WGr/5iatw3oXXmTF38kpJLjUpn2HwGfGuSq23dotuTQr9fN7OWLzTPf1vLsdLP/rfACoiGiqr9h5xAvd8u4Wg0kv7+eBbjRdZUO9nHJLcc2xCfhd0fcZyd1KroQNHdnC+0E6sge9ZL2oVWbodpcnrnqIkK6V+A8BjWuufz1N7b1NKrVRKrVy3bt18NLlHYZ+JLl773MPx3ZsfwSObUnGRhIRdEaNBko1qZ8kHPfh8ldFXU8Pvmvtyi4lop8+qSEKoavZaAs8kT7LWTN0Ftc+8wy0Pdskb7bZfZwOpP4d9hYRKmiC4fbaQ5BYl0/fWmuIbLSTZlKVuVJLzWdktqO0bV28077tZBkAZckrecU7WbbEWa5+Y7pdGFZeItZTr2MdMvxQq7llltilwj3/kxHLQIFDehlGS2X5cSd461XfyJLurNXa8lA6P/3bKm0AEw9Nu/vPQblH9Lm2WoD2wmMiZAM5RSq0C8O8AXqyU+rK3z0MADgcApVQHwFIAT0iNaa0/o7U+RWt9yvLly3fcqEcYb3zeUSi1xkVXP7DQQ0lISBAwEiSZnnGKpduynuTS+eySItTfuQ/wJrhlqcsqZy5Tr6UlYlv+1m2L1NxMqVBpq0msM06EwXwxtBUTKWsVm8ZOXlnJa0qQ0t9JfUbHFFGSOeHizZLHtdsauOcWjHHGXNjgvzZi3xb81mceaZrY0G/I99XsPQ/Sm+mXRhW3qQqrV+2QxGrj11auwRu/sMIZz3S/DEijBr+X2di985V88n7fEvhmfxLo2i2s7WHLVN87zm2PxkbKul0tYYF7wt9S9dnNkxyq1BUJl1wU0grPnlJMRGv9Xq31YVrrowC8DsCPtdbnert9G8D59fvfqffZMy7QAuDw/RbjZScejIuvewDbUnGRhIRdDiNBkm3AkwpIpB/UJQXpzTa7Rb/wcjNr7ai+hF4/VJdpjAqyJ1lrq6By4upnoYghRjD4dq4kdxo8yeZ8vXP34S9b94sSX71+TaDqA7LqG1OS20lyfEJAYx7vZqL67pO3WNuAELhXN+CrwlqDKbvKC9yrPtgMDhVe/o8/xed+en91TP3du79+M37yS3cZe7oflsKmQDX/fJryJLcVWeGQrpH/Cgh2C3a8Pxmh32XMKyYChNdGslvQ/jN9ebJWlPJEzrHzGE/yns0BlVIfVEqdU3+8EMD+Sql7ALwLwHsWbmR7Bi446xhsnurjGzc8uNBDSUhI8NBZ6AHMB/hDlGK8iKQEeZIF5XC2xUSKUpvqazQOKUAwVtlLo5JzlVBxjyvJnLj63uEY2oo9lNr2DTSngDPn66jogurqcZKvrFyDv/jWrdgy3ccFZx3tTBx4W3YJX0Nr6xewJHnA7BaCcki//0Qnx5bpXrCdn2ZrXt2yWjlQdVo3yk9M185WjbPqZs5TwBVlbdUI1efV67eb37Ypl8B0vwhJo+YEPT4R4fckJ5dD2S1Mn+F9wm0Pm6f6diwaDtPWsNeqawL3bLv++UsTSPpdt8+4QXtVW7W9owwvpDR5LfaswD0AgNb6JwB+Ur9/H/t+CsBrFmZUeyaec+S+OOnwZfj8Vffj3NOODFKFJiQkLBxGQkkmONktPAJsyukK/mPJFxqDnwaNq7F86VdS7QLiqSkFXEhUNLhP2CrJfsaLGEyAVcOOPPOCryTLdgtGwAawW9C1uuexrcH2gZRkkz2kRUluIvakWHaywO5y/+Pb8DgreS3aLdgxmyZ7lsjW49VAkN2iKJndgnmSe30dKslsMkfXt+kRWQXu+aRRs2sgj93f5pLklruJE1xmKak+y21OzhRRJZl7lMPAPXv+klpN+zSp35T+re0e7SclOWEXwQVnHY1VT2zH5Xc+ttBDSUhIYBgpJZkXbvCD8mRrBe0D9l1zXz0veC1T4fKw385MVEm2xN5/oGtt/cd//b07ce196x3Vso3MW3+svF9pFDvyJA+gJJP6lyuTI9rZ7h2z11h1exERdewWEesD/5qKibRlaW3Kk0wTlLGOa7d4bPMUXvT3P3H2lY7n5/Tghklj/VBKgVL++RMznqO3suNYJXnxWO6ck2MDqt83paWd6ZfihEqyyQSTFmEC4h8jwa+Wx9t2lWSr6hYeKeY9aM3vpTAFXOZNDqVCO03ElgL3tHAZpcnrnpLdImHXxctPPBiHLluEC6+6D2efcNBCDychIaHGSCjJ9ODjeX+JAwTFRCQleQi7BSezvULXy+/hsa7dwiXstn9de5LlFHB82e3Hdz7mZLxoeq7zfM2x06ndFmbsvlorHcd9pFIKOJ/MbK/z1xqS7Kh4Qgdae6SrVuBbSExTnmTKQNLNldP/Ry+7q3X8vG2gIskdFnhHyrdvVSnY9edlqad7hQmQ5FYarStSTcS1TUkOrDmMlJcamOoV+JtL7wgCgfj5cdW3JfmKo0jzTCSAOzGkSU3VpnY2lt57v+Iev86+khz8/LrZIlHUgXtyXmz7nv4u95TAvYRdF508w/nPOxLX3rcetz60aaGHk5CQUGMkSDI94pSynmTfbmE8lEJBDinFVQxcSS5qj6ohPJFiH1TZSwzcq5fjJU9y7imKpZY9yd+68UF8+xcPi+cQDdyDduwWgSdZUopZzmE5cM/9vL0maUSSncmI6P91idxkTbKb8vjyUtaxYiKdrCr+wfu8d91WYfzSmDhJ3m4mEwrKWG39tIOl5koymCdZs2Iitn2/2yYlebpfiEoyH+e/Xbcan/7v+/CJK+6Jnt8wdgstvKemHOLNboB+qR27hV+0hfrseoF7VGCH9xUUT4FuJLY06WjLZmKyW+yBnuSEXQ+vfe4R2Hu8g7/41i3YmjJdJCTsEhgNkmyUZBUsffNUUUCkLLX3AG+CW1DDVswDgA9//w6zTUo1JdotQIF7CLb5ARyOksye6//jK7/AH19yI+u7nfSXRNDJbjGAJ5kmCGMdOZ2af37b6qCqdVumDXEhiMVEPMI4WR/fpHS6QZfh9n6dUSLPXCVZJsTh8fxaTvVKc51MRhIdph0sS0sCnYp7/UL0JPskrcleUpFbf0IlT0A2TfaC/Qg9J3CyhSSL95P2PlsleVE3r+wOTHV2J26IZregFY7YuKvza7ZIUOCenNLPvu+zqn1tqxUJCTsaSxd18bHXnoRbH96Mt160ElO9MCg1ISFh52I0SHL96palRv1qH4SAnMmibCFaHD1PLcuVzZN8yYo1Zpvk//TFL10TLKmYSFnqoEKftLQvwV3ajijJ2k1P5ts4JC8zXyJvS68FAJMzlRoy1Svx+NYZL6hMIMneeI2S3KD09YXf09+eZ1VqwDZC3Wa3ACDbLbxiIoW27WcZTwGnraeZjcPvo9Vu4efb9iYXlA3Ef8jya75281T0HH3wzUZJpr8vwZO8eCxHvywdJdjxNcOmrCO7Es8xTasrMCTbHd8Nqzdg5QMbGsfrK8mrn9iOX/vYlVi3xQZqOmnw2paQEhJ2As4+4SD8/WueiWvuewJ/dMmNoq0tISFh52E0SDJ7vuWe3cJXkt3Apup1mBRwfmlmSgkWjslVnIGQGFIJXkon5m5DYLfQ2v3u5gc34gs/u998Puo938Xfff/OQEmWyF/lh7aTCpOTuSF1nKk4l2diCjifaGxj6bk2T/W8zALSmFxCNqySLOdJLtHNM2SZavWei0qy13eXeYopL7a1W9i23ewWNnDPKtFcSfZIclMKuF4h2nb4tSff86SXHo0f97N7HmfWnXh/dQ+sLyKz2ttiPeSLxnIUpaceO75m+7v5E5jqGM9u4Y3vn39yb+NoKyW/vp/qdu96dAt++egWrHp8m9lvmFzRCQk7C7998mH4wDlPx2W3P4o/+/rNaZUjIWEBMRIkmZ6mPJOA9SRXD0JjrRCIEud7bf8eOQU1So0sk5W/gfIkawCKylIj2ObbLXwl+ZxP/Awf+K/bnX18AsELL/jtczKWZ5n5ntr3YTzJkcA9v5vtzFc33Svbi4lA9iQ3TVzcUtfC9rLycedKiZlNYm3FviNiCWXHa7IxCHaLnKUILErtkOxqHG7O6KrphuwWRVhxjxcTAayFYdJTkm0GCY2r7nkcZz35AHN8ExwlWbvf8W1Ekvca66AoS2fCxZXkUtuUdVm9EmOvs65tQM33YhOcuAEzWa7G1ivdlSD+fqpX4Gsr17TaTxISdjTOf95ReNfZx+ObNz6ED37n9nRPJiQsEEaCJJvsFkKe5GZPcqjwtv1jxAOeqhRw1gftjsm+j+ZJriF5kqXAPceT3DDM0iOOsQAmXi2w4xM9SZWtycd4J2P5p3lf7jG80MNM4WZlKIoS///ld2PtJrvszwO6AG63aCDJAiHiqAL3aiLGSLSkTlPf37n5Ybz3m7eIfXdyG7iHWvk2GVXMxMyWRFbe/WHsFmzFYygluV8KKxLu79XtyPcIfV71xHas2zKNs44bjCTzzcFKg2O3qE56Yix3Jp7+CkGl8FbvSUnm50Q+fUmtHgSSQkzXmAf8OX/LRYnL73gM7/76zSavd0LCQuKPXnwcLjjraHzx6lX42I/uXujhJCTskRgJkkyPPQVrR6CHsJ8ezM1uUe/TojByOJ7koqwrqoX7uUUx4sSzsjxUKu0H/us23PXoluqcIkqyn8/YR565hFtDDmDSpm9ljqN+q/bDtiW7RVMwnEOS+66SfOvDm/EPl93lBBz6/RKJeWD9drz9Syvx3ZsfCcbUFnRZBe5lyDM/oFFS16vv/vDiG3HJitXifl0TeGfbC7NbuNt4dj3jaTZ9Cp7kBpKsNTDtMfwqm4b9LE3agDCg75Cli8x4myDlSeal0gkz/RJjnQydrMr77ZBr9p4ryXmm6v3d/fgZSMvNpx61H/70V48Xx+tXxQQsOe4XspL8g9vWmiwsvgKfkLAQUErhf73iaXjNcw7DP11+Ny686v6FHlJCwh6HESkmQkpymLPYLzQhZbdwlNcWwuAH7mVZTEnWwTFinuRaaXxsyzS+8LNV+MLPVuGev3o5SuMZ5kvbVl2OKd7dXAXnE/P/8uvVzV31sSmIbayTYfNULzgnn4xvm+ljopthqlcGqcsoqIwTEq1l8n/dfU9gul/iB7c9ihc/9WVYVBfkCPsPDkWf2y243UNS1z11mfLtctBkQjkk2ctuwRTxPFOOfYJSyNkCNGEfRBGlSoxA5Uvm0Fr2wAfnV+9D154KmwxltwheuZJcYDzPqkwipXYItZMhA0BZX+zcqPzWFkWBe/b4cExZFk+VN9MPVxfob3CGXRv+t/w/v3ELnnX4suD7hISFhFIKf/OqZ2DLVB9/+Z3bsc9EB797yuELPayEhD0Go6UkC3aLME9ySOqkUtUx+HaLnBFNDtluEY7bVNxjD++f3v14vc0l4BqsGllknN08c/I1a60Dzyt9T95PQCiBLfCEvpAnuSm7xORMgX0XjwGoK8UJPm2ulmto8bx4/l1+/YH2oMt+UanvgwTu+WR161Q/INPdjAfuVd/ZstT2njPZLbz7o+ulgBM9yQrOPj6meu414GWp+Th8lDWZJpJMk422wCCpip+0MjPdLzHezYIJCeBaJjRTvnPlpuerglm9wD1hTDGbE+ApyV4hH64k+8Gn6+qMH9P9RJITdh108gz/+Hsn4flPPgDv+cbN+P6t4YpaQkLCjsFokGS2ROt7dpsq7lyqBxQAACAASURBVMllqZsJA1ei+kXpEHMO3g8VE+HffeQHd+Jfr3mgogPKDSha9cQ2Q2L982zLSDCWZ0F2C1FJhks0cuW221SWeiy3eZIdT7DHLbbN9LGMkWQnLV59TXJ2jpVXVVa9Cd+/7RF8+doHgjHxsXNUSnImBO6F+/rq+eapXjBZ8O0SgEt4q1cbLKmUq3jyFHK0b5AnmdqVfDwI7QC+57epZLPWlgRaJTm6u3g8P4YfOtMvMd7J0clJSdZmH/+eNMVWMirLbrdR/m5ppYfA80/78Fd7AJgy6vza+Irx+u0z5jwSEnYljHdyfOrc5+BZhy/DH19yE666+/GFHlJCwh6B0SDJIEKiAn+o70meazER/mAtNS0XC2MSPMmc0H3yinvZmF3l7eGNk5UnWYUEoS1P8ljHJcnxogo1CSfVkvL5Cl5TglNMRKhW5vezfbrAvou79TVwPcnUVu4oye2E7X9+4xb8r/+41XzmpCeWAs4UE2lTkutTmehWP+jmqV6Y3YJyDDLia/3c9l6j91VFRnu8CdxTNOa48usHblLWiiD/sTe5aCzZzJTkwe0W8koEfwUq8j3Wycz9bJRgHQadcjtKntnxa8BMHJvSEcYmp4BrtzABvML96mdoIYU+keSEXRF7jXfwxTeeimOW74W3fWklblgdzxWekJAwPxgNksyWvf0UcH3P6tBmt2gjaf4DNI96ku37tuwWmXKXfh/aOFl7ksOsF7bintxWJ3ePafQkg9kt6lRl3JPczd3z4p7knjcJkc5v20zf2C2me252C7JQcJU15kluQpvPmFLAZZnv1Rb2rb/be7yy6m+Z6gf7UVlpTnzpHKJ2C6Y7+yngpDzJ9JFPIP7k7OPx/539ZAASSXYnQlIOa74vVcab6A6mJPNLwKvo+dtm+gXG8jpwTzNPch08yv30Jk+yCtPzKeUmwYspybEAx76Q5s0P4APi12kmeZITdlEsXdzFv15wKpbvM443fn4F7ly7eaGHlJAw0hgtkgwVzdIgRePzYL5Oi0JL8JdoY2oWJ2xTDanMekUZtPHQhkmnSAWHyWccGV83z7zqZqHnlb7ndgsKQDSeZK1NkBmByMe4oyTLxLMoNaZ6JZbVSvK0pyTTZIOfY+VJnj1Jln67XlFauwXfV8ypTDaEiiRvnuwFkxFrt7ADN6sXZiIGx27Bz9EvJvLwpimsXr/d6YOuAT/uecftj+MP3AeA4ElG+2SBUJbAVF0Zb6KbV6sVLSzZCdxjxLfaZjdy/7cUPJiZtHduMRHuF+cTXpi+QmQNSrKz2kN2i8J99feLHZ+QsKvhwH0m8OULTsPisQ7Ou3CFUyAnISFhfjEaJBkhIbH5UV0VV0r35hbpkPuY6hX45BX3YNtM4SissYc1EZ1+URoPqETipvulo4gt6uZ4qLZbSFJZG5kfy7NA+ZNKSJe1lMyDxBTLpqBZXwTRk+yQM7sv+WadwD02Lpo4ZIGSLJ5WFK2eZB64pwfLA71XrSRvFgL3bHYK+52v7jt2C6U8T7Jblvri61bjT7/6C6cPbkUgjOU5xjqy3cLPL91rsFuUzG4x0c3rlIHNF93PTFGdq/sZqP62KKVbVY3QEl8377d2SLIfuOf3JU2cfE8y/1PhRNjep6HdIkaGk90iYVfH4fstxpffcir6RYlzL7zOyTefkJAwfxgNkmyU5Hh2C3o2SoVDytIW1ogpmdfdvx4f+cEvseL+JzDesSnI2uwW2xmhkRS+mb6rJB++3yI8vrUKIJKUZPKlxtTCQT3J0K4VIK9TanEFvuPZLXieZDG7BXtP1faMktwvnO1E1PxLNxe7hexJ1qZgBd+fH0fXmfreq/bqbmGeZFN0xQu8q95795xmFeUyL7tFFh6/hVUmrNqpXjlJ7nYUxjtyJT1/ciGtHJhtWhsleqKTiYVsfPDNpUdm/XSDVEGvX3K7BZy836W2k9VMeX5xzQP3tNnfh6r7IZCNBZCLifRMnmRX+ZaQSHLC7oDjDtwHF735VGzc3sN5F16HDdtmFnpICQkjh9EgyfWrUmGKtKDinkOqUH9nyW6MpE3WhTGmeqVR9IB4lD2RgO3T1XG8Sp0Pfvzh+y522vaxZFHHGbuPbp4Fy+OxinsKnpIMV2nt5O7tQePvdjKjyBWRwD0qJLKvkN2ikylD9FwlWbcSNh9tGSv6ZYlunhlyWwjEi86TrtNiUpIn+2Zytaj275KSzNXhoJiI40l2z9EoyQ2eWltBkivJGcbrMUieZH5vNWa3KKvj80yhk2dicGhwjA7/ZkwX7NBqsom6OIgbuMdXa7S2EyqawJhUjZDyJEtKsnt9xltIsilLzdMJRpTklAIuYXfBMw9bhs++4RQ8sH473viFFdjqTbgTEhLmhtEgyYxUWH+oRsnVLG2/J5hgPq1rMhMnn9N9S0zGGUnOM3c53bZdNbRtpvpHa5+JbpQAchK1315j5r3EofaZ6Drt++jmYT5gSTHT2u3XKOlmuzaqJ6FXVNepmyn0Ch0EnTkp3gobHNbJFK68ax3+6fK76zFmZtLhj2l4T3I8uwZQ2y2Y2m9WFNi+dJ4mLVl92punemY/CnLrCkpy7inJPHtDLE8ybZMgXYJunpn7zr+PtHc+3FKwmBVeAarfaLpfYqJuyy8J3TYe7X3n+9A56XVtP+514hX3uJKsdRWwqgDcsHoDNmybEf9u/Mlpl/1N9gt3TAArSz3AZCIF7iXsTjjj2P3xz69/Nm59eDPeetHKYBKdkJAwe4wISa5eFeQHMX0G5OwW9HBvWnrm//AQYQLiSjJ1TUrykokOilKjV5T4+QPrnX1V5GEv5cldUpPkmCrdDTzJkepy2s3DTEq6taBAUJIrawh9X3ptSxk9OrnCWCfDDas3GnV5rJOZJX8nuwVmY7eg85bJXr/UdQq4en/Bk9zxLCz0yu0WRkkWA/eE7BaGcMsp4Ko2ZEirHmOdzFnB8Pd3SbJ9f/CSiWDfqV5h7uGs4Z4nOJvJbmFsFxaF1p59wsrN3G6h4SrJ/goO3RI3rN6I8z5/nTimLHN/gzF2XWckJdkE7oXbfPT6w92DCQkLjZeecBD+4TXPwrX3P4E/vPjGFHyakDBPGC2SzIKkQgJXEw+BOFPAUdbgSeYZBcY9Iit6kktfSe6g0BqX3f4oXv0v1zj7+svqTSC7RYzYSMVEwrLHoSKYKTgp4Cq7RehJJhIEVITDzRgRKnjdXDnXq/ousyng+Jj04IF7l9/xKG5/eLNRTX2bCR9HJ8tEImvH426j61fZLUhJrs5BCtyzhUFgjrd2C+WcZWcAJZmO5Xl8x5iSHEC72Tq4J/kgnySXlSeZSLIawG7h3E/eq3uvaUcZ5qs4GjaDDF+ByJWnJNdtTdWWh1sfklNcqUBJth8c3zEF7BWh3SKGmSIpcQm7H37r5EPxwXOejh/d8Sj+7Os3t64QJSQktKOz0AOYD5jsFmBBWKX20pPZ781x7DtKhxYjDFxJduwWCqK31HiSa5K890QHWmtsmuwF+/KHve939kF2ixiZV0oFgVaSYkZ2i7625aEzL3Cvm4WeZKUsqex711guFhIqoPz68VOkfLqD4IKLVgIAvnTBqQBgMir4qFLAqSADBd+ViK+fsWPLdM+8pxLORPR8FR5wFWBjt8iAjPEyR0lu8SQ7RL6TYayMK8l8IsR/k+X7jHv7VingxmvSP1h2i3BsJoCPbStKjfEOI8l0PGxQnxkvKe11CripXokHN2w3+bt59gvp/vXLUo9FPMkmX7qnKDchBe4l7K4474yjsGmyh7//4V1YMtHB/znn6aIdMCEhYTC0KslKqQml1Aql1C+UUrcppT4g7DOulPqKUuoepdR1Sqmj2Lb31t//Uin1a/M7/ApWSXYJC1fU5DzJ9oFvSbLch6sk8+wWmUhmaUzbarvFPuNdFKUWH8D8eO5Zlf5tWzLRrCT7S+8aMjGgstcmk0KVUoCRH0FJLijFV00qfSWZE7WaqHQzFZBkUmWBUEke0m1hyE+V1cPd9up/uRoPbpg0FfcAO3nh4+74SnL9E22atJ5ka7dwi4Hw91yJdu0WLMCsZRIEsPSB3G6RZ0HeakKp3UkT9yRTdhFCoTWmewUmOtxu0UKSnbHJr1XbFemlYiL8fEhlpmOcinsKuOa+J3DW316B7TOFmzJP8jKBAvfsZze7Bb8PXXLci/3hMCSSnLA74x0vOg5vff7RuOiaB/Cxy+5a6OEkJOzWGERJngbwYq31VqVUF8BVSqlLtdbXsn0uALBBa32cUup1AP4WwGuVUicAeB2ApwN4EoAfKaWO11rP63omPfYUi3gvtW+tqF4lDy1lt2haep7igXuM5FWp08L9qZ3tzG5RavkB7JJkTqLCdpcsqj3JkXHyHL30WSwrXPfrlk9WznF0Tejwfu1JdpTkwu2LwJVAPqkAbLEOYO6e5Ec2VvlBKy+2e+zPH9hQj8FOZHguY0LX9yTX2yqSXO3jB+5xGLuFadtN4xYjc1Elub5FfEuIX6aa4PvO+fuli1ySTHYLuoeVUmhzIIh5kj1vMrWdq4ooF4WGYj97qd2y1NQn2S0I22f6zsSpm2ditommyYfjO9ZEjl3bRRNS4F7C7gylFP7815+GzZN9/NOP78GSRV285fnHLPSwEhJ2S7QqybrC1vpjt/7fZzKvBHBR/f7rAF6iKvbzSgD/rrWe1lrfD+AeAKfOy8jdMQJwg6TKUovEQSpNXGpdBwLFlcyY3aITUZKpP6MkT9h8wT6iJEoI7aLAvV5E7eIlkYGKkHBFkqd449ksKCUZz9CglEvMyG6RGyXZv8a2X1LsOnkW+KwXscBHhxRHCH0TblpTEeFuHk+x1xWUZN4PbfPtFhu3W7vFREMKOEmlttktmiZBMU+ymzmkm1de+1wg6LS/GzTZQJK1xnTfKsl51p5RxPuJvPbctnOmJHO7hYa9z3nxkyxzr8NUr3CWF/KIklyle7SfOy0p4KSy1DHMpMC9hN0cSin89auegV9/xsH40HfvwFevX7PQQ0pI2C0xUOCeUipXSt0E4DEAl2mt/ZDzQwGsAQCtdR/AJgD78+9rPFh/N6/gD2put5AIsVR8oiiramBZpqKEIW63iBcTue6+J/DD29cCqJRkICwEAQxntyBv7JRAtqt+dUA83TRp5mtnUpAreJ5k7RRnAWzgHtkTekXpLO2LdotcsFuwtGTO7wH3twTigYxU8OOWOrCrk1e2gS9f+wD+8OIbnH2pLDXvj/eT1TYdX2XeMtU3yr+f3YJzN2u3sNdBynUMuJ7zuCfZHR9dg5iSzMfMzxEAzjzugKDtKnDP2kaGC9xzrxGvkFdNooRiIroO6lPWbkFj7GSZY+vZPlM4U8O43UJF/27EFHDGbuFOLqX2k5KcMArIM4WPvfYkPP/JB+A937wZ37vlkYUeUkLCboeBSLLWutBanwTgMACnKqVOnO+BKKXeppRaqZRauW7duiGPtoTEBgf5kffVaxH5jgLXYpbF6VjgXhZLAafx2s9ci2vvW18Xgqirpc1IS8f2fZvSSESDk3aOsnQzHZRaJg2VkuzaLRTcgKzMU+sKypPM7Bax7BYme4HgSV7E7CrcfkJkioNbWziozS1TVSAkleO++cGNuO5+N8VevyxttTfByqBQKc1+4RkA2LB9ph6zlyeZtU+/H6/gaDy3DWSuyZPMJx+UFjBiSQ6CM+k87v3rX8eTli5y9i3KMAVcu92Cf3C/C5RkVZelLq3tp/Ik2wmshlXJfaXdn0T6vniCX0xkjE1cxRRwxm7h3V9CxpCZyAQ0IWF3w3gnx6fPew5OPmJfvPPfb8SVdw37bE1I2LMxVAo4rfVGAFcAeJm36SEAhwOAUqoDYCmAJ/j3NQ6rv5Pa/ozW+hSt9SnLly8fZlhOnmSbjssjcIzA2O9gvstrq8bwnmS5mAjve9FYbshtu5LcrDTSd7GE8UUQuCcr6tSWa7dQbIm8TvcmKckZeXhLMYMIYMlIN88CBZR7kh0rCDQ8oc+o9j6ZocPIr0pKsh/EBgCPbp5y8iT725WiKnFURdBue6Iu9UoKPllNeA5royTTyoS251GVaebn064kF96926Yka095pvPIFKAyf1+NqX7hpIBrtVuw923ZLaiYSL/UbsBffT9Vx1JGmdrS43iS/cA9+Z+oIP90RHH2lWTfkzzezYNjUuBewihh8VgHnz//uTh2+d54+5d+HuTpT0hIiGOQ7BbLlVLL6veLAJwN4E5vt28DOL9+/zsAfqyrJ++3Abyuzn5xNIAnA1gxX4MnWKLHiomU2lNUyVoRfmeD1GaR3ULJ5YVdj2hpiIBIbtnxbSngqJ1Y6VxeZbD6DI/IcqXYLrXbID1tjsvqpXMCLafTEnXP9yRzu0V98SuF1h0rL8bC/aNah5MUsgWEJLnajxR+ypMspbx7dPO0k4HC307ea6Mkl9r0u35rRZJpYiQryaGVg65FlTRkWE+y+5vRMTF/Ls+mAVS/CwWx+kcUWjt2i8FSwIWrL/bV3cY96ybdHqr7iWe3oNzkgEv+J327RURJ5qtGgFx4B7C/Sc97JYhKcrJbJIwYli7u4ksXnIaDlozjTV+4Hnc8IucfT0hIcDGIknwIgCuUUjcDuB6VJ/k7SqkPKqXOqfe5EMD+Sql7ALwLwHsAQGt9G4CvArgdwPcBvGO+M1tU/VSv3Kc4mN1C49r7nsCGbT1TnCDuSY4oybnsSebtbJ8pzD4SSY7le5U4FO073aAkk4ZHuYOljB7kObaeZOV8psA+Tsx6ZYlM2SCpfuHlouZL/qQkZxlmvCVuHrjn2C2AYJJCJGaRV16Z+iJCQ0VU/IBNoFKSrfqtgz4Uqt+R2iy0xv57VfmFjZLsBe7BIWjumLjVwLfjtGUvAap7h6cvpImTX0DD7I/qtxljWTqIePqrHGVZ3YPjTgo4eRx2PG5f/Dt+KBFfOkVzb+g6mwqPFyitspw5SnLf+XtoSgHHN8VUdhuMKWe3mBCU5FRxL2EUsXyfcXz5Ladh8VgH5124Avc/vm2hh5SQsMtjkOwWN2utT9ZaP1NrfaLW+oP19+/TWn+7fj+ltX6N1vo4rfWpWuv72PF/pbU+Vmv9FK31pTviJEyQFCx3KXR7WeqZQuN1n7kWK1atR541BzE5JJkRnU7Ek1xojcVjboAfELNb2Peu3SJsmPaN2S140BeRD+7DfOclN+K6+54wSrKxqij3/Gk7Jx/0nQncK0svKLA69p7HtmB9TS47uQoycSwas+foq9yUv5lAJMYnM77doptntRfYzZsNAO/+taewDBZl8BtnWfU7ciV5/73HAADrt01XYx4kcI/ZLQpmt+Dn4wbuxZVhGssBe4/jmYctNdskNVnrKhUf94rTb+/vXmqNaScFXDydoGm/psLVJNL9zg8YrFK6uSn1NChwD+YzWTMAl+CWGlg8Hv7d+MiUa3NqUtkBm/GjN4AneTopyQkjisP2XYwvv+VUlFrj3M9dh0c2TS70kBISdmmMWFlqOIowEeJurhwFlcDJGwVYDWS3YITNf1gTSg0cuswGTRFpmZwZIruFMA5qp8luQedolGR2zpff+Riuu399tQ9Tzv2cyJQWz1/GzpStxFeUbp5k8vu+9KNX4q++d4cZg18KmCvJjrJXq/9cPSQSM9HxSbIl84D9jfn5ZkrhHS86Fq969mFO6Wk61lonVB3AxpXkiiT7nmR+jL0mdvXC7yMIMHNS/MkoS0sw/+RXj8c/vu7koC9pfyLg/aI05Ng3XPRLjZmitCngVDyji2mfVhsyxchx9Z1r7anIeaAk1/tZu4V27RbePUbpEqtzkcfmB/zF7Ba+F9m/FzlJPuGQJTjhkCXJk5ww0jjuwH1w0ZtOxabJHs67cIURNBISEkKMBkm2rmQAMISnMGQxs55kwTcLuEvBHFO9Ake957u4nXm43DzJoZJMS/+Lx22Ams1KIeVJZiS5xZMcs2284Ywj8bRDllTlgBnxLTUC+wGVDFawy+WUyo4+UxVCfxm7IkHkSXZVWa0RlN3u5FlIklngHlf2yG7BSRMpyH6WC5/Xdeprrtn5am2X9KnN+9Ztw6W3VGn5bM5jCtyjpXmNZYvHkCnrSfbzJLtKcvXKS16b3N2ZS1MHSQHXL0vcsXazM25znpKSDI1eUZpViL5jt3D3pUkaz27hB0sG7ZtJqF154Pm2CaWmdHr+b6XN/QQAazdNYf22GXNuPsGldIlAfDLoW08i1uWgwmLf+1sg28nLnn4wvvfO5+PoA/ZK2S0SRh7POGwpLjz/FKxZvx3nf36FyRKUkJDgYjRIsre8ToowPfw7Gcv/K6TKAmolOUNQJmW7oPxyJZnnSR7LM6z68Ctw5P6L68DBqrG/fOXTjbrWarfI4iTq5ScezOwWLnl4xqFLcdi+i5zCEnmd99knBqWuTKKu3cKdJJQ6zDxQjZVV3PM8yUWp8cimKWf/Tq6CJe5F0cA9XSvJ9hrElGQ/GHCMBe5ZNdyeF7X5xatX4U++9gszNsB6km1Z6krlXLqoiy3TVcVES5JD8pl5144HB4b5fO25xQTc61dtwJu+cL3Tn+lLIMkU6McrB2Yxktyrzme8M7zdgqvO5mdnhxpPsvKPr/6ne+mff3Iv/vOmhw2R989xyQAk2b+urYF7hUuWCePdDFe++0X4+OtOAlBNYlLgXsKegNOO2R//cu6zcccjm/GWi1ZGLXwJCXsyRoIkE+gxmWVuVS9KDwa42S34Uq7NkywQSg/ck8xJMnE7aqfUGi992oE474yjmALcVpaaL+Pbfa5894vwydc/25A+v3Ifpd7imQ5oclB6xKBf2wGUsufnTxIqFTbMzesE7pVlUNVwrU+SMxWQDu5J9rNbaO2eN5HTsU7mED7fFtPNlfnNeZo3aosIMZUJB6z1Qdfnb/MkV8ctWzxmx1GTSjp3rg/zUugApeGD6T/2+7YFzAGhkix7kis7wTi3WxCZ9+wW0/X9RysWNJFqgmbn4nNjR0muyXmeh6o/ZZDhsL7puN0ipupWNhb+uZkk+5MqQidTOGL/xfY+y7MUuJewx+DFTz0I//C7z8KKVevxjn+7IVj1S0jY0zESJNk+xD27BZHFPGMKKfMks38QiMz4xMUnmIC79N/JbGBWh+XQpQA6pVwiIGe3sO9jgV0T3cyxOvhkO6+38ewNZJ8IlOTabsHtFXlWESquJIt2C2VTwPULbRQ6VfubAyU5E+wWjifZt1top8QwkZc8U47K7qNb/8ZFieAaVOMIrxsRZyJwPPAuzxT2ZnaZJXV550UsvzDBLwroll2OB+4NUoLbzxMsZXGg1QLHbhEJ3KMJS4cR1PbsFlYV9+0W/FCrJLud6jq9RUD4jRXG7W8fdt1jqi5Nav22fMRsFuY47/omJTlhT8MrTzoUf/nKE3H5nY/hT7/2C/GZl5Cwp2IkSLItAVx9NnYLCtDKLBGQKpMBNnDNJy7SUjTPk8xJECcmulYzM2+bZLdwPMmRwC6f9PhlqSmnMffDUg7cwlPRKLCMB+qRtYI4a2mUZJd8KGXJZb/UJiBqvJOh0BprvWjpTqbw9hcc63zneJJLV0n2FccJlp84lsEAqCZC3I9t7Q52O+BOUuhal2VFRintGimiXPU96fBl+OTrn40zjt3fXC+Cr2K6gXsuSeZE31/6lzCQkoxqstHtKNOuCdzzxkZBafx+ahsHz5ZC1gse4Gn2K7UziXLb0AGRjQfuMZIc9SQPFrjnFxPx0fW8Id08S4F7CXsczj39SLz7156C/7zpYbz/27e1ri4lJOwpGAmSTH/OliTD2B2Aym9qyBMP3HOUZOUoZQSJQPhKsh8cRko2D1aih7gcuGffx4qJ2ICxuJKcKcqTbL/j6cTMeZe6LvzAylLXJNnk+tUAEFOSq7H0CltxbyzPUJRwlGQi2f/vC4/Fqg+/AicfsQx7jeUNSnKlALvZLaySHCssUfWvnGIk9LtZT3J4/bvMbpFl1m5BimjHs9W84pmHiAptkItYW1tP7mU/4WRuMCW5nSSXWjt5kntFaT3J3r6k/OfsvmwvJmL79smxk92CVh+8MdYW+IDI8r8XDm63iPF3v+Ie/VQ+6eXBmFLfnVxQkhNJTtgD8QcvPBZvf8Ex+NK1D+AffnjXQg8nIWGXQKd9l10fRPTIf0kPfiIq3SzDtrqGCS/SwAPKiGQGnmThecmzW/Bl38whHm4AFZES3ufv/8qxeNsLjsFXV64x38XKUpsgs/o735NM1giuYhr7hRC4p7UOsltUvtzqhEkFbw3cq89nrJNDa421my1J9gnIt/7gTADALQ9uMt/5acJ4RgrAksROljnXxgdZaqg5Uqh9lX+KESBqW+vKqmDLUldj4Ona/MlCk90CcFc3YgL4IGKNdP19FPWkhwfu0XH+7mR9sfmew4lhME6WJ9lP/caPrfoNx0w2mtkoyTFUnuTQblHZe+zfBt1fvuUnVwoFdFDOmuwWVb7u+KQsIWHUoJTCe17+VGye6uETV9yDpYu6eOsLjlnoYSUkLChGSkkm2cwEsAmBe9yvyYN5bJ7k4ewWPAWc9XnaLAuWpIXj3ndxF/vtNeaQKK4cOp5LIhQUuCcpybXSR3kvSfnzlWSTAk6xCYaypBqwhUN8UlbZLWzgXr8soVSl4BWlxqOcJEfYIQ/c81FqtxQxJ1Kx9siPzS02NBmi8ROx5yohnYeu/bLGalKSkmz78/mSlCeZg0hZplQQPEeYL7sFnZPJk+xkt3D3p3HNxm5RtUUrDeRJtscWWju+eQPt5kkm0EdJSf7r334GTjx0SXRMvpJMKrW/2sD/7jnyyP40AfYzsiQk7AlQSuFDv/UMvOKZh+CvvncHPvXf9y70kBISFhQjoiRXr/S4U+RJLklRzcz7fr0sPVkWQXYLIFzeFe0WHb4MnxkiwpePiaTHlpT9/QkxYtbx9vVtG0TWH9o4iXf++00AKgV9IaRjsAAAIABJREFUUhdynmTtBm0RKTRp0HRVTERSMomoF2VFPshyUmo3ZVeM1EqlgAEb8Mb7NOSnhSSTn5zuBVKSrSc8JOZEnDV0nSe5VpLrMZAy66uWQHtmBU5GY0rybOwW0jWgvkj55v5kGit1Rb9Px7tXG2FWakL1m99atAoQy8ARs1tISvLrTzsCD6zfhlsf2gwJqvbgm7bMZMj9nf1iIn7fgd2i/jzZKxzrU0LCnoI8U/jY756ETCl8+NI7sWHbDN7z8qemlZWEPRIj8hRwVcNMVQSZVOBubpeU+4V2FDcCKcmcBKzbMh0UxwBc33DOltOJh+WqKtBAHk1AVgBtzl1ZlZTK7lq7hfvQ9xW85x61L048dGlUSS7JbuF5kvuMJPtEhM6RzrPUVeBeJ8uQZfTZ9hWzRywek+dm1bK8SwTNMnquAkLD96Hfzg/U8rNbcJjAvZrA9U3gnmsrkX67psA9t/94+elZ2S1Ekkz3eagk++Oz5N163NvtFjblWuhJZkpyqc195B6vnbLU/rjCPMmVJ3mswV6Tsb+76nzktujfAP9vgHbz7RbHH7wPAODnD6yP9p2QMOoY62T4+GtPwnmnH4lPX3kf3vONW4KJZkLCnoCRIMl2Obh6tVkdartFxu0WrDIZD9yryR9/6D/3r36EV//L1QCA5xy5L/7i158GwCV/eZ6ZBWdewY2UZBNMKJFkplSacXCSzPYlokWVxvzgIiKKhD940XEY67h5kwmF5oF7tn1edY6UZp/Y8H4oKLCTWz+0Y2GJ2S2iSnLoSabrlmdZNHCPVPSCWWyCwD3hWCJUVAqbpwnMFJiSHB7r/mbhmGxqPBVYNQgDKcneuKVUZ5SyjCZvRVk6+/EjZnwlOWsfh5kwqbAstfUoazPZCFLAaTsRkc7Nn0SQJ9lPf8fhl4Ontv2JGa0O8MnbWJ5F9z/9mP2w93gHP7zt0WjfCQl7AvJM4YOvfDr++CVPxldWrsE7Lr4hFRxJ2OMwEiTZ2i34ErL9vipLXb3vFxpj9cO5x8gj+W/Xb5/BbQ/bwDLC215wDJ5//AF43rH745ClE+b7SjW2gXK2f2tpoP18+BaK6j3Ye5ldZUo5qdOAmuQHAW/xPMkmBRydRxYqyZkKia5y7BaVctyp/dB8YgLEleTxyDK2RtVeVwiY62TxPMm8EIxZMSCSzM7Ph+mn9sv62S1ou6gkIyRoHP2ybLTaUD9t8G0iUl9EfB0lOeJtDz3Jqr3inq6VZPCAPc22syqPSiH3iL1G9dsGgXtsssWxd02SuWXER6UkyysOHDQP5pO3sU5mVWzvHh3v5PiVpyzHj+98LNp3QsKeAqUU3nX28Xj/b56AH9z2KN70hetTCeuEPQojQpJt8Bm9lk4xEasS9kpmtyhc1VMphRtXb8TvfebaoI9cKTz14CW4+K2nO57aDiNXtjgCz25h2/chKcl8v5gCKS2R+8vcnSxDpqpr43uSOYG0OZXrXMEsBZwUuJcpS9RIOe7kmSGpfu5pcfyZwiVvPR2vOvlQ53uyS3SdwD3bVqw98kRrzZbXC9eTLBHsjvEk26BFzSY3XZYBIjiHlslMr2C5isVRD2a3CFPAhfsYT3JNKvusb38AYZ7k6p7/5BX3RMdQ6mpSQF5/f+yVUmztJdKEsBQC98AmW4RF3dyQ/abiMbSiQqD3gd2iLOtJof2um9spjp8yDgCO3G+xCX5NSEgA3nTm0fj4a0/C9avW4/WfvQ5PbJ1e6CElJOwUjAZJrl+5akjliYGKjNJDvShLRpJ9T3L1fuu0LV1stjcEjRWeksztFk1qonmgR3zIsYAviTzz8thARQCrcYQleWlsPHCPAq5cT7IcuEdfaV0HidVL7GXpXtOmvMZnHLs/li8Zd77TqEguV/d44J5EaGifzCNcdB7GriEcaz3J2lhN+G/ZFSYxBCUQNI5+UQbVFmeDMLtF+CdLJNkqyaXTp2O3KNx7lZr/yA9+6bSptcYPb1uLux/dgk/9973G0mGKifB9wYq3ZGGAJU08fFsFTWTo2Fc9+1D87984wWyP/d7VuN37XbJPVBlXwpWUrmCR4hjv5OiX4eQyIWFPxm+dfCg++4ZTcPdjW/CaT12DhzZOth+UkLCbYzRIsseSrd2iDmhiftMeW8537BYZ99qGWS1iFb0kkpzVFoySEQOJZEuFKVwlOd6njyxzlTWrrtp80QTKblHZLWyAG8/wYDzJAkmm74paOc5rQl59tp01KYE0Rg4KMpTyE+cNgXs8DZ8fuOcXE+Ew2S20/R1pYlWlnIvbLdCSAq5fMnvB7DlyqCR7bWVKDtxzMoQIdgvJ6rONTQ7/46aH8LYv/RwXXLQSQFVxUCkYduxU2tNubm7/b0XD5t3mMJkn6nvm5MOX4fWnHWHPvSVwj192utY8x/JYnuG7tzyMB57Y5hzbzTNnlckHFQtKRUUSEly86KkH4ksXnIZ1W6fx6n++Gnc/umWhh5SQsEMxGiTZy25BhM1UPWMkuV+Ust3CW76Vig9IyDOFo5fvBQB4x4uOZf25xEB63kvL+XwMMW4lppMT7RZk+wjLUmu4tg0iN3Ta5EmWUp9x1bZXlOhmmanWxycXTWWkacwuqhLart2iVgjrPiRQSW7A/m5+CjhJMeRKMqnodKkypYwnVpyUcCVZ2F5VvQv3HRZteZLzTAV5krkXHnDJZOBJZu09zpZQH9pQqURr6wqKX37LaY6H3bdbmImikpRkmPuNo1e6SrJ/Hflkyb9VYkryElatb9tMgUc3T+PtX/65227HkmRJrSbPvF+wJyEhAXjuUfvhq28/A4XWeM2nr8FNazYu9JASEnYYRoMkm8C9CjkpqNoqbMTdCqZU9pw8yW4wlk+SY6Joniksmehi1YdfgZedeEi1r3ItDdV3kpLcnN0itkwftVtwkpyTKh6WpTZV+ZS9dr6SXKWIC7NbcCWZ/M6dXNXFOLRzTZuWy6sxuxeV7BuO3YIp8bH2yFoC2KV1Q7walGQizpWSnGH9thn86zWr6v6sEi79Dm22mMoXPLjdIpYD2if3oUdc2ewWgk0FiGW3CO89TpJ9W4ZNs2ftOARe6VApedLAM70QfNU/yAmdx/8W+GSNj3PJoi7a0M1tZcmY3QII0ywmJCRUeNohS/D13z8DSya6eP1nr8VVdz++0ENKSNghGC2SzEgJBWEBLECrJnE2TzJLAafcVF3+A7JJSfZBAU68LHWTJzmeJ1ns0uwz5pFJPpRuXittGkIKOAC12mhU+MzN8FCl7JLy9HIluZpoUEEVnhsYaF4ur7aHiqMfuEf7dJj9gUDnnys34wbA7BZm3GEqti67LzqZwvptM/ibS++sz0GhqQx2W+BevywNWRzEkhzL+OF7qYOUcJligXv8fpDHJwXuEdZtmQ7267BzUKjui3vXbcWWKde3zzO8cLJLzfvqNmB/I5rM+tu7uXw+tK9jU1KkJIc5uI8/cJ+gXfrTb1SSe4kkJyTEcOT+e+Hrv38GjthvMd70xRX43i2PLPSQEhLmHaNBkiFkt+BLwBmpqm6eZN8awB/S26fdpdZo4J7AgDJVkQYnBZykZLbYLWIKpLEgeLYEPpa8zm4hK8klKyZiz8PPk0y5cd1zY/7f2oPczSvFmYgVFQtptVsE3lXtTCz4NcjzUEnmxT5oP5r49I3dwh7je6T97Bb+2Lga78Pxw7JjaUzbpovWFHAc45Hc0b666reVZwo9kwKO/f58P/aWVGdz77H21221GR0MmWbXWKkqpeDvfuoap8gOD5L1/44U28cH/UZEsP0JQLdBSfatQMZuISjJfuBqN8+skixMhMaS3SIhYSAcuGQCX3nbGXjWYcvwjotvwMXXrV7oISUkzCtGgyT7dotMgac+I3JEhNHaLVwlmXOo7T1XKYsG7glKFHmgyddL7Uv7VdvC75wT8mB9zpwUu2Ps1MopzzxAIE+yUix1V6aQZ5lR98hPLeVJpvRbdI2JGE3X15NS5LXZLSTvalGruv716GRhECGRGSpLDQCFCQZzJ068LYLjSRYmA2Nmezh2bs3hhy5dNAagKg++j6BqxhBVkgfwJJPFZRi7RZOSfNejWwwJpt8iV8qkDFy/3U2PptlqRaYU3NLq9UTDm/wAPHCPCLZ7DbiSvGjMnUT4KwPUJfckX/rO5wMIV4XGmP1KWi2wnuSkJCcktGHp4i6+dMFpeOHxy/Hn37oFn7ziHiePekLC7ozRIMn1Kz00yW5RenYLKpsspYCj4wjbfCU5ogZKXtLKbmErkAEuuaG3UkW3tnLHfB+eO9gPZOrktsBGWEzELn/TFutJtp5THhBnr4PdnwfuZSyAbHFNaKR0ZU5bgZJc/Sb8OFtMJAtUP17swyrJrt3CvybS8VqHk52MKclSKjD+0/A+li6qiPHDGycNYRtESR6LkOQwu0X4mdThbsxuwT6YwD1jA7L7rdsyjV5R4lc/diW+ccOD1X7cMlKvFvjPP+5JpvvIH0epQ9tJz0sB558b/72X7+2mC/RXOawn2U5MnnbIEhy8ZCKoEsaLlIh2i27yJCckDINFYzk+84ZT8FsnPQkf+cEv8aHv3hHY/BISdkcMLnXtwrDFROyDv9QI1KIqMMzaLXgKOCq9S5icmYvdQtX922Alzu/2Gutgy3TfJSBCHzFqxYuW5EqhgA6KbXSqSEQxnR2Vb66UQdtmnnueZBWWGDYe6zpQr/IQV3moZ+rlaSLJ3WHtFvVYHZJFSnKugvaIWPIUcJYk13YLdt19wtk1kyeZjCpSkkWSLFsbli0eA7AN/VIbwtZyGQDEleRgkhJ4xDnhy8T9+BE9ISCP8PjW6YAYUlEaVSvJk0JZWg0wu4U06XNXVQjmt/JsUfZ87OcDl4zj7se22vNWXiaYus+lnt0izxSmeqHdgp+fj5TdIiFheHTzDB/93ZOwbPEYLrzqfmzc3sPfvvoZrbEpCQm7MkaEJFev9Mz0SyRbT3JttzCVyezDk1fHA4BtM57dYojAvby2MXBP8iFLF+HZRyzDYfsuxsbJHq68a10QuCcFJ0ngFo48U0AhKMmZDdybnCkw1smM0tsvw+IOWVYdQ+p7rJgI5ZClcty9QmOiWx1HiiYtjTcVE5GunYbNu2z3sfv6/9iSvSDLwhRwppiI59PmsAQptALkGZDXd5RUttklaPb9MkbSiLDF8l075xIhyb6P2r9mnNyPO0qyvDrR8zzJyiPJfm5g7WVokYLZdOnZLbJQ0RaLiXiZSMJJjG1nv71cJdm/3+m67DPhkuROrgKyy20pYp7kZLdISJgVskzh/b95AvbbawwfvewubJrs4ROvP9mpUpuQsDthNEiyCdyzD3Ne4KDDFMN+qY06xu0WfsDYwEqy8D31XzD1bK/xDr75B2cCAP74khurcXlpuKTgpMv/5FcCWwgvkGEUwcwrJpIrE7i3dbqPJRNdk+KrrDN/+JaBTCkzcSCC749pok6PRWo9BUL2ijKwW0gqHUdgVdFVUKG7XG/PNUaizIQAlnBJFgl/aZ3a04KSzFPdiYF7jifZvl+62JI0a7cIDg9AaccIR+2/GP/65tNELy616Zd6drNBhJYHQPIk222TM0WQ+rBXaidLh6wkuxNS/rMrNtHw5wp+xT3f8uIqvuHfhutJrj4smYgryWN5hpmidGwpcnaL2m6RslskJAwNpRT++CVPxr6Lu3jft2/DGz6/Ap87/5TgbzMhYXfASKyD+EpyJ1Po9bVRt3jgXq8ojYe1x6Le+6V2lqV9JTm2YhRNAVfKAWEAS2vmZbcIlGMFHLt8bzzlYDeFFZEQXmVPslsoKEOSlzKvpi0m4p5H6EkOz5uU5Kzet19oQ9aJgC3qDqoke3mSUf0OkkKYZ1nQHqmvnMzThILIntQWgchSqXVA0HhZ6lLgSlLQGFCdOymVvpJ89gkHhQ3Vx/h2i4lujiP2Xxzsyz3a/jnF7BbccGGyW2Rhhhc+0SH0i9L0qaBEkkzWIiCuJFPebf84APjfv3ECnv/kA3D60fs7251Ucl6f3JOslJ08THT91QKrJNM1HmuzW3ST3SIhYa4474yj8I+vOxk3PLABr/v0tU6KyYSE3QWjRZLrJ+nSRV1smuyBBFhT/KKsSBTlEOaqWVk2e5Jj1gfpIZtnvPRzeNxY7hIcTnQH6dPxJDtKMiPJtZKsAWyZ6jupsSiokZOWLPAka0dNJZDKRhk8+nUxkUwxkjxgCrgwu0U1sXGyW9Tn1M2VUbEJXTbZsJ5kr4qbZ0GR+i+1EBCXKeZZbgnc85Rv8iLza37te1+CT7z+5KAdAHjX2cfjTWce7Y4tVjiF/d6AO+54nmT73leSuaWgX+rAYtAr7GqIUgiC4ID6d2NZUvifBF3/amVCPCUcd+De+NIFoWruWFC8Y/nESMFO5vy/tzzLjJJM5FfKw+30WzeWylInJMwN5zzrSfjc+afg/se34TWfuhpr1m9f6CElJAyF0SDJ9Ss9NPddPIYN22eskszITlUhrgpqK7jdwgssCrJbRJ7wkqMgUypaIAGwD2ZSuBVTxDhiFNPkDmYk1i+r3anXo7UGtkz1nICmQhOB523CU5JdtY6uoeNJ1hr9ojQlsEmlXEwp4IbMbkF2mDxXOOnwZXj/b57gZAc5/3lH4fwzjjT7O0oyI7yADcp0LSiep9mUcdai3cLk026xW/jWBvLFcpJ88NKJwFJB+O1nH4qXPPVAvPzEg3HyEcvq820O5KPxuoF78pj472w8yQJJ7vVDJblXlE4fnCTz/OOmBLynJPObeABrtgO3kElot+A2pacdsgQnHroEBy+dCNqgMRP55Yq7mALOKMmJJCckzBUvfMqB+PJbTsOG7T38zqeuxi/XblnoISUkDIyRIMm+0rdscRcbJ3uG3BA5KrVGryzRrW0KPLuF70kO8iQPoSQr5u2VuLVZKo/YLairWG5mvj1nZMnJaKCUsT1M90vHD1aaPMmuFSHPsjqoz3qW/Xy6pOYqpVCU5PGur2c96Rg0cC9QkmGzW/zHO87Em848mtkLFA7fbzFe+JQDzf7ck+wrwXT9+TmGSjJLASeUf6b7Rkr56Sq17nWnqm9S9TcJnawi+f9y7nNw6lH7AXAtHBx0nkcesBhLJjrO5IdbNtzsFsxuESjJFYGc6GbolTb4ktCv82DTefIVFhqL60l21W1+GoOkwuOg33fZoq4Y1Monl089eAm+80fPx97j7jXPM2XI7rjJ3x33OgOpLHVCwnzjOUfu+3/bO/MwKarr/b+nl9mHYRvGYXPYF9kFBGQREEVRcddEoyQajcbdmK9mVZNfNO5J3GLcTeJuohITRYMYkaiogLKoGIiKC6gRFQRmub8/qm7Vrapb1dVr9fScz/PM0z3VtZyuru5669R7z8EDp06GEMDRv1+GV/77v6hDYphQlIRIdtstOleVYVdLG7bvNISuPBHue80SSxCpg9QA016g7A233cJP8Om8yvGYUqdXcxJOWplkvd1CvY2s36aSSVasF+5tdasus557M8nCYxlQ7QdtQmbrnJYDK5McM8R2S6tht1BFmSWSU9gt3PEK077hqJOseJIBZzbStq14/cqt2kyySyQrHfe87Z6Dm6E461kr74nIyiDrur/pcF/cAP6DHuU+mzG4Hqsu3h/ViigMM3DPXQJOiubaiqTWk6xePLoH7skQheJJJtO2444X8D+e/ZCCvUt1mfcui5JJDqoeoqv+UZZIkUnmEnAMk3OG7FaLh0+bgi5VSRx/64tY8taWqENimJSUhEiWhguZMetiVhj4ZJvRGUyeKP+33ewiFpeVHOwUYUur05O83V3dws8frBEzMSJtCTJJwuNJdorlVCd/tVZxTBHM7m11UUSy2mShpVV4PKKq/7ilrc0q/WVPM96PzCRbdgvTvqKKoSpr4F561S10pcA89gKNB9ddm1eN15ktdzcjMUWy8JaAU+0WOvw9yXaFBXfNXj90A9387D3u/aDO5hDJGuEN2MJTHs/y/9qKBFpahdaHq4pktWiIlUlWRLJ64aYu644jDE3dqnH8pL74wwnjNXYLSnkxCTj3oxS/6sVPoEjm6hYMk1P6dK3Cg9+bgqbu1Tj5rpfx2MoPog6JYQIpCZHszSQb4uRTs+SZt1Ob0dJWrW7R5hJK2911klOIFse8RNYAMt1itk1AjjaylzPehy1KdOhKwMViXiHftdpbsxeQHdKcA/cML6mZNW6zPcnWoEdzH8tMMpm+6xazZJu67zKtk+zOcsq41GnqdtT96F6Xzu7iV0JO20wkFiyS/e0W9gVJ2JJHzm6M5Jmmm5c0AlHNkAbt+hjZx7OaSd7V2oZdrd7sqTUozi1UrTsPzhJwuu6SRszO9V54wFD/IM11/fLQkRhQX+NZVhXJ7gucHx84DMft1ReA8xhMajzJumOUiFCWiLHdgmHyQH1tOe4/dRLG9u2Cs+97Dfcs2xh1SAzjS4nUSTZQ7RYA8Jkrkyyx7RZuT7I9T9hMsk48E+mbWUikzcKvBFzc5+RvbVOKasWLGyev3UJtwKAKNl0JuFjMFmDNisD3q5Mcj5Ftt4jFHKKsyqxukcpu4bYUtLY5/bIAUFVubK+63K7PLLHaese8WXTd/vf1JEN4RKlqP9GjF4Ixogwyyd51+Q4UdYljR2bdz27h0vrqfpciuVNFAi0au4Vjm66QrAsowFHdQmcfccf0t7OmYo+eddr3qMMrkgGK6V/77vT+SozefeIYuOdjaylXmu8wDJNbOlUkcfd3JuKMP7+Knz66Gp9ta8ZZswemfbeJYfJNSWWS1eoWAPDptl2OGqqSpFkyrEUduCecWbntruoW6WSS40Se7LZKv/pq9KyrsG7rej3J5rLaLaqeZDsudzMRAOha5eNJbhOekneqtaK5pU2Z5lyn7UkmtFrNRJzd8CrLXJnyFO9D0qKxW4zt0xl3LJiAcX27AHCKLrUttTeTLO0W9jRvnWWZCfXGEjeziX6os7tFYe+uVah1DaoLQmeN8BPJ8v3ohKuv3cJ1JKnr3tli2y3aBDwtnB3bck2XF2htbcKyYRgXVkq80D9PdWx4cW5dPQ6CTqsJTTCpOu4BhkhmTzLD5I+KZBw3Hb8nDh/XC9c+/RYueXyNVZGKYYqFksgkW4OGzP9tu8UurVdXlixzrCNFdYtUmT3HNM3tc5WDRvXEQaN6euZxCyC/TLIqpBIxwzqiWiMktRUJQ8y2CWedZCHg/ilS7RbS9hAjbwbd2XHPyCTHY+Ro4lCZDJlJVgRKr86VVnkyt+icOdSuaKGuslwpAecW83Y2XM0kuxtNyJFnGpGcIpPsEGmuQXLHTuiDA0bsFiiy/dB5jZ2vOx/VbTtLwKmxOtehvi/LblFuHB9f7XQe94CdiXZnedTj3LJbkFF1IhknNLd6B4dKUtXQ9sSgySS7L4516Kwf6ufiL5LjbLdgmDyTjMdw1ZGj0aWqDLc9vwGfb9+FK48aHWh1Y5hCUhIi2Wu3ME74W79uRlki5jnBSmGp0urKrLozyX52C52QcgiDEFpAzm+d0Mk53Y0tlIwLAD97RixG6FKVxCdf7fLYLSDct+TJujXdrJRPc2fQrY57ZNotzIF7ag1gqy11ih86dfs15Qkr+xskTnV1gd2eaMCugR1ot7CqW6Q/cC/m8xnL5brXlHsXCoFO/KpYrddddx3c+0Dnc7ZeU4ShOnAP8HrxAfvYd4ckRWqbsLtbyrhqK5L4bNsuXwGb6gLKjc6TLNfRv77adzln1z7jeVi7BYtkhsk/sRjhJ/OGoWt1Ga588k1s/boZNx63p6e5EMNEQUlcrg3qUYPT9xlgeZHLE3FLqOl8tYk4eUSIp5mIZ+CeftvudsaAU1CHyZi5M8d+HlD3/HKQlC2YvAt0NStcdHK1pW5zlYADbEGxS2npXF2WcGTeKpK2J1lmIZMxctToratMoktVEr26VAa+b7n8waN7mgMp5eCvcOJU9ST7WTfU9+hpS60M3Gtx9Z6Okb7ygcQ96NGanp728yA/S7/VuNcv43B3XHQMJnStQ413p1nSTTZA+Wqn12LgF5P0kAthN1yR+1jWK/aLKd1MsnfQIFBdnsAdCybgjm9P9F3OcbFgfpypOu4BRrZ5p6a7IMMwuYeI8P2ZA/H/DhuBZ9/agm/d9iK2ft0cdVgMUxoieVhjJ/xw7lBH9q6zaS8wbv8650/EndnleSMbccURo1zVLUKWgNNM9xu45IfXbuGc7rdNKZDt/73zSn92bYqBe2rczUpliGMn9sGDp0625lEtDlY5sTh5Mskv/mhfHDyqMfB9T+zXFT+ZNwyXHz4SgF2RIqzNIbCZSJvMhsMzv/v/NuEtfaa2pdbHoY8p24EnbsuN53WNWASM90LK2/PruAc4j095F6ZGZpJ1dgvr+HTt41a7hbnqSQbgaerhjiNV5ZOgZdVYZg7tYV0I6nAOijSep6qTDBiNR9yNVRiGyS/H7bU7rv/GOKx8/3Mc8/tl2PzFjqhDYjo4JSGSdciMp67yQdI17dpjxqCpe7XjROwRyT7CTTc96Fa3DmvgXshMsmrPiJO+PJqka3UZasoTjpisEnCaDDugNEIhQm1FEqP7dLbmsferPegrGYs5PMnxmDHoLZVgjMcIJ0/rj+ryhNnBz78Bi0RfJ9lrC2nVVLdwf1ZSQAnhHMQpYwiyi/hZatJMkHqwa2D7ve6KwxTNibhTPgddqKnC8YFTJ+OMmQNRY1YPcd9BARS7hWu63GcCsO0W5rxSdPt0qM4gk+wkbPc+3V2JVG2pAdNuwXWSGabgzBvViNsXTMC7n23HkTcvw7ufbo86JKYDU7Ii2boVrxm45741LU/YanvrVkU0yYFxOnQne11JryDIJY6D7BPq9Lg5WM+deVYXG9bYCQN71DjibG0zmokQgEdOn4Lz5wx2vJddrd4srERmkuNkt/uNu+wWmQy6INgZ7EBPsvJSuSKS/WouB3msZulPAAAgAElEQVSS1ThbXFnDOJGjCoI3Xr34Trf1shvblh782be57CRuT7Ij0+1ah2oRGtGrDj/Yf4i1L7YF2S1cK2qx7BbOOskAUJvCbpFudQvPoMGQu9k5mNG+AyPxE+tc3YJhomPaoHr86eS98MWOZhxx8wtY/cHWqENiOiilK5IT9gnRfXJPxmPaTKDf3VU/q4Xfa6mqW7ixbrG7BmP5LWk3ECGHQNRllM+aPQh//f7eDlHRJozsKRFhXN8uOHP2IMd7kSXgdBcGaobe8iTHCeVJ226RbpbQ2JadmQxaXt9MRFMnWddMxKctNWCLakkqu0XMx9qQzluv0gxMCXsXQV7DqW2sHRd+GnFqD/Lzfu1l1lwO3NtnSL2yvHNbErnPVE+y3JYcCOgn3DM5RlTCZ5K9n03Q91nCA/cYJlrG9u2CB0+djDgRDr1hKa74xzrtwGKGySclK5KlEIgReTJCsi218bp98lczySqBt/9TZZLTGrjn/N+/woHxKDPJqmgG9OLa22yjzderajfi8K7H4Uk296tR3UIpq5WpSA5R3ULnMQ4euKdmDV2eZOX/5lb3wD3vOh1xKHvZz5+ciud+OBPPnD/DuV7XXQXvdg1kET9L+MbJV4Ra2WZXO3SVMvOCYNvOVtSWJ3DntyeiUhmkqW7LjXHR5RbJSfM19Y6Mf1Y/Fe79EXY3J5TYZSRhkthcAo5homdQQy0WnjUV88f0wo3PvoM51zyHp1Z/ZP3eMEy+KVmRXGYJAu8tZNWioAoGty/Vmj/NW+jOLl+p53fHkk51C2d7XufrumUkbcIbm8ysqtUt3FjVLdSBezGyphvrycRuQdo6yW6cmWSZRfXWNG7VVLdwZ4bl3QbA+9kbdyDCiXWdIA1D95pyDKivcUyzs7Z+25UXdOb/5vREzFmyThe73Ec6gSovKrftanF4vY2YjMdenau0MQkI6y6MVd3CzCRvUwYCqltN25Psmj1dT3IiFtP61P0Y2KMGQ3arTStGhmFyT/eaclx11Gg8+L3JqClP4JR7XsFJdy1nrzJTEEpWJEsBFCPy3KIx7BbejJ1fJjndE7qqxcIN3HNmji37ha8n2Xx02y0CxLXuPXg7sZl1klu8VgVJmZW9hTWwKRl3epKztVsEVT7QDdzT+c51nmTvwD075kNG93S8lkrnx3w+4+w9ycGZZDndnblNxGOIx8iycMQ1dhApjnWfTzIhPcktyn6F43FYo140trWpJeCMabK6xZc77O+ew/pSoIF78lhKxMnjmw7i3DmDccM3x6UVY3uAiCqI6CUiWklEq4noEs08C4hoCxGtMP9OjiJWhlGZ0NQVC8+aip/MG4YX//Mp5ly7BL995m0eO8DkldIVycrAPXftV7WZiKPqg08mOV3NF3dk9FLPTy4xIv/3W1RtSy0tF0DwgD/de/Bkks0Jza1eq4K1jCK0dlkD7ZzNRDKyW0AtPRckku3n5QFtqVvbNJ5k1zzqwLwRveqw8fJ5djwpPzh99jhLq22ITLLxKDyeZDODa4pTXe3mZIDdImlO27az1RbJLlE9rLGTNiYB4alu0cnMJKuWBQq4YEmFN5Mcbjm5nUQsPZFcwuwEMEsIMRrAGABziWiSZr77hRBjzL9bCxsiw+hJxmM4eVp/PHP+Pth3eAOuWfQW5l73Lzz31paoQ2NKlJIXyaq1QqIOclLFRKuf3SLtTHJ6mUV3LFa20Gd+uy11zOHHDfKOakWfa5rtSQ4jVu2Be4kYWZ34gPRr4MpY5P4Pqnygq5Ns7APnfC2aW+tuG0hQnOpnuEdPrzjMVybZupvgc4kkp0pPsu03dopk0ohkaYEIzCTvarGtSq47HL4iWdh3Ydwl4FTUzaZbT9o9f9jlLYtJPGbFmK59qpQQBl+Z/ybNPzZ4Mu2K3eoqcMM3x+Gek4xGQifc/hK+/6dX8eHWryOOjCk1Slgk24Lx1OkDrDJngBy4ZzxXB9blym6ha2AQhPsWuzyJ+w4ktLLgzkYaQZ5kHe65rI57AXYLNWaZJUzEY6hwZJIzKwEnB+4Fe5Lt50HVLZo1Zez82lLrkDG88pN98dD3pnjj9RHG2TYTcfvLPa/H5LFh/C8/q927Gq2Zq8qdg+3UdcqKE3pPsswkt3gzyebyfk07hIAnS1tTnvTOmMW+cS/JmeTMIaI4Ea0AsBnAIiHEi5rZjiCiVUT0EBH1KXCIDBOKaYPq8Y9zpuH8OYPx9NqPMfvqJbjluXc8A7EZJlNKWCTbJ/rKsrhV5sx4zRZVziYb+nWlmx101mFNPb8t2I1H+5Z6sGiPxwjDGjthj151jjjDhqurHw3o/by6GOzMrzOTnKknublNiu7gDLZEvVvgtVuk9iQng9pfm/N2qylHpaZUm7omR4bUd43hSOVHl5PlBdRJU/vhogOG4oojRwEAqpKy9Jo3plpTuGozyUqLbtuqBM/8T54zHXcsmOBYVjanAez9puu4l4029dZJTjOTHCM0dKoAAFSVeWPrSAghWoUQYwD0BjCRiEa4ZnkcQJMQYhSARQDu0q2HiE4houVEtHzLFr7dzURDecI4vz993gxM7t8Nv3piHeb99l94acNnUYfGlAAle7ZIum4ZqyRi+oF7ObNbOKodpF7WLYyshhE+ol2N/SxF/Lu9yWG3K5EZYF0W1o3D6+sauJcJBNtnG7a6haPjnivYFm11C2eMQfsp1S15h8XCkbUNXCwlttdY/7plwzDnG92ns6MjohT0jvfmySQHd6ErS7jtFvZ8Q3ar9Xw+AnaNcbnM+KYuOGrP3qitSOL2pRucsWdA1tUt4jFcfsQozBnegOEa+0xHRAjxOREtBjAXwBvK9E+V2W4FcIXP8rcAuAUAxo8fz5YNJlL6dK3CbQsmYNGaj3HxY6tx9O+X4fBxvfCjA4ehe0151OEx7ZTSzyRrTqZxxcOq6iZ/e0PmIjlcMxFnVtsWycHxuMVKqsoI3vW4/jf3hW4A3e7dnOW/3N3T1IF7mRC2hq761uoqE6gqi6NHbYW3TnKI6hZBpHKMOAfr6QVzJtheY/165Or9FEl1uV2iz71Mp8qgTLI9rdxn4J5EbUEOAKfesxw7mlsd20rGY7jyqNHoV1/tiSMT3IuG/UranmRCXWUSh4/rnXkQJQAR1RNRZ/N5JYA5ANa55mlU/j0EwNrCRcgw2TFneAOePm8Gvj9zAB5f+QFmXfUs7lm20TcJxjBBlGwmWTZH0ImWZNwWdXkZuJdmZtGvBFybj63Kz3tsrSdknG4hJjOMVp1kRQs9fd4Mh2hX32MiTh7hlC5O+0JAJlnZbk15Ei9cOAudKpLY8tVOx3y6hii7dapAbXkCX+50lgQcqqmHmyqTrL7srCSRnUiWh6CvJ1leQPkcq5VJTac78zHIk+zIJLsuMN37olfnSvzowKGoTMbx00dX4+MvdmLT58aAGff3Tf0vm12TeSbZtluovPrTOdYA1Q5GI4C7iCgOI0nygBBiIRFdCmC5EOIxAGcR0SEAWgB8BmBBZNEyTAZUlsVxwf5DcdjY3vj5Y2/gp4+uxgPL38cvDx3huPPGMKlIKZLNQRt3A2iAkcC6RQjxG9c8FwA4TlnnMAD1QojPiGgjgC8BtAJoEUKMz134/tjdxXSv2aJOPam7RfLQ3Wqx7qMvsxTJqZe1vJ/m4wEjGrHuoy/RvVY/UEqKFndc8v+wQs09m+VJbjGzmYrE8VgVXJnf7DPJ9vNgT7LzeeeqMk88gN2WWt0Xh4zuiVnDemDUxU9Z0/5xzjQ01lV6tpPqM9dVj3DHlwnSh+5rt5AXUD5JEVknWZfprtVUvpBo7RY+xxMR4ZTpA7Dyvc+taTKT7BbU6r/ZXEC4rRphx4ZadZJdC/gNQix1hBCrAIzVTP+Z8vwiABcVMi6GyQcDe9TgjyfthcdXfYhfLlyDQ29cim9O7IsL9h9inTsYJogwp5oWAOcLIYYDmATg+0Q0XJ1BCHGlrKkJ48d1iRBCdc3PNF8viEAGUnmSyarG4By451Qeo3rLAXHpbTvzttTG45mzBmLlz/ZDj9oK7fzu2rj2epyPqXALD7k+mWEL9CQrG0nGY5awyhQ1liC7hV8lCb+21G4rRKcKZ9WFobt1Ql2ltxJDKkHnl/nOxncL2J7kVBdXwsdwIatb7GpV6xMbx0S1KZJ3aUZ+qxcm7mYifoNPR/Wuw2WHjwQA7GiWdx8CrDK+r6Qm20yyu9siwzAdAyLCIaN74pnzZ+DbU/rhvpffw6yrl+DB5e/53pFjGElKZSOE+FAI8ar5/EsY/rReAYt8A8C9uQkvc6TdQtueNx5ztFeWuDPJI3sbt2XSzSSnO5BLhmANyIsR6qo0JbRc63QLkqBmIkHblaRV3UK1G2iqS6SLw74QWCfZuV07HrdITl3GLohU78fP65xtJlleqPmtxu64p39dVrfYrjTQIRgDG2Xt4h27vB2qAu0WPm+KiNDQyRgQI7teuWdVLxqyqSHtrW4RbrmgLoMMw3QcaiuS+NnBw/H4GVPRr3s1LnhoFY7+/TKs/fCLqENjipi00n9E1ATjVp2uriaIqArGSOmHlckCwFNE9AoRnRKw7pyWE5InfZ2WUEuWOeskO+drMger6U7u35jYF3sP7KbddqZ1ksOUi3PM7y7hZmWk9cut/38H4Mbj7Fa77vlkNtGukxyQ0XVlkrPFYbcImUl2WC9cIbS2ei0j6ZCOJ9lpAclOjMljMOXAPR+VLAfubVeEsJFJJqss2w5NG1fHwL1ksN3CuZwxr6yZ7RajTruF72pS4l40rHVDrW7BMAwzvGcnPHjqZFxxxCj855NtOOh3z+MXC9fgK9dYFYYB0hi4R0Q1MMTvOUIIv0uvgwEsdVktpgohNhFRDwCLiGidEOI594K5LickO4ipt1MaOpXj4y92GnYLM5OccIhkY95L5++B+ppyy8eoy0LJ28w6dC2Bg0i7KoVPqTf3AEA3iXjM8X7dAlK+z+27jB+LoLJujlJsuRDJSixhS8A5K2I4Y2jWlIBLh3SqW/j5kzNBiOAsvtyW311CWQJOfoYyJiKg2qwP/HWKTHJX06tnVV0JIZLt6hYukaw8zy6T7Pw/3TrJbLdgGEYSixGOntAH++3RgCuefBO3L92Ahas+wE8PGo55IxuzHoDNlA6h1A0RJWEI5D8JIR4JmPVYuKwWQohN5uNmAH8BMDGzUNPD9tfaauKh703BFUeOcnSI09VJHtunCw4Y2eg7Mj4VmQ7cC1s+zB7o5yek/JdVvafu+aTQlFfU8kJCh7rtZCL7H5SwmWRdaTNAk0mWnuQ0P7tu1fqBgG78RH22meRUdZLtEnA+nuQyTSYZZIjkgEyyus/loDapm4N2oZVJbs53Jjkzu4XdeIczyQzDOOlcVYZfHTYSj5w2Bd1rynHGn1/Dt257Ce9s+Sr1wkyHIOWZg4xLqtsArBVCXBMwXx2AGQAeVaZVE1GtfA5gPyhF6/NJmSaT3KdrFY4eb3RYrUh6s8S2sILjtXSFlipEQ3khpUgOm0n2sWe0pchCul9zXy1LASpFcmVZ6o50QO7tFoGDv5RNBXmS7VrP6cXx8GlTcMkhe6R8T36iPutMMryl63Qb9q9uYXqSFZEcixmfuywBJwfZOVeriGSz8L5l3wnYiWWW3UKfSfbbRrpknUlmTzLDMD6M7dsFj50xFZfO3wMr3/8cc697Dlc9+ab2rhvTsQhjt9gbwLcAvE5EK8xpPwLQFwCEEDeb0w4D8JQQYpuybAOAv5gnxwSAPwsh/pGLwFMhRU6rj3dTZknVjJwUmWrbZyC1P9WNn2821fxh7wj7iXdbJKdeFvD6PONmANt2SruFfyZZ3UZQe+ewZFLdIqhRiK6ZSBiaulejqXt1yvnU9daUJ1AWj2FXa1vOPMl+67E9yfrlBzcYNZ+nDLD98nLgnpVJTvHDL7PpoewW5l0Eq7qFa1b1c81mz7iXDbub3d9lhmEYHfEY4YTJTThgRCMue2Itrl+8Hn9dsQkXH7wH9h3eEHV4TESkFMlCiOcR4vwmhLgTwJ2uaf8BMDrD2LJCimS/Ei8ykyzFFGBnkqUoSGSaSVZEY5jsWbqeZDmbW0ymElhAcCUGub5tZmWEQLuFmknOsd0i2JOsX4bIsBRI8diapSc5FepqiYC9B3bD4je3WIMeM8UqQ+gTtxSdfgP3+nWvxis/2ddRB9jyJJuD+nR2CxXbbpH6uJTH+o6WVsRIc7yrdwiIcOLk3fF1c/rZmYwzyeaFXy7udjAMU/rU15bjmmPG4OgJffDTv76Bk+9ejn2H9cDPD94DfbpWpV4BU1KU7JlDDtRJlUlWa8ZaItMlDtLNJOffk6wXL7Ic14K9m/xjC7JbSJG8S3qSww3cy7UAcQ/C89uu+/2r761Z1nrOsm5xmDgIhFlDewAAPvpiR1brTVUnOVUmGQC61ZS7BhOSo7pFc2vAwrAzyenYLXY0t2ovbtwXE5fMH4Erjkz/utlbAi7c52pXt+BMMsMw4ZnUvxueOHsafnTgULzwzqeYc+0SXP/Pty1rGdMxKNm21Jbdwi+TbFoJ1Exy95oybPhkm1XVQZ5Y892W2qqPHHIzdpMH5wK1FUlsvHxexrFZnuQdpic5ZCZZZqDv+s7EjIuzBzUGcc5nP/e05Y6RdaVji82MwkmNmsWOGSUBv25uxTET+ma1WpHCMrPv8AY0PvM2TprWL/Q6yVyf/Dy7BNTgBoAult3C+D/oGkjeRdjZ0pbyrkk2n4U3kxxuOa6TzDBMpiTjMZwyfQAOGtUTv1i4Blc99RYeXfEBLj9iJPbcvWvU4TEFoORFsp9ok7Vgm5VM8o3H7YnFb25G7y7O+sjp2i0yziSnWQIukxO/471oRGaMwlW3kKtJxskSRzMG16cdjxWK8jyTOsmAPuOfrUfYPw77OcEor3fK9AFZr9eqk+yTAe9eU45lF81Oa50xMvYDEeH6b47FqF6dA+e3ulWGsFuoJeB0+98pnDP/LNz7I+wgQHvgXsneNGMYJs/07FyJm47fE4vXbcaP//I6jrx5Gb41aXdcsP8Q1FYEJx2Y9k3JnjlS2S3koDRVJNfXllvVLwAlC5XmuT19kZyeGE/Xw6ySCMgkG6/HrMYQgSLZqj+bm0MobHULuxa0v11EN79KqkxqGNx2hlyRjww4kX0hc9ConujbLZyvLj2R3JbSbhFJJtn88rLdgmGYbJk5tAeeOm8GTpzchHv+/V/MueY5LFrzcdRhMXmkZEVymWW30L8u/bZBA60yHRnvEKIh9rAtesOt389uEW5Zp5fWjT3QKbjVtHtwY7bItaRaX1DWXbcoafb/kh/OxEs/Ti8b61lviu1mitWWOofCW9ot0l6OUh//aiMZ7f533LjIJpPsJG1PMtstGIbJATXlCVx8yB545LQpqKtM4rt3L8f3//QqNn+Z3XgUpjgpWZFsddxLMXAvaBBTmExa0HJhl7W8n2FP/BkOKHTHpltcDu6qCCj/BtjvqyygK186hBFk6ny62cJmkjtVJNGjtiKDKPXrzeXgwANHNjoec4EcuJeKymQcA3vUWP/L5GtwMxH1glBnt7Cf5zaTnJ7dgttSMwyTS8b27YLHz5yKH+w3GIvWfox9r16C+19+17fyENM+KVlPstVxzyeVbIvk3GeS0x+4Jx/DbccSihmojlR2i9qKBDZ/uRMVZSlEshQfOfJ6hs0kA7bH1k2q2/25xJkhzd16h+xWm3LwZboQhTsO37hkf8f/YZrpxGN26b1Ux0I2+8nT7jrkYWd1zWS7BcMwOaYsEcMZswbhgJGNuOiR1/F/D7+Ov7y2CZcdPgr9QtTbZ4qfkk2vWAP3fC7qrDrJAdUYMu2458zWhvckhxXjcv5MbiGr70WXAZWDEILKvwGK3SJH4sOupBBuf+ntFuGm5YJ8ieR8QAh3HMZjpD12U3XRk9+1Ks2FlaOZSA53VNqZZLZbMAyTJwbU1+C+707CZYePxOoPvsD+1z2HGxavD0zCMe2DkhXJ0gaQqgRcEJnaGtT5wyybqSc5k0yys06y9/VOlaZITmm3MNeXM/ER/ra4IZK903VCKH/NRPJjt8gHMaJQ3ng3YY9/2fJZVzLQcTGRfgjKepxLhz3s4jm+48EwDKMjFiN8Y2JfPHPeDMwe2gNXPvkmDv7d81jx3udRh8ZkQcmeOcK2pQ4inmmdZNWnmY9mIrHMxDuQOstdW2F6klPsH3fDlWyRqwnr4dZmjUN6knNBu8ok++yvVIS9kyL9/yktOhEM3EvwwD2GYQpIj04VuOn4PXHLt/bE/7bvwuE3LsWlj6/BNrO0KtO+KGGRbJwUU7WlDsLqOJbmyT3hEKKp5w9zW1vFtmekFRYA5y1xnW7oZIrkoEYixrbTy36nIj1PMmn3q37gXpaBBcSge16MZCqS7QuX4PnkBWlliu9UNrvJvWzYdcWtEnAl+1PHMEwRst8eu2HReTPwzb364valG7Dftc9h8Zubow6LSZOSPXPkJJMcy0yMOgRUSJUWj1HozHC6zUdUqsvtsZpau4XpSS4P6UnOld0iPU+yfr5CNhMhn+fFSCIWy+hzioe8Y1FmiWSd3SI3FxOZZ5KN+ZI8cI9hmALTqSKJXx46Eg9+bzIqkjF8+46XcfZ9r+HTr3ZGHRoTkpKtbiFFsl81lvIQpctyUic55KK/OmwEJjSFa3OZ7kA/FfV96wfuJUKtOx17RBhkLGFKyvkO3CukJ7kd2S1OndEfn361K+3lwja5kQK0UjtwT/887VhcMfDAPYZh2gsTmrriibOn4cbF7+DGZ9fjube24CfzhuPwcb1yOqCZyT0lnElOJfJSH5iZ2i3SrZMMAMdM6Iv+9TWpZ4TiB87gxO/sFOd9XVa3aAmoHw1kfgHhH5fxGCbjR6T//FK3Rc4d+eq4lw/26FmH6Rm0DA/b2dG2W3ivuZ0XE7nMJIdbrmt1GX4ybxjmjshd3WmGYZh0KU/Ece6cwfjbWdPQr3s1zn9wJU64/SW8++n2qENjAihhkZz9W4uZNWDTFYK5usXshwwn2+xY0MC9XSlK18RznKGzRXKITHJMX91CXjSE9dJmiyHW87uNKJEfRaqPxBLJZd4ZnSXgMo+l3FVtJZ2a4idP64/d6rJrHsMwDJMLBjfU4qHvTcEv5u+B1979HPtdtwS3PPeOb08HJlo6tEg+bGwvXH3U6MB5ErFw3cr8yIdQs+wWGcYVJCKlJzlVfcdsGppo15d2CThNJtlcNGlWNMj3gDq/OEqF0JnkhL8nWbe+TJg3qhF3LJiQ8fIMwzDFQixG+NbkJiw6bzqmDuyOXz2xDofeuBRvbNoadWiMi5IVyWGyv9ceMwZH7Nk7cJ4YUVaWgrxkkrOwWwBAdZmRLQ7yJKcSyZlaUXwxV1MWwm6RauCebHCSbwFLKP5Be9kQtsxf6DrJWeys6vIEZg7tkfkKGIZhiozGukr84YTxuOGb4/DR1p2Yf8NSXPb3tfh6V2vUoTEmJSuSAWBEr0644ohRWa3juL12xz5D0vdzSvJpt8hUvMsycLrQaqTdoiWV3cJ8zNnAPYMwdwDIpwRczOWTzneS168UXakQtoJJmzk6VlcnWV2Sx84xDMM4ISLMG9WIZ86bgSPH9cbvl/wH+1/3HJau/yTq0BiUuEheeOY0HD2hT1br+NnBwzFtUOYimfKwh8PeBvdDimSdaJHez+YUA/dsu0VGIfiuL5zdQv/epaiTQjvvVgifAYSlQtgmN7KrZapMcmnn3RmGYTKnriqJXx85Cvd+dxLiMcJxt76I8x9Yif9tS78yEZM7SlokFwO5yrSqZJ9JllUIvMv36FQOAJg/pmfgOnJeJ9l8DGe3CB64lyhQJrnj2C2C52sJEMmO9ZXyzmIYhskBkwd0w9/PnobT9xmAR1dswr7XLMGjKzZB+NWzZfIKi+Q8kx+7RXYD94IyyZ0qklhz6f44a9agwHXEQ/pVwyJXkwiRmo4RabObhc4kdxi7RYo3aWWSNXYL9TMo5aw7wzBMrqhIxvHDuUPx2BlT0btLJc6+bwW+c+fL2PT511GH1uFgkZxn8qELsmkmAgBVZtc9P9FSVZZIeYvdnbXNFvmekiGaifi1WZYD9mTDlLxnkn3iKBXCDtyTmWRdF8vKFG3QGYZhGD3De3bCI6fvjZ8eNBwvbvgMc65Zgtuf32AlJpj8wyI5z+SnuoXxmKlIrjaFS6rBeYExZGn5cGMP3MvCbmHu6/Jk3PF/voiRrj5I6RC2416QJ1mdVtp7i2EYJvfEY4STpvbDU+dOx8R+XXHpwjU4/KYXsO6jL6IOrUPAIjnP5LVOcpae5G27WjKOQd6Cz9ktdHM1yVB2C5+BezFnJjnvzURQ2haCsM1EWtqMiy2d3ULNLpfwrmIYhskrvbtU4Y4FE/CbY8fgvc+2Y95vn8cvFq7BFzuaow6tpGGRnGfykc3cZ3AP/GC/wejfvTqj5avLDeGyfWfmIlne7MlZxz1Iu0W4TLJOnMp9XZEsjCe51Dvuha2iIluYazPJZSySGYZhcgERYf6YXnjmvBk4enwf3L50A2ZdtQSPvPo+D+zLEyyS80yuOtKp1FUlccasQRmvWwqXbVkULJc+1Jx13JOZ5FB1kvXZTTlNlrHLvye5tA0EYUWyLBeo8yRXlbHdgmEYJpd0qS7DZYePxF9P3xu9ulTivAdW4ujfL8OaD9iCkWtYJHdAZMe9bLr6yKvWnFW3MB/DiGT/ttTOTHK+rRBU8nWSw9l6WgPsFmp2OVc1tRmGYRhgdJ/O+MtpU3D54SOxfvNXOOh3/8LFj63G1q/ZgpEr+LTVATlsbC/UlCdw2LheGa9DDtYKMc4uFHYmORd2C/8Sd7nEbwBhqSCvV7Kpk1zBA/cYhmHyRixGOHZiXyz+wT44bq/dcfeyjZh11bN4YPl7aOMqGFnDIrkD0qdrFd64ZH8MqK/JeB25tltIwtotdJv1DrX8cRYAABZJSURBVNzLcyYZJZ5JDlkCTloqyjTl+8qVabk4VIY01Ga/EoZhmBKjc1UZfnHoCDx2xlQ0da/GDx9ahSNufgFvbNoadWjtmkTqWRjGi7Rb5KqjoBTdWdktZAk405Oc73EMRJT/jURIWLvFn787CU+v+Rg15d6fE/UiIheHyqNn7I2dzZmXLmQYhillRvSqw4OnTsYjr23C5X9fi4Ovfx7fnNgXF+w/BJ2ryqIOr93BIpnJiFZTp+SqTnKrJZJD2C1ienEec3mSW/J8q8kIoXQzyfGQA/cG1NdgwIzUdyVykXWvSMa1AwQZhmEYg1iMcOSevTFneAOue/ot3L3sv3ji9Q/xw7lDccz4PnkpKFCqsN2CyYhWmUnO0ZdNlhEL25Zap7fiLk+yrN+bLwy7RV43ESmW3SLHDWMYhmGY/FNXmcTPD94DC8+cikE9anHRI6/jsBuXYsV7n0cdWruBRTKTEXJAQK5EcrOZmg7TltrPbmFnkk2R3JrfTHLpd9xzPma/vlLeWwzDMMXJsMZOuP/USbjumDH4YOsOHHbjUlz48Cp8tm1X1KEVPSySmYzoZzYyGdGzLifrk3aLslDVLfTlxOw6ydJukedMsk/nv1JBXgDlyndewruKYRimqCEiHDq2F/55/gycPLUfHnrlfcy86lnc8+//WudfxguLZCYjpg+ux1PnTsdR43vnZH3NbeHtFidOacJxe+3umS6XlSI53198P9tHqWA1E8lRKrmULygYhmHaA7UVSfx43nD8/expGN7YCT/96xs45Prn8cp//xd1aEUJi2QmYwY31OasBJpsSBHGbjF/TC8cOLLRM12KsHLTbtGcZ7sFUNo+27Ad9xiGYZj2xaCGWvz5u3vh+m+Oxadf7cIRN72AHzy4Ep98tTPq0IoKFslMUSAFbRi7hR/SblGo6gel3nFP7s8QVflCwSOqGYZhigciwkGjeuKZ82fgezMG4NEVmzDzqmdx59INaGnlUpsAi2SmSJBfyDB2Cz+sgXshstG5oMPYLXLcepxhGIYpHqrLE7jwgKH4xznTMaZPZ1z8+Boc9Lvn8dKGz6IOLXJYJDNFgVUnOQuBG3fZLfJNqQ/cy5VItqtklO6+YhiGae8MqK/B3d+ZiJuPH4cvd7Tg6N8vw7n3r8DmL3ZEHVpksEhmigJpt0hmcUve3ZY635R6JtmqbpGlTULaX0p5XzEMw5QCRIS5Ixrx9HkzcOasgfjbqg8x6+oluPVf/7FKtXYkWCTniZuOG4ezZw+KOox2Q0saA/f8GN27M6YPri+YSCaUtoXAaiaSpbq1RHLWETEMwzCFoLIsjvP3G4Knzp2OCU1d8Mu/rcW83/4Ly975NOrQCgqL5DxxwMhGnDtncNRhtBtarLbUmR+S+w5vwN3fmZjVOtKixO0W0r6SbSZZXrQ0cy1OhmGYdkVT92rcvmACbj1hPL5ubsU3/vBvnHnva/hoa8ewYLBIZooCuy119qKzrIB2i1JOj04b3B3nzRmMgT1qslpP365VAOwujQzDMEz7gYiw7/AGLDp3Bs7ZdxCeWv0RZl39LG5e8g52tZS2BYNFMlMUWB33ciBwywqUSS51u0WniiTOmj0o60zyTcfviauOGo0+plhmGIZh2h8VyTjO2Xcwnj5vBqYM6I7L/74Oc3/zHBau+qBk/coskpmiQH7BcmGVyMbXnA4xopK2W+SKrtVlOHLP3HRmZBiGYaKlT9cq3HrieNyxYAIggDP+/BqmX7EYNyxej8+27Yo6vJySiDoAhgFsT3JO7BaFyiSXttuCYRiGYXyZObQHpg+ux7NvbsadL2zElU++id888zYOHdMTJ05pwh4966IOMWtYJDNFgWwm0q7sFkQgttkyDMMwHZR4jDB7WANmD2vA2x9/ibuWbcTDr2zCA8vfx8R+XfHtKU2YM7wBiUINqM8xLJKZoiCXmeRkojD5XQLX/mUYhmEYABjUUItfHjoSF+w/FA8ufw93LduI0/70KnrWVeBbk5tw7IQ+6FJdFnWYadE+pT1TcrTkoOOepKB2C1bJDMMwDGNRV5nEydP649kfzMQfThiPfvXV+PU/1mHSZc/gwodXYe2HX0QdYmg4k8wUBVZ1ixwI3GyrMYQlRgQh2G/BMAzDMG7iMcKc4Q2YM7wBb370Je58YSP+8tr7uO/l9zCpf1csmNIPc4Y3FOycnQkpFQkR9SGixUS0hohWE9HZmnn2IaKtRLTC/PuZ8tpcInqTiNYT0YW5fgNMadCaQ7tFobK7RECM78UwDMMwTCBDdqvFZYePxL8vmo2LDhiK9z77Gt/74yuYfsVi3PLcO9i6vTnqELWEySS3ADhfCPEqEdUCeIWIFgkh1rjm+5cQ4iB1AhHFAdwAYA6A9wG8TESPaZZlOjiVyTi+bm4t6itKNzxwj2EYhmHC07mqDKfOGICTpvbD02s3484XNuBXT6zDtYvexmHjemHBlCYMbqiNOkyLlCJZCPEhgA/N518S0VoAvQCEEboTAawXQvwHAIjoPgDzQy7LdCAeO2NvLF3/Sbvy+PLAPYZhGIZJn0Q8hrkjdsPcEbth7Ydf4K4XNuLhV97Hn198F3sP7IYFU/ph1tAekSfO0vIkE1ETgLEAXtS8PJmIVgL4AMAPhBCrYYjp95R53gewV0aRMiXNoIZaDCqiq8cwGHWSWSUzDMMwTKYMa+yEy48Yhf+bOxT3vvwu7ln2X3z37uXo07USJ05uwlHj+6CuMhlJbKEdlURUA+BhAOcIIdxDE18FsLsQYjSA3wH4a7qBENEpRLSciJZv2bIl3cUZpuDEiCUywzAMw+SCLtVlOH2fgfjXD2fixuPGobFTJX75t7WY9Ktn8JO/vo71m78seEyhMslElIQhkP8khHjE/boqmoUQTxDRjUTUHcAmAH2UWXub0zwIIW4BcAsAjB8/np2eTNHDdguGYRiGyS2JeAwHjmzEgSMb8camrbjrhY14YPn7+OO/38W0Qd3x7b2bsM/gHogVwIoRproFAbgNwFohxDU+8+xmzgcimmiu91MALwMYRET9iKgMwLEAHstV8AwTJTEixFglMwzDMExeGNGrDlceNRrLLpyFC/Yfgrc//grfuXM5Zl79LG5/fgO+2JHfqhhhMsl7A/gWgNeJaIU57UcA+gKAEOJmAEcCOI2IWgB8DeBYYRSQbSGiMwA8CSAO4HbTq8wweeWFC2ehPAeNSQIhcHULhmEYhskz3WrK8f2ZA3HK9P54cvVHuGPpRly6cA2ufupNHLlnb5wwpQkD6mtyvt0w1S2eB4Ktl0KI6wFc7/PaEwCeyCg6hsmQnp0r874NtlswDMMwTOFIxmM4aFRPHDSqJ1a9/znufGEj7n3pPdy17L+YMbgelx0+Mqfnf+64xzAZcvi4XmhtizoKhmEYhul4jOrdGdccPQYXHTAM9770Lp54/UN0rS7L6TZYJDNMhhwzoW/UITAMwzBMh6a+thxnzR6EM2cNzHmvBW6qyzAMwzAMw7Rr8tGMjEUywzAMwzAMw7hgkcwwDMMwDMMwLlgkMwzDMAzDMIwLFskMwzAMwzAM44JFMsMwDMMwDMO4YJHMMAzTQSCiCiJ6iYhWEtFqIrpEM085Ed1PROuJ6EUiaip8pAzDMNHDIplhGKbjsBPALCHEaABjAMwlokmueU4C8D8hxEAA1wL4dYFjZBiGKQpYJDMMw3QQhMFX5r9J80+4ZpsP4C7z+UMAZlM+CpAyDMMUOSySGYZhOhBEFCeiFQA2A1gkhHjRNUsvAO8BgBCiBcBWAN006zmFiJYT0fItW7bkO2yGYZiCwyKZYRimAyGEaBVCjAHQG8BEIhqR4XpuEUKMF0KMr6+vz22QDMMwRQCLZIZhmA6IEOJzAIsBzHW9tAlAHwAgogSAOgCfFjY6hmGY6GGRzDAM00Egonoi6mw+rwQwB8A612yPATjRfH4kgH8KIdy+ZYZhmJInEXUADMMwTMFoBHAXEcVhJEkeEEIsJKJLASwXQjwG4DYA9xDRegCfATg2unAZhmGig0UywzBMB0EIsQrAWM30nynPdwA4qpBxMQzDFCNst2AYhmEYhmEYFyySGYZhGIZhGMYFi2SGYRiGYRiGcUHFOGiZiLYA+G+ai3UH8EkewsknHHNhaG8xt7d4AY5ZZXchRIcqHJzhbzZQPMdNscQBFE8sxRIHwLHoKJY4gOKJJdM4fH+zi1IkZwIRLRdCjI86jnTgmAtDe4u5vcULcMxMZhTLZ1AscQDFE0uxxAFwLMUcB1A8seQjDrZbMAzDMAzDMIwLFskMwzAMwzAM46KURPItUQeQARxzYWhvMbe3eAGOmcmMYvkMiiUOoHhiKZY4AI5FR7HEARRPLDmPo2Q8yQzDMAzDMAyTK0opk8wwDMMwDMMwOaEkRDIRzSWiN4loPRFdGHU8fhDRRiJ6nYhWENFyc1pXIlpERG+bj10ijvF2ItpMRG8o07QxksFvzf2+iojGFUm8FxPRJnM/ryCiA5XXLjLjfZOI9i90vGYMfYhoMRGtIaLVRHS2Ob2Y97NfzEW5r4mogoheIqKVZryXmNP7EdGLZlz3E1GZOb3c/H+9+XpTIePtaBTLb7bu9yOiOLTfr4hi0X53IownTkSvEdHCiOPwnL8jjKUzET1EROuIaC0RTY4ghiHK7/4KIvqCiM4pdBxKPOeax+sbRHQvEVXkZMVCiHb9ByAO4B0A/QGUAVgJYHjUcfnEuhFAd9e0KwBcaD6/EMCvI45xOoBxAN5IFSOAAwH8HQABmATgxSKJ92IAP9DMO9w8PsoB9DOPm3gEMTcCGGc+rwXwlhlbMe9nv5iLcl+b+6rGfJ4E8KK57x4AcKw5/WYAp5nPTwdws/n8WAD3F3ofd5S/YvrN1v1+RBSH9vsVUSza706E++Y8AH8GsDDiz8hz/o4wlrsAnGw+LwPQOeJ44gA+glFvOIrt9wKwAUCl+f8DABbkYt2lkEmeCGC9EOI/QohdAO4DMD/imNJhPowDHubjoRHGAiHEcwA+c032i3E+gLuFwb8BdCaixsJEauATrx/zAdwnhNgphNgAYD2M46egCCE+FEK8aj7/EsBaGF/yYt7PfjH7Eem+NvfVV+a/SfNPAJgF4CFzunsfy33/EIDZREQFCrejUTS/2Wn+fuQzjnS/X/mMxe+7U3CIqDeAeQBujWL7xQgR1cG4uLsNAIQQu4QQn0cbFWYDeEcIkUlDoVyRAFBJRAkAVQA+yMVKS0Ek9wLwnvL/+4joxyUEAsBTRPQKEZ1iTmsQQnxoPv8IQEM0oQXiF2Mx7/szTGvC7YqFpejiNW/rj4WRrWkX+9kVM1Ck+9q8TbsCwGYAi2BkLz8XQrRoYrLiNV/fCqBbIePtQER+bBQzmu9XFDE4vjtCiKhiuQ7ADwG0RbR9Fd35Owr6AdgC4A7ThnIrEVVHGA9g3H27N6qNCyE2AbgKwLsAPgSwVQjxVC7WXQoiuT0xVQgxDsABAL5PRNPVF4Vxn6Coy420hxgB3ARgAIAxML4wV0cbjh4iqgHwMIBzhBBfqK8V637WxFy0+1oI0SqEGAOgN4zs5dCIQ2KYQIJ+EwqJ+7tDRCMKHQMRHQRgsxDilUJv24fA83cBScCwCN0khBgLYBsMe14kmOM6DgHwYIQxdIFxN6ofgJ4Aqono+FysuxRE8iYAfZT/e5vTig7zagdCiM0A/gLjxP2xvHVuPm6OLkJf/GIsyn0vhPjY/JFvA/AH2Lf5iyZeIkrCOBn+SQjxiDm5qPezLub2sK/NW5GLAUyGYVVJaGKy4jVfrwPwaYFD7SgUzbFRTPj8JkSK8t2ZG8Hm9wZwCBFthGHJmUVEf4wgDgC+5+8oeB/A+0p2/yEYojkqDgDwqhDi4whj2BfABiHEFiFEM4BHAEzJxYpLQSS/DGCQOWq9DEba/7GIY/JARNVEVCufA9gPwBswYj3RnO1EAI9GE2EgfjE+BuAEMpgE4xbHh7oVFBKXX/cwGPsZMOI91qxk0A/AIAAvRRAfwfCTrRVCXKO8VLT72S/mYt3XRFRPRJ3N55UA5sDweS4GcKQ5m3sfy31/JIB/mtl8Jve0i9/sQhLwmxBFLLrvzrpCxyGEuEgI0VsI0QTjGPmnECIn2cF0CTh/FxwhxEcA3iOiIeak2QDWRBGLyTcQodXC5F0Ak4ioyvwuzYbxe589uRj9F/UfjNH/b8HwHP446nh8YuwPYxT3SgCrZZwwfI/PAHgbwNMAukYc570wbps3w7hiPckvRhijoG8w9/vrAMYXSbz3mPGsgnHybVTm/7EZ75sADohoH0+FYaVYBWCF+Xdgke9nv5iLcl8DGAXgNTOuNwD8zJzeH4ZYXw/j9mC5Ob3C/H+9+Xr/KI6NjvJXLL/Zut+PiOLQfr8iikX73Yn4eNkHEVa38Dt/RxjPGADLzc/orwC6RBRHNYw7bnVFcIxcAuNi7g3zvFSei/Vyxz2GYRiGYRiGcVEKdguGYRiGYRiGySkskhmGYRiGYRjGBYtkhmEYhmEYhnHBIplhGIZhGIZhXLBIZhiGYRiGYRgXLJKZdgsRtRLRCiJaSUSvElFg8XAi6kxEp4dY77NEND53kTIMwzDKb7b8y1mnOCJqIqJIahczpUsi9SwMU7R8LYzWqSCi/QFcBmBGwPydAZwO4MYCxMYwDMM4sX6zGaY9wJlkplToBOB/AEBENUT0jJldfp2I5pvzXA5ggJnBuNKc9//MeVYS0eXK+o4iopeI6C0imlbYt8IwDNNxIKKNRHSF+Vv8EhENNKc3EdE/iWiV+Zve15zeQER/MX+3Vyp3EeNE9AciWk1ET5kdAxkmYziTzLRnKoloBYxuaY0AZpnTdwA4TAjxBRF1B/BvInoMwIUARijZ5wMAzAewlxBiOxF1VdadEEJMJKIDAfwcRm94hmEYJnPkb7bkMiHE/ebzrUKIkUR0AoDrABwE4HcA7hJC3EVE3wHwWwCHmo9LhBCHEVEcQA2ALgAGAfiGEOK7RPQAgCMA/LEwb40pRVgkM+0Z1W4xGcDdRDQCRhvnXxHRdABtAHoBaNAsvy+AO4QQ2wFACPGZ8toj5uMrAJryEz7DMEyHIshuca/yeK35fDKAw83n9wC4wnw+C8AJACCEaAWwlYi6ANgghJAinH+7maxhkcyUBEKIZWbWuB7AgebjnkKIZiLaCCPbnA47zcdW8PeEYRgm3wif5+mwU3neCoDtFkxWsCeZKQmIaCiAOIBPAdQB2GwK5JkAdjdn+xJArbLYIgDfJqIqcx2q3YJhGIYpHMcoj8vM5y8AONZ8fhyAf5nPnwFwGgAQUZyI6goVJNOx4AwZ055R/W0E4EQhRCsR/QnA40T0OoDlANYBgBDiUyJaapYJ+rsQ4gIiGgNgORHtAvAEgB9F8D4YhmE6Am5P8j+EELIMXBciWgUjG/wNc9qZAO4gogsAbAHwbXP62QBuIaKTYGSMTwPwYd6jZzocJESmdzUYhmEYhmGyw7TEjRdCfBJ1LAyjwnYLhmEYhmEYhnHBmWSGYRiGYRiGccGZZIZhGIZhGIZxwSKZYRiGYRiGYVywSGYYhmEYhmEYFyySGYZhGIZhGMYFi2SGYRiGYRiGccEimWEYhmEYhmFc/H8jUJGRag2ougAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch: 10 | Time: 1m 10s\n", + "\tTrain Loss: 2.998 | Train PPL: 20.040\n", + "\t Val. Loss: 4.710 | Val. PPL: 111.007\n" + ] + } + ], + "source": [ + "for epoch in range(N_EPOCHS):\n", + "\n", + " start_time = time.time()\n", + "\n", + " train_loss = train(model, train_iterator, optimizer, criterion, CLIP, train_history, valid_history)\n", + " valid_loss = evaluate(model, valid_iterator, criterion)\n", + "\n", + " end_time = time.time()\n", + "\n", + " epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n", + "\n", + " if valid_loss < best_valid_loss:\n", + " best_valid_loss = valid_loss\n", + " torch.save(model.state_dict(), 'tut1-model.pt')\n", + "\n", + " train_history.append(train_loss)\n", + " valid_history.append(valid_loss)\n", + " print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')\n", + " print(f'\\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')\n", + " print(f'\\t Val. Loss: {valid_loss:.3f} | Val. PPL: {math.exp(valid_loss):7.3f}')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5WDU5eQ75X3y" + }, + "source": [ + "__Let's take a look at our network quality__:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cOTL-zv15X3y" + }, + "outputs": [], + "source": [ + "del utils" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "M0OKfJS05X3y" + }, + "outputs": [], + "source": [ + "import utils\n", + "import imp\n", + "imp.reload(utils)\n", + "generate_translation = utils.generate_translation\n", + "remove_tech_tokens = utils.remove_tech_tokens\n", + "get_text = utils.get_text\n", + "flatten = utils.flatten" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "S8SYGrQq5X3y" + }, + "outputs": [], + "source": [ + "batch = next(iter(test_iterator))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "xlcz-CRJ5X3y", + "outputId": "f4eae206-5f9c-4762-b494-22355e8cb326" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Original: there is a 24 - hour front desk at the property .\n", + "Generated: the property offers a 24 - hour front desk . .\n", + "\n", + "Original: this property also features free wifi .\n", + "Generated: free wifi access . . . .\n", + "\n" + ] + } + ], + "source": [ + "for idx in [1,2]:\n", + " src = batch.src[:, idx:idx+1]\n", + " trg = batch.trg[:, idx:idx+1]\n", + " generate_translation(src, trg, model, TRG.vocab)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "oVKMRVa85X3y" + }, + "outputs": [], + "source": [ + "from nltk.translate.bleu_score import corpus_bleu\n", + "\n", + "# \"\"\" Estimates corpora-level BLEU score of model's translations given inp and reference out \"\"\"\n", + "# translations, _ = model.translate_lines(inp_lines, **flags)\n", + "# # Note: if you experience out-of-memory error, split input lines into batches and translate separately\n", + "# return corpus_bleu([[ref] for ref in out_lines], translations) * 100" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "-DWG8DZ45X3y" + }, + "outputs": [], + "source": [ + "import tqdm" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "OpJHC6Kn5X3y", + "outputId": "16b1e810-83a5-49a4-c20e-3d14b0aebce2" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "59it [00:03, 18.87it/s]\n" + ] + } + ], + "source": [ + "original_text = []\n", + "generated_text = []\n", + "model.eval()\n", + "with torch.no_grad():\n", + "\n", + " for i, batch in tqdm.tqdm(enumerate(test_iterator)):\n", + "\n", + " src = batch.src\n", + " trg = batch.trg\n", + "\n", + " output = model(src, trg, 0) #turn off teacher forcing\n", + "\n", + " #trg = [trg sent len, batch size]\n", + " #output = [trg sent len, batch size, output dim]\n", + "\n", + " output = output.argmax(dim=-1)\n", + "\n", + " original_text.extend([get_text(x, TRG.vocab) for x in trg.cpu().numpy().T])\n", + " generated_text.extend([get_text(x, TRG.vocab) for x in output[1:].detach().cpu().numpy().T])\n", + "\n", + "# original_text = flatten(original_text)\n", + "# generated_text = flatten(generated_text)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "lG_rvxZj5X3y", + "outputId": "0d8ab766-e4b1-4faf-db04-3cd354764939" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "14.139920232081806" + ] + }, + "execution_count": 111, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "corpus_bleu([[text] for text in original_text], generated_text) * 100" + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "colab": { + "machine_shape": "hm", + "provenance": [] + }, + "kernelspec": { + "display_name": "Py3 Research", + "language": "python", + "name": "py3_research_kernel" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.7" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/homeworks/hw2_seq2seq/my_network.py b/homeworks/hw2_seq2seq/my_network.py new file mode 100644 index 0000000..966416d --- /dev/null +++ b/homeworks/hw2_seq2seq/my_network.py @@ -0,0 +1,182 @@ +import torch +import torch.nn as nn +import torch.optim as optim + +import torchtext +from torchtext.datasets import TranslationDataset, Multi30k +from torchtext.data import Field, BucketIterator + +import random +import math +import time + + +class Encoder(nn.Module): + def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout): + super().__init__() + + self.input_dim = input_dim + self.emb_dim = emb_dim + self.hid_dim = hid_dim + self.n_layers = n_layers +# self.dropout = dropout + + self.embedding = nn.Embedding( + num_embeddings=input_dim, + embedding_dim=emb_dim + ) + # + + self.rnn = nn.LSTM( + input_size=emb_dim, + hidden_size=hid_dim, + num_layers=n_layers, + dropout=dropout + ) + # + + self.dropout = nn.Dropout(p=dropout)# + + def forward(self, src): + + #src = [src sent len, batch size] + + # Compute an embedding from the src data and apply dropout to it + embedded = self.embedding(src)# + + embedded = self.dropout(embedded) + + output, (hidden, cell) = self.rnn(embedded) + #embedded = [src sent len, batch size, emb dim] + + # Compute the RNN output values of the encoder RNN. + # outputs, hidden and cell should be initialized here. Refer to nn.LSTM docs ;) + + # + + #outputs = [src sent len, batch size, hid dim * n directions] + #hidden = [n layers * n directions, batch size, hid dim] + #cell = [n layers * n directions, batch size, hid dim] + + #outputs are always from the top hidden layer + + return hidden, cell + + +class Decoder(nn.Module): + def __init__(self, output_dim, emb_dim, hid_dim, n_layers, dropout): + super().__init__() + + self.emb_dim = emb_dim + self.hid_dim = hid_dim + self.output_dim = output_dim + self.n_layers = n_layers + self.dropout = dropout + + self.embedding = nn.Embedding( + num_embeddings=output_dim, + embedding_dim=emb_dim + ) + # + + self.rnn = nn.LSTM( + input_size=emb_dim, + hidden_size=hid_dim, + num_layers=n_layers, + dropout=dropout + ) + # + + self.out = nn.Linear( + in_features=hid_dim, + out_features=output_dim + ) + # + + self.dropout = nn.Dropout(p=dropout)# + + def forward(self, input, hidden, cell): + + #input = [batch size] + #hidden = [n layers * n directions, batch size, hid dim] + #cell = [n layers * n directions, batch size, hid dim] + + #n directions in the decoder will both always be 1, therefore: + #hidden = [n layers, batch size, hid dim] + #context = [n layers, batch size, hid dim] + + input = input.unsqueeze(0) + + #input = [1, batch size] + + # Compute an embedding from the input data and apply dropout to it + embedded = self.dropout(self.embedding(input))# + + #embedded = [1, batch size, emb dim] + + # Compute the RNN output values of the encoder RNN. + # outputs, hidden and cell should be initialized here. Refer to nn.LSTM docs ;) + # + + + #output = [sent len, batch size, hid dim * n directions] + #hidden = [n layers * n directions, batch size, hid dim] + #cell = [n layers * n directions, batch size, hid dim] + + #sent len and n directions will always be 1 in the decoder, therefore: + #output = [1, batch size, hid dim] + #hidden = [n layers, batch size, hid dim] + #cell = [n layers, batch size, hid dim] + + + output, (hidden, cell) = self.rnn(embedded, (hidden, cell)) + prediction = self.out(output.squeeze(0)) + + #prediction = [batch size, output dim] + + return prediction, hidden, cell + + +class Seq2Seq(nn.Module): + def __init__(self, encoder, decoder, device): + super().__init__() + + self.encoder = encoder + self.decoder = decoder + self.device = device + + assert encoder.hid_dim == decoder.hid_dim, \ + "Hidden dimensions of encoder and decoder must be equal!" + assert encoder.n_layers == decoder.n_layers, \ + "Encoder and decoder must have equal number of layers!" + + def forward(self, src, trg, teacher_forcing_ratio = 0.5): + + #src = [src sent len, batch size] + #trg = [trg sent len, batch size] + #teacher_forcing_ratio is probability to use teacher forcing + #e.g. if teacher_forcing_ratio is 0.75 we use ground-truth inputs 75% of the time + + # Again, now batch is the first dimention instead of zero + batch_size = trg.shape[1] + max_len = trg.shape[0] + trg_vocab_size = self.decoder.output_dim + + #tensor to store decoder outputs + outputs = torch.zeros(max_len, batch_size, trg_vocab_size).to(self.device) + + #last hidden state of the encoder is used as the initial hidden state of the decoder + hidden, cell = self.encoder(src) + + #first input to the decoder is the tokens + input = trg[0,:] + + for t in range(1, max_len): + + output, hidden, cell = self.decoder(input, hidden, cell) + outputs[t] = output + teacher_force = random.random() < teacher_forcing_ratio + top1 = output.max(1)[1] + input = (trg[t] if teacher_force else top1) + + return outputs diff --git a/homeworks/hw2_seq2seq/utils.py b/homeworks/hw2_seq2seq/utils.py new file mode 100644 index 0000000..f3691d2 --- /dev/null +++ b/homeworks/hw2_seq2seq/utils.py @@ -0,0 +1,33 @@ + +def flatten(l): + return [item for sublist in l for item in sublist] + +def remove_tech_tokens(mystr, tokens_to_remove=['', '', '', '']): + return [x for x in mystr if x not in tokens_to_remove] + + +def get_text(x, TRG_vocab): + text = [TRG_vocab.itos[token] for token in x] + try: + end_idx = text.index('') + text = text[:end_idx] + except ValueError: + pass + text = remove_tech_tokens(text) + if len(text) < 1: + text = [] + return text + + +def generate_translation(src, trg, model, TRG_vocab): + model.eval() + + output = model(src, trg, 0) #turn off teacher forcing + output = output.argmax(dim=-1).cpu().numpy() + + original = get_text(list(trg[:,0].cpu().numpy()), TRG_vocab) + generated = get_text(list(output[1:, 0]), TRG_vocab) + + print('Original: {}'.format(' '.join(original))) + print('Generated: {}'.format(' '.join(generated))) + print() diff --git a/week05_transformer/.ipynb_checkpoints/README-checkpoint.md b/week05_transformer/.ipynb_checkpoints/README-checkpoint.md new file mode 100644 index 0000000..e69de29 diff --git a/week05_transformer/.ipynb_checkpoints/transformer-checkpoint.ipynb b/week05_transformer/.ipynb_checkpoints/transformer-checkpoint.ipynb new file mode 100644 index 0000000..c598ef6 --- /dev/null +++ b/week05_transformer/.ipynb_checkpoints/transformer-checkpoint.ipynb @@ -0,0 +1,1852 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "source": [ + "# **Seminar - Attention и Transformer**" + ], + "metadata": { + "id": "jcYtDZ6yYlk7" + } + }, + { + "cell_type": "markdown", + "source": [ + "## 1. Let's build Transformer from scratch in Pytorch\n", + "\n", + "" + ], + "metadata": { + "id": "GFS003OQYv2w" + } + }, + { + "cell_type": "code", + "source": [ + "from collections import OrderedDict\n", + "import torch\n", + "import torch.nn as nn\n", + "import torch.nn.functional as F\n", + "import math\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline" + ], + "metadata": { + "id": "gEmHY574YpNu" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.1 Multi-head Attention\n", + "\n", + "#### Main class - MultiHeadAttention\n", + "\n", + "**Initialization:**\n", + "* _in_size_ ~ size of the input embeddings\n", + "* _head_size_ ~ size of the embeddings for Q, K, V matrices after transformation\n", + "* _num_heads_ ~ number of heads\n", + "* _out_size_ ~ size of the output embeddings\n", + "* _query_in_size_ ~ size of the input embeddings\n", + "\n", + "**Forward:**\n", + "* query, key, value ~ 3 tensors (one for each Q, K, and V transformation - these are not yet the tensors of shape $\\text{batch_size} \\times seq \\times d_k$, but tensors of shape $\\text{batch_size} \\times seq \\times \\text{in_size}$)\n", + "* mask ~ boolean mask for Masked Multi-head Attention (in the decoder)\n", + "\n", + "$$ Attention(Q, K, V) = softmax\\Bigg(\\frac{QK^T}{\\sqrt{d_k}}\\Bigg) \\cdot V $$\n", + "$$ MultiHead(Q, K, V) = Concat(head_1, ..., head_H) \\cdot W^O \\quad ; \\quad head_i = Attention(Q W_i^Q, K W_i^K, V W_i^V)$$" + ], + "metadata": { + "id": "ygtUy2dGZAC8" + } + }, + { + "cell_type": "code", + "source": [ + "class MultiHeadAttention(nn.Module):\n", + " \"\"\"\n", + " Class to calculate Multi-head attention (or Masked Multi-head attention for the decoder) operation\n", + " \"\"\"\n", + " def __init__(self, in_size, head_size, num_heads, out_size, query_in_size=None):\n", + " \"\"\"\n", + " Args:\n", + " in_size: embedding size of input\n", + " head_size: hidden size of Q, K, V matrices\n", + " num_heads: number of heads\n", + " out_size: output embedding size\n", + " query_in_size: embedding size of input for query (if not provided - same as in_size)\n", + " \"\"\"\n", + " super(MultiHeadAttention, self).__init__()\n", + "\n", + " # Store all passed layer hyperparameters\n", + " self.in_size = in_size\n", + " self.head_size = head_size\n", + " self.num_heads = num_heads\n", + " self.out_size = out_size\n", + " self.query_in_size = self.in_size if query_in_size is None else query_in_size\n", + "\n", + " # Linear transformations for Q, K, V matrices (get all Q, K, V matrices directly)\n", + " self.query_matrix = nn.Linear(self.query_in_size, self.num_heads * self.head_size, bias=False)\n", + " self.key_matrix = nn.Linear(self.in_size, self.num_heads * self.head_size, bias=False)\n", + " self.value_matrix = nn.Linear(self.in_size, self.num_heads * self.head_size, bias=False)\n", + " # Linear transformation for concatenating heads\n", + " self.out = nn.Linear(self.head_size * self.num_heads, self.out_size)\n", + "\n", + " def forward(self, query, key, value, mask=None):\n", + " \"\"\"\n", + " Args:\n", + " query : tensor for query\n", + " key : tensor for key\n", + " value : tensor for value\n", + " mask: mask for the decoder\n", + "\n", + " Returns:\n", + " output vector from multihead attention\n", + " \"\"\"\n", + " # Tensors come with the shape batch_size x seq_len x in_size\n", + " batch_size = key.size(0)\n", + " seq_len = key.size(1)\n", + "\n", + " # The number of tokens in the query will differ for the decoder\n", + " query_seq_len = query.size(1)\n", + "\n", + " # Apply linear transformations to the input\n", + " q = self.query_matrix(query) # (batch_size, query_seq_len, head_size * num_heads)\n", + " k = self.key_matrix(key) # (batch_size, seq_len, head_size * num_heads)\n", + " v = self.value_matrix(value) # (batch_size, seq_len, head_size * num_heads)\n", + "\n", + " q = q.view(batch_size, query_seq_len, self.num_heads, self.head_size).transpose(1,2) # (batch_size, num_heads, query_seq_len, head_size)\n", + " k = k.view(batch_size, seq_len, self.num_heads, self.head_size).transpose(1,2) # (batch_size, num_heads, seq_len, head_size)\n", + " v = v.view(batch_size, seq_len, self.num_heads, self.head_size).transpose(1,2) # (batch_size, num_heads, seq_len, head_size)\n", + "\n", + " # Считаем релевантность\n", + " relevance = q @ k.transpose(2, 3) / math.sqrt(self.head_size) # (batch_size, num_heads, query_seq_len, seq_len)\n", + "\n", + " # Если есть маска (для декодера), то заполняем значения по маске как минус бесконечность (чтобы exp(r) = 0 в softmax)\n", + " if mask is not None:\n", + " relevance = relevance.masked_fill(mask, -torch.inf)\n", + "\n", + " # Получаем вероятности\n", + " relevance = F.softmax(relevance, dim=-1)\n", + "\n", + " # Считаем выходы из каждой головы\n", + " head_i = torch.matmul(relevance, v) # (batch_size, num_heads, query_seq_len, head_size)\n", + "\n", + " # Конкатенируем выходы\n", + " concat = head_i.transpose(1,2).reshape(batch_size, query_seq_len, self.head_size * self.num_heads) # (batch_size, query_seq_len, num_heads * head_size)\n", + "\n", + " return self.out(concat) # (batch_size, query_seq_len, out_size)" + ], + "metadata": { + "id": "U2vT-vwEY6_S" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing MultiHeadAttention for the encoder" + ], + "metadata": { + "id": "xyTV7cqXay9b" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = MultiHeadAttention(\n", + " in_size=10,\n", + " head_size=4,\n", + " num_heads=3,\n", + " out_size=15,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "8A6Af8DEazNa", + "outputId": "80961b75-44ba-4ef2-839d-31f34e4e5fb9" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (out): Linear(in_features=12, out_features=15, bias=True)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 3 + } + ] + }, + { + "cell_type": "code", + "source": [ + "# Check in normal forward pass from the encoder\n", + "tmp_input = torch.rand(2, 5, 10)\n", + "\n", + "print(\"Encoder-like input, no mask\")\n", + "print(f'Input shape: {tmp_input.shape}')\n", + "tmp_output = tmp_layer(tmp_input, tmp_input, tmp_input)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "3ilYxygIa33f", + "outputId": "6412f347-246a-47c8-d7bc-3368ee9743c2" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Encoder-like input, no mask\n", + "Input shape: torch.Size([2, 5, 10])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing MultiHeadAttention for a mixture of encoder and decoder" + ], + "metadata": { + "id": "if4DQzHFbAml" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = MultiHeadAttention(\n", + " in_size=10,\n", + " head_size=4,\n", + " num_heads=3,\n", + " out_size=15,\n", + " query_in_size=12,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "ATH6d79ibCn2", + "outputId": "295cfa64-4d6b-4ad6-839d-e8879728c6fc" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=12, out_features=12, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (out): Linear(in_features=12, out_features=15, bias=True)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 5 + } + ] + }, + { + "cell_type": "code", + "source": [ + "# Check forward pass in the decoder, where we mix information from the encoder and decoder\n", + "tmp_input_q = torch.rand(2, 5, 12)\n", + "tmp_input_kv = torch.rand(2, 7, 10)\n", + "\n", + "print(\"Encoder+Decoder-like input, no mask\")\n", + "print(f'Input Q shape: {tmp_input_q.shape}')\n", + "print(f'Input KV shape: {tmp_input_kv.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input_q, tmp_input_kv, tmp_input_kv)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input_q, tmp_input_kv, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "FDNk8KctbEZj", + "outputId": "e7ecebaf-e8aa-466e-b9d8-b3652ef25a91" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Encoder+Decoder-like input, no mask\n", + "Input Q shape: torch.Size([2, 5, 12])\n", + "Input KV shape: torch.Size([2, 7, 10])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Triangular Mask in the decoder" + ], + "metadata": { + "id": "4_a1I_XnbKSJ" + } + }, + { + "cell_type": "code", + "source": [ + "def make_decoder_mask(decoder_embed):\n", + " \"\"\"\n", + " Make mask for decoder Masked Multi-head Attention based on input sequence\n", + " Args:\n", + " decoder_embed: decoder sequence after embed\n", + " Returns:\n", + " mask: mask for Masked Multi-head Attention\n", + " \"\"\"\n", + " batch_size, decoder_seq_len, _ = decoder_embed.shape\n", + " mask = torch.tril(torch.ones((decoder_seq_len, decoder_seq_len))).expand(\n", + " batch_size, 1, decoder_seq_len, decoder_seq_len\n", + " ).bool()\n", + " return mask" + ], + "metadata": { + "id": "sEZC_D24bMSe" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing MultiHeadAttention for the decoder with a mask" + ], + "metadata": { + "id": "0-KrJQ0ObP3c" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_input = torch.rand(1, 10, 256)\n", + "tmp_mask = make_decoder_mask(tmp_input)\n", + "print(f\"Mask shape: {tmp_mask.shape}\")\n", + "\n", + "# Visualize the mask\n", + "fig, ax = plt.subplots(figsize=(10, 10))\n", + "plt.imshow(tmp_mask[0, 0, :, :])\n", + "\n", + "# Add text labels\n", + "for i in range(tmp_mask.shape[-2]):\n", + " for j in range(tmp_mask.shape[-1]):\n", + " text = plt.text(j, i, tmp_mask[0, 0, i, j].item(), ha=\"center\", va=\"center\", color=\"red\")\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 848 + }, + "id": "qQjcaCqrbR-J", + "outputId": "8c8e8714-31b9-4caa-8cab-9854d8c2cf90" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Mask shape: torch.Size([1, 1, 10, 10])\n" + ] + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAygAAAMtCAYAAAB0OVVCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABPOklEQVR4nO39f5CedX0v/j+vbMgSkmxCgEASfkiU2JoIgUYoMFPqyJCeKZ3j5+NYZTxTLB2/0x7iD6IM4Bz1OCKUUhmNcPw1ih2pR+gAYvXT02+/+AEEa8mBBEGNqDSEEn4ILLshkM2PfX//gJNkIRHeyGbf2X08ZjLket/Xdd+v+5lld5973de9XSmlBAAAoAGTxnoAAACA/0NBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQjMl7+wGHh4ezYcOGzJgxI13X7e2HBwAAxkApJRs3bsy8efMyadKez5Ps9YKyYcOGHHHEEXv7YQEAgAY89NBDOfzww/d4+14vKDNmzEiSPHj369I33SvMXqn/a+Gbx3oEAAB41bZla27P/7OjD+zJXi8o/+dlXX3TJ6VvhoLySk3u9hvrEQAA4NUrz//n5S7z0BAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQeFVOaOsy43lprEeY58is3oyqyezejKrJ7N6Mqsns3rjJbPJYz3Aa23S3F/+xtvLhw9M+chBe2ma9p1fVuWMPPiS9bPzR9nQTR+Didons3oyqyezejKrJ7N6Mqsns3oTPbNXVVCuuuqqXH755Xn00Udz3HHH5fOf/3xOPPHE13q2V2X4ntft3LjpmXSXP5Vy+5E716btctKolGR7ksnd3hqvSXfm0Pxt3jJibSC9YzTNvkFm9WRWT2b1ZFZPZvVkVk9m9SZyZtUF5dprr82KFSvyxS9+MSeddFI++9nPZtmyZfn5z3+eOXPmjMaMdebs8pT6JiXdLms/fDaT3rEhw9fMTXfZU8naoZRvzUt37cZkYDjl63N3HNp97NfJT4ZSbjj8+YXhklzZn+6aweTX25MF+6WcNzs5c99vsVvTk/5u/xFr7yj3Z1lZl8OyKRszJT/K3Hwlx2Zzt/sPmQXl6fxV7snC9KckeTjT87mckPu72UmSReWJ/EXuzcL0ZyC9uSPz87Us3uP9tU5m9WRWT2b1ZFZPZvVkVk9m9SZyZtWPfsUVV+R973tf/vzP/zxJ8sUvfjHf+9738rWvfS0XXnjhaz7gaOgueTLl4wcnR+2XzJyUZOPLH7SyP931G1MuOyRZMCX50XPplj+WMrsnOWXqqM+8tw2ny1VZkkczLXOzKe/P6rwvP87nc8Ju978wd+ZXmZWVOT7D6fL6DGTbC5c4zS3P5NL8IFdncT6TpZmZoSzPmizP6pf8ZGBfJrN6Mqsns3oyqyezejKrJ7N6EyWzqoKyZcuW3HXXXbnooot2rE2aNCmnn356/vVf/3W3xwwNDWVoaGjH9uDg4Ksc9bVTzp+dnHbAKz9gqKRb2Z9y3bxk6Qtl5Kj9kjs3p7tmIGUfLyi/n0fynXLjju1VOSyf6k7esf1YpuXrZVE+mLv3+D/AnDybf8jCPNT1JUkezowdt52Vtbk5R+bG7pgdt11VluQzuSWfKydka9czGk9rVMmsnszqyayezOrJrJ7M6sms3kTOrKqgPPHEE9m+fXsOPfTQEeuHHnpo1q5du9tjLr300nzyk5989ROOhuP2f/l9drVuS7rnSvKuDSPXt5Zk8b7/WsA1OSQrd/nA3pyeHF8ey1lZmyOyMQdkW3oynN4Mp7dsy9BuTvtdn2OyInfl9LI+d2dObsvheeSFi7gWZCBHZyBvK+tHHNOTZG42ZX36RvX5jQaZ1ZNZPZnVk1k9mdWTWT2Z1ZvImY36C8wuuuiirFixYsf24OBgjjjiiNF+2N/sgBddFL+7N1vetsvfN5UkSfnGvGTui9rklH3/AvvNmTziHSEOLZtyce7IP2ZBrs7iDGZKFueJfCR3ZXKGM7Sb+/hGtyjfL0fmpDySE/No/iw/zSXlpNzRzc/UbMv3siDfzhtectzjqTiT1RCZ1ZNZPZnVk1k9mdWTWT2Z1ZvImVUVlIMPPjg9PT157LHHRqw/9thjOeyww3Z7TG9vb3p7Gz/LcFBPsnbLyLWfDO1MZ+GUlN4ueXjruLze5MWOSX+6lHwpx6V0zxew08p/vOxxD3czckNm5IYszEfLv2VZ1uWOzM8vMitHZXBcvy2ezOrJrJ7M6smsnszqyayezOpNpMyqflHjlClT8nu/93u5+eabd6wNDw/n5ptvzsknn/wbjmxbOXVqcs9Qct1g8sCWdJc/ObKwTJ+U/OWsdJ944vl91m1Nfrw5+erTz2+PMxsyPful5O35ZQ4rz+T08mDOzAN73H9K2Z7lZXWOLY9nTtmUReWJLMxTWf/C6xyvzRvzpjyZ5WV1Xl+ezvyyMSeXDVleVu+tpzTqZFZPZvVkVk9m9WRWT2b1ZFZvImVW/RKvFStW5Oyzz87SpUtz4okn5rOf/Ww2bdq041299klvnZacd2C6i59Mhkry7hnJO2ckP9t5sqxcMDs5qCfdyv5k/eNJX0/y5t6UDxw4hoOPjge6WflCOTbvys9zTu7LvTk4X8viXJBVu91/OF36siUXZFVmZSiDmZLbMz9/l0VJkn/vZuXD5bSck/tyRW5Jl5INmZ5bc/jefFqjSmb1ZFZPZvVkVk9m9WRWT2b1JlJmXSml1B505ZVX7vhFjUuWLMnKlStz0kknvaJjBwcHM3PmzPTfvyB9M6pO4Exoy+YtGesRAADgVdtWtuaW3JSBgYH09e35IvxXVVB+GwrKq6OgAACwL3ulBUVDAAAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABoxuSxHoBX5p83rBnrEfY5y+YtGesRAACo5AwKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgrsJWeUdbmx3DTWY+xTZFZPZvVkVk9m9WRWT2b1xktmk8d6gNfapLm//I23lw8fmPKRg/bSNPsGmdU5v6zKGXnwJetn54+yoZs+BhO1T2b1ZFZPZvVkVk9m9WRWb6JnNu4KyvA9r9u5cdMz6S5/KuX2I3euTdvlpFEpyfYkk7u9NV6TZFbvzhyav81bRqwNpHeMptk3yKyezOrJrJ7M6smsnszqTeTMxl1ByZxdnlLfpKTbZe2Hz2bSOzZk+Jq56S57Klk7lPKteemu3ZgMDKd8fe6OQ7uP/Tr5yVDKDYc/vzBckiv7010zmPx6e7Jgv5TzZidnjoMWK7NqW9OT/m7/EWvvKPdnWVmXw7IpGzMlP8rcfCXHZnO3+//NFpSn81e5JwvTn5Lk4UzP53JC7u9mJ0kWlSfyF7k3C9OfgfTmjszP17J4j/fXOpnVk1k9mdWTWT2Z1ZNZvYmc2b75L/Zb6i55MuXjBydH7ZfMnJRk48sftLI/3fUbUy47JFkwJfnRc+mWP5Yyuyc5ZeqozzzWZPbyhtPlqizJo5mWudmU92d13pcf5/M5Ybf7X5g786vMysocn+F0eX0Gsu2Fy8LmlmdyaX6Qq7M4n8nSzMxQlmdNlmf1S36asi+TWT2Z1ZNZPZnVk1k9mdWbKJlNyIJSzp+dnHbAKz9gqKRb2Z9y3bxk6QvfWB+1X3Ln5nTXDKSMw2+2X0xmI/1+Hsl3yo07tlflsHyqO3nH9mOZlq+XRflg7t7jJ405eTb/kIV5qOtLkjycGTtuOytrc3OOzI3dMTtuu6osyWdySz5XTsjWrmc0ntaoklk9mdWTWT2Z1ZNZPZnVm8iZTciCkuP2f/l9drVuS7rnSvKuDSPXt5Zk8cR4LaDMRlqTQ7Jyl08Gm9OT48tjOStrc0Q25oBsS0+G05vh9JZtGdrNqdLrc0xW5K6cXtbn7szJbTk8j7xw4duCDOToDORtZf2IY3qSzM2mrE/fqD6/0SCzejKrJ7N6Mqsns3oyqzeRM5uYBeWAF13gvbs3W962y983lSRJ+ca8ZO6L2uSUCXKxuMxG2JzJI95F49CyKRfnjvxjFuTqLM5gpmRxnshHclcmZzhDu7mPb3SL8v1yZE7KIzkxj+bP8tNcUk7KHd38TM22fC8L8u284SXHPZ6KM1kNkVk9mdWTWT2Z1ZNZPZnVm8iZTcyC8mIH9SRrt4xc+8nQznQWTknp7ZKHt47LaydeFZmNcEz606XkSzkupXu+gJ1W/uNlj3u4m5EbMiM3ZGE+Wv4ty7Iud2R+fpFZOSqD4/qtBGVWT2b1ZFZPZvVkVk9m9SZSZn5RY5Jy6tTknqHkusHkgS3pLn9y5Dff0yclfzkr3SeeeH6fdVuTH29Ovvr089sTkMxG2pDp2S8lb88vc1h5JqeXB3NmHtjj/lPK9iwvq3NseTxzyqYsKk9kYZ7K+hdeG3pt3pg35cksL6vz+vJ05peNOblsyPKyem89pVEns3oyqyezejKrJ7N6Mqs3kTJzBiVJ3jotOe/AdBc/mQyV5N0zknfOSH6282RZuWB2clBPupX9yfrHk76e5M29KR84cAwHH0MyG+GBbla+UI7Nu/LznJP7cm8OzteyOBdk1W73H06XvmzJBVmVWRnKYKbk9szP32VRkuTfu1n5cDkt5+S+XJFb0qVkQ6bn1hy+N5/WqJJZPZnVk1k9mdWTWT2Z1ZtImXWllLI3H3BwcDAzZ85M//0L0jfDCRxGz7J5S8Z6BAAAXrCtbM0tuSkDAwPp69vzRfgaAgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGZPHegAYLf+8Yc1Yj7DPWTZvyViPAABMcM6gAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAzTqjrMuN5aaxHmOfIrN6Mqsns3oyqyezeuMls8ljPcBrbdLcX/7G28uHD0z5yEF7aZp9g8zqyazO+WVVzsiDL1k/O3+UDd30MZiofTKrJ7N6Mqsns3oyqzfRM6sqKJdeemluuOGGrF27NlOnTs0pp5ySyy67LG984xtHa75qw/e8bufGTc+ku/yplNuP3Lk2bZeTRqUk25NM7vbWeE2SWT2Z1bszh+Zv85YRawPpHaNp9g0yqyezejKrJ7N6Mqs3kTOrKii33nprzj333LzlLW/Jtm3b8tGPfjRnnHFGfvrTn2batGmjNWOdObs8pb5JSbfL2g+fzaR3bMjwNXPTXfZUsnYo5Vvz0l27MRkYTvn63B2Hdh/7dfKToZQbDn9+YbgkV/anu2Yw+fX2ZMF+KefNTs4cBy1WZvVkVm1retLf7T9i7R3l/iwr63JYNmVjpuRHmZuv5Nhs7nb/qWlBeTp/lXuyMP0pSR7O9HwuJ+T+bnaSZFF5In+Re7Mw/RlIb+7I/Hwti/d4f62TWT2Z1ZNZPZnVk1m9iZxZ1aP/r//1v0Zsf/3rX8+cOXNy11135Q/+4A9e08FGU3fJkykfPzg5ar9k5qQkG1/+oJX96a7fmHLZIcmCKcmPnku3/LGU2T3JKVNHfeaxJrN6Mnt5w+lyVZbk0UzL3GzK+7M678uP8/mcsNv9L8yd+VVmZWWOz3C6vD4D2fbCpXRzyzO5ND/I1Vmcz2RpZmYoy7Mmy7P6JT+B2pfJrJ7M6smsnszqyazeRMnst6pHAwMDSZLZs2fvcZ+hoaEMDQ3t2B4cHPxtHvI1Uc6fnZx2wCs/YKikW9mfct28ZOkL3yQetV9y5+Z01wykjMNvHF9MZvVkNtLv55F8p9y4Y3tVDsunupN3bD+Wafl6WZQP5u49fqKdk2fzD1mYh7q+JMnDmbHjtrOyNjfnyNzYHbPjtqvKknwmt+Rz5YRs7XpG42mNKpnVk1k9mdWTWT2Z1ZvImb3qgjI8PJwPfehDOfXUU7N48eI97nfppZfmk5/85Kt9mNFx3P4vv8+u1m1J91xJ3rVh5PrWkiyeGK8FlNmrILMR1uSQrNzlE+jm9OT48ljOytockY05INvSk+H0Zji9ZVuGdnN6+fockxW5K6eX9bk7c3JbDs8jL1wsuCADOToDeVtZP+KYniRzsynr0zeqz280yKyezOrJrJ7M6sms3kTO7FUXlHPPPTf33Xdfbr/99t+430UXXZQVK1bs2B4cHMwRRxzxah/2tXHAiy5W3t2bLW/b5e+bSpKkfGNeMvdFbXLKBLnwWWb1ZDbC5kwe8c4jh5ZNuTh35B+zIFdncQYzJYvzRD6SuzI5wxnazX18o1uU75cjc1IeyYl5NH+Wn+aSclLu6OZnarble1mQb+cNLznu8VScyWqIzOrJrJ7M6smsnszqTeTMXlVBWb58eb773e/mtttuy+GHH/4b9+3t7U1vb+M//T2oJ1m7ZeTaT4Z2prNwSkpvlzy8dVxeB/CqyKyezEY4Jv3pUvKlHJfSPV/ATiv/8bLHPdzNyA2ZkRuyMB8t/5ZlWZc7Mj+/yKwclcFx/faLMqsns3oyqyezejKrN5Eyq/pFjaWULF++PDfeeGO+//3v5+ijjx6tufaqcurU5J6h5LrB5IEt6S5/cuQ3ktMnJX85K90nnnh+n3Vbkx9vTr769PPbE5DM6slspA2Znv1S8vb8MoeVZ3J6eTBn5oE97j+lbM/ysjrHlsczp2zKovJEFuaprH/h9bTX5o15U57M8rI6ry9PZ37ZmJPLhiwvq/fWUxp1Mqsns3oyqyezejKrN5EyqzqDcu655+ab3/xmbrrppsyYMSOPPvpokmTmzJmZOnUf/onvW6cl5x2Y7uInk6GSvHtG8s4Zyc92niwrF8xODupJt7I/Wf940teTvLk35QMHjuHgY0hm9WQ2wgPdrHyhHJt35ec5J/fl3hycr2VxLsiq3e4/nC592ZILsiqzMpTBTMntmZ+/y6Ikyb93s/LhclrOyX25IrekS8mGTM+t+c1nefclMqsns3oyqyezejKrN5Ey60op5RXv3O3+dfBXX3113vve976i+xgcHMzMmTPTf/+C9M2oOoEDjLJl85aM9QgAwDi1rWzNLbkpAwMD6evb80X4VWdQKroMAABANacwAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaMXmsBwDa8c8b1oz1CPucZfOWjPUIADCuOIMCAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIwjpxR1uXGctNYj7FPkVk9mdWTWT2Z1RsvmU0e6wFea5Pm/vI33l4+fGDKRw7aS9PsG2RWT2b1ZFbn/LIqZ+TBl6yfnT/Khm76GEzUPpnVk1k9mdWTWb2Jntm4KyjD97xu58ZNz6S7/KmU24/cuTZtl5NGpSTbk0zu9tZ4TZJZPZnVk1m9O3No/jZvGbE2kN4xmmbfILN6Mqsns3oyqzeRMxt3BSVzdnlKfZOSbpe1Hz6bSe/YkOFr5qa77Klk7VDKt+alu3ZjMjCc8vW5Ow7tPvbr5CdDKTcc/vzCcEmu7E93zWDy6+3Jgv1SzpudnDkOWqzM6smsnsyqbU1P+rv9R6y9o9yfZWVdDsumbMyU/Chz85Ucm83d7j+dLyhP569yTxamPyXJw5mez+WE3N/NTpIsKk/kL3JvFqY/A+nNHZmfr2XxHu+vdTKrJ7N6Mqsns3oTObN981/st9Rd8mTKxw9OjtovmTkpycaXP2hlf7rrN6ZcdkiyYEryo+fSLX8sZXZPcsrUUZ95rMmsnszqyezlDafLVVmSRzMtc7Mp78/qvC8/zudzwm73vzB35leZlZU5PsPp8voMZNsLlx/OLc/k0vwgV2dxPpOlmZmhLM+aLM/ql/zUbl8ms3oyqyezejKrN1Eym5AFpZw/OzntgFd+wFBJt7I/5bp5ydIXvuE5ar/kzs3prhlIGYffBL2YzOrJrJ7MRvr9PJLvlBt3bK/KYflUd/KO7ccyLV8vi/LB3L3HL05z8mz+IQvzUNeXJHk4M3bcdlbW5uYcmRu7Y3bcdlVZks/klnyunJCtXc9oPK1RJbN6Mqsns3oyqzeRM5uQBSXH7f/y++xq3ZZ0z5XkXRtGrm8tyeKJ8VpAmb0KMqsnsxHW5JCs3OWLzub05PjyWM7K2hyRjTkg29KT4fRmOL1lW4Z2c0r++hyTFbkrp5f1uTtzclsOzyMvXGC5IAM5OgN5W1k/4pieJHOzKevTN6rPbzTIrJ7M6smsnszqTeTMJmZBOeBFF97u7s2Wt+3y900lSVK+MS+Z+6I2OWWCXMQrs3oyqyezETZn8oh3azm0bMrFuSP/mAW5OoszmClZnCfykdyVyRnO0G7u4xvdony/HJmT8khOzKP5s/w0l5STckc3P1OzLd/Lgnw7b3jJcY+n4kxWQ2RWT2b1ZFZPZvUmcmYTs6C82EE9ydotI9d+MrQznYVTUnq75OGt4/I17a+KzOrJrJ7MRjgm/elS8qUcl9I9X8BOK//xssc93M3IDZmRG7IwHy3/lmVZlzsyP7/IrByVwXH9lpUyqyezejKrJ7N6Eykzv6gxSTl1anLPUHLdYPLAlnSXPznym6Lpk5K/nJXuE088v8+6rcmPNydfffr57QlIZvVkVk9mI23I9OyXkrfnlzmsPJPTy4M5Mw/scf8pZXuWl9U5tjyeOWVTFpUnsjBPZf0Lr0G+Nm/Mm/JklpfVeX15OvPLxpxcNmR5Wb23ntKok1k9mdWTWT2Z1ZtImTmDkiRvnZacd2C6i59Mhkry7hnJO2ckP9t5sqxcMDs5qCfdyv5k/eNJX0/y5t6UDxw4hoOPIZnVk1k9mY3wQDcrXyjH5l35ec7Jfbk3B+drWZwLsmq3+w+nS1+25IKsyqwMZTBTcnvm5++yKEny792sfLiclnNyX67ILelSsiHTc2sO35tPa1TJrJ7M6smsnszqTaTMulJK2ZsPODg4mJkzZ6b//gXpm+EEDrBvWzZvyViPAAD7hG1la27JTRkYGEhf354vwtcQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNmDzWAwDsy/55w5qxHmGfs2zekrEeAYCGOYMCAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggLAhHZGWZcby01jPcY+RWb1ZFZPZvXGS2aTx3qA19qkub/8jbeXDx+Y8pGD9tI0+waZ1ZNZPZnVk1md88uqnJEHX7J+dv4oG7rpYzBR+2RWT2b1ZFZvomf2WxWUv/7rv85FF12UD37wg/nsZz/7Go302xm+53U7N256Jt3lT6XcfuTOtWm7nDQqJdmeZHK3t8ZrkszqyayezOrJrN6dOTR/m7eMWBtI7xhNs2+QWT2Z1ZNZvYmc2asuKKtWrcqXvvSlHHvssa/lPL+9Obs8pb5JSbfL2g+fzaR3bMjwNXPTXfZUsnYo5Vvz0l27MRkYTvn63B2Hdh/7dfKToZQbDn9+YbgkV/anu2Yw+fX2ZMF+KefNTs4cBy1WZvVkVk9m9WRWbWt60t/tP2LtHeX+LCvrclg2ZWOm5EeZm6/k2Gzudv8lcEF5On+Ve7Iw/SlJHs70fC4n5P5udpJkUXkif5F7szD9GUhv7sj8fC2L93h/rZNZPZnVk1m9iZzZq3r0Z555Ju95z3vyla98JRdffPFrPdOo6y55MuXjBydH7ZfMnJRk48sftLI/3fUbUy47JFkwJfnRc+mWP5Yyuyc5ZeqozzzWZFZPZvVkVk9mL284Xa7KkjyaaZmbTXl/Vud9+XE+nxN2u/+FuTO/yqyszPEZTpfXZyDbXrhkc255JpfmB7k6i/OZLM3MDGV51mR5Vr/kJ537MpnVk1k9mdWbKJm9qoJy7rnn5o//+I9z+umnv2xBGRoaytDQ0I7twcHBV/OQr6ly/uzktANe+QFDJd3K/pTr5iVLX/jifdR+yZ2b010zkDIOv6C/mMzqyayezOrJbKTfzyP5Trlxx/aqHJZPdSfv2H4s0/L1sigfzN17/II+J8/mH7IwD3V9SZKHM2PHbWdlbW7OkbmxO2bHbVeVJflMbsnnygnZ2vWMxtMaVTKrJ7N6Mqs3kTOrLijf+ta3cvfdd2fVqlWvaP9LL700n/zkJ6sHG1XH7f/y++xq3ZZ0z5XkXRtGrm8tyeKJ8VpAmb0KMqsns3oyG2FNDsnKXb5Qb05Pji+P5ayszRHZmAOyLT0ZTm+G01u2ZWg3L2O4PsdkRe7K6WV97s6c3JbD88gLF6UuyECOzkDeVtaPOKYnydxsyvr0jerzGw0yqyezejKrN5EzqyooDz30UD74wQ/mX/7lX7L//q/si+JFF12UFStW7NgeHBzMEUccUTfla+2AF11Eurs3W962y983lSRJ+ca8ZO6L2uSUCXJBqszqyayezOrJbITNmTziHW4OLZtyce7IP2ZBrs7iDGZKFueJfCR3ZXKGM7Sb+/hGtyjfL0fmpDySE/No/iw/zSXlpNzRzc/UbMv3siDfzhtectzjqTiT1RCZ1ZNZPZnVm8iZVRWUu+66K48//nhOOGFnm9u+fXtuu+22XHnllRkaGkpPz8gveL29ventbfyncgf1JGu3jFz7ydDOdBZOSentkoe3jsvXZ78qMqsns3oyqyezEY5Jf7qUfCnHpXTPF7DTyn+87HEPdzNyQ2bkhizMR8u/ZVnW5Y7Mzy8yK0dlcFy/zafM6smsnszqTaTMqn5R49ve9rbce++9WbNmzY4/S5cuzXve856sWbPmJeVkX1FOnZrcM5RcN5g8sCXd5U+O/AI/fVLyl7PSfeKJ5/dZtzX58ebkq08/vz0ByayezOrJrJ7MRtqQ6dkvJW/PL3NYeSanlwdzZh7Y4/5TyvYsL6tzbHk8c8qmLCpPZGGeyvoXXrd9bd6YN+XJLC+r8/rydOaXjTm5bMjysnpvPaVRJ7N6Mqsns3oTKbOqMygzZszI4sWLR6xNmzYtBx100EvW9ylvnZacd2C6i59Mhkry7hnJO2ckP9t5sqxcMDs5qCfdyv5k/eNJX0/y5t6UDxw4hoOPIZnVk1k9mdWT2QgPdLPyhXJs3pWf55zcl3tzcL6Wxbkgu7+Ocjhd+rIlF2RVZmUog5mS2zM/f5dFSZJ/72blw+W0nJP7ckVuSZeSDZmeW3P43nxao0pm9WRWT2b1JlJmXSml/DZ38Id/+IdZsmTJK/5FjYODg5k5c2b671+QvhlVJ3AAGAeWzVsy1iMAMAa2la25JTdlYGAgfX17vgj/t/4tLLfccstvexcAAABJKq9BAQAAGE0KCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZkwe6wEAmFj+ecOasR5hn7Ns3pKxHgFgr3EGBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAKhyRlmXG8tNYz3GPkVm9WRWb7xkNnmsB3itTZr7y994e/nwgSkfOWgvTbNvkFk9mdWTWT2Z1ZNZnfPLqpyRB1+yfnb+KBu66WMwUftkVk9m9SZ6ZuOuoAzf87qdGzc9k+7yp1JuP3Ln2rRdThqVkmxPMrnbW+M1SWb1ZFZPZvVkVk9m9e7MofnbvGXE2kB6x2iafYPM6sms3kTObNwVlMzZ5Sn1TUq6XdZ++GwmvWNDhq+Zm+6yp5K1Qynfmpfu2o3JwHDK1+fuOLT72K+Tnwyl3HD48wvDJbmyP901g8mvtycL9ks5b3Zy5jhosTKrJ7N6Mqsns3oyq7Y1Penv9h+x9o5yf5aVdTksm7IxU/KjzM1Xcmw2d7v/tmFBeTp/lXuyMP0pSR7O9HwuJ+T+bnaSZFF5In+Re7Mw/RlIb+7I/Hwti/d4f62TWT2Z1ZvIme2b/2K/pe6SJ1M+fnBy1H7JzElJNr78QSv7012/MeWyQ5IFU5IfPZdu+WMps3uSU6aO+sxjTWb1ZFZPZvVkVk9mL284Xa7KkjyaaZmbTXl/Vud9+XE+nxN2u/+FuTO/yqyszPEZTpfXZyDbXrjMdW55JpfmB7k6i/OZLM3MDGV51mR5Vr/kp8P7MpnVk1m9iZLZhCwo5fzZyWkHvPIDhkq6lf0p181Llr7wheio/ZI7N6e7ZiBlHH5xejGZ1ZNZPZnVk1k9mY30+3kk3yk37thelcPyqe7kHduPZVq+Xhblg7l7j98Ezcmz+YcszENdX5Lk4czYcdtZWZubc2Ru7I7ZcdtVZUk+k1vyuXJCtnY9o/G0RpXM6sms3kTObEIWlBy3/8vvs6t1W9I9V5J3bRi5vrUkiyfGawFl9irIrJ7M6smsnsxGWJNDsnKXb242pyfHl8dyVtbmiGzMAdmWngynN8PpLdsytJuXflyfY7Iid+X0sj53Z05uy+F55IULeRdkIEdnIG8r60cc05NkbjZlffpG9fmNBpnVk1m9iZzZxCwoB7zogsjdvdnytl3+vqkkSco35iVzX9Qmp0yQiytlVk9m9WRWT2b1ZDbC5kwe8a5Ah5ZNuTh35B+zIFdncQYzJYvzRD6SuzI5wxnazX18o1uU75cjc1IeyYl5NH+Wn+aSclLu6OZnarble1mQb+cNLznu8VScyWqIzOrJrN5EzmxiFpQXO6gnWbtl5NpPhnams3BKSm+XPLx1XL7W+FWRWT2Z1ZNZPZnVk9kIx6Q/XUq+lONSuucL2GnlP172uIe7GbkhM3JDFuaj5d+yLOtyR+bnF5mVozI4rt8aVWb1ZFZvImXmFzUmKadOTe4ZSq4bTB7Yku7yJ0d+sZo+KfnLWek+8cTz+6zbmvx4c/LVp5/fnoBkVk9m9WRWT2b1ZDbShkzPfil5e36Zw8ozOb08mDPzwB73n1K2Z3lZnWPL45lTNmVReSIL81TWv/Ba92vzxrwpT2Z5WZ3Xl6czv2zMyWVDlpfVe+spjTqZ1ZNZvYmUmTMoSfLWacl5B6a7+MlkqCTvnpG8c0bys50ny8oFs5ODetKt7E/WP5709SRv7k35wIFjOPgYklk9mdWTWT2Z1ZPZCA90s/KFcmzelZ/nnNyXe3NwvpbFuSCrdrv/cLr0ZUsuyKrMylAGMyW3Z37+LouSJP/ezcqHy2k5J/flitySLiUbMj235vC9+bRGlczqyazeRMqsK6WUvfmAg4ODmTlzZvrvX5C+GU7gAMDLWTZvyViPAPBb21a25pbclIGBgfT17fkifA0BAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANCMyWM9AADwm/3zhjVjPcI+Z9m8JWM9AvAqOYMCAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAMMrOKOtyY7lprMfYp8is3njJbPJYD/BamzT3l7/x9vLhA1M+ctBemmbfILN6Mqsns3oyqyezejKrc35ZlTPy4EvWz84fZUM3fQwmap/M6k30zKoLysMPP5wLLrgg//RP/5Rnn302b3jDG3L11Vdn6dKlozFfteF7Xrdz46Zn0l3+VMrtR+5cm7bLSaNSku1JJnd7a7wmyayezOrJrJ7M6smsnszq3ZlD87d5y4i1gfSO0TT7BpnVm8iZVRWU/v7+nHrqqXnrW9+af/qnf8ohhxySX/ziFznwwANHa756c3Z5Sn2Tkm6XtR8+m0nv2JDha+amu+ypZO1Qyrfmpbt2YzIwnPL1uTsO7T726+QnQyk3HP78wnBJruxPd81g8uvtyYL9Us6bnZw5DlqszOrJrJ7M6smsnszqyaza1vSkv9t/xNo7yv1ZVtblsGzKxkzJjzI3X8mx2dzt/lutBeXp/FXuycL0pyR5ONPzuZyQ+7vZSZJF5Yn8Re7NwvRnIL25I/PztSze4/21Tmb1JnJmVY9+2WWX5YgjjsjVV1+9Y+3oo49+zYcabd0lT6Z8/ODkqP2SmZOSbHz5g1b2p7t+Y8plhyQLpiQ/ei7d8sdSZvckp0wd9ZnHmszqyayezOrJrJ7M6sns5Q2ny1VZkkczLXOzKe/P6rwvP87nc8Ju978wd+ZXmZWVOT7D6fL6DGTbC5cGzy3P5NL8IFdncT6TpZmZoSzPmizP6pf8RH1fJrN6EyWzqoLyne98J8uWLcs73/nO3HrrrZk/f37+63/9r3nf+963x2OGhoYyNDS0Y3twcPDVT/saKefPTk474JUfMFTSrexPuW5esvSFT6pH7ZfcuTndNQMp4/AT7YvJrJ7M6smsnszqyayezEb6/TyS75Qbd2yvymH5VHfyju3HMi1fL4vywdy9x28c5+TZ/EMW5qGuL0nycGbsuO2srM3NOTI3dsfsuO2qsiSfyS35XDkhW7ue0Xhao0pm9SZyZlUF5YEHHsgXvvCFrFixIh/96EezatWqfOADH8iUKVNy9tln7/aYSy+9NJ/85Cdfk2FfM8ft//L77GrdlnTPleRdG0auby3J4onxWkCZvQoyqyezejKrJ7N6MhthTQ7Jyl2+IdycnhxfHstZWZsjsjEHZFt6MpzeDKe3bMvQbl4uc32OyYrcldPL+tydObkth+eRFy5+XpCBHJ2BvK2sH3FMT5K52ZT16RvV5zcaZFZvImdWVVCGh4ezdOnSXHLJJUmS448/Pvfdd1+++MUv7rGgXHTRRVmxYsWO7cHBwRxxxBG/xcivgQNedHHf7t5sedsuf99UkiTlG/OSuS9qk1MmyIWCMqsns3oyqyezejKrJ7MRNmfyiHdSOrRsysW5I/+YBbk6izOYKVmcJ/KR3JXJGc7Qbu7jG92ifL8cmZPySE7Mo/mz/DSXlJNyRzc/U7Mt38uCfDtveMlxj6fiTFZDZFZvImdWVVDmzp2bN73pTSPWfvd3fzfXX3/9Ho/p7e1Nb2/jPy05qCdZu2Xk2k+GdqazcEpKb5c8vHVcvm72VZFZPZnVk1k9mdWTWT2ZjXBM+tOl5Es5LqV7voCdVv7jZY97uJuRGzIjN2RhPlr+LcuyLndkfn6RWTkqg+P67WRlVm8iZVb1ixpPPfXU/PznPx+xdv/99+eoo456TYfa28qpU5N7hpLrBpMHtqS7/MmRn3inT0r+cla6Tzzx/D7rtiY/3px89enntycgmdWTWT2Z1ZNZPZnVk9lIGzI9+6Xk7fllDivP5PTyYM7MA3vcf0rZnuVldY4tj2dO2ZRF5YkszFNZ/8L1AdfmjXlTnszysjqvL09nftmYk8uGLC+r99ZTGnUyqzeRMqs6g3LeeefllFNOySWXXJI//dM/zZ133pkvf/nL+fKXvzxa8+0db52WnHdguoufTIZK8u4ZyTtnJD/bebKsXDA7Oagn3cr+ZP3jSV9P8ubelA809BbLe5PM6smsnszqyayezOrJbIQHuln5Qjk278rPc07uy705OF/L4lyQVbvdfzhd+rIlF2RVZmUog5mS2zM/f5dFSZJ/72blw+W0nJP7ckVuSZeSDZmeW3P43nxao0pm9SZSZl0ppdQc8N3vfjcXXXRRfvGLX+Too4/OihUrfuO7eL3Y4OBgZs6cmf77F6RvRtUJHACAV2TZvCVjPQLwItvK1tySmzIwMJC+vj1fhF/9W1jOPPPMnHnmmb/VcAAAALvjFAYAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMmj/UAAACvtX/esGasR9jnLJu3ZKxHgCTOoAAAAA1RUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAADQnDPKutxYbhrrMfYp4yWzyWM9wGtt0txf/sbby4cPTPnIQXtpmn2DzOrJrJ7M6smsnszqyayezOqcX1bljDz4kvWz80fZ0E0fg4naN9EzG3cFZfie1+3cuOmZdJc/lXL7kTvXpu1y0qiUZHuSyd3eGq9JMqsns3oyqyezejKrJ7N6Mqt3Zw7N3+YtI9YG0jtG0+wbJnJm466gZM4uT6lvUtLtsvbDZzPpHRsyfM3cdJc9lawdSvnWvHTXbkwGhlO+PnfHod3Hfp38ZCjlhsOfXxguyZX96a4ZTH69PVmwX8p5s5Mzx0GLlVk9mdWTWT2Z1ZNZPZnVk1m1relJf7f/iLV3lPuzrKzLYdmUjZmSH2VuvpJjs7nb/benC8rT+avck4XpT0nycKbnczkh93ezkySLyhP5i9ybhenPQHpzR+bna1m8x/tr3UTObN/8F/stdZc8mfLxg5Oj9ktmTkqy8eUPWtmf7vqNKZcdkiyYkvzouXTLH0uZ3ZOcMnXUZx5rMqsns3oyqyezejKrJ7N6Mnt5w+lyVZbk0UzL3GzK+7M678uP8/mcsNv9L8yd+VVmZWWOz3C6vD4D2fbC5dRzyzO5ND/I1Vmcz2RpZmYoy7Mmy7P6JWch9mUTJbMJWVDK+bOT0w545QcMlXQr+1Oum5csfeETxFH7JXduTnfNQMo4/KTxYjKrJ7N6Mqsns3oyqyezejIb6ffzSL5TbtyxvSqH5VPdyTu2H8u0fL0sygdz9x6/2Z6TZ/MPWZiHur4kycOZseO2s7I2N+fI3Ngds+O2q8qSfCa35HPlhGztekbjaY2qiZzZhCwoOW7/l99nV+u2pHuuJO/aMHJ9a0kWT4zXAsrsVZBZPZnVk1k9mdWTWT2ZjbAmh2TlLt9Eb05Pji+P5ayszRHZmAOyLT0ZTm+G01u2ZWg3LzG6PsdkRe7K6WV97s6c3JbD88gLF4wvyECOzkDeVtaPOKYnydxsyvr0jerzGw0TObOJWVAOeNGFart7s+Vtu/x9U0mSlG/MS+a+qE1OmSAXvcmsnszqyayezOrJrJ7M6slshM2ZPOLdpw4tm3Jx7sg/ZkGuzuIMZkoW54l8JHdlcoYztJv7+Ea3KN8vR+akPJIT82j+LD/NJeWk3NHNz9Rsy/eyIN/OG15y3OOpOJPVkImc2cQsKC92UE+ydsvItZ8M7Uxn4ZSU3i55eOu4fA3oqyKzejKrJ7N6Mqsns3oyqyezEY5Jf7qUfCnHpXTPF7DTyn+87HEPdzNyQ2bkhizMR8u/ZVnW5Y7Mzy8yK0dlcFy/Be9EyswvakxSTp2a3DOUXDeYPLAl3eVPjvwkMn1S8pez0n3iief3Wbc1+fHm5KtPP789AcmsnszqyayezOrJrJ7M6slspA2Znv1S8vb8MoeVZ3J6eTBn5oE97j+lbM/ysjrHlsczp2zKovJEFuaprH/hmopr88a8KU9meVmd15enM79szMllQ5aX1XvrKY26iZSZMyhJ8tZpyXkHprv4yWSoJO+ekbxzRvKznSfLygWzk4N60q3sT9Y/nvT1JG/uTfnAgWM4+BiSWT2Z1ZNZPZnVk1k9mdWT2QgPdLPyhXJs3pWf55zcl3tzcL6Wxbkgq3a7/3C69GVLLsiqzMpQBjMlt2d+/i6LkiT/3s3Kh8tpOSf35Yrcki4lGzI9t+bwvfm0RtVEyqwrpZS9+YCDg4OZOXNm+u9fkL4ZTuAAALRg2bwlYz0C49y2sjW35KYMDAykr2/PF+FrCAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZkwe6wEAABh7/7xhzViPsM9ZNm/JWI8wLjmDAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAADjwBllXW4sN431GL+1yWM9wGtt0txf/sbby4cPTPnIQXtpmn2DzOrJrJ7M6smsnszqyayezOrJrM75ZVXOyIMvWT87f5QN3fQxmGjvqioo27dvz3//7/8911xzTR599NHMmzcv733ve/Pf/tt/S9d1ozVjleF7Xrdz46Zn0l3+VMrtR+5cm7bLSaNSku1JJrcx+1iRWT2Z1ZNZPZnVk1k9mdWTWT2Z1bszh+Zv85YRawPpHaNp9q6ql3hddtll+cIXvpArr7wyP/vZz3LZZZflb/7mb/L5z39+tOarN2fyzj99k5Jul7VfbsmkNzyQ3Lwp3RkPpTvqV8mdz6X74GPp3vvIiLvpPvbrdP/3f+xcGC7JyqfSnbgu3dG/Sve29cl3n9m7z220yKyezOrJrJ7M6smsnszqyayezKptTU/6u/1H/Pm/8ot8ufx/851yY/6+fC/vL3dn/7Jtj/exoDydy8utual8O98u385V5f+XheWpHbcvKk/kivL/5rvlhvx9+V7+a1nzG+9vb6k6g/LDH/4w//k//+f88R//cZLkda97Xf7n//yfufPOO0dluNHSXfJkyscPTo7aL5k5KcnGlz9oZX+66zemXHZIsmBK8qPn0i1/LGV2T3LK1FGfeazJrJ7M6smsnszqyayezOrJrJ7MXt5wulyVJXk00zI3m/L+rM778uN8Pifsdv8Lc2d+lVlZmeMznC6vz0C2vXB+Ym55JpfmB7k6i/OZLM3MDGV51mR5Vr/kzM3eVlVQTjnllHz5y1/O/fffn4ULF+aee+7J7bffniuuuGKPxwwNDWVoaGjH9uDg4Kuf9jVSzp+dnHbAKz9gqKRb2Z9y3bxk6Qsf7Eftl9y5Od01Aynj8H+AF5NZPZnVk1k9mdWTWT2Z1ZNZPZmN9Pt5JN8pN+7YXpXD8qnu5B3bj2Vavl4W5YO5e48FZU6ezT9kYR7q+pIkD2fGjtvOytrcnCNzY3fMjtuuKkvymdySz5UTsrXrGY2n9YpUFZQLL7wwg4OD+Z3f+Z309PRk+/bt+fSnP533vOc9ezzm0ksvzSc/+cnfetDX1HH71+2/bku650ryrg0j17eWZPHEeC2gzF4FmdWTWT2Z1ZNZPZnVk1k9mY2wJodk5S7FY3N6cnx5LGdlbY7IxhyQbenJcHoznN6yLUPdS7+tvz7HZEXuyullfe7OnNyWw/PICxfZL8hAjs5A3lbWjzimJ8ncbMr69I3q8/tNqgrKddddl7//+7/PN7/5zSxatChr1qzJhz70ocybNy9nn332bo+56KKLsmLFih3bg4ODOeKII367qX9bB7zooqvdXYmz68vvNpUkSfnGvGTui9rklAlyAZfM6smsnszqyayezOrJrJ7M6slshM2ZPOIduw4tm3Jx7sg/ZkGuzuIMZkoW54l8JHdlcoYztJv7+Ea3KN8vR+akPJIT82j+LD/NJeWk3NHNz9Rsy/eyIN/OG15y3OOpOJM1CqoKyvnnn58LL7ww7373u5Mkb37zm/Pggw/m0ksv3WNB6e3tTW9v4y32oJ5k7ZaRaz8Z2pnOwikpvV3y8NZx+XrGV0Vm9WRWT2b1ZFZPZvVkVk9m9WQ2wjHpT5eSL+W4lBfePfe08h8vc1TycDcjN2RGbsjCfLT8W5ZlXe7I/Pwis3JUBpt82+KqgvLss89m0qSRdbanpyfDw8Ov6VB7Wzl1arr/8XRy3WCydP901298/n+IxVOe32H6pOQvZ6X7xBMpJcmJU5PB7cmqzcmMScmfjt0psLEis3oyqyezejKrJ7N6Mqsns3oyG2lDpme/lLw9v8y/lrlZnCdzZh7Y4/5Tyvb8f/Lj3Jb5eTTTckiey8I8ldszP0lybd6Ylfl/s7yszj/l6GxOT47MxvxeHsuV3fF762ntVlVB+ZM/+ZN8+tOfzpFHHplFixZl9erVueKKK3LOOeeM1nx7x1unJecdmO7iJ5Ohkrx7RvLOGcnPdp4sKxfMTg7qSbeyP1n/eNLXk7y5N+UDB47h4GNIZvVkVk9m9WRWT2b1ZFZPZvVkNsID3ax8oRybd+XnOSf35d4cnK9lcS7Iqt3uP5wufdmSC7IqszKUwUzJ7Zmfv8uiJMm/d7Py4XJazsl9uSK3pEvJhkzPrTl8bz6t3epKKeWV7rxx48Z87GMfy4033pjHH3888+bNy1lnnZWPf/zjmTJlyiu6j8HBwcycOTP99y9I34yqX8MCAADNWDZvyViPsE/ZVrbmltyUgYGB9PXt+QxXVUF5LSgoAACMBwpKnVdaUDQEAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmTB7rAQAAYF/0zxvWjPUI+5TBjcM5cOHL7+cMCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQjMljPcBrbdLcX/7G28uHD0z5yEF7aZp9g8zqyayezOrJrJ7M6smsnszqyazeRM6sK6WUvfmAg4ODmTlzZvrvX5C+GaNwAufxbTv/ftMz6S5/KuX2I3euTZv0/J8kKSXZnmRy99rPsS+RWT2Z1ZNZPZnVk1k9mdWTWT2Z1RuHmQ1uHM6BCx/IwMBA+vr69rjf+HuJ15zJO//0TUq6XdZ+uSWT3vBAcvOmdGc8lO6oXyV3Ppfug4+le+8jI+6m+9iv0/3f/7FzYbgkK59Kd+K6dEf/Kt3b1ifffWbvPrfRIrN6Mqsns3oyqyezejKrJ7N6Mqs3gTMbdy/xeiW6S55M+fjByVH7JTMnJdn48get7E93/caUyw5JFkxJfvRcuuWPpczuSU6ZOuozjzWZ1ZNZPZnVk1k9mdWTWT2Z1ZNZvfGa2YQsKOX82clpB7zyA4ZKupX9KdfNS5a+8A931H7JnZvTXTOQ0sg/5miSWT2Z1ZNZPZnVk1k9mdWTWT2Z1RuvmU3IgpLj9q/bf92WdM+V5F0bRq5vLcni3tdurpbJrJ7M6smsnszqyayezOrJrJ7M6o3TzCZmQTngRRcQ7e5KnF2uS8qm599HoHxjXjK3Z+R+U9q+GOk1I7N6Mqsns3oyqyezejKrJ7N6Mqs3TjObmAXlxQ7qSdZuGbn2k6Gd6SycktLbJQ9vbea1eWNOZvVkVk9m9WRWT2b1ZFZPZvVkVm+cZKagJCmnTk33P55OrhtMlu6f7vqNz//jLp7y/A7TJyV/OSvdJ55IKUlOnJoMbk9WbU5mTEr+dM9vkzZeyayezOrJrJ7M6smsnszqyayezOqNl8wUlCR567TkvAPTXfxkMlSSd89I3jkj+dnQjl3KBbOTg3rSrexP1j+e9PUkb+5N+cCBYzj4GJJZPZnVk1k9mdWTWT2Z1ZNZPZnVGyeZjb9f1AgAADRn4v6iRgAAYJ+loAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaMbkvf2ApZQkyeAzw3v7oQEAgDHyf77//z99YE/2ekHZuHFjkuSoE9bt7YcGAADG2MaNGzNz5sw93t6Vl6swr7Hh4eFs2LAhM2bMSNd1e/Ohf6PBwcEcccQReeihh9LX1zfW4zBO+Thjb/Bxxt7g44y9wcfZ+FJKycaNGzNv3rxMmrTnK032+hmUSZMm5fDDD9/bD/uK9fX1+R+AUefjjL3Bxxl7g48z9gYfZ+PHbzpz8n+4SB4AAGiGggIAADRDQXlBb29vPvGJT6S3t3esR2Ec83HG3uDjjL3Bxxl7g4+ziWmvXyQPAACwJ86gAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAXlBVdddVVe97rXZf/9989JJ52UO++8c6xHYhy59NJL85a3vCUzZszInDlz8va3vz0///nPx3osxrG//uu/Ttd1+dCHPjTWozAOPfzww/kv/+W/5KCDDsrUqVPz5je/Of/7f//vsR6LcWT79u352Mc+lqOPPjpTp07N61//+nzqU5+KN5+dGBSUJNdee21WrFiRT3ziE7n77rtz3HHHZdmyZXn88cfHejTGiVtvvTXnnntufvSjH+Vf/uVfsnXr1pxxxhnZtGnTWI/GOLRq1ap86UtfyrHHHjvWozAO9ff359RTT81+++2Xf/qnf8pPf/rTfOYzn8mBBx441qMxjlx22WX5whe+kCuvvDI/+9nPctlll+Vv/uZv8vnPf36sR2Mv8HtQkpx00kl5y1vekiuvvDJJMjw8nCOOOCLvf//7c+GFF47xdIxHv/71rzNnzpzceuut+YM/+IOxHodx5JlnnskJJ5yQ//E//kcuvvjiLFmyJJ/97GfHeizGkQsvvDB33HFHfvCDH4z1KIxjZ555Zg499NB89atf3bH2jne8I1OnTs0111wzhpOxN0z4MyhbtmzJXXfdldNPP33H2qRJk3L66afnX//1X8dwMsazgYGBJMns2bPHeBLGm3PPPTd//Md/POJzGryWvvOd72Tp0qV55zvfmTlz5uT444/PV77ylbEei3HmlFNOyc0335z7778/SXLPPffk9ttvz3/6T/9pjCdjb5g81gOMtSeeeCLbt2/PoYceOmL90EMPzdq1a8doKsaz4eHhfOhDH8qpp56axYsXj/U4jCPf+ta3cvfdd2fVqlVjPQrj2AMPPJAvfOELWbFiRT760Y9m1apV+cAHPpApU6bk7LPPHuvxGCcuvPDCDA4O5nd+53fS09OT7du359Of/nTe8573jPVo7AUTvqDA3nbuuefmvvvuy+233z7WozCOPPTQQ/ngBz+Yf/mXf8n+++8/1uMwjg0PD2fp0qW55JJLkiTHH3987rvvvnzxi19UUHjNXHfddfn7v//7fPOb38yiRYuyZs2afOhDH8q8efN8nE0AE76gHHzwwenp6cljjz02Yv2xxx7LYYcdNkZTMV4tX7483/3ud3Pbbbfl8MMPH+txGEfuuuuuPP744znhhBN2rG3fvj233XZbrrzyygwNDaWnp2cMJ2S8mDt3bt70pjeNWPvd3/3dXH/99WM0EePR+eefnwsvvDDvfve7kyRvfvOb8+CDD+bSSy9VUCaACX8NypQpU/J7v/d7ufnmm3esDQ8P5+abb87JJ588hpMxnpRSsnz58tx44435/ve/n6OPPnqsR2Kcedvb3pZ77703a9as2fFn6dKlec973pM1a9YoJ7xmTj311Je8Tfr999+fo446aowmYjx69tlnM2nSyG9Te3p6Mjw8PEYTsTdN+DMoSbJixYqcffbZWbp0aU488cR89rOfzaZNm/Lnf/7nYz0a48S5556bb37zm7npppsyY8aMPProo0mSmTNnZurUqWM8HePBjBkzXnJN07Rp03LQQQe51onX1HnnnZdTTjkll1xySf70T/80d955Z7785S/ny1/+8liPxjjyJ3/yJ/n0pz+dI488MosWLcrq1atzxRVX5Jxzzhnr0dgLvM3wC6688spcfvnlefTRR7NkyZKsXLkyJ5100liPxTjRdd1u16+++uq8973v3bvDMGH84R/+obcZZlR897vfzUUXXZRf/OIXOfroo7NixYq8733vG+uxGEc2btyYj33sY7nxxhvz+OOPZ968eTnrrLPy8Y9/PFOmTBnr8RhlCgoAANCMCX8NCgAA0A4FBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA04/8Pp2TqsxxSD9QAAAAASUVORK5CYII=\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = MultiHeadAttention(\n", + " in_size=10,\n", + " head_size=4,\n", + " num_heads=3,\n", + " out_size=15,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "cu4gFFYBbWMa", + "outputId": "58aba7a8-a103-4f17-b658-ff445ab6ac8d" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (out): Linear(in_features=12, out_features=15, bias=True)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 9 + } + ] + }, + { + "cell_type": "code", + "source": [ + "tmp_input = torch.rand(2, 5, 10)\n", + "tmp_mask = make_decoder_mask(tmp_input)\n", + "\n", + "print(\"Decoder-like input, with mask\")\n", + "print(f'Input shape: {tmp_input.shape}')\n", + "print(f'Mask shape: {tmp_mask.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input, tmp_input, tmp_input, tmp_mask)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input, tmp_mask, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "gPA5Ba6UbYFP", + "outputId": "7790ce5c-c428-42e0-8d84-da9994d01c1e" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Decoder-like input, with mask\n", + "Input shape: torch.Size([2, 5, 10])\n", + "Mask shape: torch.Size([2, 1, 5, 5])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.2 Positional Encoding\n", + "\n", + "#### Main class - PositionalEncoding\n", + "\n", + "**Initialization:**\n", + "* max_seq_len ~ the maximum token length of the sequence\n", + "* emb_size ~ the embedding size of the input\n", + "\n", + "**Forward:**\n", + "* _decoder_emb_ ~ embeddings of tokens from the decoder input\n", + "\n", + "$$\\text{PE}_{(\\text{pos}, 2i)} = sin\\Bigg( \\frac{\\text{pos}}{10000^{\\frac{2i}{\\text{emb_size}}}} \\Bigg) \\quad ; \\quad \\text{PE}_{(\\text{pos}, 2i + 1)} = cos\\Bigg( \\frac{\\text{pos}}{10000^{\\frac{2i}{\\text{emb_size}}}} \\Bigg)$$" + ], + "metadata": { + "id": "nqpfRJPlbVpf" + } + }, + { + "cell_type": "code", + "source": [ + "class PositionalEncoding(nn.Module):\n", + " \"\"\"\n", + " Class to calculate Positional Encodings, suggested in `Attention is all you need [Vaswaniet al., 2017]`\n", + " \"\"\"\n", + " def __init__(self, max_seq_len, emb_size):\n", + " \"\"\"\n", + " Args:\n", + " max_seq_len: max length of input sequence\n", + " emb_size: demension of embedding\n", + " \"\"\"\n", + " super(PositionalEncoding, self).__init__()\n", + "\n", + " # Запишем все переданые гиперпараметры слоя\n", + " self.max_seq_len = max_seq_len\n", + " self.emb_size = emb_size\n", + "\n", + " # Посчитаем позиционные эмбеддинги в тензорном виде\n", + " pos = torch.arange(max_seq_len)[:, None]\n", + " inds = torch.arange(emb_size)[None, ::2]\n", + "\n", + " pe = torch.zeros(max_seq_len, self.emb_size)\n", + " pe[:, ::2] = torch.sin(pos / (10000 ** ((2 * inds) / self.emb_size)))\n", + " pe[:, 1::2] = torch.cos(pos / (10000 ** ((2 * inds) / self.emb_size)))\n", + " pe = pe.unsqueeze(0)\n", + "\n", + " # Добавляем полученный тензор как параметр, который будет сохранятся вместе с моделью, но не будет обучаться\n", + " self.register_buffer('pe', pe)\n", + "\n", + "\n", + " def forward(self, decoder_emb):\n", + " \"\"\"\n", + " Args:\n", + " decoder_emb: decoder sequence after embed\n", + " Returns:\n", + " output: input with positional encodings\n", + " \"\"\"\n", + " # Тензоры приходят размера batch_size x seq_len x emb_size\n", + " seq_len = decoder_emb.size(1)\n", + "\n", + " # Прибавляем позиционные эмбеддинги\n", + " return decoder_emb + self.pe[:, :seq_len]" + ], + "metadata": { + "id": "oo22_W7gbql2" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing PositionalEncoding" + ], + "metadata": { + "id": "CUlw9fUJbvW0" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = PositionalEncoding(\n", + " max_seq_len=5,\n", + " emb_size=10,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Qq1zwI4dbysA", + "outputId": "d4f63ad9-9541-4cfb-c9f3-e80528fba9d8" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "PositionalEncoding()" + ] + }, + "metadata": {}, + "execution_count": 15 + } + ] + }, + { + "cell_type": "code", + "source": [ + "tmp_input = torch.rand(2, 5, 10)\n", + "\n", + "print(f'Input shape: {tmp_input.shape}')\n", + "tmp_output = tmp_layer(tmp_input)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "xUDvRmxLb1RM", + "outputId": "5eae2649-e3de-4500-b55e-0d3471d8b2d5" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Input shape: torch.Size([2, 5, 10])\n", + "Output shape: torch.Size([2, 5, 10])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let’s examine the positional encodings." + ], + "metadata": { + "id": "b1SMX9aWb4-a" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer.pe.shape" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "zLpl1jWzb9As", + "outputId": "68c293f3-800c-45eb-bec3-86db51edbe24" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "torch.Size([1, 5, 10])" + ] + }, + "metadata": {}, + "execution_count": 17 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "the article substantiates [Attention is all you need [Vaswaniet al., 2017]](https://www.semanticscholar.org/reader/204e3073870fae3d05bcbc2f6a8e263d9b72e776):\n", + "\n", + "We chose this function because we hypothesized it would allow the model to easily learn to attend by\n", + "relative positions, since for any fixed offset k, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$." + ], + "metadata": { + "id": "iMmyRsQ1b_rF" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = PositionalEncoding(\n", + " max_seq_len=200,\n", + " emb_size=100,\n", + ")\n", + "\n", + "fig, ax = plt.subplots(figsize=(10, 10))\n", + "plt.imshow(tmp_layer.pe[0, :, :], aspect=\"auto\")\n", + "plt.xlabel(\"emb_size\")\n", + "plt.ylabel(\"max_seq_len\")\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 853 + }, + "id": "G5gnKgeGcQPR", + "outputId": "ffa750f7-0827-4793-a10a-1ced840035db" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1IAAANECAYAAAC6qR6BAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAChSElEQVR4nOzdeZgdZZn//7uqztLb6e50Z+lOJ2QjC0sCCUuIgIKgkCiIoCKoLCKLsiiZGZ38xnGEcQyCIioRhvmi6FcQVBYRFAWURQhhCSGsgSxk7yzd6fR+lqr6/cE3sTOc535SXel0d/J+XVeuS3LnrlN9uuOV59TzeW4nDMNQAAAAAAC7ze3vGwAAAACAwYaFFAAAAABExEIKAAAAACJiIQUAAAAAEbGQAgAAAICIWEgBAAAAQEQspAAAAAAgIhZSAAAAABARCykAAAAAiIiFFAAAAABEtM8spBYsWCBjx46VkpISmTlzpjz//PP9fUsAAAAA9lFOGIZhf99EXPfcc4+cd955cuutt8rMmTPlpptukt/+9reybNkyGT58uLU/CALZsGGDZDIZcRxnL9wxAAAAgIEoDENpa2uTkSNHiuuanzvtEwupmTNnylFHHSU333yziLy3MBo9erRceeWV8q//+q/W/nXr1sno0aP7+jYBAAAADBJr166VUaNGGeuJvXgvfSKXy8lLL70k8+bN2/l7ruvKySefLAsXLizak81mJZvN7vzvHWvJUd/+prglJUV7XjnzZ8Z7OOy+L6r3qPXG7ee1eW1em9fmtXltXpvX5rV5bV57z/W2tgcyZsa7kslk1GsM+oXU1q1bxfd9GTFixC6/P2LECHnrrbeK9syfP1+uueaa9/2+W1JiXEhVZsyP9Uw9u9Mbt5/X5rV5bV6b1+a1eW1em9fmtXntPfvaImKN/Owzh01EMW/ePNm+ffvOX2vXru3vWwIAAAAwiAz6J1JDhw4Vz/Nk06ZNu/z+pk2bpK6urmhPOp2WdDr9vt+/42O3SoVhdXrF+uON93DJyY+r9/hwp74inn7kcrW+It9urNVOalJ7twddaj3Z0KHWs2HeWAtrc2qvHwZqPcgU1LraW6pf2yZM9T4aGCbixQpDL8Zrx/3oI85ZKpzDAgAAsNOgfyKVSqXkiCOOkMcf/8diJggCefzxx2XWrFn9eGcAAAAA9lWD/omUiMjcuXPl/PPPlyOPPFKOPvpouemmm6Sjo0MuvPDC/r41AAAAAPugfWIhdfbZZ8uWLVvkW9/6ljQ2Nsrhhx8ujzzyyPsOoAAAAACAPWGfWEiJiFxxxRVyxRVXxLrGcC8nGa/4bscXb5pu7Pvh9QvU6x70xJfU+i9m3a7Wb20y57M+Nup1tXdRd6VanzJis1rfUMgaa0OGmLNbIiLtoblXRCRZoWestHyWU6rnq2z5rDDV+4xVmIw5es2L0evGzGfF6A/jZqTIZwEAgH3IoM9IAQAAAMDexkIKAAAAACJiIQUAAAAAEe0zGak9YfZTXxa3tPjMp4l3LTL2rfput3rduvtSav2YE/T7+vwrM4y1351wi9r7863mfJWIyKyalWr9jfxQY21cdbPau9X31Xp1Rp9x1R2ac1DJEj0jVRD9tZ20XlczVjHmQInEmyMV+6OPWDmlmF93f+acyFgBAIA9jCdSAAAAABARCykAAAAAiIitfT1MuqldEl7xI7cLs6YZ+7701lj1uplHXlPrrYG+NbD2WfPWwEM/qu9Z+uvqiWr9J9N/rdafap9irE2q0I9OX+9XqPXh5frx6dsD8/a78lL9aPV8qG/dSyT1eiDmbWxOUj863Xb0epwjzOMcX/7ea8fo7cftcf26LRAAAKAInkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakeglVrJXCSRWub7hlv7PMeqFOvW+5vVOv3tOk5pmGLthlracP97uAvy6j1yUdvV+s3NI011s4daT4SXkTknaz+vtSXtqr1lsD841lZomekOi0ZqVRaPz5dy1i5loyUlq8SEZGEXlczVjEzUmGcI8zj5pRiHp/eb8hnAQCAIngiBQAAAAARsZACAAAAgIhYSAEAAABARGSketh0weHipUuK1p458kZj39mXn6let+OkqWp9wTJz/kpEpOHN5cbaxoI+i6n6LbUsI7xStb68cZixNmGsPkfqwe0z1PqoEnP2S0Rki19urFWnu9TejkDP46STxeeF7ZAXJSPl2WZQ6Rkqx7PMmdLE/eijP+dIxemPma/q1zlUZKwAANgn8UQKAAAAACJiIQUAAAAAEbGQAgAAAICIyEj18LmL/iIlFcXfkse6hhr7Chv0OVGr/3OkWq9YWK3WQ9+cyXmya7TaW71Mz1AlHU+tO+vMGaqRXqfau6x1hFqfM/xVtd5YqDbWhqT11+4M9a+rNGmbI2XOMSWTekbKD/U8j+PFyPvEnCMVR6wZVCJkhQAAwD6FJ1IAAAAAEBELKQAAAACIiIUUAAAAAERERqqHS6velcpM8bXl1FuvMPYdcGSbet35x96r1m//2SfUujdxnLH2wNYqvXdVo1rvDHJqvWy9OdgyzNN/fNZsr1brDQ3Nav01Jf9Vk+xQe9uClFqvSGXVereSc0pY5kDZ5ki5lpxTIErd0usr2a7d6VcxiwkAAGAnnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakePv3OqZIoTxetjf3J68a+5f96sHrdz2a2qfVfLl2p1pvOOMRYe2tFpdo7cevLan2dn1frmfXmmUmljp5DammuUOvDPD1btj5bbazVpVr11w7K1HpFUs9IZZUoUdoyg8rXMk4i4sbIWDlx50jFyRrFzSn1a8aq9+9bSDYMAAAUwRMpAAAAAIiIhRQAAAAARMTWvh66f1wviWRJ0Vp5xQZj36dOeUa97lPd+uv6rfo2tc0fMG+vK3ur+P3urjdyI9R62XrzzXuOvg53m5JqvdbVt9dt6DIf7X5o+Xq1t8nXtxVmLFv7ukPPWEt55u+HiEjecgS5Z9na5ytHrztezK19cbcGxhDG2F7HFjcAADDQ8EQKAAAAACJiIQUAAAAAEbGQAgAAAICIyEj1kP7LYkk4xXM9b/3wGGPffcMfUK97+LNfVOvjG5rU+kdnvGqsvf7oNLXXGzpUrS9s13MryY3mo9t9SxYo3aSv06tcPfjS2GE+2r16RIfauylfrdbLvZxa1zJS6YR+/HleyTiJ2DNSGtvx54Hl6HVb1kj9nsbJOO3GawMAAAwmPJECAAAAgIhYSAEAAABARCykAAAAACAiMlI9ZE+ZIb5hjtS9Z/zI2LfJz6vXrb6vXK23HVmq1i8ffp+x9o1lY9Ref1ydWn9+a5laL2lqNtbaQ30WU0mTnqkpM+TRdtjWYX5far12tffNrga1XpnoUusdYcpYK03o3299ypRI0jKHKhBzTsmNmVOyZaz05lgvHU/c1+7Hew/JhgEAsE/iiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkeqh7KoNkihPF60N9cy5mEtWfEa97pA/vanWl//rwWp9Wqp4bktExFmzQe1tOeMQtb5lo55Tmti+xljb5OvzkEqbLHOmHP3Hr6vN/HVn3G61tymv59LqUq1qvTMo/nMgIlKi/CyIiOQtMaSEZY6Ur8yCss+RssyoipOxGsQ5pVjizs/qT4P1PQcAYBDgiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkerhngP/LJWZ4mvLA//yVWNf5RJznkZEpL7jRbU+7qi1an1dwTwzyW/Vsz7bpqhlSazX711Ccz5kQyGjtpY06Vkiz7Gs49vMP54ZR792c07PSE0s3azWWwNzPqvEK6i93aH+dVnnSCnvuRtnDpTEnSPVf1mhcDDnlAAAwD6JJ1IAAAAAEBELKQAAAACIiIUUAAAAAERERqqHW1rGS0mh+Fty0PXmLJLTlVWv2/WhqWr962N+pdbvaDnS/NrJlNqbnKxnqFJ/rVLrTtqcoXo7V6e/dnOXWrdJtJnX+RlXn5fUnC1T6xlPvzdtjlSpbY6U5fOJpOXetTlSrq1XyVeJiHWuUKC8dvw5Uv2Yc2KeEgAA2MN4IgUAAAAAEbGQAgAAAICIWEgBAAAAQERkpHq4+2cni5cqPj9oxMqXjH1hVs9IrfpXPUt0UmmnWr/i+WOMtcl1W/Vrj3lbrS9ZP12te0OqjbVlnXpWyG0xz78SEcmH+jylZJs52FLi6KGX7VnzHCgRkYzbrdY35IcYa6VeTu3Nx5wjlVdyTp6nZ6Rs4s2R0st+GO/e4rw2AADA3sYTKQAAAACIiIUUAAAAAETE1r4ehv/sZUk4yaK1xsvMR5DXP7VNve68Dz2k1rOhvkWuYqH5KO/sgcPV3k8M+ZNaf3v9FLUeDDdvcXunXd9vFbbqW/tsX3eqzVwrcTy1t63bfHy5iEi5q2/HbPPNWwNtx593h/pfq6Srb+3Tqp7lCPFA9O11cXbIxdoWGPfF+9NgvW8AANCneCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6cCccIK5XPFtzwWV/NPbdOnSOet1Lqjao9TvbGtR63cJWY23zkRm198i0nlPyGvV8V+ch9cbahm16eKS+c6VabwsKaj3Vas7kpA1Zth26u1JqvcySkWr3zRmrEteWkdLvLeHqOSat6lp6fdFzTI6lX2/ux4xU3JxSf+acYrxvYb/edz++NgAAgwBPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVA9vX10ubmnx+UF/qDbnfZbPeUm97kvZnFr/0fIPq/Xa15Yba9s/f7jaW+WWqvWgWc9ItTWMNtZam8vV3rqc/nW3BPo6PtVuzvO4lgBHvkvPKZU7+r21Fszv2+iSZrXXlpFKuXo2LK9EaryYs5wcS+5FnUNFZgYAAGAnnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGake/nT8rZLJFF9bfmzZ2ca+3026V73u8S9doNazL9ao9SFZc0Zq2KStau82v1OtB516vWOUORjjNelZIAn1PE9zUDyPtkOy1Zwl8hz9MwCny1PrZY6eU2otmOdIpW1zpAJ9hlXK9dW6r4SRPNsMKst77sbIWDmx50jF7I8h7MfXBgAA+yaeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR42+ynp9IuvLbtvGGns67xNz7wk7x+i1mvWZtV6YuwBxtrHR72m9r6QrVLrthxTd4M5D1S2ypKRcvWcUmOhWq0n2/UskvrSXfrQoxJHzxq1580ZqTJXn0FlmyOV9mxzpMyfb3iWrI8vet2Wc/K1n4e4c6QG6xyqwXrfAACgT/FECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9XDhHy4Tt6T4bKMJf3rO2HfF6tPV6w596G39hS05pW0nTzLWZmd+q/bevvWDat3VRznJsIYWYy14Yajl2uackYjIhryeHfPa9OyY2mvNSOn9HUpGqsSxZKQCPSOVcnufkUrY5kip1bhzpPR6YMln2bJGfqjcPXOgAADAAMMTKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1cPEm9dKwi2ejen42FHGvjf+oGdiGpoX6S8c6HOoNs0y1w5N6cGTv707Ua2Pr21S69OHrTPWXm+uUXud8nK1vi6nZ6Scji5jTc3TiEiiU39fkpbAT0c+ZayVuPp8q+15/etOWmZY+UqYyLNkpNQ5UGKfI6U3D+KcErOgAADAHsYTKQAAAACIiIUUAAAAAETE1r4ewq5uCQ3brir+xbzFreySMvW6+eOnqfXkG+Zri4gcdZT5+PS0o28rDJZVqHV/uL5VbFbli8baiqaD1F6pzqjl9V36Frmwq9tYK4i+HTJhbhURkaSjf4bQlTf/1Shx9PvuDvW/VmnL1sB86BlrCcu2QBvXsj0vUA5Qj707Lta2wrgv3o8G870DAAAjnkgBAAAAQEQDfiE1f/58OeqooySTycjw4cPljDPOkGXLlu3yZ0444QRxHGeXX5dddlk/3TEAAACAfd2AX0g9+eSTcvnll8tzzz0njz76qOTzefnoRz8qHR0du/y5iy++WDZu3Ljz1/XXX99PdwwAAABgXzfgM1KPPPLILv99xx13yPDhw+Wll16SD37wgzt/v6ysTOrq6mK91ruXThavpKRo7bWJNxv75rw9Q73uisuPUetjSsao9W/V/7extq5gPiJcRGTIW3oupatBP6r78PRaY+2e5k61N6jWr72xs6DW010txlp3qPd6XfrXnRRzDklEpDtnzp7ZMlKdfvEj9HdIu/q955R7sx5/rlbtGSlNrKPTRcRy4vzANVjvW0TCQXzvAAAMdAP+idT/tn37dhERqanZdYbRnXfeKUOHDpVDDz1U5s2bJ52d5n/kZ7NZaW1t3eUXAAAAAOyuAf9EqqcgCORrX/uaHHvssXLooYfu/P1zzz1XxowZIyNHjpSlS5fKN77xDVm2bJncd999Ra8zf/58ueaaa/bWbQMAAADYxwyqhdTll18ur732mvz973/f5fcvueSSnf976tSpUl9fLyeddJKsWLFCJkyY8L7rzJs3T+bOnbvzv1tbW2X06NF9d+MAAAAA9imDZiF1xRVXyEMPPSRPPfWUjBo1Sv2zM2fOFBGR5cuXF11IpdNpSaffn2O59uw7pSxTPJ9y47aJxtfzDjLXRESuOOkvav1/tp2q1o8pMWdm7m7TF4DVyzrU+pYZ+pypcUlzJsfd1q72dh5Sr9ab2/WsUH12k/nagZ4GSurxLUk6ekYqn1PmSFnmQGUD/a9V0tXvXZ0jZclI2aZMudaMlZKDipmR6tc5UoM1KxT3Pe9Pg/U9BwBgNw34hVQYhnLllVfK/fffL0888YSMGzfO2rNkyRIREamv1/8hDwAAAAC9MeAXUpdffrncdddd8vvf/14ymYw0NjaKiEhVVZWUlpbKihUr5K677pI5c+ZIbW2tLF26VK6++mr54Ac/KNOmTevnuwcAAACwLxrwC6lbbrlFRN4butvTz3/+c7ngggsklUrJY489JjfddJN0dHTI6NGj5ayzzpJvfvOb/XC3AAAAAPYHA34hFYZ6RmD06NHy5JNP7pHX+mDpNqksLX4i/H/cdIGxL/dJ/bpfHbJcrd97TKNa3+qbc073bTlJ7XVXbVDr7adPUutVbqmxFra1qb1dtXqWraOj+MyundfPmzNUnZboSKJb/wOuJcBRyJlzStY5UkFKrQ/39PctH5r/WiYcPV/lW94XN0ZuxTYHKrAmtAAAAPYdg26OFAAAAAD0NxZSAAAAABARCykAAAAAiGjAZ6T2ppMXf0G8svfPlxIRabjtRWPf8Kf0rM9b+axa/+r4x9X6r1sPNtZeXDFG7Z3Y9LJazzXoeR8/NOdego4utbe7Vl+n+21JtS7KrKi2QO9NdOp5Hc/R7y3MmjNSKUsWqMvX7y2Z1udn5fpwjpRn61cyiU7sOVJxegfxPCUAALBP4okUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqoe6mhCQSxfMt7qRxxr4bR/1Mve4nXv+CWn/00LvV+lGLPm6slb6l57PEkgVqGNms1rcF5hxUmM+pvd21alkSreYskE1rWDzLtvPaXfq8JRsnb37f0pZZTraMVImjZ6T0OVJ6xsm3BJHi5Jxsc6Ts/YMz5xQO0vsGAAB9iydSAAAAABARCykAAAAAiIitfT04L7wujlN8W9Zbd0w39nmW7VTd941Q695Uvd9dWGWsVa/Ut5l5w/T9dUcOW6PWVxZSal2Tq9XvrWSTZWufspesxS9TW73umFv7subXTlm213Xbjj+3bu3Tjj/Xv658qH824lm2qfnSh8efxxFzW2G/Gsz3DgAAjHgiBQAAAAARsZACAAAAgIhYSAEAAABARGSketh+9lHipYofJ/7MiTcY+76+4RT1unV/WKXW//wv5gyUiEj9wk5jLdFiPp5cRMQfV6fWj6l4Qa0v7hprLrp6xsmryar15HI95+QYjqIXEWn2K9RetzOv1m28nDnYkrRkXuwZKT3nlFMyUsmYx5+7sY4/t+SrQsu1Le9boOSzAAAABhqeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR6O/PLLkqoonm9Z66eNfc88dJh63dGNC9X6D9/9iFovWbrCXFRmLYmIbPvkIWp9anqDWv/eRnP+yy3Vsz4jalrVendbqVp3UkpGqmDJSGXjZaTcnLmmJ6BEugr6nyhx9XvrCMw/a7Y5UkGo/zx4rp6x0qqWHzWrWP2WXj/Uvy7pzxlYAABgn8QTKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1cP19YulMlN8bXngr6809k26r0m9rj9zqlrfsKhcrY9tW63WNS1T9PqYhB4+eXXzSGOtvqJZ7R1ftVWtr2gbrtad0uIzvUREmgv6e+Z0KyEnsWdq3Kw2R0p/z7IF/a9V0imo9Xxonq81kOdI2S8wSHNKMbNhAABg38QTKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1cNX1x8lqYpU0drkG9cY+wrr9VlMK783S603PKVnZhIN5pySv1nPIaUm67OcKlxzDklEpGVDpbFWX6XPQ5pSrme71rRNVutOqXnOVHNez0hJVs9IBaLndTyl3bOEZnK+p9ZTjj4LKh+a+21zpOJmpPzQXLdlpAJ1ClVMgzVfBQAA9lk8kQIAAACAiFhIAQAAAEBELKQAAAAAICIyUj28fstU8ZLFM0NDOpcZ+7wDx6nXPffUp9T6iz8cpdbbjhljrGVe0tfCJ48x37eIfZ5S6Trzj4hfo+eUDixpVOtPtOoZq7DMnN/amtVfO8xm1Xo+1LNGrpKRSjr6e57Nx5sjlQ2SSq9+30Go35tnnUNlFnecUr+OYxqss6AG630DALAf4IkUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqIfPbFyXhFM+nrPqWeRZUokO/7v1Df6PWT280Z2JERDYeM95YS20frvaeUf1Htd4UdKn1inXm+T3ZmrTaOyG5Ra177XqOKciYM1Lbs+YZUyIiXk6fn5VX00AiXtb8dbuWzx/yljlSSctra3OkbHOgcqK/tq2/r3pF7HOo9N5YL92/Bum9h4P0vgEA2Ft4IgUAAAAAEbGQAgAAAICI2NrXQzjzUAkTxbeTfe/zdxj77th4rHrdDQV9C5tXXaXWJ85cbaw1rjYfjS4iMj2t7zt8LadvkatYbz4HvLNO35I4wlPOEBcRp6NbredHDTHW2rPmbX8iIkMKzfq1Lce+a7fuWfaZ5fP69rqU5Qjz7j48/jzh6l+3VrVtzfPFsnUv5tbAWOJsU2OLGwAAKIInUgAAAAAQEQspAAAAAIiIhRQAAAAARERGqoetX8uKV1a89uFSc+bmqLH3q9e94J3PqnX/cP0I86+P/r/G2mVTLlJ7q1w9A/VC11i1XrKhzVhrOnSo2lvjptS606EfvZ7PDDPW2rr0o9er8wW13h3qeR0v1/vjz31frycd/d6ygfmvZYWn5+18S6DHteSYfKUc5/jy3RFoCa3+zFcNZoP1fSOXBgAYBHgiBQAAAAARsZACAAAAgIhYSAEAAABARGSkevjL4b+WykzxteXMF75o7Ft01B3qdTc9cIBaz8/S7+vEUvO8pRGTtqi92/xOtf5cy3j9xbdsM5ayNbVqa5klIxV26RmpnOF7ISKS7bZc29fnLWUt0RFtjpRrCXAEeVtGSr+3fGieQ2Xv1f9Ku44+R0rv7XWriIhYxm8BAAAMKjyRAgAAAICIWEgBAAAAQEQspAAAAAAgIjJSPTzaWStlXvF8ysjvmd+qf19wjHrdhgfXqvUNPy5X69kwb6yd1vCa2rsoO0Stv7apXq2PannHfF9D9byOTditz0TKZcyhGr/T8qMb6PfWreSQRES8nDlL5Dn65w9hQa+ntHlJIpJT5kh5loyTbY5UwtX7taptjlRgmc0Vaw5V3HzVYJ2nBAAABiyeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR6+8+uzxUuXFK2NXrTQ2PfQn/SM1NjV5l4RkSsmNar1+9pHGWtzMkvV3tu2fkitd66vUOth3jxQKVljnm8lIpIP9ZxSkDNnv0REcpXmYIzTHe8zAFtGyrUNmtJY5ki5lrxO1jf/tYw/R0p/bS1jZesFAADYn/BECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9TDmv9+QhJMqWmv6vDkHNe7eVv3C0w9Ry5/K6Bmqk5ecb6z9ffqdau9fV01U62Xr9KyQuOZ63ZA2tXV7oGeobLOe8hlzJsfrjPcZQEdY/Pu88/rKHCkbp6APPUpa5kjlQ/PXZstI+UqviIgrvc85xZoDJSJOjFlQcV+7P4WD+N4BAIAZT6QAAAAAICIWUgAAAAAQEQspAAAAAIiIjFQPTmVGHDddtDbray8Y+948oqBe993/nKXWK5zir7lDx8Khxlp6RlLtDZfpc6Iq1ut5Ha/S3D+haqvau8mPt07XMlKp7fGu3Rno77mb1bNIGltGyrNkZnKB+a+l69jyVXrmLeHaMla9nyPlW/JX/ZpzipHP6leD9b4BANgP8EQKAAAAACJiIQUAAAAAEbG1r4dlVzaIW1pStPZw/UPGvo+NOU297uw55m2BIiJPd+vfhvqFWWNt3aXtau+Qt/TtVBUbcmpdhtYYS1PKX1Nb3y0M0a9tEWTMWya9Tfrx5dqx7SIirUHx7/PO9lyMrX152/Hnlq19vvnercefWz4bsW0rDJS9ZHF3mdm29vmhUmeLGwAAGGB4IgUAAAAAEbGQAgAAAICIWEgBAAAAQERkpHr41ZxbpCJTfG156boTjH3rTx+tXveeEb9V6+euOEOtp19eaaw92TVG7a1epmeo3LZutZ6vqzLWJpdsVHvfydapdSeh//ilMub8VqJTz0g5np6Rsh5/HiMj5eqn4Ytnyftox5+nHP3ituPPXUs+y1fCSLbjz/tTYPm6AAAA9jSeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR6GeHnJeMXXlkt/eJixb9iX1qrXbQv0XMsbz49T6xO2LTTW7t88Xe1139VzTKLN7hGRjulDjbWxiSa19y8th6p1x/LTV1XRZawFXeX6tZP6xdv8Ur0/H2OOlCUjlbT055U5Uq4Eem+of90JV/+6grD3GSn9zkScGLOg4vS+dwG97IfK3Q/gbBgAAOg/PJECAAAAgIhYSAEAAABARCykAAAAACAiMlI9zPnbV8QtLSlam/Tr54x9t173d/W6V64+Q63XPaenS7xhw4y1xSsOUHsnNi1W67Z5S22jzGvtEV5e7X2n1XzfIiLJUn3G1fByc31rV63a66T0JNJ2S0ZK8uagk5qnERE3rwdyPEvgJxeYvycpR8842eZjeTHyPk7MrFDcfgAAgIGEJ1IAAAAAEBELKQAAAACIiIUUAAAAAERERqqHyTdtl4TXXbSWP+5wY9/IxAvqdZf9fpJaP+D5NWq948ixxlrpMstUIkdfK4cFfehRZ4M5D1Tj6Xmc9duq1Pq4Ej1jVV/aaqy1dFnyNsmUWm7zi2fhdnCUjJSNa2m1fXqhzpFy4s2RckV/33xl4JJtjpRvmUkWy2DOV8WdgQUAAAYknkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakegjXrJXCKZ46abhhr7Ltu62HqdUc/sFGtF9auU+uNF4821oa9rM8V8obp85aCpma1nmzoMNbShvdqh64mfVZTWK7XR5WYs2PvdOlZISetZ6Q6Cnq+S5sjFVhyRraMlGcJzRQC8+cbtjlSWsZJRCTh9r7flpGysc2RCkT/nsZ78UGcsQIAAAMST6QAAAAAIKIBv5D69re/LY7j7PJrypQpO+vd3d1y+eWXS21trVRUVMhZZ50lmzZt6sc7BgAAALCvGxRb+w455BB57LHHdv53IvGP27766qvl4Ycflt/+9rdSVVUlV1xxhZx55pnyzDPPRH6dxi8eLl66+LHYC2fcZOw77K6vqtcdv3yhWnfLy9X68JmNxlrF3fr2usL4erWesBx/PmXEZrWuSTbpP15hhWVrX8q87TDRpW9RC0v0rX2tlq19Yd58NLttC5pjO/7c0bff5X3z5xuu5bXzofnodBERz9IfhNprx9za10+9sQ3m48sH870DADDADYqFVCKRkLq6uvf9/vbt2+X222+Xu+66Sz784Q+LiMjPf/5zOeigg+S5556TY445puj1stmsZLPZnf/d2mqeVwQAAAAA/9uA39onIvLOO+/IyJEjZfz48fK5z31O1qx57xCCl156SfL5vJx88sk7/+yUKVPkgAMOkIULzU+B5s+fL1VVVTt/jR5tPswBAAAAAP63Ab+Qmjlzptxxxx3yyCOPyC233CKrVq2S448/Xtra2qSxsVFSqZRUV1fv0jNixAhpbDRvh5s3b55s375956+1a9f28VcBAAAAYF8y4Lf2zZ49e+f/njZtmsycOVPGjBkjv/nNb6S0VM/YmKTTaUmn35+ROf/CP0tJRfG35KHOYcbrjbu/U30975DJar1QWTyXtcPc8fcaa/+9drra23L0oWp9aHONWp9V84qxlg3NOSIRkZImyzHfFXqOqS7ZYqx5XZYgkuX48/a85fhzJTuWD/V8lqu/Lfbjz31zzsl2/Hk+0P9K244w144/tx1frt9Z/OPTAQAABpIB/0Tqf6uurpZJkybJ8uXLpa6uTnK5nLS0tOzyZzZt2lQ0UwUAAAAAe8KgW0i1t7fLihUrpL6+Xo444ghJJpPy+OOP76wvW7ZM1qxZI7NmzerHuwQAAACwLxvwW/v++Z//WU477TQZM2aMbNiwQf7jP/5DPM+Tc845R6qqquSiiy6SuXPnSk1NjVRWVsqVV14ps2bNMp7YBwAAAABxDfiF1Lp16+Scc86RpqYmGTZsmBx33HHy3HPPybBh72WWfvjDH4rrunLWWWdJNpuVU045RX7605/26rW+VLVaKjPFH9JN/ekVxr7Rz+lzot79N/3pWEKPWMnssm3G2i1tbWpvyxS1LJWrK9X6kWUrjbXtQU7tLWnSMzH5jD4Da5hn/trcLj2IFJTq1+6wZKS8vPlIfN8yT8ktWOqWB8F+0Ps5UoElf2XPSCmv3Y8ZJ1s+y96/h25kbxus9y0i4SC+dwAAdseAX0jdfffdar2kpEQWLFggCxYs2Et3BAAAAGB/N+gyUgAAAADQ31hIAQAAAEBEA35r39505ttzJFFePDsz9ievGfucuhHqdWd9fKlaf2bNOPvNmV47qc9LSk/ertbbX9MzUpOT5v5GX//xKW2yzDzKmOcliYgMc7PGmtOtZ6QKGT0D1ZXX37eMb773ILRkoGxzpCyBHd83f77hOXpGKh/q72nSMocqCHufkdLvLH7OCQAAYCDhiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkeqh8KMRIsmSorVUlTnf0Th7tHrdXzX8QK3/vOJQtf6nziHGmjdSz2d99IC31PqfR+mDi0d4pcba4mxG7U0363OmOqaYry0iknHNWSInq1/bL9N/tDtz+pwpLSOVt86RUsv2OVK++eu2ZZzygSUj5er9vjK4KO4cKVtGSp3P1Z/5qrjzkAbrPCUybQAAqHgiBQAAAAARsZACAAAAgIhYSAEAAABARGSkekg9tlgSTvHszJs/MmeJ6idvivW6n696Va1/5s3PGWvehFq197Tqh9X6/Q1HqfWkY87cLM/Wqb2J5g61nq8sU+tlhu+FiIhYMlKFUv0zgmxe/9EPffNUpJxtjlQhXrYkCJQ5UpZpTbY5UmmxBLgUriUb5vdjpCawTbEi77N/Gay5NADAoMITKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1UP37CMkYZgj9cAnbjL2eZbsyNz1p6j1nx/wtFrfuKjeWCuboocBjkx3qvWyhna13hmYs0hvdpjvS0TEaWlT6/mKYWo97Zh/PMOcJSNVon9GkMvqc6RCbY6UJW7j5i11S4AjKGgZKf3Fg1C/tufoWSI/NL+2a+m1UcaCWTlkXgAAwADDEykAAAAAiIiFFAAAAABExNa+HiqvWifJ8lTRWrVrPja6wdOP8V70h6lq/aUvPa7W6xeaX3vNqfpauMItvlVxh0NHbFTr63zzPrXlrUPV3pL2ZrWez+jb1DzH/LWFOX3/XKFU3wvm5y2fIYTmbWx5y9a8uMefh775+knHvOVQxH78ua0/UD5bcS1HiNs2/jn9eQQ5WwMBAMAexhMpAAAAAIiIhRQAAAAARMRCCgAAAAAiIiPVwz0THpPKTPG15bg/fc3Yd+XMv6rXHXv/VrX+w499VK2XvbTaWKu/qkLt3ep3qPVjqleq9TdyI4y1jdsq1d6xXevVeqFSz+uo8raMlN4eZvUskYTmPE9eOSJcxJ6R0rJfIiKiZKRsx58XLBkp2xHmvhImsmWk4gqU97x/81X9+NoAAGDA4okUAAAAAETEQgoAAAAAImIhBQAAAAARkZHq4YfN46UklyxaO+j6FmPfzV/RM04T33hOrT/70ky9f9MiY+30hlVq76JsrVo/qlTPSD24fYaxlm3Wg0hhwTz/SkTEy+g5p3xozlDZrm2bI+Xkev8ZQrcth5SPmalRMlK2nFI+0O/NlrHylfyX7bW1fNXu9AMAAAwmPJECAAAAgIhYSAEAAABARCykAAAAACAiMlI93Pd/ThQvVVK0NmL1y8a+Cfdl1Ot6E8er9REL9WyJN2SIsXZKhTk/JSJyy5YT1Pp/1j+m1p/fOsZYSzVZZjFZZCq61Ho2NGeoQl+fQWWbI+Xk9Pdck7dlpCxzpKyCOHOkLDOuLHOkAuWzFdfy2jZxZkE5vf92xX7t/hQO0vsGAGB/wBMpAAAAAIiIhRQAAAAARMRCCgAAAAAiIiPVw7BfLJGEU3yO1IavHGnsq/uJnlNa+8/6nKjR921U67nDxxlrB6X0tfATqyaq9dqGZ9T66o3mOVQVTZZZTQn9x2toRYdabwuUWVGhZaZRmV73YmSkusPiPyM7ODEzUk5BmSNlySn16xypkDlSAABg/8ETKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1YMzcaw4Xrpo7ZJL/2Dse/jOSep1Dzp9mVrffkOzWt/4mXpjLW3IdO3gvFWh1r3j9bV0Yn3x90NEpHSrnnlxy8rU+vCy7Wq9Jej9Ot8v0e8t0d77a9syUm5en9Vk4/jKHClLzijuHClflHyWpTcuX8lv9escqJgzrPrVYL53AAAGOJ5IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpHpZ/rUTcspKitcuqVhv7fvGxj6vXvWPM99X6+emT1Hr6mCZjbU2hXe0dskzPtbQH3Wq9fJ05ZFHSrMx5EhEno+ez6ks2qPUtgZ6x0vhl+tedarZ8huCYv277HKm4GSlzzTYHqmDJlXmi31ugZKxs+azAEsjpy7iOb5krRlYIAADsaTyRAgAAAICIWEgBAAAAQERs7evhz8f9t2QyxdeWp751jrEvf+Y29bpljqfWw4MmqPWrJj1qrD3ROVbtrVrWptZXF/QtUZn15u176aas2htW6Vv7RqX0921LodJcVLbeiYg4pcr+OBHxcvqPvuOZv2fdgeX489hb+5QjyC29hUD/WUtq+wZFJBfq/XG4MY4w79fjzwEAAIrgiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkephbaFUygvF15aFG+qMff99y0/V6/5/jcer9U2zqtT6pyrWGGvnrTxN7XXf3ajWX82OVOtl67vM124110RE8sP0jNTIpJ6R2pAfYqxpGSYRkURJXq172bRaF8f8GYPt+HOJm5FS2j3LMd4F5fjy9/otx58rn624lqPX/UF8xnhg+doAAAD+N55IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpHi5+8BJxS0qK1iY8stDYd3Raz8yc98cj1Xr+A3rWqNRJGWsvrzhA7Z3UvFitP9euz7DyGpUcU17PIWWn1Kj1ukSLWn++Y7y5aMlIlZfm1LqT0zMxjmf+jKEj0PNVTl6f1WTjmEd37cYcKf1PuKJnpPKh+f8S4syB2p1+7c4sY8OsYvVbev3QkoljBhYAAPsknkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakeJt68WhJu8TxS+2lHG/tu396oXnfcfdvV+tG/eEWtP9FtzmCVL+v9PCQRkRe26BmryuZN+vUVXTV6jqnG7Vbr67uqjTUnoeezMiVZtd6lR6jUDFY20DNxjh8zI+WbQzn6OyriW+dI6XmdbqXftc2gCvUwkUNWCAAA7EN4IgUAAAAAEbGQAgAAAICIWEgBAAAAQERkpHoIczkxRUSG/PNqY993njxdve6kJc+r9YuGLFLrX1n5aWOt5i1l6JCIJIYPVesbNuiznio6VxlrjmWWU/dQPTNT7eqZm42dlcZaOtWmX7tEn82Vy1rmSCXNfzU6A/NcLxERKegZKdvcIUdp9ywDkWxzpDzLHCkt52TLV/Ul8lUAAGCg4YkUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqYdVlk8QrKSlae2viT419H/6vL6nXTYzVZzUdkKhQ68ueH2usTXy7Se3Nj69T66n1+kwkCc3ZlLCg57OytXquJePqP35b28uNtVFpfU5UdapTrTflLJmbhPneuq1zpPQcko2WkbJ98uFbMlK2WVC+8gqu6O+ZL3p+y+3PnNNgzVjpbykAAOhHPJECAAAAgIhYSAEAAABARGzt6+F7n/mllGWKH+n9vaaJxr7U35aq1133lSPV+ps5fRta3ULzdqxw7Ua1d9vMQ9V6xTq1LK5hq6OISNDdrfbma/Stf2lH3yLX3m5+bUnrR5APS21R6+/mLcefK1v7unzLdkhfP/48sGyR03bfeZa9Xr5yfPl7/fprB6bz/2V3tgXG29rnK9tI+/X488G6LRAAAPQpnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakejirZLpUlxdeW/98Pvmjsqyt/Q71u/SdWq/UFW05U65UvmINMhbY2tbdlslqWkX/X8zxubY2xFm7eqvaW1erZr6RTPI+2Q9BmziKFKT2nNCSpv7abteRekubrdwV6PksK+ntq4+rRMpXt+HMvRs6pX48vtwgk3pHzAAAAUfFECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9fCRF88XryxdtDb6/7xk7Gs5a4Z63bsP/L5a/9AjV6v1SeteMNacpJ7XKZ3Sotfv0dfS/vAhxprb1q72jqrertZtEq3mDFVYqn/dQxN6dszLWjI1yb6cI6W/tqO0u45ljpQlI+VaXjsfmt9zz9KrzaB677V7n7HSv+q+7x+8Lx7DYL1vAAD2Ep5IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpHhp+6EgiUXxt6Uweb+zbdlaHet0RXvHc1Q5DF+rfBjeTMddqzRkmEZFTDnhLrb+xuV6tt08fZaxVbDHfl4jIxMotat0P9cxNss0c0ghK9JxSTULPb7k5PccUxshIhb5lVlOoZ4W0jJRnCa74gV63zZHSck5x50g5lv5407cAAAD2Lp5IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpnha/KeIUz7+8/cvDjW23z/iFetn/3KLPmRr23Fa1XphqzmflSvVv4WnVD6r115r0/FZ7wxhjrXxVhdo7uew1tZ4NC2o9qcScAsvXXe12qvU4GanugmWOVJDXy5Z5TG6MsFDBMkfKs8xy8i2zoNTePhw8FDefBQAAsKfxRAoAAAAAIhrwC6mxY8eK4zjv+3X55ZeLiMgJJ5zwvtpll13Wz3cNAAAAYF824Lf2vfDCC+L7/9jr9Nprr8lHPvIR+fSnP73z9y6++GK59tprd/53WVnZXr1HAAAAAPuXAb+QGjZs2C7/fd1118mECRPkQx/60M7fKysrk7q6ut2+ZjablWw2u/O/W1tbRUSk5ZyjxEuVFO35+4duMF6vPqFnhS7+7XFqfcI7L6j1xq8ebax53WqrzEjpfyDo1uvt5jFSUlOjL1gnphvVemeoZ4lSreZcTMGWkfJiZqTSnrHW7Vvmfvn6e+pbckqOb667lofIgSUj5VryWYGSc7LllHzLvfVnzsk2w0rv3YM3srcN0nsPB+l9AwD2LwN+a19PuVxOfvWrX8kXv/hFcXr86+bOO++UoUOHyqGHHirz5s2Tzk79H9Hz58+Xqqqqnb9Gjx7d17cOAAAAYB8y4J9I9fTAAw9IS0uLXHDBBTt/79xzz5UxY8bIyJEjZenSpfKNb3xDli1bJvfdd5/xOvPmzZO5c+fu/O/W1lYWUwAAAAB226BaSN1+++0ye/ZsGTly5M7fu+SSS3b+76lTp0p9fb2cdNJJsmLFCpkwYULR66TTaUmn9WO/AQAAAMBk0CykVq9eLY899pj6pElEZObMmSIisnz5cuNCymTWZS9KuqL4jKDVhVJj39PdVep1x9+nDETaDYVZrcZa+8ZytbfCLZ752skSAMk1mHNM3UNTau+YxDa1vj3QcyupNnOep1Cq70rNOHr+ysnrGSm/1Py+5SxzpEp9Sw4ptGWkzDXP8v0KAr3uOfq9aXOkPEu+ysaWkdKuHifjBAAA0BcGTUbq5z//uQwfPlw+9rGPqX9uyZIlIiJSX1+/F+4KAAAAwP5oUDyRCoJAfv7zn8v5558vicQ/bnnFihVy1113yZw5c6S2tlaWLl0qV199tXzwgx+UadOm9eMdAwAAANiXDYqF1GOPPSZr1qyRL37xi7v8fiqVkscee0xuuukm6ejokNGjR8tZZ50l3/zmN3v1Ot+te0UqM8Uf0h145xXGPn9oTr3upBcWq3X34Elq/ZKD/m6s/a5ihtq71e/QX9syc6thZLOx1lUzQu0d5unbsVYW9JyatrWvu8Z8PLmISLlr2YaW07f+hSnz+5L19dcu9S3bBq3Hn6tllXVrn+W186H5a7NuzVO2BQIAAOxrBsVC6qMf/aiERXIlo0ePlieffLIf7ggAAADA/oyPkAEAAAAgIhZSAAAAABDRoNjat7dcvvYYSVUUP9J78o2rjH3tRxygXjcxUj9BcNMxNWr985WvG2v5UXpeZ1G2Vq27NUPU+pHD1hhrjw2tU3urLEevb/H1o9uTbeYcU1uD/qNbZjkm3MkX1LqfNH/GkM1b/toEluPP9W5xlYyUa/nsw5aRci3HnwehuT+p3ZiI+GJ77d4fYW47/tyWO5P+PD5df1v6rre/cWQ9AGAfxxMpAAAAAIiIhRQAAAAARMRCCgAAAAAiIiPVw7JbDpZEsniupzr3trGv7LGl6nU3XjBdrW87Up9pNNQzZ4lOqTDnp0REfrLpJLXu1+kZqWMqnjHWHq45Su1NOnp+a31ez4Z57VljzS/V51+VWF5bLBmpIG3+jCFvmSMV+noOyS9ylH9PceZIhYH+2YhnSWj5ymcradHfMxvXlmPqR4H2vpD12b8M5lwaAGCv4okUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqIfO7FyThJIvWVl7zAWPfuOtfUa9b8clGtf7BoWvV+vNZc4Zqeqr4/e7w5KoJan3EqFK1PjW9wVgr1OrZLpv1OT2f5XR0m19bv21JG76PO4QFyxwpJSNVKFjyV6GeQ7K9a67f+0yObY6UtV+ZI+VZZlD5YbzPZbQv2yW3AgAABhieSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR6CD0yTIFF8jtRNn7vd2Pdfr1ygXve2yT9U655lTs23133cWLtr3N/0a79VodbbR6plGZMwh1MqajvV3s4gp9Y3dFerdaejy1izZaRc2zAYW0YqZe4v5C1zpAL9+2mLQGlzpGxfV2jJSHmWWU750JL/isG1ZKz6kkPGCgAA7GE8kQIAAACAiFhIAQAAAEBELKQAAAAAICIyUj1sv7pTvLLiAZVZJS3Gvo1n6VmgA5NptZ509FzKcy9ONtbWjP6D2lu9TM+lNE3TwyMVbvHMmIjI6OoWtbfZkpFa31ml1qWrw1jyS/Wsj+dYPiPI9z4j5Rcs17bMkbIlheLMkRJLRsqWU9LmSLmWfFVg+VzGtWQBNU6MXgAAgL7AEykAAAAAiIiFFAAAAABExEIKAAAAACIiI9XDH6feI5WZ4mvLI5+72Nj3o5m/Vq9707ZJav1TmVfUet1Cc+2vp4xXe6vealPrm2br+S1fyftMqtys9m7w9Wtv6dBnXA3tbjbflyUjZRP6yrAmEfFTSq8tI2WRC/X+OOOW4s6RCpR78yw35ttmd1n05ZSpWBkrZlABAIAieCIFAAAAABGxkAIAAACAiNja18MfO4dLmVv8KPIDvmfe3/Ox33er173qDx9R69tPLFPrQ55vNNYe2DRd7XXXbFTr40bq2+uagi5jbWLpJrV3ZW64Wt/ebj5aXUSkJpc31oJSfWueTVjo/fHnkrfs9Qr1bWR5y+cXjnL8ufVYd8v+OOvWPmUfm+3odJs4x5/begPLe96vOLodAIB9Ek+kAAAAACAiFlIAAAAAEBELKQAAAACIiIxUD9ff+Snx0sVzO6NeNJ9B/kinfsz3+Hv1DNVdtUep9YmrXjbW3lhxhNo7qfkltX5krfmIcRGRt/Ll5vtKmbNbIiLPd05Q6/k2/X2TwJyDciwZKe3Y9veubcncaLfmxzzmO9T73UKMTI3l3mxZo3xQPCMoYs9X+ZZj3W2vHff4dAAAgL2JJ1IAAAAAEBELKQAAAACIiIUUAAAAAERERqqH0be9LgknVbTWdN4xxr4rnj9Mve6EZ19V69WHHq3W3bQ5sFO+rPj97mSZOzSrYplaX9w11lg7pfwNtfeX7XVq3W0z53Fs0qXmGVMiIgXRM1Shb8lYKW+rk4/3+UPOOkcqxsUt+SsbW35L7R3An8s4xK8AAMAeNnD/5QMAAAAAAxQLKQAAAACIiIUUAAAAAERERqoHd0iVuG7xPNLxVy0y9i36rj4HyqutUesjntuu1sNDzPOYat4qqL2J4UPV+sGpp9T63ZvN+a3zK/WM1IrttWo92db7dXxZSVatd4f6+yKWOVOBkpFy9XiWVT7Us2GOZcaVyjI+yzYLKlBmObmOfnHrHCnLa2scywyqvtSfrx1XOIjvHQCAgY4nUgAAAAAQEQspAAAAAIiIhRQAAAAARERGqodlV9aLW1pStPZw/R+MfbP/Uq5ed8unDlXrtXctVuubLj7CWKt/fIvamx+vz3Ia5SXV+quN9cZaxVjzfCsRkS3bMmq9tE0ti5Mw/3hWWjJSWUsGSkI9O+KnzHWnEG8oUT7U/9o5hRhZokC/N1tOKR+Y81u2fJWNa8nr+DFnYAEAAOxNPJECAAAAgIhYSAEAAABARCykAAAAACAiMlI93DP7p1KRKb62vGjNR8yNeX2wUHBmk1oPf6HPPGqb1WWs1f18g9q7bdY0tV7mKgOTRKRrQ4WxlnT0eUhBk56hSrXpmRknZb63IelOtbcjziwm6ds5Urn+nCNlySkFSk7JOkdKmUEVly1fZTOYZ0EBAICBiSdSAAAAABBRr55IdXR0yHXXXSePP/64bN68WYJg10+qV65cuUduDgAAAAAGol4tpL70pS/Jk08+KV/4whekvr5eHGffOLa41PGlzLAF6M0fmo8wT53kq9e97dAFav3fJ3xWrV8wdaGx9nRH8ePad2iZrJYlH+r3XrbOvA3Ntxwxnt6qb2FLten9Ttq8NbDWsrWvzXLEuE2QNm8FS3RaHuRa/j7kxbK1r2DZn6f1+vpr244wD5TteZ5l32AQ6u+LbWtgHH7Mo9n71GD9v8fBet8AAOwlvfrX5p/+9Cd5+OGH5dhjj93T9wMAAAAAA16vMlJDhgyRmpqaPX0vAAAAADAo9Goh9Z//+Z/yrW99Szo79e1VAAAAALAv6tXWvh/84AeyYsUKGTFihIwdO1aSyeQu9cWLF++Rm9vbTvvr5eKWFs8cTbrnOWPf27cfqV73iLR+xHjz0cPV+gXVvzTW/p4+Se0tm9yi1jf55qPVRUQq1ptzLV1hTu0t0U99l1SrJSNVVmqs1aS2qb1t2vnluyFIm+/NKcQ77DJvyW/FOf48bgypEPTdQZ7Wo9eVUE6/xnXICgEAgCJ6tZA644wz9vBtAAAAAMDg0auF1H/8x3/s6fsAAAAAgEGj1/t4Wlpa5P/8n/8j8+bNk+bmZhF5b0vf+vXr99jNAQAAAMBA1KsnUkuXLpWTTz5Zqqqq5N1335WLL75Yampq5L777pM1a9bIL39pzvQMZJN/tE0SXvHZRfnjpxv75h93r3rdW1sa1PrmD+jZkVGJCmPNqx+h9p465k21vixfpdYr1plzUM1BQe0tadYDO8l2vT8sM8/IGppsU3tbgjK1bpU037ur37aIo38+kQv7bo6UBHqgx/bJSRCa+11Lxsnv/ecyfc6x3Xs4gOdQAQCAAalX//KZO3euXHDBBfLOO+9ISck//rE7Z84ceeqpp/bYzQEAAADAQNSrhdQLL7wgl1566ft+v6GhQRobG2PfFAAAAAAMZL1aSKXTaWltbX3f77/99tsybNiw2DcFAAAAAANZrzJSp59+ulx77bXym9/8RkREHMeRNWvWyDe+8Q0566yz9ugN7k3hug0SOsXnD7XcaM61fDajzzQa98Cn1foJR7yh1l/PmWc9dR2oL1xPr3pArf+9Y7JaT23YbqxtKJjnPImIlDTpYaJEW1ath+XmjFSN16H2NhXMuTIREXEsWaK0b65ZMlKOq1+7P+dIeZaZSAUlv+WJfnFfyVeJiLjS+6/Lls/qT0GMrwsAAAxevXoi9YMf/EDa29tl+PDh0tXVJR/60IfkwAMPlEwmI//1X/+1p+8RAAAAAAaUXj2RqqqqkkcffVT+/ve/y9KlS6W9vV1mzJghJ5988p6+PwAAAAAYcHq1kNrhuOOOk+OOO25P3QsAAAAADAq7vZD68Y9/vNsXveqqq3p1M/1t40WHi5cunst5fvqPjH1PdSfV646/15y3ERH56uzH1PqCLScaa9umFJ97tcNhKfMcKBGRG9aOV+vSZM5/rcgPV1vTTd1q3W3XM1L5YeacU7XXqfauzdeodcfTZzklUkpGKq+2WudI5S1zpMTvu4yUjTZHyrNcPIg5R8oXS4ALAABgANnthdQPf/jD3fpzjuMM2oUUAAAAAOyO3V5IrVq1qi/vAwAAAAAGjXh7cSwqKytl5cqVffkSAAAAALDXxTpswiYMB9d8lYsueFhKK4q/Jfe3m/NA1776MfW6Bzz5ilo/PK3nnP700jRjzZmi568qXPMsJhGRNxtHqPVx280zrt7urlN73RZ91pPTree38uOHGGu1Xrvau7RrtFq35ZhSKfOwKMc2R8rTr90d6Jk6x9e/p2qvJSNl++REy0jFFWcWlK3XFg2zjA3rs97d4YfK3Q/g+VkAAOzv+vSJFAAAAADsi1hIAQAAAEBELKQAAAAAIKI+zUg5fR0u2MPOq1wnlZnia8vDb77S2Fe1Wk9ouJmMWt/s61miYQvNc4dKzm1Ue7darl1YX6bWw4I5ELSsXc9XyXY9xxQGehYoV2n+ujOuPqOqKWeeQSUi4nj6z2ZJ0vx1h3lLbsW1zZHS/9o5MeZI2cJClglW4gfme3ctF7d9Xa5tDlUf5rP61T76ZQEAsL/r0ydSg+2wCQAAAADYHX26kPrTn/4kDQ0NffkSAAAAALDX9Wpr39y5c3f7zx533HG9eYl+ccZbp0mivPhR5Afc/Kqxz0nob+O22ZPV+i+2r1HrQ5/bYqyd+C9L1d5nu4ep9fL1+lpa+9re3V6j9g7p2KDWbedK5yrM9YybV3u35sr1107q/eUp89HsHZbjz8XTN9B1W7bASWA7zNvMCeLtIwuUfWie5Sju7rDvPpdxOAYcAAAMML1aSL388svy8ssvSz6fl8mT31skvP322+J5nsyYMWPnnxtsGSkAAAAA2B29Wkiddtppkslk5Be/+IUMGfLe0NRt27bJhRdeKMcff7z80z/90x69SQAAAAAYSHq1F+cHP/iBzJ8/f+ciSkRkyJAh8p3vfEd+8IMf7LGbAwAAAICBqFdPpFpbW2XLlvfndrZs2SJtbW2xb6q/OD8eKk6ipGjNrTVnNPz1+hHkrWfp78n/vHasWh+/4g1jbU7Fa2rvjY0fUesV6y1Ht1dVGmtbmvVj3au6utS6Y8kS5SvNW0PLLJmZlmypWnc9/ej1sqQ5I9VlyUjZtrTmA8tfO9/8PfFD/ftlOWFcPMu9FZTjzz3L8ee248ttGau+RMYKAADsab16IvXJT35SLrzwQrnvvvtk3bp1sm7dOrn33nvloosukjPPPHNP3yMAAAAADCi9eiJ16623yj//8z/LueeeK/n8e6efJRIJueiii+SGG27YozcIAAAAAANNrxZSZWVl8tOf/lRuuOEGWbFihYiITJgwQcrLLUdOAwAAAMA+oFcLqR02btwoGzdulA9+8INSWloqYRgO6iPPk4+/LAknWbT25s0zjX1jH9BnNd0y/Xa1fuWPv6LfmGPegTkpmVJbn141Qa2PWdetv/ZQ86yosKn4zK1//AE9lxL6ek4pp0SwypX3RERke7Z41m2HIQn9665IZo21lrwlb2PJfmX7co6U/pZahZacUxyu6O+br8ywcvsz40S+CgAAFNGrjFRTU5OcdNJJMmnSJJkzZ45s3LhRREQuuugijj4HAAAAsM/r1ULq6quvlmQyKWvWrJGysrKdv3/22WfLI488stvXeeqpp+S0006TkSNHiuM48sADD+xSD8NQvvWtb0l9fb2UlpbKySefLO+8884uf6a5uVk+97nPSWVlpVRXV8tFF10k7e3tvfmyAAAAAGC39Goh9Ze//EW+973vyahRo3b5/YkTJ8rq1at3+zodHR1y2GGHyYIFC4rWr7/+evnxj38st956qyxatEjKy8vllFNOke7uf2zL+tznPievv/66PProo/LQQw/JU089JZdccklvviwAAAAA2C29ykh1dHTs8iRqh+bmZkmnLbmZHmbPni2zZ88uWgvDUG666Sb55je/KZ/4xCdEROSXv/yljBgxQh544AH57Gc/K2+++aY88sgj8sILL8iRRx4pIiI/+clPZM6cOfL9739fRo4cGenr6vr4EZJIFs/W/OHjNxn7Pu59Vb3uB/W4jtQ9q8+ZciaPN9ZceU7t9d7SDwBJNm5U67nRQ4y1dJNlHW7Ly1kyVPmMuZ529B/dti79Ta9J6P2ZRIex5hYsmRnLtfOBnqFylDlSNrY5UrZPTrRZUK7l4n7vPpfZI3zLz9J+a/BGVgEAGPB69S+f448/Xn75y1/u/G/HcSQIArn++uvlxBNP3CM3tmrVKmlsbJSTTz555+9VVVXJzJkzZeHChSIisnDhQqmurt65iBIROfnkk8V1XVm0aJHx2tlsVlpbW3f5BQAAAAC7q1dPpK6//no56aST5MUXX5RcLidf//rX5fXXX5fm5mZ55pln9siNNTY2iojIiBEjdvn9ESNG7Kw1NjbK8OHDd6knEgmpqanZ+WeKmT9/vlxzzTV75D4BAAAA7H969UTq0EMPlbfffluOO+44+cQnPiEdHR1y5plnyssvvywTJujHbQ8E8+bNk+3bt+/8tXbt2v6+JQAAAACDSK/nSFVVVcm//du/7cl72UVdXZ2IiGzatEnq6+t3/v6mTZvk8MMP3/lnNm/evEtfoVCQ5ubmnf3FpNPpolmu2itWS7K8+FymMmVAz7wTHjLWRETubKtV6+5rK9T6lnMPM9bWFDrV3iHLLKGZrc1quf2YEcZaSZN+aSelz7gKs+ZZTSIifsb8nicdPWfU3V18Htg/LqDXM0nznClbRsrx9M8nskHv50gFlllMtrKNr2SkPMvFg1D/uu0Zq76bI+UwCwoAAOxhvXoi9cgjj8jf//73nf+9YMECOfzww+Xcc8+Vbdu27ZEbGzdunNTV1cnjjz++8/daW1tl0aJFMmvWLBERmTVrlrS0tMhLL72088/89a9/lSAIZOZM8wBdAAAAAIijVwupf/mXf9l5QMOrr74qc+fOlTlz5siqVatk7ty5u32d9vZ2WbJkiSxZskRE3jtgYsmSJbJmzRpxHEe+9rWvyXe+8x158MEH5dVXX5XzzjtPRo4cKWeccYaIiBx00EFy6qmnysUXXyzPP/+8PPPMM3LFFVfIZz/72cgn9gEAAADA7urV1r5Vq1bJwQcfLCIi9957r5x22mny3e9+VxYvXixz5szZ7eu8+OKLu5zyt2MRdv7558sdd9whX//616Wjo0MuueQSaWlpkeOOO04eeeQRKSn5x9HWd955p1xxxRVy0kknieu6ctZZZ8mPf/zj3nxZAAAAALBberWQSqVS0tn5Xjbnsccek/POO09ERGpqaiIdJX7CCSdIqMx/cRxHrr32Wrn22muNf6ampkbuuuuu3X5NzZ3j/yaVmeIP6cY9fLWxb9XH/ke97rg/fUmtT+p6Sa03fSBvrD3ZZZ4xJSJSuUyfUeW3tqv19lHmh5aV7+qZF7dCn2Hl5wtq3cuYv25bRsrvtPxoJ/V6hWfOb7l5S97G0+8tH+p16cM5Up5lsJA2R8qLkXEa6ALp/XtuRT4LAIB9Uq8WUscdd5zMnTtXjj32WHn++eflnnvuERGRt99+W0aNGrVHbxAAAAAABppeZaRuvvlmSSQS8rvf/U5uueUWaWhoEBGRP/3pT3Lqqafu0RsEAAAAgIGmV0+kDjjgAHnoofcf+f3DH/5wl/++7rrr5LLLLpPq6upe3RwAAAAADES9niO1O7773e/KZz7zmUGzkLq+aaKUZIvPFzroevPQpJdOzqnXHXOvnh1JHKBvhzzzsMXG2n2bZqi97rsb1LofmGc1iYh0Npjrw1/UM05OpkKvt3eo9UxFl1pXr92t55BCS0Yq45nnSDmWOVKS0F/bOkdKyQ3asjy2jJSNH/TqIbWI7M4cqd5nhdy4A7IGq8EbOwMAYJ/X+3817QbtIAkAAAAAGKz6dCEFAAAAAPuiPt3aN9g89D8fFC9VUrQ2fO3Lxr6LX/2Cet3hjy9V69tPO1ytf2Xor421k5d8Te2dtO1Fte4kU2q9rMF8PHpqm9oqQbW+tc9tblHrQyv0rX8ar1P/jMC2ta9C2drn2rb2ufpr52xb+3x9u6XGtrXPdfS9YqFy/Lkb84hwz9Jv2xoYR5wdcuyu6yXeOADAPo4nUgAAAAAQEQspAAAAAIioTxdSxx9/vJSWlvblSwAAAADAXterjNQdd9whF1xwwft+v1AoyL//+7/L/PnzRUTkj3/8Y6yb29uG/nKxJJzix5+vv+poY1/JfXpmxknpR5A3ztLDBBOS5qxRxTI94ySufhS3W12l1g8dsdFYa9tWqfZ2j61V6+mNxfNoO9SXbVfrGq/LkgVK6+9LxjUfve4W9KxPaDv+3Nf/2oW++fq+5SRMp/fxqveuH5jfN88SwPL7MRQT88sGAACIrFdPpK666ir59Kc/Ldu2/eO0gWXLlsnMmTPl1782H4wAAAAAAPuCXi2kXn75ZVm3bp1MnTpVHn30UVmwYIHMmDFDpkyZIq+88sqevkcAAAAAGFB6tbVvwoQJ8swzz8jXvvY1OfXUU8XzPPnFL34h55xzzp6+PwAAAAAYcHo9R+rhhx+Wu+++W2bNmiVvv/223H777fKhD31IRo4cuSfvb69ypowXx0sXrX3lS7839j34oYPV67aeotenHbVCra/Km2c51bxZUHsTI4ap9XCInnM6pvp5Y+0vrRPU3u6hdWq9pEw/iGRUSaOxlg/1VIwyBkpERIKU/qNfqVzAyVvmKVkyUrlAr0vY+3lNtjlSniXHpM2RsrHNgXIdPd+lZawcSy8AAMDe1qutfZdeeql8+tOflm984xvy9NNPy9KlSyWVSsnUqVPlN7/5zZ6+RwAAAAAYUHr1ROqZZ56RRYsWyWGHHSYiInV1dfLHP/5RFixYIF/84hflM5/5zB69SQAAAAAYSHq1kHrppZcknX7/FrjLL79cTj755Ng3BQAAAAADWa8WUsUWUTtMnjy51zfT31Z8rUTcsuKzjS6qWmPsu79JzwJtOrNBrX931KNq/dbm44y18reb1d78eP3eCqV6Xueo0pXG2p/b9Wt31eh5m8pyPSPVkN5mrNkyUolOtSxhSt/VWuZkjTXHNkfK09/Tgi0j5eeMpUD017ZlpGx8JSPliSXjZMlI9SdbPgsAACCqXh828bvf/U5+85vfyJo1aySX2/UffosXL459YwAAAAAwUPXqI+Qf//jHcuGFF8qIESPk5ZdflqOPPlpqa2tl5cqVMnv27D19jwAAAAAwoPRqIfXTn/5UbrvtNvnJT34iqVRKvv71r8ujjz4qV111lWzfvn1P3yMAAAAADCi92tq3Zs0a+cAHPiAiIqWlpdLW1iYiIl/4whfkmGOOkZtvvnnP3eFe9Oixt0kmU3xt+dE3zjX2JY6tUa/7k6PvUuvHpvVgywWvHGGsTV73ltrbfNw0te7qUSOZmOwy1oKsOUckItJdq2ek/Iw5ayciUpcwL8qzoWV+Vpclz5PWc0rlrpKR8m1zpPTPJ/pyjpQlQiWu5bMTbY6Ua7l4YJlRFSen1J8Zp7gzrJzej+bqX4P1vgEA2Et69USqrq5OmpvfO+TggAMOkOeee05ERFatWiVhSKgbAAAAwL6tVwupD3/4w/Lggw+KiMiFF14oV199tXzkIx+Rs88+Wz75yU/u0RsEAAAAgIGmV1v7brvtNgmC97b5XH755TJ06FB55pln5PTTT5fLLrtsj94gAAAAAAw0vVpIua4ruVxOFi9eLJs3b5bS0tKdg3gfeeQROe200/boTe4tK/KlUp4v/pDOuWGYsW/lWfrbeGqZniXa5ner9SHPpoy1oKND7d1uGeuV3K4HIWpdZdaTZRtntlbP1BQySbVel2gx1rotOaKEOdolIiJBUv+6y528sebk9WBZmNa/LtscqWRgfl99yywnR+ndHUHQ+2CMbY6UZ8lY+b17QA68jxL1AwBgj+nVQuqRRx6RL3zhC9LU1PS+muM44vuWEwwAAAAAYBDr1UfAV155pXzmM5+RjRs3ShAEu/xiEQUAAABgX9erJ1KbNm2SuXPnyogRI/b0/fSrLz9wsbglJUVr4/+y0Nh32Y1t6nUf7ChT6xvyDWp9+KIWYy1M60eIV0zeptZbNlSqdc9R1tqWc53D2pxaz1m29g3zzPvz2ix7d5Jdlm1kaf0zhLRj/kDAKViuXa5fO+/rW/sSypbJwLKd0rEcf+7ZvmfK++rZLt6HbMef2+4s7hHmAAAA/1uvnkh96lOfkieeeGIP3woAAAAADA69eiJ18803y6c//Wl5+umnZerUqZJM7vpk4aqrrtojNwcAAAAAA1GvFlK//vWv5S9/+YuUlJTIE088IU6P7UKO47CQAgAAALBP69VC6t/+7d/kmmuukX/9138V1913jiw+cMG7knCLHzXedsbRxr65Q25Vrzvpbxep9fIK/fjzkW+sMNa8UfVq7+wD3lDrf5KD1Xp7YL4315LPGlLTrtZzFbVqvVr50dpQ0H90E5aMVC6j55RKtDxQwXL8ecKSkQr0eqlyYIv9+HO1bKUdf+5ZXjsQPX9lzTlZjk8HAAAYSHr1L5dcLidnn332PrWIAgAAAIDd1auV0Pnnny/33HPPnr4XAAAAABgUerW1z/d9uf766+XPf/6zTJs27X2HTdx444175OYAAAAAYCDq1ULq1VdflenTp4uIyGuvvbZLzbHMqRnIwkJBQsN2xeH/tNLY93beknG6t3juaodtk0r1+1IyM10Th6m9H6u6X61vHFGl1lcXzLkWp1yfjzWuulmtr8gMVetljjnH1BLo75lnmyNVq//olyg/xo41I6X/HbDNkZKg90GnuBkpbY6Ujd+HGSdbviouNXvWnzOo4v7f6eD9v2MAAAa8Xi2k/va3v+3p+wAAAACAQYPTIgAAAAAgIhZSAAAAABBRr7b27atWfflAcUtKitaWTbjF2Hfc0s+r18385TW1XrJ1klr3DhxrrG2arOevDkvl1PrMSnP2S0Tk1exIY82pzKi9kypWqfU3M/rXXeqYv7aWQM9neV0Fte6n9PetxFE+Y7BkmIKk/vlEwdfrYWjO5NgiUI4e3xLX8tlJqMyRci0BrMCSr0q6+s35fRjocfoz54ToBvP3i1waAOw3eCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6uP5Tv5DyTPEZP9/ZOsXY13X/CPW6GX+DWk++skKtN33iEGOtdYqeO6lwi2e+dphR+q5a/1XTB4w1v7pC7Z1U2qjW85V6DsJTckpNBf21va68WvfTljyPlpGyzJEKLHOkCoHl84vA/L74Sn5KpJ/nSFk+l0mLnlvTuNqcp34W2JJrgznvAwAAjHgiBQAAAAARsZACAAAAgIhYSAEAAABARGSkejiypE0qS4qvLefdcKKxr+5BfV5S+0emqfWSPzyv1jd/wJzJGTNhs97rd6j18ZafgJe2jjbW0rV6/mp8Sr+3Qqb3gZ5mv1ytO1k9IxXoY6Qk6RTPyomIiG/JSCX1nJFvmSOl9lrqjpKv2h1aRsqzZIFsc6Rs/DDG+0IMCQAA7GU8kQIAAACAiFhIAQAAAEBELKQAAAAAICIyUj2c9PwF4pUVz/2Muf0FY1/BkplZe1aDWj/oxTq1fvKM18295RvV3me79RlXp5W1qvUNG2qMtfpa/cdntNeu1iVjmfUUmjM5W/MZtdfpzunXTqtlcZXPGMK4c6QKSv5KRET5um2pMtscKVf0e9PGVHmWWU62jJQXY8iVG3NAlhsjvuXEi34BAIB9FE+kAAAAACAiFlIAAAAAEBFb+3o44KZQEl7x7UvOIRONfWFafxuv+8Dv1PqCI89W61eO+LGxZttu9f3Gj6r1M8qfVuup9UljratW3/M0zNPfl7JMVq1nw4Kx1pzTjz+Xbv3agfnLEhHLFrgg3ta+wHL8eagcYW475jvmDjgJA/axAQAA7A6eSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR7CJW9L6BQPzyz/1VRjn7O2VL3uZyq2q/VvfkA/DntaqviR7CIi+VDP6zy9coJa3z7qL2q9Yp251jVcbZVSJ6XWa8o71XpnaD4efVtOf8/DvOVo9bQeNvIc5TMGy3H3geVvVRAjh2Q//twSorJR2m1HkPuWz2Vc2/HpfK4DAAAGEf7lAgAAAAARsZACAAAAgIhYSAEAAABARGSketh27hHiGfJIzx7/fWPfguaj1es+0aWvV8ccrQSRRGRjod1YG+6Vqb3Jt/T6qln6vVWsN+eBWg7Se9WckYjUlbeq9e1K3mdbVv+6Utltaj3Q41uq0NezQkFSz0CFviUjFZqv72vzraR/50gFYd/NoHIdW75K51j6AQAAouKJFAAAAABExEIKAAAAACJiIQUAAAAAEZGR6uGDlz0v6Yric6TeyZvnFv1/Q5eo1z3smS+q9Ztm/Eat/7zlCGPtnKqX1N4hy/T0yJLsaLVeur7DXKzVg0a+kvURERlZqs/XagrSxlprt3m2lojIUMscqSAVIzMTWDJSlr9Vod/7zy98Sw4pdhRIub5nmwMV2jJztjlUfZexiiN2vmpgflkAACAmnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakerh3+qlRmiq8tJ/7ycmPfX865Qb1uzX3lav2kYzvV+lUvHGOsjThczxlVLdNnNS1snaDWvc3meUy1NfrX1R5m1XpDWp/11FioMtY6uvV8Vm2hoNaDdIyBS755tpaIPSMl1jlS5kxOYJ0jped5bLO9tIFM1oyU5d7cGEOubHOk4gqU97xfMf8KAIABiydSAAAAABARCykAAAAAiIiFFAAAAABEREaqh0vXfkCS5cWzN5NuWmns+/LRZ6vXrX7kTbWe/76euSlbaM4i3V8/Xe11Vm9U64s363OkhjatMdbGVukZqK2WLNHIZItabyxUG2vZruLzvnYILa8tqd7ndUJLniZIWjJQtoyU1mrLSFm+bCvLnCqNbcaVvb/vPtfp64wVAADY//BECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9bBywRRJJEuK1qoK7xj7Gh/QZzHVdbyo1h/sGKH3L2wz1l6f1qD2Ttqmv/bWDQeq9ZrubvO1KzrU3rWFSrVel2hR60+1TzHWgi7Lj64lx+Sm9DCRHyoZKkv+KrTcmhMjI5W35Ihsc6SslC/bljMKLPdmm0Olsb22LTsGAACwp/FECgAAAAAiYiEFAAAAABGxta+HivtekIRT/FjtFd+ZZew78H/Wqtft/PA0tX7TigPU+pDXVxhrFcsOV3udhP4tLlmvHyMujnnL1EGlG9TWt3N1av2YUvOR8iIi67qGmG+ry1N7bRKWrX0FMddDy/a5wPKWKpe2sm5h6/2p7u+JcYR5EPP484DPdQAAwCDCv1wAAAAAIKJ+XUg99dRTctppp8nIkSPFcRx54IEHdtby+bx84xvfkKlTp0p5ebmMHDlSzjvvPNmwYdenIGPHjhXHcXb5dd111+3lrwQAAADA/qRfF1IdHR1y2GGHyYIFC95X6+zslMWLF8u///u/y+LFi+W+++6TZcuWyemnn/6+P3vttdfKxo0bd/668sor98btAwAAANhP9WtGavbs2TJ79uyitaqqKnn00Ud3+b2bb75Zjj76aFmzZo0ccMA/ckWZTEbq6vQ8zu7wjz9MnETx489/8tn/Y+z7wTcPUa/77rx6tV7y3HC1XtVlzkjVvFVQe71hQ9V6+TrLMeFlZcba2OQWtfeBliPU+pzyt9X6xi7z8emJrnh5nFRKf9987fh07Wh0EQks8a04x5/bjhiPffy50m47vjyw5Ldcp/cBLjfG0ekiIo7l+HS9N9ZLAwCAfdSgykht375dHMeR6urqXX7/uuuuk9raWpk+fbrccMMNUijo/0jOZrPS2tq6yy8AAAAA2F2D5tS+7u5u+cY3viHnnHOOVFb+40nFVVddJTNmzJCamhp59tlnZd68ebJx40a58cYbjdeaP3++XHPNNXvjtgEAAADsgwbFQiqfz8tnPvMZCcNQbrnlll1qc+fO3fm/p02bJqlUSi699FKZP3++pNPpotebN2/eLn2tra0yevTovrl5AAAAAPucAb+Q2rGIWr16tfz1r3/d5WlUMTNnzpRCoSDvvvuuTJ48ueifSafTRRdZHV9tE688V7TnqPR242u6hx+s3tM3P/SgWr/zyo+p9cQY8yKv/O1mtTc3Qc+OZdbr2yDdGvMsp9GJTrV3WdsI/bWH62GiLR0VxprXaQmuWIItpam8Ws/HGPZkmyPl9OEcKcfvu4yUjR9zjpRvyX8NVnHyWf0pHKT3DQDA3jKgF1I7FlHvvPOO/O1vf5Pa2lprz5IlS8R1XRk+XD/AAQAAAAB6q18XUu3t7bJ8+fKd/71q1SpZsmSJ1NTUSH19vXzqU5+SxYsXy0MPPSS+70tjY6OIiNTU1EgqlZKFCxfKokWL5MQTT5RMJiMLFy6Uq6++Wj7/+c/LkCHmJykAAAAAEEe/LqRefPFFOfHEE3f+947c0vnnny/f/va35cEH39sSd/jhh+/S97e//U1OOOEESafTcvfdd8u3v/1tyWazMm7cOLn66qt3yT8BAAAAwJ7WrwupE044QUJlXo9WExGZMWOGPPfcc3vsfv449XdSmSme0zjsmUvNjWeaszwiIhdVNar1e5e8q9ZbPjzRWKt8+FW1d9vx09T6sOe3qXW/zvxkr8bVf3xWb9OfCpY5KbW+va3UWEt3q60ijp63KUvqGamsNivK8nMZJPW6U+h9ligf6u95jFFN/6/ffG+2WU62GVe2OVRxxM1nAQAARLVvprsBAAAAoA+xkAIAAACAiFhIAQAAAEBEA/r4873t9x3DpdQp/paMvd6c75jy3y+o130+q+dx/K1Nar1xljkjVfGbDrW3pfgorZ1GPKRnpFpnjTXWSi0Zp9ZtZWo96ehzpPLt5uuXdamt4iT1H+2yZPF5YTvkLDkojSXGFCvHZJ0jFeO+RSTWHKm4tK/NjTnTKG4/AADA/8YTKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1cMPfnWWeOmSorWGxYuMfd8Z8Yx63WNfvFCtNwzdrNaPOOodY601nVZ7Kyc3q3V/W4tab28w55g8y6wmb6ueobJx28yvnei0zGpK6D/aFcmsWs/GiNQElr9Vnm0GlsK3zGpygphZIGWOlGfJGQWWWU5u3CFX/cTpz3wV47EAABiweCIFAAAAABGxkAIAAACAiNja18Oo/35VEoYjvbeef7Sxry14Ur1uyX3Var17Rrla/7eRtxlr3x35SbX31NFvqvWXsvpauqPBvK0pH/pqb7o53r6kZJv53pJd+jYxJ6VvK8wk9K19HbYzzBVhQt8K5vq9//wiL/qR8eLH24YWZxdbYNmH5on+PQuUbYt9vS3Q789z3/dVbEsEAOzjeCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6cIfViusWP078w1ctNPZdufoM9bq1f3xbra+6fLJaP0Y54bxz0jC19/Sq+9T6S86Rat0fZT6ruz3Qc0YlTXruJBvm1XqyzVxLdFsyLamkWs5oFxeR7tCSRVKEScvR7Hq0TMQxh0v6/vhzc8mz5Ihsx5/3JVs+i7gOAADY03giBQAAAAARsZACAAAAgIhYSAEAAABARGSkelh25QhxS0uK1h4e8Xtj39S7vqJed2TzIrVefcwmtd4amHNK2ybr85KmpfRAjltRodbH1jcZa1sseZzSJn32T2egZ6RSrebrJzrjzZGq8PR8V2egBNMsbHOknEKvLy1523yrmBGpOOOabBkp1zKkylc+1/HiDLjqY35ouTcCWgAA7JN4IgUAAAAAEbGQAgAAAICIWEgBAAAAQERkpHr47SkLpCJTfG153upTjX2jH2hUr5v74GFq/eoJ96r1u9rMc6a2T9EDN2WunhVyh9ao9SNrVxtrawuVam9Jk56Bagv1QE6qzZw98br17FdYon/dVYlOtd4R6v2qhCW/ZcshOebPN3zbvCQ/RshJRETJOdk+denPOVJ9yRnA+SwAANB/eCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6SDmBpAwxjxU3HmTsq3hHnxO18sqZav3jZVvU+tHPf9xYGztRn0G12e9Q6/m6arU+q+JxY+2dbJ3am2zWc0jNflKtp9rMeR+vU89fhWk945RxzbO5RETa/FJz0bHklJKWjJQe7xLHNV/fNkcqbpwn3hwp/XMZT/SL+2o+K94XZpthBQAAEBVPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVA+nP3a5uKUlRWuTfmvOQXkHT1Kve9VJf1brScdT685zVcbaJ85/Ru19tnuEWu9sKP717nBwypzBeqJ1itrrbtfzWZv9CrWeajPnoNxuS0aqVM9flblZtd4RpM1FZc6TiIhryUi5loyURssRiYhIEHeOlLnk2V7aMuOqL9nmaw1kQcz8FwAA6B88kQIAAACAiFhIAQAAAEBELKQAAAAAICIyUj1M/lGTJLzi2ZjcCTOMfWuP1XNGV1avVOuPdJWr9bpnzfOYTvnKG2rv/I2nqvW2UXo+a5Rnzhotbxum9oat7Wq9sWDOfomIeO05Y82xZKQKVcocKBGp9PQ5Uu/mhppfW5nzJCLiefHmSGkZLOscKT9e3ibOuKXAkt/yLEOqAj7XAQAAgwj/cgEAAACAiFhIAQAAAEBEbO3rIWzcLKGTKlpr/7F5C9ysmrfV676V14/avvHdM9R6eql5a+CBSeWYbhF5duUEtS4N+l6uMrf4+yEisqalWu0d2b5CrW/MD1Hrbpt5+52TNW/7ExEplOlbFssc/XuyvVBmLlqOP08m9b171q19ytZB3/bZRxjzKG2l3fapi21rXxxunD2Hu9GvbTp0Bu/J6vH2agIAABVPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVA8bLpomXrr4UeYvHPYjY19noB/FPee1L6j1ppeHq/VxrWuMNVf0AEfqLf0Y8PxB5qPVRUTyoTnQ09akH9se5vUc09ruGrXudCpHlBcKaq9fqn9GUO7qGal2X8meWY4/T3q2jJSeW3GUUE4+1LNfEvf4c/2EcpUtI+WqSST70e6DVayMlaXXD2N8wwAAQCw8kQIAAACAiFhIAQAAAEBELKQAAAAAIKJ9M5TQS18+/0EprSj+ltzTVm/sO63cnGESEcnfp2eg6jfqmZpEw0hj7d3C39XeIcv0DEXVhzer9U1+l/m+mpJqr82Grkq1HnaZX1sCPQuUt2WkHD3X1loonpUTEXE8PbiSTur5rYJ1jpT53m0ZKacP50hZ0lkS9ukcKf3nuC9nWAEAABTDEykAAAAAiIiFFAAAAABExEIKAAAAACIiI9XDOZmNUpkpvrac/uMrjX1PfnKpet0Rf1gZ677aZo4x1h7vnKT2Vi7brtZn1ryr1pflq4y1dJMll+LqqZpNnRm1Xtm9Sb++olBqyTE5elCpo6DMkbLkq1KWOVK+LSOlDB4KQstnH0G8uUJO0PusUWAZeuQ5en6r2/a1xeBYXhsAACAqnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakeTnvjE5IoL56NGX3zK8a+hYnD1OuO3rRQf2HL7J/GWeONtQcaD1d7ndUb1PrM8hVq/cVO82uXNOn37ZaaZzGJiGxrL1PrmWzWXPT0/JUtI1Xu6lmi9rw5I+V4em86oc+R6rZkpBzla7PNkbLN17KJEyXqy1lOtnwVAADA3sYTKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1UPyR0MkkSie63GHm3MxY+/fql63cMw0tZ548121Pv7oNcbam8sb1N5JLS+o9cnJJrV+6/oTjLXSZj0r5FSUq/WuNmVWk4iEysAlWxqnUKrXk5b+jkLK3Ovl1N60p2ekXN+S93HNX509I6V/T/zQMmdKKXvKfCsRe0bK0y6+G/39hRlUAACgGJ5IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpHhJPLJGEUzw988ZPjzb2TfrK8+p1V94wS62Pe8g8q0lE5F8O+JWx9k9/vFjtdRL6t3hkQs8pvdk4wlg7oEnPCkllhVp22iw/fsp8LS0/JSLiWzJSJY7+GUJ7zpyRGuLpvWUJ/X1xLHOkRLk3P7R89mGZSWYTJw4U9mPGybek5lxyTgAAYA/jiRQAAAAARMRCCgAAAAAiYmtfD52nHymJZPHjz/8454fGvn+a+Hn1ul/46JNq/bdNJ6j1E0u7jbWat/Sjtr0Rw9V62rCVcYfC+jJjLdHUovb6Q/Tjz5NtMdbxli1shVK9nrRs7evOKe+Lpx9BXuKZv18iIq5ta5+yddB2/LkTxNzCppxQbvtu2Y4vdx3L0ezKK7gyiLfmDdZthQPzNHoAAAYMnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGake6q5YKcny4sdep5R8x9oz6tTr/n7ob9X6fcccptazYd5YK3+7Se3NjTcfX267tohI+Xolt7K9Xe3tGl2v1pNtlhCGq+SBAtvx57aMlJ410jJStiPlSz39PXV8S2ZGyW/ZMlIS6DkkmzhxHltGyt7ff5/r+DGPjQcAAPsfnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakefjH2SanMFF9bjntwrrHviNPfVq+7ptCl1i+frM+Z+n1Hg7EWrtuo9m77kD5HakMhq9Yr1pszN2Fbm9rbXTtKrSf1dnGS5h/PMKtnpIJSPSvkWj5DyOeVLJKr99oyUq4lI+Uoc6QKgeWzD0vWJ7DNY4oRFbK1ejEubp9BZZth1fvXdgbrHCgAANCneCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6+O7WKZLuLj4/6KDvbzH2/eSvD6jXPe+dc9T67yb/Rq1/5JXzjLUhne+ovS2T9XzHG/mhar18XbexFnTo2a+uWn2dXtKs35ubThtrfi6n9jqlBf3alkyNn1UyUgl9llPa1V/b0csijvne7HOk4uV5tDiQZ3nP4s6RsuWc9kvkswAAGLB4IgUAAAAAEbGQAgAAAICIWEgBAAAAQERkpHr4823HipcqKVobtuEVY1/G1d/GrfePVusV88xZIBGR9oXDjLXakrVqb9XkZrX+fMcEtZ5s3G6s+Xk9p5St1fMdlWv02UBSWvx7ISLidHaqralSfZaT51g+Q8iZ66Ey50lEpMy15LdsOSbPnIMKwnhzpGws45osL61nnLwYF48zB6qvBRLjTQMAAIMWT6QAAAAAICIWUgAAAAAQEQspAAAAAIiIjFQPtb9aLAmn+BypdV872tj3zU16Jmbk79eo9Wfm6uvZ+oVZY80dWaf2zh79plpf1DRWrbtb9YyVJlerZ0dSrZZ5S2Wl5mKLPk+prET/ntg4SkZKyzCJiJR55u+XiIjj2zJS5tfOx8xIWfM8StlV5luJxJ8jZc1/xeDKwM1Y9SlGcwEA0Gf69YnUU089JaeddpqMHDlSHMeRBx54YJf6BRdcII7j7PLr1FNP3eXPNDc3y+c+9zmprKyU6upqueiii6S9vX0vfhUAAAAA9jf9upDq6OiQww47TBYsWGD8M6eeeqps3Lhx569f//rXu9Q/97nPyeuvvy6PPvqoPPTQQ/LUU0/JJZdc0te3DgAAAGA/1q9b+2bPni2zZ89W/0w6nZa6uuLb195880155JFH5IUXXpAjjzxSRER+8pOfyJw5c+T73/++jBw5MtL9OAcfKI5X/Cjyr110n7Hvht9+Ur3umLUL1fpN6z6i1tNLVhlrnUfpx5d/vPJetX73G0eo9Qmtjeaiq29xS9R06/U2fZtZUG7e2uekUmpvZYm+vc7GzSp7ohL6113i6FsW3YJlm5lr/nyjEOivLb6v1y3inDJuO/7c7cNjwvtyW2Dc3XHsrusF3jQAwCAw4A+beOKJJ2T48OEyefJk+fKXvyxNTU07awsXLpTq6uqdiygRkZNPPllc15VFixYZr5nNZqW1tXWXXwAAAACwuwb0QurUU0+VX/7yl/L444/L9773PXnyySdl9uzZ4v+/T90bGxtl+PDhu/QkEgmpqamRxkbzk5T58+dLVVXVzl+jR+sDcwEAAACgpwF9at9nP/vZnf976tSpMm3aNJkwYYI88cQTctJJJ/X6uvPmzZO5c+fu/O/W1lYWUwAAAAB224BeSP1v48ePl6FDh8ry5cvlpJNOkrq6Otm8efMuf6ZQKEhzc7MxVyXyXu4qnX5/FmrlV5PilhXP3pxXud54vd/dt1297/CIQ9T60uczan1C03PGWvPkKWrvtJSemXHWKUeMi4gE5n63vFxtravRt0x67cWPmt+hUGW+t0RK761Od6l1GzdnDmmEScvx567l+PNADyKFMY4/Dy3Hn9s4MWJMli/LyleCMV4f5qsAAAB6Y0Bv7fvf1q1bJ01NTVJfXy8iIrNmzZKWlhZ56aWXdv6Zv/71rxIEgcycObO/bhMAAADAPq5fn0i1t7fL8uXLd/73qlWrZMmSJVJTUyM1NTVyzTXXyFlnnSV1dXWyYsUK+frXvy4HHnignHLKKSIictBBB8mpp54qF198sdx6662Sz+fliiuukM9+9rORT+wDAAAAgN3Vr0+kXnzxRZk+fbpMnz5dRETmzp0r06dPl29961vieZ4sXbpUTj/9dJk0aZJcdNFFcsQRR8jTTz+9y7a8O++8U6ZMmSInnXSSzJkzR4477ji57bbb+utLAgAAALAf6NcnUieccIKa6fjzn/9svUZNTY3cdddde+R+HvvA/0gmU3xteeKrXzD2lS95Q73uu/81S63XPafnP7xhw4y11in6zKIyV5+3VLZeH9jiJM39bqZC7Z1QtVWtb+4cotYLoyqNtWRSz0jVpjvUuo2nZaQS+ucPJW5erTt9OUfKElTybRkqpexZhvvY5kh5cQJYMTmWAVnxpm8BAID90aDKSAEAAADAQMBCCgAAAAAiYiEFAAAAABENqjlSfe2tfLmU54uvLdM3mPM8iTH6MN+Pn7pIf93bGtR65xFjjbXxExvV3s2+nhXKrNfTIW51lbEWVunzr6aUr1TrmzpK1HouY84DlZa8fw5YTzWpzWrdD/W8jjYKypqRciwZKdvAJWWOVMEyR0osX5dVjFlQtoyUTaB8ba414xTvtQEAAKLiiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkerhyvsvErekeG5n3OMLjX3rr/qAet3fjrhXrX9y9dFqvfFCcwbrK/WvqL1Pd9Wr9bL13WpdhpmzYYWqUrX1wPQmtf5Ed51az1WY1/lhSp8jNTTZrtYLlslB2iiowDpHKqfWHd+SY1LnSNkyUvrXFYj+2nFGPQW2OVKWAJZvy3/1E1s+CwAA7J8G5r9cAAAAAGAAYyEFAAAAABGxkAIAAACAiMhI9TB+wSpJuKmitbYzZxr76s5YrV63OdAzM255uVqvPcY8K+rk8jfV3v/aMEetJzduU+tdBw4z1oK0vg4fm9yq1sOc/r7kKs2Zm7C0+PdphyEJfX5W3pIl8vpyjpSvZ25Cz/x1F0LzbC0REQkKet0iThyoL5NE+2tOyWE8FgAAAxZPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVE9BKKakR8M/vWNsmz/qQfWyV6w6S3/ZqSPU+tXjHzDWDkym1d6FK8ep9UlNy9V6+wkNxppt5tBIT89A2TJS+QpzLSjV50jVevocqXyo37yXM2dygoQeXLFlpCToyzlSMQZBiYg2Zsq1fO4SWuZIuZYZVoH0XSCoPzNWzn6a7+rDbycAAAMCT6QAAAAAICIWUgAAAAAQEVv7elj1lfHilpQUrS0bd4uxLxvq2+tWPjhBrTvH6Pc1u8x8jLhr+Ram3ypV60G7vgWuvcG8PyfZqbZKjae/LxLqW57ylea6X6J/3dWe5fhzy2Hd2vHnQVL//CHp6EerOwV9i1uYNB9xHli29nmB/nX5lq/bsXxPNLatfX3J5zMh7CH9+GMMABhk+NcHAAAAAETEQgoAAAAAImIhBQAAAAARkZHq4Qdn/VzKM8XzKf+x5ZBeX3f0/RvU+vL5lWrdc8yb9t8t6EGlIcv0vI4tp9Q1qmCs+U3mLI+ISNrRjyi3KWTMWaJCqf7a1W6XWu+2fN3a8eeh9fhzy3vu668dps3XL4T6Zx9ujIyTiP1Ie03MlxZf+do8y9HpAAAAextPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVA+Hp9olkyq+tvzGzR809m0/UL/u+JUL1fqXDmlT63/urDLWNuSHqL2Vy7ar9TCtz3qqHtlqrLWInu2yUrJfIiKSyRtLhTI9f5Vxzb0iIt2WYTFaRipI6r1JS57HCSxzpDzz5xu+ZY6UTWALMillLasnYp8j5cUJYMXkOvrXrd2ZY+kFAAD7J55IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpHk5a9EVxy0qK1sb9/AVjX+2Mg9TruodOUevnV92u1s9+61xjLe2Z5zyJiLhrNur1IdVq/fAR6421Z7tTam821HNKTkLPOZVlssZaoUR/7TJLrqU50H/0vay5v1Cqf/6QsmWBfFtGSpkjZctIWfJXNvHmSFkybxaBmPutGSfLfC0AAIA9jX99AAAAAEBELKQAAAAAICIWUgAAAAAQERmpHsb8KJCE5xetOYdOMvaFL7ymXnf1N2ep9eFeuVrfsGiksZYbrmekJrWYs10iIu60erU+s9L8ta0aUqv2bg9yat1J6RmpmvJOY62zVJ9hVe7onxGsC/TXdvPKHKmM2iqWMVP2HJOSkfJtOSTLnChfGxQlos6RsrGNqPIsF/fJOQEAgEGEf7kAAAAAQEQspAAAAAAgIhZSAAAAABARGakewlfeltApnp1Zeeehxr6Jc0eo1z3246+o9ZeyepaofqE5B9U8Rc/6OAn9W9zVoAd+ZpS+a6y9VDVW7W30PbXulBaf2bVDXXmrsfZ2qZ7tSjv6190d6u+bly2elRMRCSzvqX5lEccyRyrwzJ9v+PvwHKm+ZJtDFYc1d9aHr20V51sycL+dAAAMCDyRAgAAAICIWEgBAAAAQEQspAAAAAAgIjJSPTR//gjxUsVzO88e931j3ymn/7N63bsablTrV649Ra2Xv7RGqR6g9np1en6rvUH/ERifMOe3JpU3qr3v5mvUulNaqtZHlpqv/0aZ2ipJR89ntQZ6PsvNmcNCviUElXQs4RJLjkkbp1Tw4332YYtAaXEe1/K5i22OlGsJYAVKxirpmjNrIiJ+HwZ6nP7MOKF3+J4BAPYCnkgBAAAAQEQspAAAAAAgIrb29XDypQslXVF839ZrOfMx4aWf3KReNx/qW5r+/tJBan1i4yJjreztcrU3N2G4Wu9o0LdEDXHN2+8mpvWve3m2Tq2HFfrWvob0NmPN13fmiWvZ6tUZpPX+nHkrWWj5W5N0LJ9PWI4/DxO9P/48tOyv82377+LsiBrAx58PWmxR27/wVwgABhWeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR7+fdgbUpkpvrac9IsvG/t+f84P1OvOXTdbrQ9fqG+M96qrjLVwvX4EefOJ+vHn3Q15/bWVvM/YRJPa+6dtU9V6UKEHnUYmW4y1QqmeHdHuW0SkNdDzWU6uYKwFScv3yxZ0sOSUQuXkdu2I8P/3B/S6heWEclVouTfPcvi6r3yukxbz92N3uLHCX30rsB5KDwAABiKeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR4uWn28JMtTRWsH3rjc2DfpPD3r89KDh6r1MYs2qvXc4ROMNe+JxWrv9sl6NmRYQ4veH3QZayMTerZjeeswte4YZnbt0JBQ5kiVxcuVtFsGUTl58xypwPK3xrV9PuGbry0iErrmrFEQxBs0o7+yiGObM6WIm0Ky5r/6kD9wI1QAAGCA4okUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqYe2CAyWRLJ6dqZSVxr4bt01UrzvmgS1q3V+5Rq1v/HS9sXbAc3rWZ8jkZrU+fdg6tb4qb15rH5TSf3zWN5vnX4mIDK/U+2u8TmMtLI2XkWqzZKQkZ56vFejRLvEcPesTBvq9BwktI2X57CO0XFvvjjVHSvo149R3nwkpkTUAALAf44kUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqofyBFyXhFA/ALP/uLGPfssfMc55ERA588zm17qTTaj11jDnn5Iwy56dEROaMfl2tj0vr+a0l2dHG2uHpzWpvtrlUrecq9HV8jVsw1txSc01EJB/qE5Paff09d/Lm68eeIxXoQ4tCT2mNOUfKSrk1VyzZL8ssJs8yaUqbI+XFCm+JuDH747BE5gAAwCDFEykAAAAAiIiFFAAAAABExEIKAAAAACIiI9WD/8HDxEkUny9029n/bez7rwvOV6/rTdIzVGGZnte5YtJfjbX/O/E0tfdjlb9R6ynLZKFbtpxgrF1QqWekUk1K2EdEchm1LBnX3F9SmlN7bRmp1oJljlTB3B9aM1KWUEyg31vgmftDS0YqtOSvfEuOybHU9ReP0QsAADDI8EQKAAAAACJiIQUAAAAAEbG1r4eur22XRHl30dphqXZjn/vMUvW6a/5lplpPtep7oj6VWWWs/Xhy8ePadzg0qV87b9nat3iz+fjzbMNTam+6Sd+Glq9Uy1LmpIy18hLL1j7Rt8+15fWtfWEub6zZjj/3nL48/jzeZx+2Q8DjnBJu23ZoO4LcVz7XcW1Hp/OZEAAA2Mv41wcAAAAARMRCCgAAAAAiYiEFAAAAABGRkerhj4c8IJWZ4mvLQ56+zNh3YO1G9bqHnv6WWn9+xVi1XuGYj0dvnVJQe8tcc85od2zdUGWsNU/Lqr0lTXquZfuB+msnHXNYqKqkeJZth07LEePtBf3IeSmYv7bAkjuz8uMcf265tvUP2PpjfG2h5dh3iyBmv8aNda47AADA+/FECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9fCb9hFSanhLxn/PnGvZ+jE97HPHAd9X6z8uP0atP91t/jYdOFHPZ20smOdfiYgM98rUesl685yqTb4+w6q0Sc8CNU3v/Tp+SLpTrXdaIjEdeT07lvbN1w9j/q0JLTkkbY6UbVaTjS96f6w5UoM4hqR92Q75KgAAUARPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVA8/+b9niJcuKVob+coiY1/hGj0jVabMQxIRuaRmoVq/YtWnjLUz6peovU93N6j1D5duUOvl68z5kHcLtWpvulmfMxVk9IyVZkhKz0i1Bfq1O/N6PVUwz+cKEjEzM4EeRAr6MiNlmdUUKw5kubYn+sWD0Py5jmcJb9myX4PWPvplAQCwL+CJFAAAAABExEIKAAAAACJiIQUAAAAAEZGR6qHhf16VhFN8vtCWLx5t7PufqT9Sr/uvGz+k1m9uMOevRETeeH6csfb9z9yr9n573cfV+tiRW9V6Zr05K/ROdoTam2jqUOvpTKlaz4Z5Y602qV+7NUyr9c6sPkeqyjfPwAqTMTNSMeZIiS0jFXeYk9LuOZbPXQbwuCW3H2dB9escKmZgAQDQZ3giBQAAAAAR9etC6qmnnpLTTjtNRo4cKY7jyAMPPLBL3XGcor9uuOGGnX9m7Nix76tfd911e/krAQAAALA/6deFVEdHhxx22GGyYMGCovWNGzfu8utnP/uZOI4jZ5111i5/7tprr93lz1155ZV74/YBAAAA7Kf6NSM1e/ZsmT17trFeV1e3y3///ve/lxNPPFHGjx+/y+9nMpn3/VlNNpuVbPYfM45aW1tFRMQdPlRct3i25tQr/m683hFpPW/z+YePUOtvXvA3tV6/0DxDZ8K5es5o0cqxan3xEL1esqHNWHurvV7tdVrb1Xp1hVqWzsCckRqa1K/d4pep9Vze8qPflxkpyxwpNSPlxxssFFgGEzlBjK/N0mqdI6Xcm2uZI9Wfgri5NAAAMCgNmozUpk2b5OGHH5aLLrrofbXrrrtOamtrZfr06XLDDTdIQRmmKiIyf/58qaqq2vlr9OjRfXXbAAAAAPZBg+bUvl/84heSyWTkzDPP3OX3r7rqKpkxY4bU1NTIs88+K/PmzZONGzfKjTfeaLzWvHnzZO7cuTv/u7W1lcUUAAAAgN02aBZSP/vZz+Rzn/uclJSU7PL7PRdE06ZNk1QqJZdeeqnMnz9f0uni2/TS6bSxBgAAAAA2g2Ih9fTTT8uyZcvknnvusf7ZmTNnSqFQkHfffVcmT54c6XXe+uowcUtLitYeHv6Ase+27SPV6469f5tav+Xj+pypyhfWq3VN+i09Q/XsgRPUurvZfO8rWoeqvaXtTWp9eLlalrbQnIupSegZqWZfD2Dlc/qPfqhlhbx4mZjQOkdKyTHZ5khZ+NaMVIyLhzHvLUa/H/bdLuX+nEEFAAAGrkGRkbr99tvliCOOkMMOO8z6Z5csWSKu68rw4cP3wp0BAAAA2B/16xOp9vZ2Wb58+c7/XrVqlSxZskRqamrkgAMOEJH38ku//e1v5Qc/+MH7+hcuXCiLFi2SE088UTKZjCxcuFCuvvpq+fznPy9DhgzZa18HAAAAgP1Lvy6kXnzxRTnxxBN3/veOvNP5558vd9xxh4iI3H333RKGoZxzzjnv60+n03L33XfLt7/9bclmszJu3Di5+uqrd8lNRXH/RxZIRab4Q7pzV33c2Ldw6UT1upNeeV6t/+Hlo/T+tS8YaysKXWpvzTLzMd4iIq8c1aDW67etNNY2bhuh9o7rWqdfu1Q/XbHZTxprNZ6+tW95Vj8O389ZHsYq2wqdpL7/zVd6RUTEcsR4oB1/Hvfkddv2uTjXZwccAADYj/TrQuqEE06w5kUuueQSueSSS4rWZsyYIc8991xf3BoAAAAAGA2KjBQAAAAADCQspAAAAAAgokFx/PlAsObGScbauO161icxboxaH7pQ/za45eZzwv/aYb4vEZHM29vV+vqNlWq9Lps11nLNxY+K3yEs6O/L6BI957RZOcK82utUe7fm9ePPJasFkURE2XLqJPQMVGALC1kyVKFya05fH39u2WqrsrTajhEPlCPMvZgBLNtr294XAACA/40nUgAAAAAQEQspAAAAAIiIhRQAAAAARERGqodPPnq5uKXFcz+T7l1k7HOSKfW66y4/Uq03/GWrWi9MO9BYu3+jPnjYW7NRrZetrVHr4pizI6mtlpyRRX2yRa03FqqMtWnp9WpvU96cKxMRcXK9z8R4CX02l01omSOlZaQk3kvbs0CWEVgq24wqC+uMK62Xz4QAAMBexr8+AAAAACAiFlIAAAAAEBELKQAAAACIiIxUD1N+tEUSbrporfvDRxj70i+vVK87+oxVar1w8xa13njV0cZa23I9nzWpZZ1aL1+v53XcCvM8ppImy0wiS3asIblNrS/uHGusHVv6rtq7LVem1t0YGamEZY5UPoyZoerDOVI2lnFLscSZBeU6ccJb8Tgx3xQlZggAAAYxnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakegs1bJXCK53q6f2pec3b+crJ63XvGf1+tXyInqPX8rDZjLbMwo/Y6Cf1bXLE+p9bdoeY5UyVNlnyVYSbXDsM889clIrIhW22slVlyJ9uyfZeRSlrmSAW2YUyhXg+1jzdiRoUC9eIiEsbIA8XMVwW2GVcxuHFvrp/EzWcBAIC+wxMpAAAAAIiIhRQAAAAARMRCCgAAAAAiIiPVw/qLp4qXLp7rWTz1J8a+w848X73uCK/4bKod3Enj1fqlBz9trD3w84+ovV7dCLVeskHPKeXrqo210qaC2utU6vmtGlfPZ23oqjTWylxl2JKIbO/W81le1pLHUYb/pCwZKT9mHidQ/lbGHafkW3JIjt9/GSk/7H1Gyrdlv7DXheS7AAD7OP71AQAAAAARsZACAAAAgIhYSAEAAABARGSkerjyCw9IaUXxt+TOtnpj360zfqVe95rNR6j1LcfUqvXPV75urP3lnRlqb/ZAPSOVem21Wu88+EBjrXxdl9obVlWo9WpXX8dv6jRnrJKiZ6Q6uovPA9vBEs8Sccz3lk7o2bC8ZU6UbVZTqHxpThBv1pJv+ewkTqzFsWSc4sxy8gbpHCgAALDv4okUAAAAAETEQgoAAAAAImJrXw+fqdgklZnia8sZN11p7Hv16p+q1/3Sb45V6/lZ+laxoV65sRaub1R7t31Y39o37Bn9+PO2UeZ9ZpVLO9Te/Ajz8eUiImVuUq1vay8z1pKOvrWvu0vf2ldu2drnuOZtaratfTFPKJfQM29ji3v8eWA7JjzODrqYu++s9xaDa9mzGOfodQAAsH/iiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkephzutnSKI8XbQ2asESY9/dXxqiXnf8ve1qfdItb6n157N5Yy3o7FR7Wybr2ZCheT0s1NGg5HVa9a8re7B+rHvCcoR5V1vx74WIPSMVdOk/2tbjzz3z9dOenpHKWY43t9GOP48bwPJFzwI5ce69H08ot31dcdjyVQAAYP/EEykAAAAAiIiFFAAAAABExEIKAAAAACIiI9VD6U1VkkiUFK25deZwyry/H6xed9ILL6r1Lw99Vq3/14Y5xppb0q321k5uUuvi6lkjp6HLWAva9IxUZ61+bc/R1/FOW+9/PJ0u/dpeVs+9OEpGqsQzZ9ZERPIxIzXaOCVXf2kr3zKryQn6LiPlWbJGgTLLyY07QKsfOWSsAADYJ/FECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9eA99Yp4TrJo7Y1bjzb2jblPz0AkRjWo9YNSZWr92ZcmG2tTRm1Wez826nW1vqhymFqfWG++vt+l57OytfFm+yTber/O9zotGakYc6TKEnpQqduSQ7IJE8rsrmy89zSwfXYSI4rkBHHvrS9nQfVdxsrvzwFaNn33lgIAsN/jiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkeqh44wjJZEsPkfqz7NvNPZ99WsnqdfdeOF0tb6moM9jGvGsOejQOXGo2vvxynvU+qKhZ6v1mTXLjLVng5Ta212jZ0fyoa/Wk229D3gkuvVeL6dnZpyE+a9GqSVglY/5+USojN9yfMt74uh13xaaCftwjlSMLJFnCW8FMXNpGGDIdgEABgH+9QEAAAAAEbGQAgAAAICIWEgBAAAAQERkpHoYfflySZbruZ+iksVnT+1Q+cmNav225llqvWZRo7G27hMj1d5DU3rYIF9frdZnlq8w1p6Vg9TeQm1BrWdDfR5TstVcs+WrvC61LF7OktdRM1J6pi0fd46Up8yRijkOybfcmxNnJFLMcUpBODCDMQPzrgAAQH/jiRQAAAAARMRCCgAAAAAiYmtfD7ePeVoqM8XXluN+f7Wxr+Gj+uafn002H50uIjL7qSvU+oGrlhhrrZOHq71pR9922NGQVuuTk03GmpPUt0GW13aq9bZA3/qXajPvFbNt7UvYtvZl9X1oTsJ8Bnna1e+7O4z310rb2mc5BdwqsH12EuP481jbAqV/t/YFA3UD3wC9LQAAwBMpAAAAAIiMhRQAAAAARMRCCgAAAAAiIiPVw39uOVjSXcUzRQffsNnYt+w71ep1xyVK1HrVs3rdTZtzTJMmblB7Nxb0o7rbRulr6ZEJ82u75aVq76jqFrXeHJhzSCIiqXZzICgb6jmlRKce2HFzlrBRypwtK/Nyamt3qOfSrJRvifX4c0f/ftqOP491hLmlN86nNq4lgOVbru7FDXD1IT9GLg0AAPQfnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakeHvvvWeKliueVhm1aauy7+eg/qde9sXmKWh/+XKtaDw+aYKydXv+U2vt0d4Na72zQQzfaHConk1F7J1VuUutNgZ6xSrWac1AdoX7fiS49d+JZMlJh0vxXo8zt44xUwnxvcaM+vmUwkePHGFQVcw5Uf86R6kvOAM5nAQCA3uOJFAAAAABExEIKAAAAACJiIQUAAAAAEZGR6qHmVy9JwpAJWvfPM419p5Y9o1738gdPUusT33hZrW/60hHG2kfK31J7v7XuNLWebOhQ69kwb6wFVRVq7+SyV9R6Y6FKrSfazFmkTkueJtEdc46UkpFKu+b3RESkO0jp13YsWSDPfO/WOVIWgWWOVJw4T9wokO3e+otthhUAANg/Dcx/uQAAAADAAMZCCgAAAAAiYiEFAAAAABGRkerBOWySOF66aO1fvvgbY9/DncVnT+0w4d5utR7mzfOSRERaZ3WZr53QZzE9v2KsWp82Zr1a31DIGmuFWv21J6Ya1fobWX3Glddufu22QJ/VlOjUw0RuzlfrYdIz1vp6jpSjZaT02xbH1fNXvu2zkzBGHsjS6sUYE+WJ/v30LZk513ZzAAAAEfFECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9bD6q664ZcWzMWdnNhr7DnniEvW6ExYuVeve+APU+nlTF6l1Telben5r1vSVav2N/FBjrbtWn5c0OrFdrT/aeqhadzrN2bKWQM9nJbr1MJGT03NpYcr8V6PEsWSkLPktcfTPL9w+nCNlyxL1ZUbKJpAYIaqY/H587TgCsl8AAPQbnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakeHp/5M8lkiq8tT1j6BWNf/b16VsgbVqvWtx09Qq1fNOT/GmsrCnq2Y8gyPSt0ZJmekXqqfYqx1lWjr8PrzKOYRERkfVe1Wg87zPOzWvxytdfr0jNQtoyUX27+npa4ebW3raDnt2xcVwlC2TJSlvxVYPvsJEYGK27KKFDyW54lHGb9ugAAAPYw/vUBAAAAABGxkAIAAACAiNja18OSXIWU54rvRyu/vsrYl3z+NfW6W86eptabZujblkYlKoy1BS2j1d7MMv0I8slJvX5D01hjrXuovpmrwk2r9Y2dlWq9pLvZWGvyze+JiIjbpW+/k7y+tS9Imfclljj6tTsDfaun4+rvm+uZfx6sx59bru2H+mcnTj8ef96XXGcA3xwAABiUeCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6+Kd7LxS3pKRobezfFpobDT07ndmklo+rbVTrb+Y6jbUHNh6u9ibXbFDrIzz9qO7ljcOMNadGz50kHf3886aOMrU+Mmd+X7YV9OPPnW49x+RYMlJ+2vwZgzUj5evZMNsR5YlEjIyUhW/77KQPM1K2T22048/7ky1fZT2RfmB+WXbkygAAUPFECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9TDupysl4RafAdT6qZnGPq9bzxL8zyE/Ueu+6CGKBVtONNaWv1Ov9k5qXavWbTkmZ505Q1Wotcxqsuho07NlYS5nrG3N63OknKy5V0RECr5aDpJKRsrVv+5sYPlrZZn15LlaRkr/WXMsgRxrDilGRipupEa7N9eSRMqH++b/lcXOV1n6/TBm6A4AgP0YT6QAAAAAICIWUgAAAAAQ0b65Hyai8P9tZyoE5u1ghXy3uT+v72lqb9O3z9i29uXazVvJgi7zfYmIFEJ9G1qr5d6CbvP1g66Y1+7s/b1n29VWKQRZ/Q8E+vHn2ve7w/J1ZZXvl4hIwbINze9U7j1ne8/0LY1d7Zav2ze/dpyfFRGRNkt/ocP82rb7tvwVlFy7/r60l5nvTbsvEfvXpX4/RX9fbX9HrN8Ty/8/qK8do3e/fm3L3wP1tWP08tq8Nq/Na/Pae663tf29WmiJPDih7U/sB9atWyejR4/u79sAAAAAMECsXbtWRo0aZayzkBKRIAhkw4YNksn8/+3de1BU5R/H8c9yWxblIqiLVBAUE94DCQVNLTAta8ZbTpMmecksTC5T3hpE84La6Jhomv1RTaaljY2XsvKSpKaCkk6mYWkNpiLTRVERL7vn98cv9+fmpd8acYDer5mdYZ/nOTvf3fkwy5dz9tlAWSwWVVZW6o477tDRo0cVFBRkdnlowMgaagtZQ20ha6gtZA3/FMMwdObMGUVERMjL68afhOLSPkleXl7X7TaDgoL4xUStIGuoLWQNtYWsobaQNfwTgoOD/3INm00AAAAAgIdopAAAAADAQzRS12G1WpWXlyer1Wp2KWjgyBpqC1lDbSFrqC1kDWZjswkAAAAA8BBnpAAAAADAQzRSAAAAAOAhGikAAAAA8BCNFAAAAAB4iEbqTxYuXKg777xT/v7+6tixo4qKiswuCfVcfn6+7rvvPgUGBqp58+bq06ePSktL3dZUV1crIyNDYWFhaty4sfr376+TJ0+aVDEaipkzZ8pisSgrK8s1RtZQU44dO6bBgwcrLCxMNptNbdu21e7du13zhmFo0qRJatGihWw2m9LS0vT999+bWDHqI4fDodzcXEVHR8tms+muu+7S1KlTdfVeaWQNZqGRusoHH3ygnJwc5eXlqaSkRO3bt1fPnj1VUVFhdmmoxwoLC5WRkaGdO3dqw4YNunTpkh566CGdO3fOtSY7O1tr167VypUrVVhYqOPHj6tfv34mVo36rri4WG+88YbatWvnNk7WUBN+//13de7cWb6+vlq/fr0OHDigOXPmqEmTJq41s2fP1vz587V48WLt2rVLjRo1Us+ePVVdXW1i5ahvZs2apUWLFmnBggU6ePCgZs2apdmzZ6ugoMC1hqzBNAZckpKSjIyMDNd9h8NhREREGPn5+SZWhYamoqLCkGQUFhYahmEYp06dMnx9fY2VK1e61hw8eNCQZOzYscOsMlGPnTlzxoiNjTU2bNhgdOvWzcjMzDQMg6yh5owbN87o0qXLDeedTqcRHh5uvPrqq66xU6dOGVar1Vi+fHltlIgGonfv3sawYcPcxvr162cMGjTIMAyyBnNxRuoPFy9e1J49e5SWluYa8/LyUlpamnbs2GFiZWhoTp8+LUkKDQ2VJO3Zs0eXLl1yy15cXJwiIyPJHm5JRkaGevfu7ZYpiayh5qxZs0aJiYl6/PHH1bx5c8XHx+vNN990zf/4448qLy93y1pwcLA6duxI1uCRlJQUbdq0SYcOHZIk7du3T9u2bdPDDz8siazBXD5mF1BX/PLLL3I4HLLb7W7jdrtd3333nUlVoaFxOp3KyspS586d1aZNG0lSeXm5/Pz8FBIS4rbWbrervLzchCpRn73//vsqKSlRcXHxNXNkDTXlyJEjWrRokXJycjRx4kQVFxdrzJgx8vPzU3p6uitP13tPJWvwxPjx41VZWam4uDh5e3vL4XBo+vTpGjRokCSRNZiKRgqoRRkZGdq/f7+2bdtmdilogI4eParMzExt2LBB/v7+ZpeDBszpdCoxMVEzZsyQJMXHx2v//v1avHix0tPTTa4ODcmKFSv03nvvadmyZWrdurX27t2rrKwsRUREkDWYjkv7/tC0aVN5e3tfs3vVyZMnFR4eblJVaEhGjx6tdevW6YsvvtDtt9/uGg8PD9fFixd16tQpt/VkD57as2ePKioqlJCQIB8fH/n4+KiwsFDz58+Xj4+P7HY7WUONaNGihVq1auU21rJlS5WVlUmSK0+8p+LveumllzR+/Hg98cQTatu2rZ566illZ2crPz9fElmDuWik/uDn56cOHTpo06ZNrjGn06lNmzYpOTnZxMpQ3xmGodGjR+ujjz7S5s2bFR0d7TbfoUMH+fr6umWvtLRUZWVlZA8eSU1N1TfffKO9e/e6bomJiRo0aJDrZ7KGmtC5c+drvsbh0KFDioqKkiRFR0crPDzcLWuVlZXatWsXWYNHqqqq5OXl/ueqt7e3nE6nJLIGc3Fp31VycnKUnp6uxMREJSUlad68eTp37pyGDh1qdmmoxzIyMrRs2TKtXr1agYGBrmu2g4ODZbPZFBwcrOHDhysnJ0ehoaEKCgrSCy+8oOTkZHXq1Mnk6lGfBAYGuj57d0WjRo0UFhbmGidrqAnZ2dlKSUnRjBkzNHDgQBUVFWnJkiVasmSJJLm+v2zatGmKjY1VdHS0cnNzFRERoT59+phbPOqVxx57TNOnT1dkZKRat26tr7/+WnPnztWwYcMkkTWYzOxtA+uagoICIzIy0vDz8zOSkpKMnTt3ml0S6jlJ17299dZbrjXnz583nn/+eaNJkyZGQECA0bdvX+PEiRPmFY0G4+rtzw2DrKHmrF271mjTpo1htVqNuLg4Y8mSJW7zTqfTyM3NNex2u2G1Wo3U1FSjtLTUpGpRX1VWVhqZmZlGZGSk4e/vb8TExBgvv/yyceHCBdcasgazWAzjqq+GBgAAAAD8JT4jBQAAAAAeopECAAAAAA/RSAEAAACAh2ikAAAAAMBDNFIAAAAA4CEaKQAAAADwEI0UAAAAAHiIRgoAAAAAPEQjBQD4V+jevbuysrLq3WMDAOomH7MLAACgvlu1apV8fX3NLgMAUItopAAA+JtCQ0PNLgEAUMu4tA8AUOc4nU7l5+crOjpaNptN7du314cffihJ2rJliywWiz777DPFx8fLZrPpwQcfVEVFhdavX6+WLVsqKChITz75pKqqqtwe9/Llyxo9erSCg4PVtGlT5ebmyjCM/6um119/XbGxsfL395fdbteAAQNcc1df2nelvj/fnn76adf61atXKyEhQf7+/oqJidGUKVN0+fLlv/eiAQBqFWekAAB1Tn5+vpYuXarFixcrNjZWX375pQYPHqxmzZq51kyePFkLFixQQECABg4cqIEDB8pqtWrZsmU6e/as+vbtq4KCAo0bN851zDvvvKPhw4erqKhIu3fv1siRIxUZGalnnnnmpvXs3r1bY8aM0bvvvquUlBT99ttv2rp163XXpqSk6MSJE677Bw8e1COPPKKuXbtKkrZu3aohQ4Zo/vz5uv/++3X48GGNHDlSkpSXl3fLrxkAoHZZjP/3X3EAANSCCxcuKDQ0VBs3blRycrJrfMSIEaqqqtLIkSP1wAMPaOPGjUpNTZUkzZw5UxMmTNDhw4cVExMjSRo1apR++uknffrpp5L+e9aooqJC3377rSwWiyRp/PjxWrNmjQ4cOHDTmlatWqWhQ4fq559/VmBg4DXz3bt317333qt58+a5jf/6669KSkpSr169tHDhQklSWlqaUlNTNWHCBNe6pUuXauzYsTp+/LiHrxYAwCyckQIA1Ck//PCDqqqq1KNHD7fxixcvKj4+3nW/Xbt2rp/tdrsCAgJcTdSVsaKiIrfH6NSpk6uJkqTk5GTNmTNHDodD3t7eN6ypR48eioqKUkxMjHr16qVevXqpb9++CggIuOExly5dUv/+/RUVFaXXXnvNNb5v3z5t375d06dPd405HA5VV1erqqrqpo8JAKg7aKQAAHXK2bNnJUkff/yxbrvtNrc5q9Wqw4cPS5LbLnkWi+WaXfMsFoucTmeN1BQYGKiSkhJt2bJFn3/+uSZNmqTJkyeruLhYISEh1z3mueee09GjR1VUVCQfn/+93Z49e1ZTpkxRv379rjnG39+/RuoFAPzzaKQAAHVKq1atZLVaVVZWpm7dul0zf6WRuhW7du1yu79z507Fxsbe9GzUFT4+PkpLS1NaWpry8vIUEhKizZs3X7chmjt3rlasWKGvvvpKYWFhbnMJCQkqLS3V3XfffcvPAwBgPhopAECdEhgYqBdffFHZ2dlyOp3q0qWLTp8+re3btysoKEhRUVG3/NhlZWXKycnRs88+q5KSEhUUFGjOnDl/edy6det05MgRde3aVU2aNNEnn3wip9Ope+6555q1Gzdu1NixY7Vw4UI1bdpU5eXlkiSbzabg4GBNmjRJjz76qCIjIzVgwAB5eXlp37592r9/v6ZNm3bLzw0AULtopAAAdc7UqVPVrFkz5efn68iRIwoJCVFCQoImTpz4ty7XGzJkiM6fP6+kpCR5e3srMzPTtWPezYSEhGjVqlWaPHmyqqurFRsbq+XLl6t169bXrN22bZscDodGjRqlUaNGucbT09P19ttvq2fPnlq3bp1eeeUVzZo1S76+voqLi9OIESNu+XkBAGofu/YBAAAAgIf4Ql4AAAAA8BCNFADgX2/r1q1q3LjxDW8AAPwZl/YBAP71zp8/r2PHjt1wnh32AAB/RiMFAAAAAB7i0j4AAAAA8BCNFAAAAAB4iEYKAAAAADxEIwUAAAAAHqKRAgAAAAAP0UgBAAAAgIdopAAAAADAQ/8Bh9wLd2HKtcoAAAAASUVORK5CYII=\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "code", + "source": [ + "fig, ax = plt.subplots(figsize=(10, 10))\n", + "plt.imshow(tmp_layer.pe[0, :, 50:], aspect=\"auto\")\n", + "plt.xlabel(\"emb_size\")\n", + "plt.ylabel(\"max_seq_len\")\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 853 + }, + "id": "ovX65ZQUcVSJ", + "outputId": "cae6d154-c719-4272-d1de-19de4b4f43d9" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1IAAANECAYAAAC6qR6BAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA4kUlEQVR4nO3de7hXdZ33/9cXkQ3aBtwopwRPmdYoKJpEoWFgAnNbJqWpJRqKzgWa7Ltb457yNM1AOpajot5d02jdpZaNWWOJoSaoIQpIHu6GFCl05NBIgmBy2vv3h5f7146DfLabfYDH47q+18V3rfVd6725Vuqz9V1rV+rr6+sDAADAduvQ2gMAAAC0N0IKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoNBOE1LTpk3L/vvvn86dO2fw4MF54oknWnskAABgJ1Wpr6+vb+0h3q0f/vCHOeuss3LLLbdk8ODBue6663LXXXdl4cKF6dmz5zt+vq6uLq+88kqqq6tTqVRaYGIAAKAtqq+vz+uvv56+ffumQ4etX3faKUJq8ODB+dCHPpQbb7wxyVth1K9fv1x44YX5yle+8o6ff/nll9OvX78dPSYAANBOvPTSS9l33323ur5jC86yQ6xfvz7z5s3L5MmTG5Z16NAhI0aMyOzZs7f4mXXr1mXdunUN799uyaEZnY7ZvXiGn/zumeLP/KVPv//wJn/WsR3bsR3bsR3bsR3bsR3bsZvv2KvX1GW/Qb9PdXX1Nrdr9yH13//939m0aVN69erVaHmvXr3yn//5n1v8zJQpU3LllVdutrxjdk/HSnlIda1+d7eaNeWYju3Yju3Yju3Yju3Yju3Yjr1jjp3kHW/52WkeNlFi8uTJWbVqVcPrpZdeau2RAACAdqTdX5Hae++9s9tuu2X58uWNli9fvjy9e/fe4meqqqpSVVXVEuMBAAA7oXZ/RapTp0456qij8uCDDzYsq6ury4MPPpghQ4a04mQAAMDOqt1fkUqS2trajB07NkcffXSOOeaYXHfddVm7dm3OOeec1h4NAADYCe0UIXXaaaflj3/8Yy677LIsW7YsRxxxRKZPn77ZAygAAACaw04RUkkyceLETJw4sbXHAAAAdgHt/h4pAACAliakAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKNTmQ2rKlCn50Ic+lOrq6vTs2TMnn3xyFi5c2GibYcOGpVKpNHpdcMEFrTQxAACws2vzITVz5sxMmDAhjz/+eGbMmJENGzbkE5/4RNauXdtou/POOy9Lly5teF199dWtNDEAALCz69jaA7yT6dOnN3p/2223pWfPnpk3b16OO+64huV77LFHevfu3dLjAQAAu6A2f0Xqr61atSpJUlNT02j5D37wg+y999457LDDMnny5Lzxxhtb3ce6deuyevXqRi8AAIDt1eavSP2lurq6XHzxxfnoRz+aww47rGH5GWeckf322y99+/bN008/nUsvvTQLFy7M3XffvcX9TJkyJVdeeWVLjQ0AAOxk2lVITZgwIc8++2weffTRRsvHjx/f8OfDDz88ffr0yfDhw7No0aIcdNBBm+1n8uTJqa2tbXi/evXq9OvXb8cNDgAA7FTaTUhNnDgx9957b2bNmpV99913m9sOHjw4SfLCCy9sMaSqqqpSVVW1Q+YEAAB2fm0+pOrr63PhhRfmJz/5SR5++OEccMAB7/iZBQsWJEn69Omzg6cDAAB2RW0+pCZMmJDbb789P/3pT1NdXZ1ly5YlSbp165YuXbpk0aJFuf322zN69Oj06NEjTz/9dCZNmpTjjjsuAwYMaOXpAQCAnVGbD6mbb745yVu/dPcv3XrrrTn77LPTqVOnPPDAA7nuuuuydu3a9OvXL2PGjMlXv/rVVpgWAADYFbT5kKqvr9/m+n79+mXmzJktNA0AAEA7/D1SAAAArU1IAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUKhjaw/QplQqb70AAAC2wRUpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACrX5kLriiitSqVQavQ499NCG9W+++WYmTJiQHj165D3veU/GjBmT5cuXt+LEAADAzq7Nh1SS/M3f/E2WLl3a8Hr00Ucb1k2aNCn/8R//kbvuuiszZ87MK6+8klNOOaUVpwUAAHZ2HVt7gO3RsWPH9O7de7Plq1atyne+853cfvvt+fjHP54kufXWW/OBD3wgjz/+eD784Q9vcX/r1q3LunXrGt6vXr16xwwOAADslNrFFannn38+ffv2zYEHHpgzzzwzS5YsSZLMmzcvGzZsyIgRIxq2PfTQQ9O/f//Mnj17q/ubMmVKunXr1vDq16/fDv8ZAACAnUebD6nBgwfntttuy/Tp03PzzTdn8eLFOfbYY/P6669n2bJl6dSpU7p3797oM7169cqyZcu2us/Jkydn1apVDa+XXnppB/8UAADAzqTNf7Vv1KhRDX8eMGBABg8enP322y8/+tGP0qVLlybts6qqKlVVVc01IgAAsItp81ek/lr37t3z/ve/Py+88EJ69+6d9evX57XXXmu0zfLly7d4TxUAAEBzaHchtWbNmixatCh9+vTJUUcdld133z0PPvhgw/qFCxdmyZIlGTJkSCtOCQAA7Mza/Ff7vvzlL+ekk07Kfvvtl1deeSWXX355dtttt5x++unp1q1bxo0bl9ra2tTU1KRr16658MILM2TIkK0+sQ8AAODdavMh9fLLL+f000/Pq6++mn322SdDhw7N448/nn322SdJ8q1vfSsdOnTImDFjsm7dupx44om56aabWnlqAABgZ9bmQ+rOO+/c5vrOnTtn2rRpmTZtWgtNBAAA7Ora3T1SAAAArU1IAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQqGNrD0CSSqW1JwAAAAq4IgUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhdp8SO2///6pVCqbvSZMmJAkGTZs2GbrLrjgglaeGgAA2Jl1bO0B3smTTz6ZTZs2Nbx/9tlnc8IJJ+Szn/1sw7LzzjsvV111VcP7PfbYo0VnBAAAdi1tPqT22WefRu+nTp2agw46KB/72Mcalu2xxx7p3bv3du9z3bp1WbduXcP71atXv/tBAQCAXUab/2rfX1q/fn2+//3v54tf/GIqlUrD8h/84AfZe++9c9hhh2Xy5Ml54403trmfKVOmpFu3bg2vfv367ejRAQCAnUibvyL1l+6555689tprOfvssxuWnXHGGdlvv/3St2/fPP3007n00kuzcOHC3H333Vvdz+TJk1NbW9vwfvXq1WIKAADYbu0qpL7zne9k1KhR6du3b8Oy8ePHN/z58MMPT58+fTJ8+PAsWrQoBx100Bb3U1VVlaqqqh0+LwAAsHNqN1/t+8Mf/pAHHngg55577ja3Gzx4cJLkhRdeaImxAACAXVC7Calbb701PXv2zN/+7d9uc7sFCxYkSfr06dMCUwEAALuidvHVvrq6utx6660ZO3ZsOnb8/0detGhRbr/99owePTo9evTI008/nUmTJuW4447LgAEDWnFiAABgZ9YuQuqBBx7IkiVL8sUvfrHR8k6dOuWBBx7Iddddl7Vr16Zfv34ZM2ZMvvrVr7bSpAAAwK6gXYTUJz7xidTX12+2vF+/fpk5c2YrTAQAAOzK2s09UgAAAG2FkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKNQuHn/eYiod3noBAABsg2oAAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQh1bewBaWaXS2hMAAEC744oUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFOjblQ2vXrs3UqVPz4IMPZsWKFamrq2u0/sUXX2yW4QAAANqiJoXUueeem5kzZ+YLX/hC+vTpk0ql0txzAQAAtFlNCqn77rsvP//5z/PRj360uecBAABo85p0j9Ree+2Vmpqa5p4FAACgXWhSSP3DP/xDLrvssrzxxhvNPQ8AAECb16Sv9l177bVZtGhRevXqlf333z+77757o/Xz589vluEAAADaoiaF1Mknn9zMYwAAALQfTQqpyy+/vLnnAAAAaDea/At5X3vttfzrv/5rJk+enJUrVyZ56yt9//Vf/9VswwEAALRFTboi9fTTT2fEiBHp1q1bfv/73+e8885LTU1N7r777ixZsiTf+973mntOAACANqNJV6Rqa2tz9tln5/nnn0/nzp0blo8ePTqzZs1qtuEAAADaoiaF1JNPPpnzzz9/s+Xvfe97s2zZsnc9FAAAQFvWpJCqqqrK6tWrN1v+u9/9Lvvss8+7HgoAAKAta1JIffKTn8xVV12VDRs2JEkqlUqWLFmSSy+9NGPGjGnWAQEAANqaJoXUtddemzVr1qRnz57585//nI997GN53/vel+rq6vzjP/5jc88IAADQpjTpqX3dunXLjBkz8uijj+bpp5/OmjVrMmjQoIwYMaK55wMAAGhzmhRSbxs6dGiGDh3aXLMAAAC0C9sdUtdff/127/Siiy5q0jAAAADtwXaH1Le+9a3t2q5SqQgpAABgp7bdIbV48eIdOQcAAEC70aSn9m2vrl275sUXX9yRhwAAAGhxOzSk6uvrd+TuAQAAWsUODSkAAICdkZACAAAoJKQAAAAK7dCQqlQqO3L3AAAArcLDJgAAAArt0JC677778t73vndHHgIAAKDFbfcv5P1LtbW1273t0KFDm3IIAACANqtJIfXUU0/lqaeeyoYNG3LIIYckSX73u99lt912y6BBgxq2c48UAACwM2pSSJ100kmprq7Od7/73ey1115Jkj/96U8555xzcuyxx+Z//s//2axDAgAAtCVNukfq2muvzZQpUxoiKkn22muvfP3rX8+1117bbMMBAAC0RU0KqdWrV+ePf/zjZsv/+Mc/5vXXX3/XQwEAALRlTQqpT3/60znnnHNy99135+WXX87LL7+cf//3f8+4ceNyyimnNPeMAAAAbUqT7pG65ZZb8uUvfzlnnHFGNmzY8NaOOnbMuHHjcs011zTrgAAAAG1Nk0Jqjz32yE033ZRrrrkmixYtSpIcdNBB2XPPPZt1OAAAgLboXf1C3qVLl2bp0qU5+OCDs+eee6a+vr655gIAAGizmhRSr776aoYPH573v//9GT16dJYuXZokGTdunEefAwAAO70mhdSkSZOy++67Z8mSJdljjz0alp922mmZPn36du9n1qxZOemkk9K3b99UKpXcc889jdbX19fnsssuS58+fdKlS5eMGDEizz//fKNtVq5cmTPPPDNdu3ZN9+7dM27cuKxZs6YpPxYAAMB2aVJI/fKXv8w3vvGN7Lvvvo2WH3zwwfnDH/6w3ftZu3ZtBg4cmGnTpm1x/dVXX53rr78+t9xyS+bMmZM999wzJ554Yt58882Gbc4888w899xzmTFjRu69997MmjUr48ePb8qPBQAAsF2a9LCJtWvXNroS9baVK1emqqpqu/czatSojBo1aovr6uvrc9111+WrX/1qPvWpTyVJvve976VXr16555578rnPfS6//e1vM3369Dz55JM5+uijkyQ33HBDRo8enX/+539O3759m/DTAQAAbFuTrkgde+yx+d73vtfwvlKppK6uLldffXWOP/74Zhls8eLFWbZsWUaMGNGwrFu3bhk8eHBmz56dJJk9e3a6d+/eEFFJMmLEiHTo0CFz5szZ6r7XrVuX1atXN3oBAABsryZdkbr66qszfPjwzJ07N+vXr88ll1yS5557LitXrsxjjz3WLIMtW7YsSdKrV69Gy3v16tWwbtmyZenZs2ej9R07dkxNTU3DNlsyZcqUXHnllc0yJwAAsOtp0hWpww47LL/73e8ydOjQfOpTn8ratWtzyimn5KmnnspBBx3U3DM2u8mTJ2fVqlUNr5deeqm1RwIAANqRJl2RSt76mt3f//3fN+csjfTu3TtJsnz58vTp06dh+fLly3PEEUc0bLNixYpGn9u4cWNWrlzZ8PktqaqqKrqXCwAA4C816YrU9OnT8+ijjza8nzZtWo444oicccYZ+dOf/tQsgx1wwAHp3bt3HnzwwYZlq1evzpw5czJkyJAkyZAhQ/Laa69l3rx5Dds89NBDqaury+DBg5tlDgAAgL/WpJD6X//rfzU8oOGZZ55JbW1tRo8encWLF6e2tna797NmzZosWLAgCxYsSPLWAyYWLFiQJUuWpFKp5OKLL87Xv/71/OxnP8szzzyTs846K3379s3JJ5+cJPnABz6QkSNH5rzzzssTTzyRxx57LBMnTsznPvc5T+wDAAB2mCZ9tW/x4sX54Ac/mCT593//95x00kn5p3/6p8yfPz+jR4/e7v3MnTu30VP+3o6wsWPH5rbbbssll1yStWvXZvz48XnttdcydOjQTJ8+PZ07d274zA9+8INMnDgxw4cPT4cOHTJmzJhcf/31TfmxAAAAtkuTQqpTp0554403kiQPPPBAzjrrrCRJTU1N0aPEhw0blvr6+q2ur1Qqueqqq3LVVVdtdZuamprcfvvt231MAACAd6tJITV06NDU1tbmox/9aJ544on88Ic/TJL87ne/y7777tusAwIAALQ1TbpH6sYbb0zHjh3z4x//ODfffHPe+973Jknuu+++jBw5slkHBAAAaGuadEWqf//+uffeezdb/q1vfavR+6lTp+aCCy5I9+7dmzQcAABAW9SkK1Lb65/+6Z+ycuXKHXkIAACAFrdDQ2pbD5IAAABor3ZoSAEAAOyMhBQAAEAhIQUAAFBISAEAABTaoSF17LHHpkuXLjvyEAAAAC2uSSF12223bXH5xo0bM3ny5Ib3v/jFL9KnT58mDQYAANBWNSmkLrroonz2s5/Nn/70p4ZlCxcuzODBg3PHHXc023AAAABtUZNC6qmnnsrLL7+cww8/PDNmzMi0adMyaNCgHHroofnNb37T3DMCAAC0KR2b8qGDDjoojz32WC6++OKMHDkyu+22W7773e/m9NNPb+75AAAA2pwmP2zi5z//ee68884MGTIk3bt3z3e+85288sorzTkbAABAm9SkkDr//PPz2c9+NpdeemkeeeSRPP300+nUqVMOP/zw/OhHP2ruGQEAANqUJn2177HHHsucOXMycODAJEnv3r3zi1/8ItOmTcsXv/jFnHrqqc06JAAAQFvSpJCaN29eqqqqNls+YcKEjBgx4l0PBQAA0JY16at9W4qotx1yyCFNHgYAAKA9aNIVqST58Y9/nB/96EdZsmRJ1q9f32jd/Pnz3/VgAAAAbVWTrkhdf/31Oeecc9KrV6889dRTOeaYY9KjR4+8+OKLGTVqVHPPCAAA0KY0KaRuuummfPvb384NN9yQTp065ZJLLsmMGTNy0UUXZdWqVc09IwAAQJvSpJBasmRJPvKRjyRJunTpktdffz1J8oUvfCF33HFH800HAADQBjUppHr37p2VK1cmSfr375/HH388SbJ48eLU19c333QAAABtUJNC6uMf/3h+9rOfJUnOOeecTJo0KSeccEJOO+20fPrTn27WAQEAANqaJj2179vf/nbq6uqSvPW7o/bee+889thj+eQnP5kLLrigWQcEAABoa5oUUh06dMj69eszf/78rFixIl26dGn4RbzTp0/PSSed1KxDAgAAtCVNCqnp06fnC1/4Ql599dXN1lUqlWzatOldDwYAANBWNekeqQsvvDCnnnpqli5dmrq6ukYvEQUAAOzsmhRSy5cvT21tbXr16tXc8wAAALR5TQqpz3zmM3n44YebeRQAAID2oUn3SN1444357Gc/m0ceeSSHH354dt9990brL7roomYZDgAAoC1qUkjdcccd+eUvf5nOnTvn4YcfTqVSaVhXqVSEFAAAsFNrUkj9/d//fa688sp85StfSYcOTfp2IAAAQLvVpApav359TjvtNBEFAADskppUQmPHjs0Pf/jD5p4FAACgXWjSV/s2bdqUq6++Ovfff38GDBiw2cMmvvnNbzbLcAAAAG1Rk0LqmWeeyZFHHpkkefbZZxut+8sHTwAAAOyMmhRSv/rVr5p7DgAAgHbD0yIAAAAKCSkAAIBCTfpqH82somcBAKA98V/wAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEChjq09QFtS6VBJpVJp7TF2Hf6uAQBop1yRAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoFCrhtSsWbNy0kknpW/fvqlUKrnnnnsa1m3YsCGXXnppDj/88Oy5557p27dvzjrrrLzyyiuN9rH//vunUqk0ek2dOrWFfxIAAGBX0qohtXbt2gwcODDTpk3bbN0bb7yR+fPn52tf+1rmz5+fu+++OwsXLswnP/nJzba96qqrsnTp0obXhRde2BLjAwAAu6iOrXnwUaNGZdSoUVtc161bt8yYMaPRshtvvDHHHHNMlixZkv79+zcsr66uTu/evXforAAAAG9rV/dIrVq1KpVKJd27d2+0fOrUqenRo0eOPPLIXHPNNdm4ceM297Nu3bqsXr260QsAAGB7teoVqRJvvvlmLr300px++unp2rVrw/KLLroogwYNSk1NTX79619n8uTJWbp0ab75zW9udV9TpkzJlVde2RJjAwAAO6F2EVIbNmzIqaeemvr6+tx8882N1tXW1jb8ecCAAenUqVPOP//8TJkyJVVVVVvc3+TJkxt9bvXq1enXr9+OGR4AANjptPmQejui/vCHP+Shhx5qdDVqSwYPHpyNGzfm97//fQ455JAtblNVVbXVyAIAAHgnbTqk3o6o559/Pr/61a/So0ePd/zMggUL0qFDh/Ts2bMFJgQAAHZFrRpSa9asyQsvvNDwfvHixVmwYEFqamrSp0+ffOYzn8n8+fNz7733ZtOmTVm2bFmSpKamJp06dcrs2bMzZ86cHH/88amurs7s2bMzadKkfP7zn89ee+3VWj8WAACwk2vVkJo7d26OP/74hvdv37c0duzYXHHFFfnZz36WJDniiCMafe5Xv/pVhg0blqqqqtx555254oorsm7duhxwwAGZNGlSo/ufAAAAmlurhtSwYcNSX1+/1fXbWpckgwYNyuOPP97cYwEAAGxTu/o9UgAAAG2BkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKBQq4bUrFmzctJJJ6Vv376pVCq55557Gq0/++yzU6lUGr1GjhzZaJuVK1fmzDPPTNeuXdO9e/eMGzcua9asacGfAgAA2NW0akitXbs2AwcOzLRp07a6zciRI7N06dKG1x133NFo/ZlnnpnnnnsuM2bMyL333ptZs2Zl/PjxO3p0AABgF9axNQ8+atSojBo1apvbVFVVpXfv3ltc99vf/jbTp0/Pk08+maOPPjpJcsMNN2T06NH553/+5/Tt27fZZwYAAGjz90g9/PDD6dmzZw455JD83d/9XV599dWGdbNnz0737t0bIipJRowYkQ4dOmTOnDlb3ee6deuyevXqRi8AAIDt1aZDauTIkfne976XBx98MN/4xjcyc+bMjBo1Kps2bUqSLFu2LD179mz0mY4dO6ampibLli3b6n6nTJmSbt26Nbz69eu3Q38OAABg59KqX+17J5/73Oca/nz44YdnwIABOeigg/Lwww9n+PDhTd7v5MmTU1tb2/B+9erVYgoAANhubfqK1F878MADs/fee+eFF15IkvTu3TsrVqxotM3GjRuzcuXKrd5Xlbx131XXrl0bvQAAALZXuwqpl19+Oa+++mr69OmTJBkyZEhee+21zJs3r2Gbhx56KHV1dRk8eHBrjQkAAOzkWvWrfWvWrGm4upQkixcvzoIFC1JTU5OamppceeWVGTNmTHr37p1Fixblkksuyfve976ceOKJSZIPfOADGTlyZM4777zccsst2bBhQyZOnJjPfe5zntgHAADsMK16RWru3Lk58sgjc+SRRyZJamtrc+SRR+ayyy7Lbrvtlqeffjqf/OQn8/73vz/jxo3LUUcdlUceeSRVVVUN+/jBD36QQw89NMOHD8/o0aMzdOjQfPvb326tHwkAANgFtOoVqWHDhqW+vn6r6++///533EdNTU1uv/325hwLAABgm9rVPVIAAABtgZACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACjUsbUHaFMqHd56AQAAbINqAAAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACnVs7QFoZRUtDQAApfxXNAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFCoY2sPQFLpUGntEQAAgAKuSAEAABQSUgAAAIWEFAAAQCH3SLFrqrgvDQCApnNFCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoFCrhtSsWbNy0kknpW/fvqlUKrnnnnsara9UKlt8XXPNNQ3b7L///putnzp1agv/JAAAwK6kVUNq7dq1GThwYKZNm7bF9UuXLm30+rd/+7dUKpWMGTOm0XZXXXVVo+0uvPDClhgfAADYRXVszYOPGjUqo0aN2ur63r17N3r/05/+NMcff3wOPPDARsurq6s323Zb1q1bl3Xr1jW8X7169XZ/FgAAoN3cI7V8+fL8/Oc/z7hx4zZbN3Xq1PTo0SNHHnlkrrnmmmzcuHGb+5oyZUq6devW8OrXr9+OGhsAANgJteoVqRLf/e53U11dnVNOOaXR8osuuiiDBg1KTU1Nfv3rX2fy5MlZunRpvvnNb251X5MnT05tbW3D+9WrV4spAABgu7WbkPq3f/u3nHnmmencuXOj5X8ZRAMGDEinTp1y/vnnZ8qUKamqqtrivqqqqra6DgAA4J20i6/2PfLII1m4cGHOPffcd9x28ODB2bhxY37/+9/v+MEAAIBdUrsIqe985zs56qijMnDgwHfcdsGCBenQoUN69uzZApMBAAC7olb9at+aNWvywgsvNLxfvHhxFixYkJqamvTv3z/JW/cv3XXXXbn22ms3+/zs2bMzZ86cHH/88amurs7s2bMzadKkfP7zn89ee+3VYj8HAACwa2nVkJo7d26OP/74hvdv3+80duzY3HbbbUmSO++8M/X19Tn99NM3+3xVVVXuvPPOXHHFFVm3bl0OOOCATJo0qdF9UwAAAM2tVUNq2LBhqa+v3+Y248ePz/jx47e4btCgQXn88cd3xGgAAABb1S7ukQIAAGhLhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAU6tjaA7QpHSpJpdLaUwAAAG2cK1IAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFOrb2AG1BfX19kmRj/YYmfX7163Xv6vhNPa5jO7ZjO7ZjO7ZjO7ZjO7ZjN++xV69567NvN8LWVOrfaYtdwMsvv5x+/fq19hgAAEAb8dJLL2Xffffd6nohlaSuri6vvPJKqqurU6lUNlu/evXq9OvXLy+99FK6du3aChOyq3Cu0VKca7QU5xotxblGc6mvr8/rr7+evn37pkOHrd8J5at9STp06LDN2nxb165d/Q+TFuFco6U412gpzjVainON5tCtW7d33MbDJgAAAAoJKQAAgEJCajtUVVXl8ssvT1VVVWuPwk7OuUZLca7RUpxrtBTnGi3NwyYAAAAKuSIFAABQSEgBAAAUElIAAACFhBQAAEAhIfUOpk2blv333z+dO3fO4MGD88QTT7T2SLRzs2bNykknnZS+ffumUqnknnvuabS+vr4+l112Wfr06ZMuXbpkxIgRef7551tnWNq1KVOm5EMf+lCqq6vTs2fPnHzyyVm4cGGjbd58881MmDAhPXr0yHve856MGTMmy5cvb6WJaa9uvvnmDBgwoOEXoQ4ZMiT33Xdfw3rnGTvC1KlTU6lUcvHFFzcsc67RkoTUNvzwhz9MbW1tLr/88syfPz8DBw7MiSeemBUrVrT2aLRja9euzcCBAzNt2rQtrr/66qtz/fXX55ZbbsmcOXOy55575sQTT8ybb77ZwpPS3s2cOTMTJkzI448/nhkzZmTDhg35xCc+kbVr1zZsM2nSpPzHf/xH7rrrrsycOTOvvPJKTjnllFacmvZo3333zdSpUzNv3rzMnTs3H//4x/OpT30qzz33XBLnGc3vySefzP/5P/8nAwYMaLTcuUaLqmerjjnmmPoJEyY0vN+0aVN9375966dMmdKKU7EzSVL/k5/8pOF9XV1dfe/eveuvueaahmWvvfZafVVVVf0dd9zRChOyM1mxYkV9kvqZM2fW19e/dW7tvvvu9XfddVfDNr/97W/rk9TPnj27tcZkJ7HXXnvV/+u//qvzjGb3+uuv1x988MH1M2bMqP/Yxz5W/6Uvfam+vt4/02h5rkhtxfr16zNv3ryMGDGiYVmHDh0yYsSIzJ49uxUnY2e2ePHiLFu2rNF5161btwwePNh5x7u2atWqJElNTU2SZN68edmwYUOj8+3QQw9N//79nW802aZNm3LnnXdm7dq1GTJkiPOMZjdhwoT87d/+baNzKvHPNFpex9YeoK367//+72zatCm9evVqtLxXr175z//8z1aaip3dsmXLkmSL593b66Ap6urqcvHFF+ejH/1oDjvssCRvnW+dOnVK9+7dG23rfKMpnnnmmQwZMiRvvvlm3vOe9+QnP/lJPvjBD2bBggXOM5rNnXfemfnz5+fJJ5/cbJ1/ptHShBTALmDChAl59tln8+ijj7b2KOykDjnkkCxYsCCrVq3Kj3/844wdOzYzZ85s7bHYibz00kv50pe+lBkzZqRz586tPQ542MTW7L333tltt902e9LL8uXL07t371aaip3d2+eW847mNHHixNx777351a9+lX333bdhee/evbN+/fq89tprjbZ3vtEUnTp1yvve974cddRRmTJlSgYOHJh/+Zd/cZ7RbObNm5cVK1Zk0KBB6dixYzp27JiZM2fm+uuvT8eOHdOrVy/nGi1KSG1Fp06dctRRR+XBBx9sWFZXV5cHH3wwQ4YMacXJ2JkdcMAB6d27d6PzbvXq1ZkzZ47zjmL19fWZOHFifvKTn+Shhx7KAQcc0Gj9UUcdld13373R+bZw4cIsWbLE+ca7VldXl3Xr1jnPaDbDhw/PM888kwULFjS8jj766Jx55pkNf3au0ZJ8tW8bamtrM3bs2Bx99NE55phjct1112Xt2rU555xzWns02rE1a9bkhRdeaHi/ePHiLFiwIDU1Nenfv38uvvjifP3rX8/BBx+cAw44IF/72tfSt2/fnHzyya03NO3ShAkTcvvtt+enP/1pqqurG+4R6NatW7p06ZJu3bpl3Lhxqa2tTU1NTbp27ZoLL7wwQ4YMyYc//OFWnp72ZPLkyRk1alT69++f119/Pbfffnsefvjh3H///c4zmk11dXXDPZ5v23PPPdOjR4+G5c41WpKQ2obTTjstf/zjH3PZZZdl2bJlOeKIIzJ9+vTNHgQAJebOnZvjjz++4X1tbW2SZOzYsbnttttyySWXZO3atRk/fnxee+21DB06NNOnT/d9cIrdfPPNSZJhw4Y1Wn7rrbfm7LPPTpJ861vfSocOHTJmzJisW7cuJ554Ym666aYWnpT2bsWKFTnrrLOydOnSdOvWLQMGDMj999+fE044IYnzjJbjXKMlVerr6+tbewgAAID2xD1SAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgDsEoYNG5aLL7643e0bgLapY2sPAADt3d13353dd9+9tccAoAUJKQB4l2pqalp7BABamK/2AdDm1NXVZcqUKTnggAPSpUuXDBw4MD/+8Y+TJA8//HAqlUruv//+HHnkkenSpUs+/vGPZ8WKFbnvvvvygQ98IF27ds0ZZ5yRN954o9F+N27cmIkTJ6Zbt27Ze++987WvfS319fXbNdNNN92Ugw8+OJ07d06vXr3ymc98pmHdX3617+35/vp19tlnN2z/05/+NIMGDUrnzp1z4IEH5sorr8zGjRvf3V8aAC3KFSkA2pwpU6bk+9//fm655ZYcfPDBmTVrVj7/+c9nn332adjmiiuuyI033pg99tgjp556ak499dRUVVXl9ttvz5o1a/LpT386N9xwQy699NKGz3z3u9/NuHHj8sQTT2Tu3LkZP358+vfvn/POO2+b88ydOzcXXXRR/u///b/5yEc+kpUrV+aRRx7Z4rYf+chHsnTp0ob3v/3tbzN69Ogcd9xxSZJHHnkkZ511Vq6//voce+yxWbRoUcaPH58kufzyy5v8dwZAy6rUb+//FQcALWDdunWpqanJAw88kCFDhjQsP/fcc/PGG29k/PjxOf744/PAAw9k+PDhSZKpU6dm8uTJWbRoUQ488MAkyQUXXJDf//73mT59epK3rhqtWLEizz33XCqVSpLkK1/5Sn72s5/l//2//7fNme6+++6cc845efnll1NdXb3Z+mHDhuWII47Idddd12j5q6++mmOOOSYjR47MtGnTkiQjRozI8OHDM3ny5Ibtvv/97+eSSy7JK6+8Uvi3BUBrcUUKgDblhRdeyBtvvJETTjih0fL169fnyCOPbHg/YMCAhj/36tUre+yxR0NEvb3siSeeaLSPD3/4ww0RlSRDhgzJtddem02bNmW33Xbb6kwnnHBC9ttvvxx44IEZOXJkRo4cmU9/+tPZY489tvqZDRs2ZMyYMdlvv/3yL//yLw3Lf/Ob3+Sxxx7LP/7jPzYs27RpU95888288cYb29wnAG2HkAKgTVmzZk2S5Oc//3ne+973NlpXVVWVRYsWJUmjp+RVKpXNnppXqVRSV1fXLDNVV1dn/vz5efjhh/PLX/4yl112Wa644oo8+eST6d69+xY/83d/93d56aWX8sQTT6Rjx///X7dr1qzJlVdemVNOOWWzz3Tu3LlZ5gVgxxNSALQpH/zgB1NVVZUlS5bkYx/72Gbr3w6pppgzZ06j948//ngOPvjgbV6NelvHjh0zYsSIjBgxIpdffnm6d++ehx56aItB9M1vfjM/+tGP8utf/zo9evRotG7QoEFZuHBh3ve+9zX55wCg9QkpANqU6urqfPnLX86kSZNSV1eXoUOHZtWqVXnsscfStWvX7Lfffk3e95IlS1JbW5vzzz8/8+fPzw033JBrr732HT9377335sUXX8xxxx2XvfbaK7/4xS9SV1eXQw45ZLNtH3jggVxyySWZNm1a9t577yxbtixJ0qVLl3Tr1i2XXXZZ/sf/+B/p379/PvOZz6RDhw75zW9+k2effTZf//rXm/yzAdCyhBQAbc4//MM/ZJ999smUKVPy4osvpnv37hk0aFD+9//+3+/q63pnnXVW/vznP+eYY47Jbrvtli996UsNT8zblu7du+fuu+/OFVdckTfffDMHH3xw7rjjjvzN3/zNZts++uij2bRpUy644IJccMEFDcvHjh2b2267LSeeeGLuvffeXHXVVfnGN76R3XffPYceemjOPffcJv9cALQ8T+0DAAAo5BfyAgAAFBJSAOzyHnnkkbznPe/Z6gsA/pqv9gGwy/vzn/+c//qv/9rqek/YA+CvCSkAAIBCvtoHAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIX+Px+eCHVQDvPgAAAAAElFTkSuQmCC\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "code", + "source": [ + "fig, ax = plt.subplots(figsize=(10, 10))\n", + "plt.imshow(tmp_layer.pe[0, :, 50:51], aspect=\"auto\")\n", + "plt.xlabel(\"emb_size\")\n", + "plt.ylabel(\"max_seq_len\")\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 853 + }, + "id": "Emdh9j6acXH5", + "outputId": "516ee9f3-2db2-4fb0-a0f9-51054bd52c84" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1IAAANECAYAAAC6qR6BAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABA4klEQVR4nO3de5hVdb0/8M/aAwzYgcFRAVG8ZlrHK5YcDE2DEuhQJqapeYu89Hj5CaefyjnmrTqQl/KolE/nWNovr3XUfDQpwBAlxAuSlwoTMTQBS5LhkgPMrN8f6LRHQPd3y8yemf16nef7PO611l7z2T2rOu++6/P9Znme5wEAAEDJCpUuAAAAoLMRpAAAABIJUgAAAIkEKQAAgESCFAAAQCJBCgAAIJEgBQAAkEiQAgAASCRIAQAAJBKkAAAAEnWZIDVlypTYZZddomfPnjFkyJB47LHHKl0SAADQRWV5nueVLuL9uuOOO+Kkk06KG264IYYMGRLXXHNN/PSnP40FCxZEv3793vP7zc3N8eqrr0bv3r0jy7J2qBgAAOiI8jyPlStXxsCBA6NQ2Py8U5cIUkOGDImPfexjcf3110fEhmA0aNCgOOecc+LCCy98z++/8sorMWjQoLYuEwAA6CRefvnl2HHHHTd7vls71tIm1q5dG08++WRMnDix5VihUIgRI0bEnDlzNvmdxsbGaGxsbPn8dpYcFqOjW3Rv24IBAIAOa32si0fiF9G7d+93va7TB6m//vWv0dTUFP379291vH///vGHP/xhk9+ZNGlSXHbZZRsd7xbdo1smSAEAQNV6632992r56TKLTaSYOHFirFixomW8/PLLlS4JAADoRDr9jNS2224bNTU1sWzZslbHly1bFgMGDNjkd2pra6O2tnbjE1m2YQAAAFUqa5mVejedfkaqR48eceCBB8aMGTNajjU3N8eMGTNi6NChFawMAADoqjr9jFRExIQJE+Lkk0+Oj370o3HQQQfFNddcE6tXr45TTz210qUBAABdUJcIUscee2z85S9/iYsvvjiWLl0a+++/f0ydOnWjBSgAAAC2hC6xj9T71dDQEHV1dXFY4Sir9gEAQBVbn6+Lmc13xYoVK6JPnz6bva7T90gBAAC0N0EKAAAgkSAFAACQqEssNrGlZIXsPXcwBgAAuq4szyKa3/s6M1IAAACJBCkAAIBEXu0rVlMTkdVUugoAAKBS8uaI9e99mRkpAACARIIUAABAIkEKAAAgkR6pIlkWlj8HAIAqVmoaMCMFAACQSJACAABIJEgBAAAk0iNVrFCIyGRLAACoWnlpeUBqAAAASCRIAQAAJBKkAAAAEumRKlZTE5HVVLoKAACgUvLS8oAZKQAAgESCFAAAQCJBCgAAIJEeqSJZlkWWZZUuAwAAqJAsSssDZqQAAAASCVIAAACJBCkAAIBEeqSKFbIIPVIAAFC9cj1SAAAAbUKQAgAASOTVvmJZYcMAAACqVGl5QGoAAABIJEgBAAAkEqQAAAAS6ZEqVihsGAAAQHVq1iMFAADQJgQpAACARIIUAABAIj1SxewjBQAA1a3EPCA1AAAAJBKkAAAAEglSAAAAifRIFStkGwYAAFClSssDZqQAAAASCVIAAACJBCkAAIBEeqSKZIVCZAXZEgAAqlVW4lyT1AAAAJBIkAIAAEgkSAEAACTSI1WskEXokQIAgCpmHykAAIA2IUgBAAAkEqQAAAAS6ZEqlmUbBgAAUJ1KzANmpAAAABIJUgAAAIm82lfMq30AAFDdvNoHAADQNgQpAACARIIUAABAIj1SxWoKEQXZEgAAqlZWWh6QGgAAABIJUgAAAIkEKQAAgER6pIrZRwoAAKqbfaQAAADahiAFAACQSJACAABIpEeqmB4pAACobnqkAAAA2oYgBQAAkEiQAgAASKRHqkheKEReI1sCAEC1ykuca5IaAAAAEglSAAAAiQQpAACARHqkitlHCgAAqpt9pAAAANqGIAUAAJBIkAIAAEikR6qYHikAAKhueqQAAADahiAFAACQyKt9xQrZhgEAAFSn3Kt9AAAAbaLDB6lJkybFxz72sejdu3f069cvjjzyyFiwYEGraw477LDIsqzVOPPMMytUMQAA0NV1+CD10EMPxVlnnRWPPvpoTJs2LdatWxef/vSnY/Xq1a2uO+2002LJkiUt44orrqhQxQAAQFfX4Xukpk6d2urzTTfdFP369Ysnn3wyDj300JbjW221VQwYMOD9/bFCYcMAAACqU15aHuh0qWHFihUREVFfX9/q+C233BLbbrtt7L333jFx4sRYs2bNZu/R2NgYDQ0NrQYAAECpOvyMVLHm5uY477zz4uMf/3jsvffeLcePP/742HnnnWPgwIHx9NNPxwUXXBALFiyIu+66a5P3mTRpUlx22WXtVTYAANDFZHme55UuolRf/epX44EHHohHHnkkdtxxx81e9+CDD8bw4cPjhRdeiN13332j842NjdHY2NjyuaGhIQYNGhTDP/y16FZT2ya1AwAAHd/6psaY8furYsWKFdGnT5/NXtdpZqTOPvvsuO+++2LWrFnvGqIiIoYMGRIRsdkgVVtbG7W1GwemPMsiz+wjBQAA1arUPNDhg1Se53HOOefE3XffHTNnzoxdd931Pb8zf/78iIjYfvvt27g6AACgGnX4IHXWWWfFrbfeGj//+c+jd+/esXTp0oiIqKuri169esXChQvj1ltvjdGjR8c222wTTz/9dIwfPz4OPfTQ2HfffStcPQAA0BV1+CD1/e9/PyI2bLpb7Ec/+lGccsop0aNHj5g+fXpcc801sXr16hg0aFCMHTs2LrroogpUCwAAVIMOH6Teay2MQYMGxUMPPbRl/lghOuGC8AAAwBZT4lJ8YgMAAEAiQQoAACCRIAUAAJCow/dItatCYcMAAACqU15aHpAaAAAAEglSAAAAiQQpAACARHqkiuRZFnmWVboMAACgQkrNA2akAAAAEglSAAAAibzaV6wQoiUAAFSzvLTLxAYAAIBEghQAAEAiQQoAACCRHqliWURY/hwAAKpXiXHAjBQAAEAiQQoAACCRIAUAAJBIj1SRPMsiL+iRAgCAapU3l5YHzEgBAAAkEqQAAAASCVIAAACJ9EgVK2QbBgAAUJ1KzANmpAAAABIJUgAAAIkEKQAAgER6pIrkWRZ5pkcKAACqVal5wIwUAABAIkEKAAAgkSAFAACQSI9UMftIAQBAdbOPFAAAQNsQpAAAABIJUgAAAIn0SBXJsw0DAACoTqXmATNSAAAAiQQpAACARF7tK5ZlGwYAAFCdSswDZqQAAAASCVIAAACJBCkAAIBEeqSKFUK0BACAalZiHhAbAAAAEglSAAAAiQQpAACARHqkiuSFLPKCfaQAAKBalZoHzEgBAAAkEqQAAAASCVIAAACJ9EgVybMs8kyPFAAAVKtS84AZKQAAgESCFAAAQCJBCgAAIJEeqWLZWwMAAKhOJeYBM1IAAACJBCkAAIBEghQAAEAiPVJF8pos8hpNUgAAUK3y3D5SAAAAbUKQAgAASCRIAQAAJNIjVSTPNgwAAKA6lZoHzEgBAAAkEqQAAAASebWvlSwi824fAABUL8ufAwAAtAlBCgAAIJEgBQAAkEiPVBHLnwMAQHWz/DkAAEAbEaQAAAASCVIAAACJ9EgVK4RoCQAA1azEPCA2AAAAJBKkAAAAEglSAAAAifRIFcmzLPLMRlIAAFCtSs0DZqQAAAASCVIAAACJBCkAAIBEeqSKZW8NAACgOpWYB8xIAQAAJBKkAAAAEglSAAAAifRIFclrNgwAAKA65c2lXWdGCgAAIFGHD1KXXnppZFnWauy1114t5998880466yzYptttol/+qd/irFjx8ayZcsqWDEAANDVdYpX+/75n/85pk+f3vK5W7d/lD1+/Pi4//7746c//WnU1dXF2WefHUcddVTMnj07+e/kWRZ5Zv1zAACoVqXmgU4RpLp16xYDBgzY6PiKFSvixhtvjFtvvTU++clPRkTEj370o/jwhz8cjz76aPzLv/zLJu/X2NgYjY2NLZ8bGhrapnAAAKBL6vCv9kVE/PGPf4yBAwfGbrvtFieccEIsXrw4IiKefPLJWLduXYwYMaLl2r322it22mmnmDNnzmbvN2nSpKirq2sZgwYNavPfAAAAdB0dPkgNGTIkbrrpppg6dWp8//vfj0WLFsUhhxwSK1eujKVLl0aPHj2ib9++rb7Tv3//WLp06WbvOXHixFixYkXLePnll9v4VwAAAF1Jh3+1b9SoUS3/vO+++8aQIUNi5513jjvvvDN69epV1j1ra2ujtrZ24xPZWwMAAKhOJeaBDj8j9U59+/aND33oQ/HCCy/EgAEDYu3atfHGG2+0umbZsmWb7KkCAADYEjpdkFq1alUsXLgwtt9++zjwwAOje/fuMWPGjJbzCxYsiMWLF8fQoUMrWCUAANCVdfhX+772ta/FmDFjYuedd45XX301LrnkkqipqYnjjjsu6urqYty4cTFhwoSor6+PPn36xDnnnBNDhw7d7Ip9AAAA71eHD1KvvPJKHHfccfH666/HdtttF8OGDYtHH300tttuu4iI+O53vxuFQiHGjh0bjY2NccQRR8T3vve9sv5Wnm0YAABAdSo1D2R5nudtW0rH19DQEHV1dfGxz30junXvWelyAACAClm/7s14/OdfjxUrVkSfPn02e12n65ECAACoNEEKAAAgUYfvkWpPeWHDAAAAqlOpeUBsAAAASCRIAQAAJBKkAAAAEumRKpZlGwYAAFCdSswDZqQAAAASCVIAAACJBCkAAIBEeqSK5NmGAQAAVKdS84AZKQAAgESCFAAAQCJBCgAAIJEeqWKFiFy0BACA6lViHhAbAAAAEglSAAAAibzaVyx7awAAANXJ8ucAAABtQ5ACAABIJEgBAAAk0iNVJI+IXI8UAABUrbzE68xIAQAAJBKkAAAAEglSAAAAifRIFbOPFAAAVDf7SAEAALQNQQoAACCRIAUAAJBIj1SRvJBFXtAkBQAA1arUPGBGCgAAIJEgBQAAkEiQAgAASKRHqkiebRgAAEB1KjUPmJECAABIJEgBAAAkEqQAAAAS6ZEqlr01AACA6qRHCgAAoG0IUgAAAIkEKQAAgER6pIrkhQ0DAACoTqXmAbEBAAAgkSAFAACQyKt9RfJswwAAAKpTqXnAjBQAAEAiQQoAACCRIAUAAJBIj1Sx7K0BAABUJz1SAAAAbUOQAgAASCRIAQAAJNIjVUyPFAAAVDc9UgAAAG1DkAIAAEgkSAEAACTSI1UkL2wYAABAdSo1D4gNAAAAiQQpAACARIIUAABAIj1SRfJswwAAAKpTqXnAjBQAAEAiQQoAACCRIAUAAJBIj1Sx7K0BAABUJz1SAAAAbUOQAgAASOTVviJ5IYu84N0+AACoVqXmATNSAAAAiQQpAACARIIUAABAIj1SxSx/DgAA1c3y5wAAAG1DkAIAAEgkSAEAACTSI1Ukj4hcjxQAAFStvMTrzEgBAAAkEqQAAAASCVIAAACJ9EgVs48UAABUN/tIAQAAtA1BCgAAIJEgBQAAkEiPVJG8sGEAAADVqdQ8IDYAAAAk6vBBapdddoksyzYaZ511VkREHHbYYRudO/PMMytcNQAA0JV1+Ff7Hn/88Whqamr5/Oyzz8anPvWp+MIXvtBy7LTTTovLL7+85fNWW23VrjUCAADVpcMHqe22267V58mTJ8fuu+8en/jEJ1qObbXVVjFgwICS79nY2BiNjY0tnxsaGjb8g32kAACgunXFfaTWrl0bP/nJT+LLX/5yZNk/fuEtt9wS2267bey9994xceLEWLNmzbveZ9KkSVFXV9cyBg0a1NalAwAAXUiHn5Eqds8998Qbb7wRp5xySsux448/PnbeeecYOHBgPP3003HBBRfEggUL4q677trsfSZOnBgTJkxo+dzQ0CBMAQAAJetUQerGG2+MUaNGxcCBA1uOnX766S3/vM8++8T2228fw4cPj4ULF8buu+++yfvU1tZGbW1tm9cLAAB0TZ0mSP3pT3+K6dOnv+tMU0TEkCFDIiLihRde2GyQ2pw82zAAAIDqVGoe6DQ9Uj/60Y+iX79+8ZnPfOZdr5s/f35ERGy//fbtUBUAAFCNOsWMVHNzc/zoRz+Kk08+Obp1+0fJCxcujFtvvTVGjx4d22yzTTz99NMxfvz4OPTQQ2PfffetYMUAAEBX1imC1PTp02Px4sXx5S9/udXxHj16xPTp0+Oaa66J1atXx6BBg2Ls2LFx0UUXlfV38sKGAQAAVKdS80CnCFKf/vSnI8/zjY4PGjQoHnrooQpUBAAAVDPzLwAAAIkEKQAAgESd4tW+dpO9NQAAgOrU1ZY/BwAA6CgEKQAAgESCFAAAQCI9UsX0SAEAQHXTIwUAANA2BCkAAIBEghQAAEAiPVJF8mzDAAAAqlOpecCMFAAAQCJBCgAAIJEgBQAAkEiPVJG8sGEAAADVqdQ8IDYAAAAkEqQAAAASCVIAAACJBCkAAIBEghQAAEAiQQoAACCRIAUAAJDIPlLFsrcGAABQnUrMA2akAAAAEglSAAAAibzaVyQvbBgAAEB1KjUPiA0AAACJBCkAAIBEghQAAEAiPVLFLH8OAADVzfLnAAAAbUOQAgAASCRIAQAAJNIjVSTPNgwAAKA6lZoHzEgBAAAkEqQAAAASCVIAAACJ9Ei9kx4pAADgPZiRAgAASCRIAQAAJBKkAAAAEumRKpIXNgwAAKA6lZoHxAYAAIBEghQAAEAiQQoAACCRHqliWb5hAAAA1anEPGBGCgAAIFFZM1KrV6+OyZMnx4wZM+K1116L5ubmVudffPHFLVIcAABAR1RWkPrKV74SDz30UJx44omx/fbbR5ZlW7quysgi8i7yUwAAgDKUmAfKClIPPPBA3H///fHxj3+8nK8DAAB0amX1SG299dZRX1+/pWsBAADoFMoKUt/4xjfi4osvjjVr1mzpegAAADq8sl7tu/rqq2PhwoXRv3//2GWXXaJ79+6tzs+bN2+LFNfuCmEdQwAAqGYl5oGygtSRRx5ZztcAAAC6hLKC1CWXXLKl6wAAAOg0yn6R7Y033oj/+Z//iYkTJ8by5csjYsMrfX/+85+3WHEAAAAdUVkzUk8//XSMGDEi6urq4qWXXorTTjst6uvr46677orFixfHj3/84y1dZ7vI7SMFAABVrdQ8UNaM1IQJE+KUU06JP/7xj9GzZ8+W46NHj45Zs2aVc0sAAIBOo6wg9fjjj8cZZ5yx0fEddtghli5d+r6LAgAA6MjKClK1tbXR0NCw0fHnn38+tttuu/ddFAAAQEdWVo/UZz/72bj88svjzjvvjIiILMti8eLFccEFF8TYsWO3aIHtKntrAAAA1akte6SuvvrqWLVqVfTr1y/+/ve/xyc+8Yn44Ac/GL17945vfetb5dwSAACg0yhrRqquri6mTZsWjzzySDz99NOxatWqGDx4cIwYMWJL1wcAANDhlBWk3jZs2LAYNmzYlqoFAACgUyg5SF177bUl3/Tcc88tq5iK0yMFAADVrcQ8UHKQ+u53v1va382yzhukAAAASlBykFq0aFFb1gEAANBplLVqX6n69OkTL774Ylv+CQAAgHb3vhabeC95nrfl7be4PNswAACA6lRqHmjTGSkAAICuSJACAABIJEgBAAAkatMeqSzrZA1H9pECAIDq1hF6pDrbYhMAAAClaNMg9cADD8QOO+zQln8CAACg3ZX1at+ECRNKvnbYsGHl/ImKsPw5AABUt1LzQFlB6qmnnoqnnnoq1q1bF3vuuWdERDz//PNRU1MTgwcPbrmu0/VIAQAAlKCsIDVmzJjo3bt33HzzzbH11ltHRMTf/va3OPXUU+OQQw6Jf/u3f9uiRQIAAHQkZfVIXX311TFp0qSWEBURsfXWW8c3v/nNuPrqq7dYcQAAAB1RWTNSDQ0N8Ze//GWj43/5y19i5cqV77uoiinkGwYAAFCdSswDZc1Iff7zn49TTz017rrrrnjllVfilVdeif/93/+NcePGxVFHHVXOLQEAADqNsmakbrjhhvja174Wxx9/fKxbt27Djbp1i3HjxsWVV165RQsEAADoaMoKUltttVV873vfiyuvvDIWLlwYERG77757fOADH9iixQEAAHREZQWpty1ZsiSWLFkShx56aPTq1SvyPO/US57bRwoAAKpbqXmgrB6p119/PYYPHx4f+tCHYvTo0bFkyZKIiBg3bpylzwEAgC6vrCA1fvz46N69eyxevDi22mqrluPHHntsTJ06teT7zJo1K8aMGRMDBw6MLMvinnvuaXU+z/O4+OKLY/vtt49evXrFiBEj4o9//GOra5YvXx4nnHBC9OnTJ/r27Rvjxo2LVatWlfOzAAAASlJWkPrVr34V3/72t2PHHXdsdXyPPfaIP/3pTyXfZ/Xq1bHffvvFlClTNnn+iiuuiGuvvTZuuOGGmDt3bnzgAx+II444It58882Wa0444YR47rnnYtq0aXHffffFrFmz4vTTTy/nZwEAAJSkrB6p1atXt5qJetvy5cujtra25PuMGjUqRo0atclzeZ7HNddcExdddFF87nOfi4iIH//4x9G/f/+455574otf/GL8/ve/j6lTp8bjjz8eH/3oRyMi4rrrrovRo0fHVVddFQMHDkz7YdlbAwAAqE5t2SN1yCGHxI9//ON//K0si+bm5rjiiivi8MMPL+eWG1m0aFEsXbo0RowY0XKsrq4uhgwZEnPmzImIiDlz5kTfvn1bQlRExIgRI6JQKMTcuXM3e+/GxsZoaGhoNQAAAEpV1ozUFVdcEcOHD48nnngi1q5dG+eff34899xzsXz58pg9e/YWKWzp0qUREdG/f/9Wx/v3799ybunSpdGvX79W57t16xb19fUt12zKpEmT4rLLLtsidQIAANWnrBmpvffeO55//vkYNmxYfO5zn4vVq1fHUUcdFU899VTsvvvuW7rGLW7ixImxYsWKlvHyyy9XuiQAAKATKXsfqbq6uviP//iPLVlLKwMGDIiIiGXLlsX222/fcnzZsmWx//77t1zz2muvtfre+vXrY/ny5S3f35Ta2tpN93LpkQIAgOrWlj1SU6dOjUceeaTl85QpU2L//feP448/Pv72t7+Vc8uN7LrrrjFgwICYMWNGy7GGhoaYO3duDB06NCIihg4dGm+88UY8+eSTLdc8+OCD0dzcHEOGDNkidQAAALxTWUHq//7f/9uyQMMzzzwTEyZMiNGjR8eiRYtiwoQJJd9n1apVMX/+/Jg/f35EbFhgYv78+bF48eLIsizOO++8+OY3vxn33ntvPPPMM3HSSSfFwIED48gjj4yIiA9/+MMxcuTIOO200+Kxxx6L2bNnx9lnnx1f/OIX01fsAwAAKFFZr/YtWrQoPvKRj0RExP/+7//GmDFj4j//8z9j3rx5MXr06JLv88QTT7Ra5e/tEHbyySfHTTfdFOeff36sXr06Tj/99HjjjTdi2LBhMXXq1OjZs2fLd2655ZY4++yzY/jw4VEoFGLs2LFx7bXXlvOzAAAASlJWkOrRo0esWbMmIiKmT58eJ510UkRE1NfXJy0lfthhh0We55s9n2VZXH755XH55Zdv9pr6+vq49dZbS/6b7ybP8sgLm68HAADo2vKstDxQVpAaNmxYTJgwIT7+8Y/HY489FnfccUdERDz//POx4447lnNLAACATqOsHqnrr78+unXrFj/72c/i+9//fuywww4REfHAAw/EyJEjt2iBAAAAHU1ZM1I77bRT3HfffRsd/+53v9vq8+TJk+PMM8+Mvn37llUcAABAR1T2PlKl+M///M845phjOk+Qso8UAABUt7bcR6pU77aQBAAAQGfVpkEKAACgK2rTV/s6Ha/2AQBAdesIr/YBAAB0RYIUAABAojYNUoccckj06tWrLf8EAABAuyurR+qmm26KU045ZaPj69evj69//esxadKkiIj4xS9+8b6Ka295IY+8YKVBAACoVqXmgbJmpM4999z4whe+EH/7299aji1YsCCGDBkSt912Wzm3BAAA6DTKClJPPfVUvPLKK7HPPvvEtGnTYsqUKTF48ODYa6+94re//e2WrhEAAKBDKevVvt133z1mz54d5513XowcOTJqamri5ptvjuOOO25L1wcAANDhlL2P1P333x+33357DB06NJ5//vm48cYb4xOf+EQMHDhwS9bXvuwjBQAA1a0t95E644wz4gtf+EJccMEF8fDDD8fTTz8dPXr0iH322SfuvPPOcm4JAADQaZQ1IzV79uyYO3du7LfffhERMWDAgPjFL34RU6ZMiS9/+ctxzDHHbNEiAQAAOpKygtSTTz4ZtbW1Gx0/66yzYsSIEe+7KAAAgI6srCC1qRD1tj333LPsYipOjxQAAFS3EvNA2YtN/OxnP4s777wzFi9eHGvXrm11bt68eeXeFgAAoMMra7GJa6+9Nk499dTo379/PPXUU3HQQQfFNttsEy+++GKMGjVqS9cIAADQoZQVpL73ve/FD37wg7juuuuiR48ecf7558e0adPi3HPPjRUrVmzpGgEAADqUsoLU4sWL4+CDD46IiF69esXKlSsjIuLEE0+M2267bctVBwAA0AGVFaQGDBgQy5cvj4iInXbaKR599NGIiFi0aFHkeb7lqgMAAOiAygpSn/zkJ+Pee++NiIhTTz01xo8fH5/61Kfi2GOPjc9//vNbtEAAAICOpqxV+37wgx9Ec3NzRGzYO2rbbbeN2bNnx2c/+9k488wzt2iBAAAAHU1ZQapQKMTatWtj3rx58dprr0WvXr1aNuKdOnVqjBkzZosW2W4K+YYBAABUpxLzQFlBaurUqXHiiSfG66+/vtG5LMuiqampnNsCAAB0CmX1SJ1zzjlxzDHHxJIlS6K5ubnVEKIAAICurqwZqWXLlsWECROif//+W7qeysreGgAAQHUqMQ+UNSN19NFHx8yZM8v5KgAAQKdX1ozU9ddfH1/4whfi4Ycfjn322Se6d+/e6vy55567RYoDAADoiMoKUrfddlv86le/ip49e8bMmTMjy/4x/5VlmSAFAAB0aWUFqf/4j/+Iyy67LC688MIoFMp6O7BjyvINAwAAqE4l5oGyUtDatWvj2GOP7VohCgAAoERlJaGTTz457rjjji1dCwAAQKdQ1qt9TU1NccUVV8Qvf/nL2HfffTdabOI73/nOFikOAACgIyorSD3zzDNxwAEHRETEs88+2+pc8cITnU4hypyjAwAAuoQS80BZQerXv/51OV8DAADoEsy/AAAAJBKkAAAAEpX1al+XZR8pAACobm25jxQAAEA1E6QAAAASCVIAAACJ9EgVybINAwAAqE6l5gEzUgAAAIkEKQAAgESCFAAAQCI9UsXsIwUAANXNPlIAAABtQ5ACAABIJEgBAAAk0iNVJMvyyAp6pAAAoFpleqQAAADahiAFAACQyKt9xbK3BgAAUJ1KzANmpAAAABIJUgAAAIkEKQAAgER6pIpk2YYBAABUp1LzgBkpAACARIIUAABAIkEKAAAgkR6pIlmWR5bllS4DAACokFLzgBkpAACARIIUAABAIkEKAAAgkR6pInqkAACguumRAgAAaCOCFAAAQCJBCgAAIJEeqWJZvmEAAADVSY8UAABA2xCkAAAAEglSAAAAifRIFcmyDQMAAKhOpeYBM1IAAACJBCkAAIBEghQAAEAiPVJFskIehYJ9pAAAoFrlJeYBM1IAAACJBCkAAIBEXu0rkmV5ZJlX+wAAoFqVmgfMSAEAACSqaJCaNWtWjBkzJgYOHBhZlsU999zTcm7dunVxwQUXxD777BMf+MAHYuDAgXHSSSfFq6++2uoeu+yyS2RZ1mpMnjy5nX8JAABQTSoapFavXh377bdfTJkyZaNza9asiXnz5sXXv/71mDdvXtx1112xYMGC+OxnP7vRtZdffnksWbKkZZxzzjntUT4AAFClKtojNWrUqBg1atQmz9XV1cW0adNaHbv++uvjoIMOisWLF8dOO+3Ucrx3794xYMCA911PlpX+TiQAAND1ZFlp13WqHqkVK1ZElmXRt2/fVscnT54c22yzTRxwwAFx5ZVXxvr169/1Po2NjdHQ0NBqAAAAlKrTrNr35ptvxgUXXBDHHXdc9OnTp+X4ueeeG4MHD476+vr4zW9+ExMnTowlS5bEd77znc3ea9KkSXHZZZe1R9kAAEAXlOV53iHeZcuyLO6+++448sgjNzq3bt26GDt2bLzyyisxc+bMVkHqnX74wx/GGWecEatWrYra2tpNXtPY2BiNjY0tnxsaGmLQoEGxx08ujJqtNv0dAACg62ta0xh//NLkWLFixbvmjg4/I7Vu3bo45phj4k9/+lM8+OCD7/pjIiKGDBkS69evj5deein23HPPTV5TW1u7yZBVKORRKHSIXAkAAFRAXmIe6NBB6u0Q9cc//jF+/etfxzbbbPOe35k/f34UCoXo169fO1QIAABUo4oGqVWrVsULL7zQ8nnRokUxf/78qK+vj+233z6OPvromDdvXtx3333R1NQUS5cujYiI+vr66NGjR8yZMyfmzp0bhx9+ePTu3TvmzJkT48ePjy996Uux9dZbV+pnAQAAXVxFg9QTTzwRhx9+eMvnCRMmRETEySefHJdeemnce++9ERGx//77t/rer3/96zjssMOitrY2br/99rj00kujsbExdt111xg/fnzLfQAAANpCRYPUYYcdFu+21sV7rYMxePDgePTRR7dYPVmW20cKAACqWKl5oFPtIwUAANARCFIAAACJBCkAAIBEHXr58/aWvTUAAIDqVGoeMCMFAACQSJACAABIJEgBAAAk0iNVpJDlUbCPFAAAVK3cPlIAAABtQ5ACAABI5NW+IoVCcxQKzZUuAwAAqJC8xDxgRgoAACCRIAUAAJBIkAIAAEikR6pIIdswAACA6pSXmAfMSAEAACQSpAAAABIJUgAAAIn0SBXJsjyyLK90GQAAQIWUmgfMSAEAACQSpAAAABIJUgAAAIn0SBWpyZqjptBc6TIAAIBKyUrLA2akAAAAEglSAAAAiQQpAACARHqkimRZHgX7SAEAQNVqto8UAABA2xCkAAAAEglSAAAAifRIFcneGgAAQHUqNQ+YkQIAAEgkSAEAACQSpAAAABLpkSpSsI8UAABUtVLzgBkpAACARIIUAABAIq/2FakpNEdNobnSZQAAABWSl5gHzEgBAAAkEqQAAAASCVIAAACJ9EgVsfw5AABUN8ufAwAAtBFBCgAAIJEgBQAAkEiPVBE9UgAAUN30SAEAALQRQQoAACCRIAUAAJBIj1SRmqw5arLmSpcBAABUSF5iHjAjBQAAkEiQAgAASCRIAQAAJNIjVcQ+UgAAUN3sIwUAANBGBCkAAIBEghQAAEAiPVJFsqz0dyIBAICuJ8tKu86MFAAAQCJBCgAAIJEgBQAAkEiPVBH7SAEAQHWzjxQAAEAbEaQAAAASebWvSLesKbplTZUuAwAAqJC8xDxgRgoAACCRIAUAAJBIkAIAAEikR6qI5c8BAKC6Wf4cAACgjQhSAAAAiQQpAACARHqkiuiRAgCA6qZHCgAAoI0IUgAAAIkEKQAAgER6pIp0y5qjW9Zc6TIAAIAKyUvMA2akAAAAEglSAAAAiQQpAACARHqkithHCgAAqpt9pAAAANqIIAUAAJBIkAIAAEikR6pIIWuOgn2kAACgapWaByo6IzVr1qwYM2ZMDBw4MLIsi3vuuafV+VNOOSWyLGs1Ro4c2eqa5cuXxwknnBB9+vSJvn37xrhx42LVqlXt+CsAAIBqU9EgtXr16thvv/1iypQpm71m5MiRsWTJkpZx2223tTp/wgknxHPPPRfTpk2L++67L2bNmhWnn356W5cOAABUsYq+2jdq1KgYNWrUu15TW1sbAwYM2OS53//+9zF16tR4/PHH46Mf/WhERFx33XUxevTouOqqq2LgwIFJ9RQijxrLnwMAQNVqii6y/PnMmTOjX79+seeee8ZXv/rVeP3111vOzZkzJ/r27dsSoiIiRowYEYVCIebOnbvZezY2NkZDQ0OrAQAAUKoOHaRGjhwZP/7xj2PGjBnx7W9/Ox566KEYNWpUNDU1RUTE0qVLo1+/fq2+061bt6ivr4+lS5du9r6TJk2Kurq6ljFo0KA2/R0AAEDX0qFX7fviF7/Y8s/77LNP7LvvvrH77rvHzJkzY/jw4WXfd+LEiTFhwoSWzw0NDcIUAABQsg4dpN5pt912i2233TZeeOGFGD58eAwYMCBee+21VtesX78+li9fvtm+qogNfVe1tbUbHe+WNUc3y58DAEDVau4My5+neuWVV+L111+P7bffPiIihg4dGm+88UY8+eSTLdc8+OCD0dzcHEOGDKlUmQAAQBdX0RmpVatWxQsvvNDyedGiRTF//vyor6+P+vr6uOyyy2Ls2LExYMCAWLhwYZx//vnxwQ9+MI444oiIiPjwhz8cI0eOjNNOOy1uuOGGWLduXZx99tnxxS9+MXnFPgAAgFJVdEbqiSeeiAMOOCAOOOCAiIiYMGFCHHDAAXHxxRdHTU1NPP300/HZz342PvShD8W4cePiwAMPjIcffrjVa3m33HJL7LXXXjF8+PAYPXp0DBs2LH7wgx9U6icBAABVIMvzvOo3TmpoaIi6uro46dfHRo9/6lHpcgAAgApZu2pt/PjwO2LFihXRp0+fzV7XqXqkAAAAOgJBCgAAIJEgBQAAkKhT7SPV1mqyPGqyqm8ZAwCAqlVqHjAjBQAAkEiQAgAASCRIAQAAJNIjVaRb1hzdsqZKlwEAAFRIc9Zc0nVmpAAAABIJUgAAAIkEKQAAgER6pIoUsjwK9pECAICqVWoeMCMFAACQSJACAABIJEgBAAAk0iNVpBDNUROlrRsPAAB0PYUS84AZKQAAgESCFAAAQCKv9hWx/DkAAFQ3y58DAAC0EUEKAAAgkSAFAACQSI9Uke5ZU3TPZEsAAKhWzVlTSddJDQAAAIkEKQAAgESCFAAAQCI9UkXsIwUAANXNPlIAAABtRJACAABIJEgBAAAk0iNVpCZrjpqsudJlAAAAFVJqHjAjBQAAkEiQAgAASCRIAQAAJNIjVaRb1hTdM9kSAACqVVPWVNJ1UgMAAEAiQQoAACCRIAUAAJBIj1SRQuRRiLzSZQAAABVSah4wIwUAAJBIkAIAAEgkSAEAACTSI1WkJmuOmqy50mUAAAAVUmoeMCMFAACQSJACAABI5NW+IoUsj4JX+wAAoGoVMsufAwAAtAlBCgAAIJEgBQAAkEiPVJHuWVN0z7JKlwEAAFRIU9ZU0nVmpAAAABIJUgAAAIkEKQAAgER6pIrURB41Udq68QAAQNdTah4wIwUAAJBIkAIAAEgkSAEAACTSI1Uky5qjkDVXugwAAKBCshLzgBkpAACARIIUAABAIkEKAAAgkR6pIt2zpuieZZUuAwAAqJD1WVNJ15mRAgAASCRIAQAAJBKkAAAAEumRKlITzVET9pECAIBqVWoeMCMFAACQSJACAABI5NW+IjVZc9RkXu0DAIBqVWoeMCMFAACQSJACAABIJEgBAAAk0iNVpBDNUbD8OQAAVK1S84AZKQAAgESCFAAAQCJBCgAAIJEeqSLds+bokTVVugwAAKBC1ttHCgAAoG0IUgAAAIkEKQAAgER6pIoUojkKWVbpMgAAgAqxjxQAAEAbEaQAAAASCVIAAACJ9EgVqYk8aiKvdBkAAECFlJoHzEgBAAAkqmiQmjVrVowZMyYGDhwYWZbFPffc0+p8lmWbHFdeeWXLNbvssstG5ydPntzOvwQAAKgmFQ1Sq1evjv322y+mTJmyyfNLlixpNX74wx9GlmUxduzYVtddfvnlra4755xz2qN8AACgSlW0R2rUqFExatSozZ4fMGBAq88///nP4/DDD4/ddtut1fHevXtvdO27aWxsjMbGxpbPDQ0NERHRPVsf3TM9UgAAUK26Z00lXddpeqSWLVsW999/f4wbN26jc5MnT45tttkmDjjggLjyyitj/fr173qvSZMmRV1dXcsYNGhQW5UNAAB0QZ1m1b6bb745evfuHUcddVSr4+eee24MHjw46uvr4ze/+U1MnDgxlixZEt/5znc2e6+JEyfGhAkTWj43NDQIUwAAQMk6TZD64Q9/GCeccEL07Nmz1fHiQLTvvvtGjx494owzzohJkyZFbW3tJu9VW1u72XMAAADvpVMEqYcffjgWLFgQd9xxx3teO2TIkFi/fn289NJLseeeeyb9nZqsOWqyrNwyAQCATq4may7puk7RI3XjjTfGgQceGPvtt997Xjt//vwoFArRr1+/dqgMAACoRhWdkVq1alW88MILLZ8XLVoU8+fPj/r6+thpp50iYkP/0k9/+tO4+uqrN/r+nDlzYu7cuXH44YdH7969Y86cOTF+/Pj40pe+FFtvvXW7/Q4AAKC6VDRIPfHEE3H44Ye3fH673+nkk0+Om266KSIibr/99sjzPI477riNvl9bWxu33357XHrppdHY2Bi77rprjB8/vlXfVIqayKMmLH8OAADVqtQ8kOV5XvXJoaGhIerq6uKn8/eKrXrXVLocAACgQtasbIov7P+HWLFiRfTp02ez13WKHikAAICORJACAABI1CmWP28vWTRHISx/DgAA1SqLLrT8OQAAQEciSAEAACQSpAAAABLpkSrSI2uKHlqkAACgaq3Lmkq6zowUAABAIkEKAAAgkSAFAACQSI9UkUKWRyErbd14AACg6ylkeWnXtXEdAAAAXY4gBQAAkEiQAgAASKRHqkhNNEdNpYsAAAAqpiZKWzPBjBQAAEAiQQoAACCRIAUAAJBIj1SR7llTdC9x3XgAAKDr6V7ivrJmpAAAABIJUgAAAIkEKQAAgER6pIoUIo+a0CMFAADVqlBiHjAjBQAAkEiQAgAASOTVviI1Xu0DAICqVmoeMCMFAACQSJACAABIJEgBAAAk0iNVpJDlUcj0SAEAQLUqNQ+YkQIAAEgkSAEAACQSpAAAABLpkSrSPZqje6WLAAAAKqZ7NJd0nRkpAACARIIUAABAIkEKAAAgkR6pIjVZHjX2kQIAgKpVah4wIwUAAJBIkAIAAEgkSAEAACTSI1WkJvKoCT1SAABQrUrNA2akAAAAEglSAAAAiQQpAACARHqkinTLIrpnla4CAAColG4l5gEzUgAAAIkEKQAAgERe7StSCMkSAACqWal5QG4AAABIJEgBAAAkEqQAAAAS6ZEqUvPWAAAAqlOpecCMFAAAQCJBCgAAIJEgBQAAkEiPVJGaLIuaLKt0GQAAQIWUmgfMSAEAACQSpAAAABIJUgAAAIn0SBXpHll0Dz1SAABQrUrNA2akAAAAEglSAAAAiQQpAACARHqkimRZFgX7SAEAQNXK7CMFAADQNgQpAACARIIUAABAIj1SRWoiixr7SAEAQNUqNQ+YkQIAAEgkSAEAACQSpAAAABLpkSrSPauJ7plsCQAA1aq7faQAAADahiAFAACQyKt9RQpv/R8AAFCdSk0DUgMAAEAiQQoAACCRIAUAAJBIj1SRQmRRiNKWOwQAALqeUvOAGSkAAIBEghQAAEAiQQoAACCRHqkiNVkhajLZEgAAqlVNiUsmSA0AAACJBCkAAIBEXu2LiDzPIyKiYVVzhSsBAAAq6e1M8HZG2BxBKiJWrlwZERE7D36psoUAAAAdwsqVK6Ourm6z57P8vaJWFWhubo5XX301evfuHVlmQ95SNDQ0xKBBg+Lll1+OPn36VLocujDPGu3Fs0Z78azRXjxr5cnzPFauXBkDBw6MQmHznVBmpCKiUCjEjjvuWOkyOqU+ffr4NybtwrNGe/Gs0V48a7QXz1q6d5uJepvFJgAAABIJUgAAAIkEKcpSW1sbl1xySdTW1la6FLo4zxrtxbNGe/Gs0V48a23LYhMAAACJzEgBAAAkEqQAAAASCVIAAACJBCkAAIBEghQlW758eZxwwgnRp0+f6Nu3b4wbNy5WrVpV0nfzPI9Ro0ZFlmVxzz33tG2hdHqpz9ry5cvjnHPOiT333DN69eoVO+20U5x77rmxYsWKdqyazmDKlCmxyy67RM+ePWPIkCHx2GOPvev1P/3pT2OvvfaKnj17xj777BO/+MUv2qlSOruUZ+2///u/45BDDomtt946tt566xgxYsR7PpvwttT/XHvb7bffHlmWxZFHHtm2BXZhghQlO+GEE+K5556LadOmxX333RezZs2K008/vaTvXnPNNZFlWRtXSFeR+qy9+uqr8eqrr8ZVV10Vzz77bNx0000xderUGDduXDtWTUd3xx13xIQJE+KSSy6JefPmxX777RdHHHFEvPbaa5u8/je/+U0cd9xxMW7cuHjqqafiyCOPjCOPPDKeffbZdq6czib1WZs5c2Ycd9xx8etf/zrmzJkTgwYNik9/+tPx5z//uZ0rp7NJfdbe9tJLL8XXvva1OOSQQ9qp0i4qhxL87ne/yyMif/zxx1uOPfDAA3mWZfmf//znd/3uU089le+www75kiVL8ojI77777jauls7s/Txrxe688868R48e+bp169qiTDqhgw46KD/rrLNaPjc1NeUDBw7MJ02atMnrjznmmPwzn/lMq2NDhgzJzzjjjDatk84v9Vl7p/Xr1+e9e/fOb7755rYqkS6inGdt/fr1+cEHH5z/z//8T37yySfnn/vc59qh0q7JjBQlmTNnTvTt2zc++tGPthwbMWJEFAqFmDt37ma/t2bNmjj++ONjypQpMWDAgPYolU6u3GftnVasWBF9+vSJbt26tUWZdDJr166NJ598MkaMGNFyrFAoxIgRI2LOnDmb/M6cOXNaXR8RccQRR2z2eogo71l7pzVr1sS6deuivr6+rcqkCyj3Wbv88sujX79+3trYAvx/GJRk6dKl0a9fv1bHunXrFvX19bF06dLNfm/8+PFx8MEHx+c+97m2LpEuotxnrdhf//rX+MY3vlHyq6d0fX/961+jqakp+vfv3+p4//794w9/+MMmv7N06dJNXl/qc0h1KudZe6cLLrggBg4cuFGQh2LlPGuPPPJI3HjjjTF//vx2qLDrMyNV5S688MLIsuxdR6n/wf9O9957bzz44INxzTXXbNmi6ZTa8lkr1tDQEJ/5zGfiIx/5SFx66aXvv3CAdjR58uS4/fbb4+67746ePXtWuhy6kJUrV8aJJ54Y//3f/x3bbrttpcvpEsxIVbl/+7d/i1NOOeVdr9ltt91iwIABGzUurl+/PpYvX77ZV/YefPDBWLhwYfTt27fV8bFjx8YhhxwSM2fOfB+V09m05bP2tpUrV8bIkSOjd+/ecffdd0f37t3fb9l0Edtuu23U1NTEsmXLWh1ftmzZZp+rAQMGJF0PEeU9a2+76qqrYvLkyTF9+vTYd99927JMuoDUZ23hwoXx0ksvxZgxY1qONTc3R8SGNz8WLFgQu+++e9sW3cUIUlVuu+22i+222+49rxs6dGi88cYb8eSTT8aBBx4YERuCUnNzcwwZMmST37nwwgvjK1/5Sqtj++yzT3z3u99t9W9iqkNbPmsRG2aijjjiiKitrY17773X/5JLKz169IgDDzwwZsyY0bLUb3Nzc8yYMSPOPvvsTX5n6NChMWPGjDjvvPNajk2bNi2GDh3aDhXTWZXzrEVEXHHFFfGtb30rfvnLX7bqEYXNSX3W9tprr3jmmWdaHbvoooti5cqV8V//9V8xaNCg9ii7a6n0ahd0HiNHjswPOOCAfO7cufkjjzyS77HHHvlxxx3Xcv6VV17J99xzz3zu3LmbvUdYtY8SpD5rK1asyIcMGZLvs88++QsvvJAvWbKkZaxfv75SP4MO5vbbb89ra2vzm266Kf/d736Xn3766Xnfvn3zpUuX5nme5yeeeGJ+4YUXtlw/e/bsvFu3bvlVV12V//73v88vueSSvHv37vkzzzxTqZ9AJ5H6rE2ePDnv0aNH/rOf/azVf36tXLmyUj+BTiL1WXsnq/a9P2akKNktt9wSZ599dgwfPjwKhUKMHTs2rr322pbz69atiwULFsSaNWsqWCVdQeqzNm/evJYV/T74wQ+2uteiRYtil112abfa6biOPfbY+Mtf/hIXX3xxLF26NPbff/+YOnVqS6P24sWLo1D4R+vwwQcfHLfeemtcdNFF8e///u+xxx57xD333BN77713pX4CnUTqs/b9738/1q5dG0cffXSr+1xyySV6PXlXqc8aW1aW53le6SIAAAA6ExEVAAAgkSAFAACQSJACAABIJEgBAAAkEqQAAAASCVIAAACJBCkAAIBEghQAAEAiQQqAqnDYYYfFeeed1+nuDUDH1K3SBQBAZ3fXXXdF9+7dK10GAO1IkAKA96m+vr7SJQDQzrzaB0CH09zcHJMmTYpdd901evXqFfvtt1/87Gc/i4iImTNnRpZl8ctf/jIOOOCA6NWrV3zyk5+M1157LR544IH48Ic/HH369Injjz8+1qxZ0+q+69evj7PPPjvq6upi2223ja9//euR53lJNX3ve9+LPfbYI3r27Bn9+/ePo48+uuVc8at9b9f3znHKKae0XP/zn/88Bg8eHD179ozddtstLrvssli/fv37+xcNgHZlRgqADmfSpEnxk5/8JG644YbYY489YtasWfGlL30ptttuu5ZrLr300rj++utjq622imOOOSaOOeaYqK2tjVtvvTVWrVoVn//85+O6666LCy64oOU7N998c4wbNy4ee+yxeOKJJ+L000+PnXbaKU477bR3reeJJ56Ic889N/7f//t/cfDBB8fy5cvj4Ycf3uS1Bx98cCxZsqTl8+9///sYPXp0HHrooRER8fDDD8dJJ50U1157bRxyyCGxcOHCOP300yMi4pJLLin7XzMA2leWl/o/xQFAO2hsbIz6+vqYPn16DB06tOX4V77ylVizZk2cfvrpcfjhh8f06dNj+PDhERExefLkmDhxYixcuDB22223iIg488wz46WXXoqpU6dGxIZZo9deey2ee+65yLIsIiIuvPDCuPfee+N3v/vdu9Z01113xamnnhqvvPJK9O7de6Pzhx12WOy///5xzTXXtDr++uuvx0EHHRQjR46MKVOmRETEiBEjYvjw4TFx4sSW637yk5/E+eefH6+++mriv1oAVIoZKQA6lBdeeCHWrFkTn/rUp1odX7t2bRxwwAEtn/fdd9+Wf+7fv39stdVWLSHq7WOPPfZYq3v8y7/8S0uIiogYOnRoXH311dHU1BQ1NTWbrelTn/pU7LzzzrHbbrvFyJEjY+TIkfH5z38+ttpqq81+Z926dTF27NjYeeed47/+679ajv/2t7+N2bNnx7e+9a2WY01NTfHmm2/GmjVr3vWeAHQcghQAHcqqVasiIuL++++PHXbYodW52traWLhwYUREq1XysizbaNW8LMuiubl5i9TUu3fvmDdvXsycOTN+9atfxcUXXxyXXnppPP7449G3b99NfuerX/1qvPzyy/HYY49Ft27/+K/bVatWxWWXXRZHHXXURt/p2bPnFqkXgLYnSAHQoXzkIx+J2traWLx4cXziE5/Y6PzbQaocc+fObfX50UcfjT322ONdZ6Pe1q1btxgxYkSMGDEiLrnkkujbt288+OCDmwxE3/nOd+LOO++M3/zmN7HNNtu0Ojd48OBYsGBBfPCDHyz7dwBQeYIUAB1K796942tf+1qMHz8+mpubY9iwYbFixYqYPXt29OnTJ3beeeey77148eKYMGFCnHHGGTFv3ry47rrr4uqrr37P7913333x4osvxqGHHhpbb711/OIXv4jm5ubYc889N7p2+vTpcf7558eUKVNi2223jaVLl0ZERK9evaKuri4uvvji+Nd//dfYaaed4uijj45CoRC//e1v49lnn41vfvObZf82ANqXIAVAh/ONb3wjtttuu5g0aVK8+OKL0bdv3xg8eHD8+7//+/t6Xe+kk06Kv//973HQQQdFTU1N/J//839aVsx7N3379o277rorLr300njzzTdjjz32iNtuuy3++Z//eaNrH3nkkWhqaoozzzwzzjzzzJbjJ598ctx0001xxBFHxH333ReXX355fPvb347u3bvHXnvtFV/5ylfK/l0AtD+r9gEAACSyIS8AAEAiQQqAqvfwww/HP/3TP212AMA7ebUPgKr397//Pf785z9v9rwV9gB4J0EKAAAgkVf7AAAAEglSAAAAiQQpAACARIIUAABAIkEKAAAgkSAFAACQSJACAABI9P8BsQHM0k+0a/gAAAAASUVORK5CYII=\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "code", + "source": [ + "fig, ax = plt.subplots(figsize=(10, 10))\n", + "plt.imshow(tmp_layer.pe[0, 50:51, :], aspect=\"auto\")\n", + "plt.xlabel(\"emb_size\")\n", + "plt.ylabel(\"max_seq_len\")\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 850 + }, + "id": "NDEk6GqycZRX", + "outputId": "fdf0ca04-be31-4076-818f-eedbdc176aaf" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1kAAANBCAYAAAAShHTFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA1v0lEQVR4nO3deZjWdb3/8dcMywwoi7iAC4gaJzRX5IcHl9PCpEQ/Oy6ppbkQaXYkl2lRTj/BMkVNzFTSn5462knNMjt5LEnEMPWQKKhpx6XjcnADMtJhUUDm/v3R5fyaZAYcPjDe8nhc11yX93e5v+/7vr6IT7/3/Z2aSqVSCQAAAEXUdvYAAAAA7yUiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgrp29gDvds3NzXnppZfSq1ev1NTUdPY4AABAJ6lUKlm8eHG22Wab1Na2fb1KZK3BSy+9lIEDB3b2GAAAwLvE888/n+22267N9SJrDXr16pUk+btxE9Ole/1qt7nztO+2uf9+P/lCu88/+OzZ7a6v3aRnu+uX/nDLNtdtOqFbu/u+2Xv1r+ctXf/7xXbXd7u+7ee/cacZ7e477Ppx7a7f4WdN7a7/x+/f3ea6mxsPbHff7i+82u76za9Z1O76P3267fft+nt+1e6+H76s/fPh9i9e3u76g393dJvrzhoyrd19/7BiQLvrFyzv3e76Q/vObXPd+f8zpt19r9zpp+2uP+yRE9pdf8deP2xz3QH3fbbdfWfv/2/trh92V/vHfvgj17W5bs9fjW1330dHt71vkuz2i/aP/ejHr21z3e63tn/s333iX9tdv/vP17D/P7a9/+4/a/89/92h3293/R5r2P+RdvZfl32TZI9b1rD/Ye0cex32dewqPPZP17Dv4Ws49jrs79iO7djv3mM3LWnO9sOea2mEtoisNXjrI4JdutenS93q/+O6d6+2LxXW1q8hZGraD6Hamu7t779JXdvrurT/3Om6htlq2z92t03afv723pNkLd6XLsvbXd9j07ZP3a5rel1d2n7PkqTbJmt4z9t5X9b0uts6h9Z6/55tz96zV5d2961f3v4f97pu7Z8vm7QzW3vnYZL0WofXlazhz1jPdXtPa3t0fP912XejPvYa/vyvy79THduxHduxHdux1/exk6zxa0RufAEAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFVU1kLVq0KMccc0x69+6dvn37Zty4cVmyZMla7VupVPKxj30sNTU1+fd///f1OygAALBRq5rIOuaYY/L73/8+06dPz2233Zbf/OY3Oemkk9Zq30svvTQ1NTXreUIAAICka2cPsDYef/zxTJs2LQ888ECGDx+eJLn88sszZsyYXHzxxdlmm23a3Pfhhx/OlClT8uCDD2brrbfeUCMDAAAbqaq4kjVr1qz07du3JbCSpKGhIbW1tbn//vvb3G/ZsmU5+uijM3Xq1AwYMGBDjAoAAGzkquJK1vz587PVVlu1Wta1a9f069cv8+fPb3O/M844I/vuu2/+8R//ca2PtXz58ixfvrzlcVNT0zsfGAAA2Gh16pWss846KzU1Ne3+PPHEEx167ltvvTV33XVXLr300ne03+TJk9OnT5+Wn4EDB3bo+AAAwMapU69kfelLX8oJJ5zQ7jY77rhjBgwYkIULF7Za/uabb2bRokVtfgzwrrvuytNPP52+ffu2Wn744YfngAMOyMyZM1e734QJE9LY2NjyuKmpSWgBAABrrVMja8stt8yWW265xu1GjhyZV199NXPmzMnee++d5C8R1dzcnH322We1+5x11ln53Oc+12rZbrvtlm9/+9s5+OCD2zxWXV1d6urq3sGrAAAA+P+q4jtZO++8c0aPHp0TTzwxV111VVauXJnx48fnU5/6VMudBV988cWMGjUqP/jBDzJixIgMGDBgtVe5Bg0alB122GFDvwQAAGAjURV3F0yS66+/PkOHDs2oUaMyZsyY7L///rn66qtb1q9cuTJPPvlkli1b1olTAgAAG7uquJKVJP369csNN9zQ5vrBgwenUqm0+xxrWg8AALCuquZKFgAAQDUQWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAACioaiJr0aJFOeaYY9K7d+/07ds348aNy5IlS9rd/otf/GLe//73p0ePHhk0aFBOPfXUvPbaaxtwagAAYGNTNZF1zDHH5Pe//32mT5+e2267Lb/5zW9y0kkntbn9Sy+9lJdeeikXX3xxHnvssVx77bWZNm1axo0btwGnBgAANjZdO3uAtfH4449n2rRpeeCBBzJ8+PAkyeWXX54xY8bk4osvzjbbbPO2fXbdddf89Kc/bXm800475bzzzstnPvOZvPnmm+natSpeOgAAUGWq4krWrFmz0rdv35bASpKGhobU1tbm/vvvX+vnee2119K7d2+BBQAArDdVURvz58/PVltt1WpZ165d069fv8yfP3+tnuOVV17Jueee2+5HDJNk+fLlWb58ecvjpqamdz4wAACw0erUK1lnnXVWampq2v154okn1vk4TU1N+fjHP55ddtkl55xzTrvbTp48OX369Gn5GThw4DofHwAA2Hh06pWsL33pSznhhBPa3WbHHXfMgAEDsnDhwlbL33zzzSxatCgDBgxod//Fixdn9OjR6dWrV372s5+lW7du7W4/YcKENDY2tjxuamoSWgAAwFrr1Mjacssts+WWW65xu5EjR+bVV1/NnDlzsvfeeydJ7rrrrjQ3N2efffZpc7+mpqYcdNBBqaury6233pr6+vo1Hquuri51dXVr/yIAAAD+SlXc+GLnnXfO6NGjc+KJJ2b27Nm57777Mn78+HzqU59qubPgiy++mKFDh2b27NlJ/hJYBx54YJYuXZrvfe97aWpqyvz58zN//vysWrWqM18OAADwHlYVN75Ikuuvvz7jx4/PqFGjUltbm8MPPzyXXXZZy/qVK1fmySefzLJly5Ikc+fObbnz4Pve975Wz/Xss89m8ODBG2x2AABg41E1kdWvX7/ccMMNba4fPHhwKpVKy+MPfehDrR4DAABsCFXxcUEAAIBqIbIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAV17chOS5cuzQUXXJAZM2Zk4cKFaW5ubrX+mWeeKTIcAABAtelQZH3uc5/L3XffnWOPPTZbb711ampqSs8FAABQlToUWbfffnt+8YtfZL/99is9DwAAQFXr0HeyNttss/Tr16/0LAAAAFWvQ5F17rnnZuLEiVm2bFnpeQAAAKpahz4uOGXKlDz99NPp379/Bg8enG7durVaP3fu3CLDAQAAVJsORdYhhxxSeAwAAID3hg5F1qRJk0rPAQAA8J7Q4V9G/Oqrr+Zf/uVfMmHChCxatCjJXz4m+OKLLxYbDgAAoNp06ErW7373uzQ0NKRPnz557rnncuKJJ6Zfv3655ZZbMm/evPzgBz8oPScAAEBV6NCVrMbGxpxwwgn5wx/+kPr6+pblY8aMyW9+85tiwwEAAFSbDkXWAw88kM9//vNvW77ttttm/vz56zwUAABAtepQZNXV1aWpqelty5966qlsueWW6zwUAABAtepQZH3iE5/IN77xjaxcuTJJUlNTk3nz5uXMM8/M4YcfXnRAAACAatKhyJoyZUqWLFmSrbbaKq+//no++MEP5n3ve1969eqV8847r/SMAAAAVaNDdxfs06dPpk+fnnvvvTe/+93vsmTJkgwbNiwNDQ2l5wMAAKgqHYqst+y///7Zf//9S80CAABQ9dY6si677LK1ftJTTz21Q8MAAABUu7WOrG9/+9trtV1NTY3IAgAANlprHVnPPvvs+pwDAADgPaFDdxdcW717984zzzyzPg8BAADwrrJeI6tSqazPpwcAAHjXWa+RBQAAsLERWQAAAAWJLAAAgILWa2TV1NSsz6cHAAB413HjCwAAgILWa2Tdfvvt2XbbbdfnIQAAAN5V1vqXEf+1xsbGtd52//3378ghAAAAqlKHIuuhhx7KQw89lJUrV+b9739/kuSpp55Kly5dMmzYsJbtfCcLAADY2HQosg4++OD06tUr1113XTbbbLMkyZ///OeMHTs2BxxwQL70pS8VHRIAAKBadOg7WVOmTMnkyZNbAitJNttss3zzm9/MlClTig0HAABQbToUWU1NTfnjH//4tuV//OMfs3jx4nUeCgAAoFp1KLIOPfTQjB07NrfcckteeOGFvPDCC/npT3+acePG5bDDDis9IwAAQNXo0Heyrrrqqnz5y1/O0UcfnZUrV/7libp2zbhx4/Ktb32r6IAAAADVpEOR1bNnz3z3u9/Nt771rTz99NNJkp122imbbLJJ0eEAAACqzTr9MuKXX345L7/8coYMGZJNNtkklUql1FwAAABVqUOR9ac//SmjRo3K3/3d32XMmDF5+eWXkyTjxo1b77dvnzp1agYPHpz6+vrss88+mT17drvb/+QnP8nQoUNTX1+f3XbbLb/85S/X63wAAMDGrUORdcYZZ6Rbt26ZN29eevbs2bL8qKOOyrRp04oN97duuummNDY2ZtKkSZk7d2722GOPHHTQQVm4cOFqt//P//zPfPrTn864cePy0EMP5ZBDDskhhxySxx57bL3NCAAAbNw6FFl33HFHLrzwwmy33Xatlg8ZMiT/8z//U2Sw1bnkkkty4oknZuzYsdlll11y1VVXpWfPnvn+97+/2u2/853vZPTo0fnKV76SnXfeOeeee26GDRuWK664Yr3NCAAAbNw6FFlLly5tdQXrLYsWLUpdXd06D7U6K1asyJw5c9LQ0NCyrLa2Ng0NDZk1a9Zq95k1a1ar7ZPkoIMOanP7JFm+fHmamppa/QAAAKytDkXWAQcckB/84Actj2tqatLc3JyLLrooH/7wh4sN99deeeWVrFq1Kv3792+1vH///pk/f/5q95k/f/472j5JJk+enD59+rT8DBw4cN2HBwAANhoduoX7RRddlFGjRuXBBx/MihUr8tWvfjW///3vs2jRotx3332lZ9ygJkyYkMbGxpbHTU1NQgsAAFhrHYqsXXfdNU899VSuuOKK9OrVK0uWLMlhhx2WU045JVtvvXXpGZMkW2yxRbp06ZIFCxa0Wr5gwYIMGDBgtfsMGDDgHW2fJHV1devtI48AAMB7X4ciK0n69OmTr33tayVnaVf37t2z9957Z8aMGTnkkEOSJM3NzZkxY0bGjx+/2n1GjhyZGTNm5PTTT29ZNn369IwcOXIDTAwAAGyMOvSdrGnTpuXee+9teTx16tTsueeeOfroo/PnP/+52HB/q7GxMddcc02uu+66PP744/nCF76QpUuXZuzYsUmS4447LhMmTGjZ/rTTTsu0adMyZcqUPPHEEznnnHPy4IMPthllAAAA66pDkfWVr3yl5a57jz76aBobGzNmzJg8++yzrb7PVNpRRx2Viy++OBMnTsyee+6Zhx9+ONOmTWu5ucW8efNafjFykuy777654YYbcvXVV2ePPfbIzTffnH//93/Prrvuut5mBAAANm4d+rjgs88+m1122SVJ8tOf/jQHH3xwzj///MydOzdjxowpOuDfGj9+fJtXombOnPm2ZUcccUSOOOKI9ToTAADAWzp0Jat79+5ZtmxZkuTOO+/MgQcemCTp16+f3ysFAABs1Dp0JWv//fdPY2Nj9ttvv8yePTs33XRTkuSpp57KdtttV3RAAACAatKhK1lXXHFFunbtmptvvjlXXnlltt122yTJ7bffntGjRxcdEAAAoJp06ErWoEGDctttt71t+be//e1Wjy+44IKcfPLJ6du3b4eGAwAAqDYdupK1ts4///wsWrRofR4CAADgXWW9RlalUlmfTw8AAPCus14jCwAAYGMjsgAAAAoSWQAAAAWJLAAAgILWa2QdcMAB6dGjx/o8BAAAwLtKhyLr2muvXe3yN998MxMmTGh5/Mtf/jJbb711hwYDAACoRh2KrFNPPTVHHHFE/vznP7cse/LJJ7PPPvvkxhtvLDYcAABAtelQZD300EN54YUXsttuu2X69OmZOnVqhg0blqFDh+aRRx4pPSMAAEDV6NqRnXbaaafcd999Of300zN69Oh06dIl1113XT796U+Xng8AAKCqdPjGF7/4xS/yox/9KCNHjkzfvn3zve99Ly+99FLJ2QAAAKpOhyLr85//fI444oiceeaZueeee/K73/0u3bt3z2677ZYf//jHpWcEAACoGh36uOB9992X+++/P3vssUeSZMCAAfnlL3+ZqVOn5rOf/WyOPPLIokMCAABUiw5F1pw5c1JXV/e25aecckoaGhrWeSgAAIBq1aGPC64usN7y/ve/v8PDAAAAVLsOXclKkptvvjk//vGPM2/evKxYsaLVurlz567zYAAAANWoQ1eyLrvssowdOzb9+/fPQw89lBEjRmTzzTfPM888k4997GOlZwQAAKgaHYqs7373u7n66qtz+eWXp3v37vnqV7+a6dOn59RTT81rr71WekYAAICq0aHImjdvXvbdd98kSY8ePbJ48eIkybHHHpsbb7yx3HQAAABVpkORNWDAgCxatChJMmjQoPz2t79Nkjz77LOpVCrlpgMAAKgyHYqsj3zkI7n11luTJGPHjs0ZZ5yRj370oznqqKNy6KGHFh0QAACgmnTo7oJXX311mpubk/zld2NtscUWue+++/KJT3wiJ598ctEBAQAAqkmHIqu2tjYrVqzI3Llzs3DhwvTo0aPllxBPmzYtBx98cNEhAQAAqkWHImvatGk59thj86c//elt62pqarJq1ap1HgwAAKAadeg7WV/84hdz5JFH5uWXX05zc3OrH4EFAABszDoUWQsWLEhjY2P69+9feh4AAICq1qHI+uQnP5mZM2cWHgUAAKD6deg7WVdccUWOOOKI3HPPPdltt93SrVu3VutPPfXUIsMBAABUmw5F1o033pg77rgj9fX1mTlzZmpqalrW1dTUiCwAAGCj1aHI+trXvpavf/3rOeuss1Jb26FPHAIAALwndaiQVqxYkaOOOkpgAQAA/I0OVdLxxx+fm266qfQsAAAAVa9DHxdctWpVLrroovzqV7/K7rvv/rYbX1xyySVFhgMAAKg2HYqsRx99NHvttVeS5LHHHmu17q9vggEAALCx6VBk/frXvy49BwAAwHuCO1cAAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKCgqousqVOnZvDgwamvr88+++yT2bNnt7ntNddckwMOOCCbbbZZNttsszQ0NLS7PQAAwLqqqsi66aab0tjYmEmTJmXu3LnZY489ctBBB2XhwoWr3X7mzJn59Kc/nV//+teZNWtWBg4cmAMPPDAvvvjiBp4cAADYWFRVZF1yySU58cQTM3bs2Oyyyy656qqr0rNnz3z/+99f7fbXX399/umf/il77rlnhg4dmn/5l39Jc3NzZsyYsYEnBwAANhZVE1krVqzInDlz0tDQ0LKstrY2DQ0NmTVr1lo9x7Jly7Jy5cr069dvfY0JAABs5Lp29gBr65VXXsmqVavSv3//Vsv79++fJ554Yq2e48wzz8w222zTKtT+1vLly7N8+fKWx01NTR0bGAAA2ChVzZWsdXXBBRfkRz/6UX72s5+lvr6+ze0mT56cPn36tPwMHDhwA04JAABUu6qJrC222CJdunTJggULWi1fsGBBBgwY0O6+F198cS644ILccccd2X333dvddsKECXnttddafp5//vl1nh0AANh4VE1kde/ePXvvvXerm1a8dROLkSNHtrnfRRddlHPPPTfTpk3L8OHD13icurq69O7du9UPAADA2qqa72QlSWNjY44//vgMHz48I0aMyKWXXpqlS5dm7NixSZLjjjsu2267bSZPnpwkufDCCzNx4sTccMMNGTx4cObPn58k2XTTTbPpppt22usAAADeu6oqso466qj88Y9/zMSJEzN//vzsueeemTZtWsvNMObNm5fa2v9/ce7KK6/MihUr8slPfrLV80yaNCnnnHPOhhwdAADYSFRVZCXJ+PHjM378+NWumzlzZqvHzz333PofCAAA4K9UzXeyAAAAqoHIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEFVF1lTp07N4MGDU19fn3322SezZ89eq/1+9KMfpaamJocccsj6HRAAANioVVVk3XTTTWlsbMykSZMyd+7c7LHHHjnooIOycOHCdvd77rnn8uUvfzkHHHDABpoUAADYWFVVZF1yySU58cQTM3bs2Oyyyy656qqr0rNnz3z/+99vc59Vq1blmGOOyde//vXsuOOOG3BaAABgY1Q1kbVixYrMmTMnDQ0NLctqa2vT0NCQWbNmtbnfN77xjWy11VYZN27chhgTAADYyHXt7AHW1iuvvJJVq1alf//+rZb3798/TzzxxGr3uffee/O9730vDz/88FofZ/ny5Vm+fHnL46ampg7NCwAAbJyq5krWO7V48eIce+yxueaaa7LFFlus9X6TJ09Onz59Wn4GDhy4HqcEAADea6rmStYWW2yRLl26ZMGCBa2WL1iwIAMGDHjb9k8//XSee+65HHzwwS3LmpubkyRdu3bNk08+mZ122ult+02YMCGNjY0tj5uamoQWAACw1qomsrp375699947M2bMaLkNe3Nzc2bMmJHx48e/bfuhQ4fm0UcfbbXs//yf/5PFixfnO9/5TpvhVFdXl7q6uuLzAwAAG4eqiawkaWxszPHHH5/hw4dnxIgRufTSS7N06dKMHTs2SXLcccdl2223zeTJk1NfX59dd9211f59+/ZNkrctBwAAKKWqIuuoo47KH//4x0ycODHz58/PnnvumWnTprXcDGPevHmprX3Pfs0MAACoAlUVWUkyfvz41X48MElmzpzZ7r7XXntt+YEAAAD+iss+AAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKKhrZw/wblepVJIkq1a80eY2TYub21zX/Ebb+yXJm5WV7a6vraxof/+ly9tet6rtuZLkzTdr2l2f5vaPXbO00ua69t6TZC3el1Vtv64keX3Jm23v+2b7z127hudeuXQN73lz2/9vYk2ve9Xy9mdb4/7L2p592eJV7e77xoq237MkWb68/XNxade2Z2vvPEySxevwupI1/Blbtm7vafPrHd9/XfbdqI+9hj//6/LvVMd2bMd2bMd27PV57KYlf1n3ViO0paaypi02ci+88EIGDhzY2WMAAADvEs8//3y22267NteLrDVobm7OSy+9lF69eqWmpiZNTU0ZOHBgnn/++fTu3buzx+M9zLnGhuJcY0NxrrGhONdYXyqVShYvXpxtttkmtbVtf7rJxwXXoLa2drWV2rt3b39o2SCca2wozjU2FOcaG4pzjfWhT58+a9zGjS8AAAAKElkAAAAFiax3qK6uLpMmTUpdXV1nj8J7nHONDcW5xobiXGNDca7R2dz4AgAAoCBXsgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEi6x2YOnVqBg8enPr6+uyzzz6ZPXt2Z49ElZs8eXL+1//6X+nVq1e22mqrHHLIIXnyySdbbfPGG2/klFNOyeabb55NN900hx9+eBYsWNBJE/NeccEFF6Smpiann356yzLnGqW8+OKL+cxnPpPNN988PXr0yG677ZYHH3ywZX2lUsnEiROz9dZbp0ePHmloaMgf/vCHTpyYarRq1aqcffbZ2WGHHdKjR4/stNNOOffcc/PX93RzrtFZRNZauummm9LY2JhJkyZl7ty52WOPPXLQQQdl4cKFnT0aVezuu+/OKaeckt/+9reZPn16Vq5cmQMPPDBLly5t2eaMM87If/zHf+QnP/lJ7r777rz00ks57LDDOnFqqt0DDzyQ//t//2923333Vsuda5Tw5z//Ofvtt1+6deuW22+/Pf/1X/+VKVOmZLPNNmvZ5qKLLspll12Wq666Kvfff3822WSTHHTQQXnjjTc6cXKqzYUXXpgrr7wyV1xxRR5//PFceOGFueiii3L55Ze3bONco9NUWCsjRoyonHLKKS2PV61aVdlmm20qkydP7sSpeK9ZuHBhJUnl7rvvrlQqlcqrr75a6datW+UnP/lJyzaPP/54JUll1qxZnTUmVWzx4sWVIUOGVKZPn1754Ac/WDnttNMqlYpzjXLOPPPMyv7779/m+ubm5sqAAQMq3/rWt1qWvfrqq5W6urrKjTfeuCFG5D3i4x//eOWzn/1sq2WHHXZY5ZhjjqlUKs41OpcrWWthxYoVmTNnThoaGlqW1dbWpqGhIbNmzerEyXivee2115Ik/fr1S5LMmTMnK1eubHXuDR06NIMGDXLu0SGnnHJKPv7xj7c6pxLnGuXceuutGT58eI444ohstdVW2WuvvXLNNde0rH/22Wczf/78Vudanz59ss8++zjXeEf23XffzJgxI0899VSS5JFHHsm9996bj33sY0mca3Surp09QDV45ZVXsmrVqvTv37/V8v79++eJJ57opKl4r2lubs7pp5+e/fbbL7vuumuSZP78+enevXv69u3batv+/ftn/vz5nTAl1exHP/pR5s6dmwceeOBt65xrlPLMM8/kyiuvTGNjY/75n/85DzzwQE499dR07949xx9/fMv5tLq/U51rvBNnnXVWmpqaMnTo0HTp0iWrVq3Keeedl2OOOSZJnGt0KpEF7xKnnHJKHnvssdx7772dPQrvQc8//3xOO+20TJ8+PfX19Z09Du9hzc3NGT58eM4///wkyV577ZXHHnssV111VY4//vhOno73kh//+Me5/vrrc8MNN+QDH/hAHn744Zx++unZZpttnGt0Oh8XXAtbbLFFunTp8ra7bC1YsCADBgzopKl4Lxk/fnxuu+22/PrXv852223XsnzAgAFZsWJFXn311VbbO/d4p+bMmZOFCxdm2LBh6dq1a7p27Zq77747l112Wbp27Zr+/fs71yhi6623zi677NJq2c4775x58+YlScv55O9U1tVXvvKVnHXWWfnUpz6V3XbbLccee2zOOOOMTJ48OYlzjc4lstZC9+7ds/fee2fGjBkty5qbmzNjxoyMHDmyEyej2lUqlYwfPz4/+9nPctddd2WHHXZotX7vvfdOt27dWp17Tz75ZObNm+fc4x0ZNWpUHn300Tz88MMtP8OHD88xxxzT8s/ONUrYb7/93varKJ566qlsv/32SZIddtghAwYMaHWuNTU15f7773eu8Y4sW7YstbWt/1O2S5cuaW5uTuJco3P5uOBaamxszPHHH5/hw4dnxIgRufTSS7N06dKMHTu2s0ejip1yyim54YYb8vOf/zy9evVq+Yx4nz590qNHj/Tp0yfjxo1LY2Nj+vXrl969e+eLX/xiRo4cmb//+7/v5OmpJr169Wr5rt9bNtlkk2y++eYty51rlHDGGWdk3333zfnnn58jjzwys2fPztVXX52rr746SVp+P9s3v/nNDBkyJDvssEPOPvvsbLPNNjnkkEM6d3iqysEHH5zzzjsvgwYNygc+8IE89NBDueSSS/LZz342iXONTtbZtzesJpdffnll0KBBle7du1dGjBhR+e1vf9vZI1Hlkqz251//9V9btnn99dcr//RP/1TZbLPNKj179qwceuihlZdffrnzhuY9469v4V6pONco5z/+4z8qu+66a6Wurq4ydOjQytVXX91qfXNzc+Xss8+u9O/fv1JXV1cZNWpU5cknn+ykaalWTU1NldNOO60yaNCgSn19fWXHHXesfO1rX6ssX768ZRvnGp2lplL5q1+LDQAAwDrxnSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQBs9D70oQ/l9NNPr7rnBuDdqWtnDwAA72W33HJLunXr1tljALABiSwAWI/69evX2SMAsIH5uCAAVaW5uTmTJ0/ODjvskB49emSPPfbIzTffnCSZOXNmampq8qtf/Sp77bVXevTokY985CNZuHBhbr/99uy8887p3bt3jj766CxbtqzV87755psZP358+vTpky222CJnn312KpXKWs303e9+N0OGDEl9fX369++fT37yky3r/vrjgm/N97c/J5xwQsv2P//5zzNs2LDU19dnxx13zNe//vW8+eab6/amAbBBuZIFQFWZPHlyfvjDH+aqq67KkCFD8pvf/Caf+cxnsuWWW7Zsc8455+SKK65Iz549c+SRR+bII49MXV1dbrjhhixZsiSHHnpoLr/88px55pkt+1x33XUZN25cZs+enQcffDAnnXRSBg0alBNPPLHdeR588MGceuqp+bd/+7fsu+++WbRoUe65557Vbrvvvvvm5Zdfbnn8+OOPZ8yYMfmHf/iHJMk999yT4447LpdddlkOOOCAPP300znppJOSJJMmTerwewbAhlVTWdv/TQcAnWz58uXp169f7rzzzowcObJl+ec+97ksW7YsJ510Uj784Q/nzjvvzKhRo5IkF1xwQSZMmJCnn346O+64Y5Lk5JNPznPPPZdp06Yl+cvVpoULF+b3v/99ampqkiRnnXVWbr311vzXf/1XuzPdcsstGTt2bF544YX06tXrbes/9KEPZc8998yll17aavmf/vSnjBgxIqNHj87UqVOTJA0NDRk1alQmTJjQst0Pf/jDfPWrX81LL730Dt8tADqLK1kAVI3//u//zrJly/LRj3601fIVK1Zkr732anm8++67t/xz//7907Nnz5bAemvZ7NmzWz3H3//937cEVpKMHDkyU6ZMyapVq9KlS5c2Z/roRz+a7bffPjvuuGNGjx6d0aNH59BDD03Pnj3b3GflypU5/PDDs/322+c73/lOy/JHHnkk9913X84777yWZatWrcobb7yRZcuWtfucALx7iCwAqsaSJUuSJL/4xS+y7bbbtlpXV1eXp59+Okla3c2vpqbmbXf3q6mpSXNzc5GZevXqlblz52bmzJm54447MnHixJxzzjl54IEH0rdv39Xu84UvfCHPP/98Zs+ena5d//9fxUuWLMnXv/71HHbYYW/bp76+vsi8AKx/IguAqrHLLrukrq4u8+bNywc/+MG3rX8rsjri/vvvb/X4t7/9bYYMGdLuVay3dO3aNQ0NDWloaMikSZPSt2/f3HXXXauNpUsuuSQ//vGP85//+Z/ZfPPNW60bNmxYnnzyybzvfe/r8OsAoPOJLACqRq9evfLlL385Z5xxRpqbm7P//vvntddey3333ZfevXtn++237/Bzz5s3L42Njfn85z+fuXPn5vLLL8+UKVPWuN9tt92WZ555Jv/wD/+QzTbbLL/85S/T3Nyc97///W/b9s4778xXv/rVTJ06NVtssUXmz5+fJOnRo0f69OmTiRMn5n//7/+dQYMG5ZOf/GRqa2vzyCOP5LHHHss3v/nNDr82ADYskQVAVTn33HOz5ZZbZvLkyXnmmWfSt2/fDBs2LP/8z/+8Th8BPO644/L6669nxIgR6dKlS0477bSWO/u1p2/fvrnllltyzjnn5I033siQIUNy44035gMf+MDbtr333nuzatWqnHzyyTn55JNblh9//PG59tprc9BBB+W2227LN77xjVx44YXp1q1bhg4dms997nMdfl0AbHjuLggAAFCQX0YMAABQkMgCgHbcc8892XTTTdv8AYC/5eOCANCO119/PS+++GKb690JEIC/JbIAAAAK8nFBAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAF/T++xgIS+CWqCAAAAABJRU5ErkJggg==\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.3 Encoder\n", + "\n", + "#### Picture\n", + "\n", + "" + ], + "metadata": { + "id": "n9MLJLySca14" + } + }, + { + "cell_type": "markdown", + "source": [ + "#### TransformerEncoderBlock\n", + "\n", + "**Initialization:**\n", + "\n", + "* in_size ~ input embedding size\n", + "* head_size ~ size of the Q, K, V matrices embeddings after transformation\n", + "* num_heads ~ number of attention heads\n", + "* out_size ~ output embedding size for attention and the block\n", + "* ff_hidden_size ~ hidden size for feed-forward layers\n", + "* dropout_p ~ dropout probability\n", + "* query_in_size ~ input embedding size for the query (if None, defaults to in_size)\n", + "\n", + "Forward:\n", + "\n", + "*query, key, value ~ 3 tensors (one for each Q, K, and V transformation - these are not yet the tensors $\\text{batch_size} \\times seq \\times d_k$, but tensors of shape $\\text{batch_size} \\times seq \\times \\text{in_size}$)" + ], + "metadata": { + "id": "B24kUNvlckeC" + } + }, + { + "cell_type": "code", + "source": [ + "class TransformerEncoderBlock(nn.Module):\n", + " \"\"\"\n", + " Class with one full block within transformer's encoder\n", + " \"\"\"\n", + " def __init__(self, in_size, head_size, num_heads, out_size, ff_hidden_size, dropout_p=0.2, query_in_size=None):\n", + " \"\"\"\n", + " Args:\n", + " in_size: input embedding size\n", + " head_size: size of each attention head\n", + " num_heads: number of attention heads\n", + " out_size: output embedding size\n", + " ff_hidden_size: hidden size for feed forward net\n", + " dropout_p: probability for dropout\n", + " query_in_size: embedding size of input for query (if not provided - same as in_size)\n", + " \"\"\"\n", + " super(TransformerEncoderBlock, self).__init__()\n", + "\n", + " # Запишем все переданые гиперпараметры слоя\n", + " self.in_size = in_size\n", + " self.head_size = head_size\n", + " self.num_heads = num_heads\n", + " self.out_size = out_size\n", + " self.ff_hidden_size = ff_hidden_size\n", + " self.dropout_p = dropout_p\n", + " self.query_in_size = in_size if query_in_size is None else query_in_size\n", + "\n", + " self.attention = MultiHeadAttention(self.in_size, self.head_size, self.num_heads, self.out_size, self.query_in_size)\n", + " # Если выход и вход attention-а имеют разный размер, то используем линейный слой на residual connection-е\n", + " self.adapt_residual = nn.Linear(self.query_in_size, self.out_size) if self.query_in_size != self.out_size else nn.Identity()\n", + "\n", + " self.norm_1 = nn.LayerNorm(self.out_size)\n", + " self.dropout_1 = nn.Dropout(self.dropout_p)\n", + "\n", + " self.feed_forward = nn.Sequential(OrderedDict([\n", + " (\"lin_1\", nn.Linear(self.out_size, self.ff_hidden_size)),\n", + " (\"act\", nn.ReLU()),\n", + " (\"lin_2\", nn.Linear(self.ff_hidden_size, self.out_size)),\n", + " ]))\n", + "\n", + " self.norm_2 = nn.LayerNorm(self.out_size)\n", + " self.dropout_2 = nn.Dropout(self.dropout_p)\n", + "\n", + "\n", + " def forward(self, query, key, value):\n", + " \"\"\"\n", + " Args:\n", + " block_input: input to corresponding block\n", + " \"\"\"\n", + " # Получаем на вход 3 тензора batch_size x seq_len x in_size\n", + " attention_out = self.attention(query, key, value) # (batch_size, seq_len, out_size)\n", + " attention_residual_out = attention_out + self.adapt_residual(query)\n", + " norm_1_out = self.dropout_1(self.norm_1(attention_residual_out))\n", + "\n", + " # (batch_size, seq_len, out_size) -> (batch_size, seq_len, ff_hidden_size) -> (batch_size, seq_len, out_size)\n", + " ff_out = self.feed_forward(norm_1_out)\n", + " ff_residual_out = ff_out + norm_1_out\n", + " return self.dropout_2(self.norm_2(ff_residual_out))" + ], + "metadata": { + "id": "vAsmZjqEceXe" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing TransformerEncoderBlock for the encoder" + ], + "metadata": { + "id": "yTrJWG_bdA9f" + } + }, + { + "cell_type": "code", + "source": [ + "# We check the standard forward pass from the encoder\n", + "tmp_layer = TransformerEncoderBlock(\n", + " in_size=10,\n", + " head_size=7,\n", + " num_heads=2,\n", + " out_size=15,\n", + " ff_hidden_size=20,\n", + " dropout_p=0.1,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Wne9dA-BdDEI", + "outputId": "dbabfef9-790e-40fb-dc14-a372179c88d0" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "TransformerEncoderBlock(\n", + " (attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Linear(in_features=10, out_features=15, bias=True)\n", + " (norm_1): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_1): Dropout(p=0.1, inplace=False)\n", + " (feed_forward): Sequential(\n", + " (lin_1): Linear(in_features=15, out_features=20, bias=True)\n", + " (act): ReLU()\n", + " (lin_2): Linear(in_features=20, out_features=15, bias=True)\n", + " )\n", + " (norm_2): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_2): Dropout(p=0.1, inplace=False)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 23 + } + ] + }, + { + "cell_type": "code", + "source": [ + "tmp_input = torch.rand(2, 5, 10)\n", + "\n", + "print(\"Encoder-like input\")\n", + "print(f'Input shape: {tmp_input.shape}')\n", + "tmp_output = tmp_layer(tmp_input, tmp_input, tmp_input)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "yFAF2RRGdHGu", + "outputId": "dba6ef98-a7ec-4d34-ec49-c6831f278590" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Encoder-like input\n", + "Input shape: torch.Size([2, 5, 10])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing TransformerEncoderBlock for the decoder" + ], + "metadata": { + "id": "enhFxwBtdIlQ" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = TransformerEncoderBlock(\n", + " in_size=10,\n", + " head_size=7,\n", + " num_heads=2,\n", + " out_size=15,\n", + " ff_hidden_size=20,\n", + " dropout_p=0.1,\n", + " query_in_size=12,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "NFSh5Kzbdas5", + "outputId": "bb29185c-a9a4-4426-860f-e87467c50ae3" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "TransformerEncoderBlock(\n", + " (attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Linear(in_features=12, out_features=15, bias=True)\n", + " (norm_1): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_1): Dropout(p=0.1, inplace=False)\n", + " (feed_forward): Sequential(\n", + " (lin_1): Linear(in_features=15, out_features=20, bias=True)\n", + " (act): ReLU()\n", + " (lin_2): Linear(in_features=20, out_features=15, bias=True)\n", + " )\n", + " (norm_2): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_2): Dropout(p=0.1, inplace=False)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 25 + } + ] + }, + { + "cell_type": "code", + "source": [ + "# We check the forward pass from the decoder, where we mix information from the encoder and decoder." + ], + "metadata": { + "id": "ASr5ZUnWdbuw" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "tmp_input_q = torch.rand(2, 5, 12)\n", + "tmp_input_kv = torch.rand(2, 7, 10)\n", + "\n", + "print(\"Encoder+Decoder-like input\")\n", + "print(f'Input Q shape: {tmp_input_q.shape}')\n", + "print(f'Input KV shape: {tmp_input_kv.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input_q, tmp_input_kv, tmp_input_kv)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input_q, tmp_input_kv, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "zpId_xxbdfGb", + "outputId": "45b71cd9-3c28-4b4c-ebba-ee3d2ff3d084" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Encoder+Decoder-like input\n", + "Input Q shape: torch.Size([2, 5, 12])\n", + "Input KV shape: torch.Size([2, 7, 10])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### TransformerEncoder\n", + "\n", + "**Initialization:**\n", + "\n", + "* max_seq_len ~ maximum sequence length in tokens\n", + "* vocab_size ~ vocabulary size\n", + "* emb_size ~ input embedding size\n", + "* num_layers ~ number of TransformerEncoderBlocks\n", + "* att_out_size ~ output embedding size from attention and the block\n", + "* att_head_size ~ embedding size of Q, K, V matrices after transformation\n", + "* num_heads ~ number of attention heads\n", + "* ff_hidden_size ~ hidden size for the feed-forward layers\n", + "* dropout_p ~ dropout probability\n", + "\n", + "**Forward:**\n", + "\n", + "* encoder_input ~ tokens input to the encoder before embedding" + ], + "metadata": { + "id": "bAAU2hKcdh4z" + } + }, + { + "cell_type": "code", + "source": [ + "class TransformerEncoderBlock(nn.Module):\n", + " \"\"\"\n", + " Class with one full block within transformer's encoder\n", + " \"\"\"\n", + " def __init__(self, in_size, head_size, num_heads, out_size, ff_hidden_size, dropout_p=0.2, query_in_size=None):\n", + " \"\"\"\n", + " Args:\n", + " in_size: input embedding size\n", + " head_size: size of each attention head\n", + " num_heads: number of attention heads\n", + " out_size: output embedding size\n", + " ff_hidden_size: hidden size for feed-forward net\n", + " dropout_p: probability for dropout\n", + " query_in_size: embedding size for the query input (if not provided, use in_size)\n", + " \"\"\"\n", + " super(TransformerEncoderBlock, self).__init__()\n", + "\n", + " # Store all passed layer hyperparameters\n", + " self.in_size = in_size\n", + " self.head_size = head_size\n", + " self.num_heads = num_heads\n", + " self.out_size = out_size\n", + " self.ff_hidden_size = ff_hidden_size\n", + " self.dropout_p = dropout_p\n", + " self.query_in_size = in_size if query_in_size is None else query_in_size\n", + "\n", + " self.attention = ...\n", + " self.adapt_residual = ...\n", + "\n", + " self.norm_1 = ...\n", + " self.dropout_1 = ...\n", + "\n", + " self.feed_forward = nn.Sequential(OrderedDict([\n", + " (\"lin_1\", ...),\n", + " (\"act\", ...),\n", + " (\"lin_2\", ...),\n", + " ]))\n", + "\n", + " self.norm_2 = ...\n", + " self.dropout_2 = ...\n", + "\n", + "\n", + " def forward(self, query, key, value):\n", + " \"\"\"\n", + " Args:\n", + " block_input: input to corresponding block\n", + " \"\"\"\n", + " # Input of 3 tensors batch_size x seq_len x in_size\n", + " attention_out = ...\n", + " attention_residual_out = ...\n", + " norm_1_out = ...\n", + "\n", + " ff_out = ...\n", + " ff_residual_out = ...\n", + " norm_2_out = ...\n", + " return norm_2_out" + ], + "metadata": { + "id": "vg8IP5CqdwAb" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing TransformerEncoder" + ], + "metadata": { + "id": "1GO52uVYd-Qb" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = TransformerEncoder(\n", + " max_seq_len=20,\n", + " vocab_size=10000,\n", + " emb_size=10,\n", + " num_layers=2,\n", + " att_head_size=7,\n", + " num_heads=2,\n", + " att_out_size=15,\n", + " ff_hidden_size=20,\n", + " dropout_p=0.1,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "id": "BBlrE6acd9fP" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "tmp_input = torch.randint(10000, (2, 5))\n", + "\n", + "print(f'Input shape: {tmp_input.shape}')\n", + "tmp_output = tmp_layer(tmp_input)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input, tmp_output" + ], + "metadata": { + "id": "6HJUmeR8eFe8" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.4 Decoder\n", + "\n", + "#### Picture\n", + "\n", + "\n", + "\n", + "#### TransformerDecoderBlock\n", + "\n", + "**Initialization:**\n", + "\n", + "* in_size ~ input embedding size\n", + "* head_size ~ size of Q, K, V matrix embeddings after transformation\n", + "* num_heads ~ number of attention heads\n", + "* out_size ~ output embedding size for attention and the block\n", + "* ff_hidden_size ~ hidden size for feed-forward layers\n", + "* dropout_p ~ dropout probability\n", + "* encoder_out_size ~ encoder output embedding size (if None, defaults to in_size)\n", + "\n", + "**Forward:**\n", + "\n", + "* decoder_emb ~ tensor from the previous block or embeddings with positional encodings\n", + "* encoder_output ~ output tensor from the corresponding encoder" + ], + "metadata": { + "id": "RYI4RNxyeGR-" + } + }, + { + "cell_type": "code", + "source": [ + "class TransformerDecoderBlock(nn.Module):\n", + " \"\"\"\n", + " Class with one full block within transformer's decoder\n", + " \"\"\"\n", + " def __init__(self, in_size, head_size, num_heads, out_size, ff_hidden_size, dropout_p=0.2, encoder_out_size=None):\n", + " \"\"\"\n", + " Args:\n", + " in_size: input embedding size\n", + " head_size: size of each attention head\n", + " num_heads: number of attention heads\n", + " out_size: output embedding size\n", + " ff_hidden_size: hidden size for feed forward net\n", + " dropout_p: probability for dropout\n", + " encoder_out_size: embedding size of outputs from encoder (if not provided - same as in_size)\n", + " \"\"\"\n", + " super(TransformerDecoderBlock, self).__init__()\n", + "\n", + " # Запишем все переданые гиперпараметры слоя\n", + " self.in_size = in_size\n", + " self.head_size = head_size\n", + " self.num_heads = num_heads\n", + " self.out_size = out_size\n", + " self.ff_hidden_size = ff_hidden_size\n", + " self.dropout_p = dropout_p\n", + " self.encoder_out_size = in_size if encoder_out_size is None else encoder_out_size\n", + "\n", + "\n", + " self.masked_attention = MultiHeadAttention(self.in_size, self.head_size, self.num_heads, self.out_size)\n", + " # Если выход и вход attention-а имеют разный размер, то используем линейный слой на residual connection-е\n", + " self.adapt_residual = nn.Linear(self.in_size, self.out_size) if self.in_size != self.out_size else nn.Identity()\n", + " self.norm = nn.LayerNorm(self.out_size)\n", + " self.dropout = nn.Dropout(self.dropout_p)\n", + " self.encoder_block = TransformerEncoderBlock(self.encoder_out_size, self.head_size, self.num_heads, self.out_size, self.ff_hidden_size, self.dropout_p, self.out_size)\n", + "\n", + "\n", + " def forward(self, decoder_emb, encoder_output):\n", + " \"\"\"\n", + " Args:\n", + " decoder_emb: decoder sequence after embed\n", + " encoder_output: output from encoder\n", + " \"\"\"\n", + " # Получаем на вход тензор batch_size x seq_len x in_size и тензор batch_size x encoder_seq_len x encoder_out_size\n", + " mask = make_decoder_mask(decoder_emb) # batch_size x 1 x seq_len x seq_len\n", + " attention = self.masked_attention(decoder_emb, decoder_emb, decoder_emb, mask=mask) # batch_size x seq_len x out_size\n", + " mmha_out = self.dropout(self.norm(attention + self.adapt_residual(decoder_emb)))\n", + "\n", + " return self.encoder_block(mmha_out, encoder_output, encoder_output) # batch_size x seq_len x out_size" + ], + "metadata": { + "id": "WI673PnTeOn8" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing TransformerDecoderBlock" + ], + "metadata": { + "id": "EBJy06RwfHeY" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = TransformerDecoderBlock(\n", + " in_size=10,\n", + " head_size=7,\n", + " num_heads=2,\n", + " out_size=15,\n", + " ff_hidden_size=20,\n", + " dropout_p=0.1,\n", + " encoder_out_size=12,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "zsXTUhlIfAkd", + "outputId": "1c2ea4d0-e0c1-4c05-e2a0-1ec30768c896" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "TransformerDecoderBlock(\n", + " (masked_attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Linear(in_features=10, out_features=15, bias=True)\n", + " (norm): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (encoder_block): TransformerEncoderBlock(\n", + " (attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Identity()\n", + " (norm_1): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_1): Dropout(p=0.1, inplace=False)\n", + " (feed_forward): Sequential(\n", + " (lin_1): Linear(in_features=15, out_features=20, bias=True)\n", + " (act): ReLU()\n", + " (lin_2): Linear(in_features=20, out_features=15, bias=True)\n", + " )\n", + " (norm_2): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_2): Dropout(p=0.1, inplace=False)\n", + " )\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 29 + } + ] + }, + { + "cell_type": "code", + "source": [ + "# Testing the forward pass in the decoder, where we mix information from the encoder and decoder\n", + "tmp_input_decoder = torch.rand(2, 5, 10)\n", + "tmp_output_encoder = torch.rand(2, 7, 12)\n", + "\n", + "print(\"Encoder+Decoder-like input\")\n", + "print(f'Decoder input shape: {tmp_input_decoder.shape}')\n", + "print(f'Encoder output shape: {tmp_output_encoder.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input_decoder, tmp_output_encoder)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input_decoder, tmp_output_encoder" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "zulYoZ6GfKVP", + "outputId": "1c43ddb6-0bee-4885-8516-b023c3ec874f" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Encoder+Decoder-like input\n", + "Decoder input shape: torch.Size([2, 5, 10])\n", + "Encoder output shape: torch.Size([2, 7, 12])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### TransformerDecoder\n", + "\n", + "**Initialization:**\n", + "\n", + "* max_seq_len ~ maximum token length of the sequence\n", + "* vocab_size ~ size of the vocabulary\n", + "*\temb_size ~ input embedding size\n", + "* num_layers ~ number of TransformerEncoderBlocks\n", + "*\tatt_out_size ~ output embedding size for attention and the block\n", + "*\tatt_head_size ~ embedding size of the Q, K, V matrices after transformation\n", + "*\tnum_heads ~ number of attention heads\n", + "*\tff_hidden_size ~ hidden size for the feed-forward layers\n", + "*\tdropout_p ~ dropout probability\n", + "*\tencoder_out_size ~ encoder output embedding size (if None, defaults to in_size)\n", + "\n", + "**Forward:**\n", + "\n", + "*\tdecoder_input ~ input tokens to the decoder before embeddings\n", + "*\tencoder_output ~ output tensor from the corresponding encoder" + ], + "metadata": { + "id": "xRLiJy5FfOUq" + } + }, + { + "cell_type": "code", + "source": [ + "class TransformerDecoder(nn.Module):\n", + " \"\"\"\n", + " Class for decoder within transformer.\n", + " \"\"\"\n", + " def __init__(self, max_seq_len, vocab_size, emb_size, num_layers, att_out_size, att_head_size, num_heads, ff_hidden_size, dropout_p, encoder_out_size=None):\n", + " \"\"\"\n", + " Args:\n", + " max_seq_len : maximum length of input sequence\n", + " vocab_size: size of the vocabulary\n", + " emb_size: embeddings size\n", + " num_layers: number of encoder layers\n", + " att_out_size: output size for attention and each encoder block\n", + " att_head_size: size of each attention head\n", + " num_heads: number of heads in multihead attention\n", + " ff_hidden_size: hidden size for feed forward net\n", + " dropout_p: probability for dropout\n", + " encoder_out_size: embedding size of outputs from encoder (if not provided - same as in_size)\n", + " \"\"\"\n", + " super(TransformerDecoder, self).__init__()\n", + "\n", + " # Запишем все переданые гиперпараметры слоя\n", + " self.max_seq_len = max_seq_len\n", + " self.vocab_size = vocab_size\n", + " self.emb_size = emb_size\n", + " self.num_layers = num_layers\n", + " self.att_out_size = att_out_size\n", + " self.att_head_size = att_head_size\n", + " self.num_heads = num_heads\n", + " self.ff_hidden_size = ff_hidden_size\n", + " self.dropout_p = dropout_p\n", + " self.encoder_out_size = in_size if encoder_out_size is None else encoder_out_size\n", + "\n", + " self.embedding_layer = nn.Embedding(self.vocab_size, self.emb_size)\n", + " self.positional_encoder = PositionalEncoding(self.max_seq_len, self.emb_size)\n", + " self.dropout = nn.Dropout(self.dropout_p)\n", + "\n", + " self.decoder_blocks = nn.ModuleDict({\n", + " f\"decoder_block_{i}\": TransformerDecoderBlock(\n", + " in_size=self.emb_size if i==0 else self.att_out_size,\n", + " head_size=self.att_head_size,\n", + " num_heads=self.num_heads,\n", + " out_size=self.att_out_size,\n", + " ff_hidden_size=self.ff_hidden_size,\n", + " dropout_p=self.dropout_p,\n", + " encoder_out_size=self.encoder_out_size,\n", + " ) for i in range(self.num_layers)\n", + " })\n", + "\n", + " self.fc = nn.Linear(self.att_out_size, self.vocab_size)\n", + "\n", + " def forward(self, decoder_input, encoder_output):\n", + " \"\"\"\n", + " Args:\n", + " decoder_input:\n", + " encoder_output:\n", + " Returns:\n", + " out: output vector\n", + " \"\"\"\n", + " # Получаем на вход batch_size x seq_len и batch_size x encoder_seq_len x encoder_out_size\n", + " decoder_emb = self.embedding_layer(decoder_input) # batch_size x seq_len x emb_size\n", + " decoder_emb = self.positional_encoder(decoder_emb)\n", + "\n", + " out = self.dropout(decoder_emb)\n", + "\n", + " for block in self.decoder_blocks.values():\n", + " out = block(out, encoder_output) # batch_size x seq_len x att_out_size\n", + "\n", + " return self.fc(out)" + ], + "metadata": { + "id": "aLuT6JaQfby_" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing TransformerDecoder" + ], + "metadata": { + "id": "3vBYyZrXzQHW" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = TransformerDecoder(\n", + " max_seq_len=20,\n", + " vocab_size=10000,\n", + " emb_size=10,\n", + " num_layers=2,\n", + " att_head_size=7,\n", + " num_heads=2,\n", + " att_out_size=15,\n", + " ff_hidden_size=20,\n", + " dropout_p=0.1,\n", + " encoder_out_size=12,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "WOnV1Y20zV9G", + "outputId": "2375326f-a3f6-4e95-dea0-aba9ca82b975" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "TransformerDecoder(\n", + " (embedding_layer): Embedding(10000, 10)\n", + " (positional_encoder): PositionalEncoding()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (decoder_blocks): ModuleDict(\n", + " (decoder_block_0): TransformerDecoderBlock(\n", + " (masked_attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Linear(in_features=10, out_features=15, bias=True)\n", + " (norm): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (encoder_block): TransformerEncoderBlock(\n", + " (attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Identity()\n", + " (norm_1): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_1): Dropout(p=0.1, inplace=False)\n", + " (feed_forward): Sequential(\n", + " (lin_1): Linear(in_features=15, out_features=20, bias=True)\n", + " (act): ReLU()\n", + " (lin_2): Linear(in_features=20, out_features=15, bias=True)\n", + " )\n", + " (norm_2): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_2): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (decoder_block_1): TransformerDecoderBlock(\n", + " (masked_attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Identity()\n", + " (norm): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (encoder_block): TransformerEncoderBlock(\n", + " (attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Identity()\n", + " (norm_1): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_1): Dropout(p=0.1, inplace=False)\n", + " (feed_forward): Sequential(\n", + " (lin_1): Linear(in_features=15, out_features=20, bias=True)\n", + " (act): ReLU()\n", + " (lin_2): Linear(in_features=20, out_features=15, bias=True)\n", + " )\n", + " (norm_2): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_2): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " )\n", + " (fc): Linear(in_features=15, out_features=10000, bias=True)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 44 + } + ] + }, + { + "cell_type": "code", + "source": [ + "# We will test the Transformer model by passing through both the encoder and decoder, ensuring that information from the encoder is correctly used in the decoder for sequence generation.\n", + "tmp_input_decoder = torch.randint(10000, (2, 5))\n", + "tmp_output_encoder = torch.rand(2, 7, 12)\n", + "\n", + "print(\"Encoder+Decoder-like input\")\n", + "print(f'Decoder input shape: {tmp_input_decoder.shape}')\n", + "print(f'Encoder output shape: {tmp_output_encoder.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input_decoder, tmp_output_encoder)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input_decoder, tmp_output_encoder" + ], + "metadata": { + "id": "WU6z4-uczXom" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.5 Transformer" + ], + "metadata": { + "id": "X7PUNDaT0OBW" + } + }, + { + "cell_type": "code", + "source": [ + "class Transformer(nn.Module):\n", + " \"\"\"\n", + " Class for full encoder-decoder transformer\n", + " \"\"\"\n", + " def __init__(\n", + " self,\n", + " max_seq_len,\n", + " vocab_size,\n", + " emb_size,\n", + "\n", + " num_encoder_layers,\n", + " enc_att_out_size,\n", + " enc_att_head_size,\n", + " enc_num_heads,\n", + " enc_ff_hidden_size,\n", + " enc_dropout_p,\n", + "\n", + " num_decoder_layers,\n", + " dec_att_out_size,\n", + " dec_att_head_size,\n", + " dec_num_heads,\n", + " dec_ff_hidden_size,\n", + " dec_dropout_p,\n", + " ):\n", + " super(Transformer, self).__init__()\n", + "\n", + " # Store all the passed hyperparameters of the model\n", + " self.max_seq_len = max_seq_len\n", + " self.vocab_size = vocab_size\n", + " self.emb_size = emb_size\n", + "\n", + " self.num_encoder_layers = num_encoder_layers\n", + " self.enc_att_out_size = enc_att_out_size\n", + " self.enc_att_head_size = enc_att_head_size\n", + " self.enc_num_heads = enc_num_heads\n", + " self.enc_ff_hidden_size = enc_ff_hidden_size\n", + " self.enc_dropout_p = enc_dropout_p\n", + "\n", + " self.num_decoder_layers = num_decoder_layers\n", + " self.dec_att_out_size = dec_att_out_size\n", + " self.dec_att_head_size = dec_att_out_size\n", + " self.dec_num_heads = dec_num_heads\n", + " self.dec_ff_hidden_size = dec_ff_hidden_size\n", + " self.dec_dropout_p = dec_dropout_p\n", + "\n", + " # Encoder\n", + " self.encoder = TransformerEncoder(\n", + " max_seq_len=self.max_seq_len,\n", + " vocab_size=self.vocab_size,\n", + " emb_size=self.emb_size,\n", + " num_layers=self.num_encoder_layers,\n", + " att_head_size=self.enc_att_head_size,\n", + " num_heads=self.enc_num_heads,\n", + " att_out_size=self.enc_att_out_size,\n", + " ff_hidden_size=self.enc_ff_hidden_size,\n", + " dropout_p=self.enc_dropout_p,\n", + " )\n", + "\n", + " # Decoder\n", + " self.decoder = TransformerDecoder(\n", + " max_seq_len=self.max_seq_len,\n", + " vocab_size=self.vocab_size,\n", + " emb_size=self.emb_size,\n", + " num_layers=self.num_decoder_layers,\n", + " att_head_size=self.dec_att_head_size,\n", + " num_heads=self.dec_num_heads,\n", + " att_out_size=self.dec_att_out_size,\n", + " ff_hidden_size=self.dec_ff_hidden_size,\n", + " dropout_p=self.dec_dropout_p,\n", + " encoder_out_size=self.enc_att_out_size,\n", + " )\n", + "\n", + " def forward(self, encoder_input, decoder_input):\n", + " \"\"\"\n", + " Args:\n", + " encoder_input: input to encoder\n", + " decoder_input: input to decoder\n", + " out:\n", + " out: final tensor with logits of each word in vocab\n", + " \"\"\"\n", + " # Input has shape batch_size x enc_seq_len and batch_size x dec_seq_len\n", + " encoder_output = self.encoder(encoder_input) # (batch_size, enc_seq_len, enc_att_out_size)\n", + "\n", + " return self.decoder(decoder_input, encoder_output) # (batch_size, dec_seq_len, vocab_size)" + ], + "metadata": { + "id": "nkvAAfu20OYC" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.6 Testing" + ], + "metadata": { + "id": "6ZRK9UIn0Yvg" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = Transformer(\n", + " max_seq_len=20,\n", + " vocab_size=10000,\n", + " emb_size=10,\n", + "\n", + " num_encoder_layers=3,\n", + " enc_att_head_size=7,\n", + " enc_num_heads=3,\n", + " enc_att_out_size=20,\n", + " enc_ff_hidden_size=30,\n", + " enc_dropout_p=0.2,\n", + "\n", + " num_decoder_layers=2,\n", + " dec_att_head_size=7,\n", + " dec_num_heads=2,\n", + " dec_att_out_size=15,\n", + " dec_ff_hidden_size=20,\n", + " dec_dropout_p=0.1,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "id": "365uJEiR0bNS" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "tmp_input_encoder = torch.randint(10000, (2, 9))\n", + "tmp_input_decoder = torch.randint(10000, (2, 5))\n", + "\n", + "print(f'Encoder input shape: {tmp_input_encoder.shape}')\n", + "print(f'Decoder input shape: {tmp_input_decoder.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input_encoder, tmp_input_decoder)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input_decoder, tmp_input_encoder" + ], + "metadata": { + "id": "7eEBPO9F0y3r" + }, + "execution_count": null, + "outputs": [] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/week05_transformer/README.md b/week05_transformer/README.md new file mode 100644 index 0000000..a4c803d --- /dev/null +++ b/week05_transformer/README.md @@ -0,0 +1,2 @@ +Transformer: +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1L4RtAHb_vbfz20lxGXto1FBk5Mo74YzQ?usp=sharing) \ No newline at end of file diff --git a/week05_transformer/transformer.ipynb b/week05_transformer/transformer.ipynb new file mode 100644 index 0000000..c598ef6 --- /dev/null +++ b/week05_transformer/transformer.ipynb @@ -0,0 +1,1852 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "source": [ + "# **Seminar - Attention и Transformer**" + ], + "metadata": { + "id": "jcYtDZ6yYlk7" + } + }, + { + "cell_type": "markdown", + "source": [ + "## 1. Let's build Transformer from scratch in Pytorch\n", + "\n", + "" + ], + "metadata": { + "id": "GFS003OQYv2w" + } + }, + { + "cell_type": "code", + "source": [ + "from collections import OrderedDict\n", + "import torch\n", + "import torch.nn as nn\n", + "import torch.nn.functional as F\n", + "import math\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline" + ], + "metadata": { + "id": "gEmHY574YpNu" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.1 Multi-head Attention\n", + "\n", + "#### Main class - MultiHeadAttention\n", + "\n", + "**Initialization:**\n", + "* _in_size_ ~ size of the input embeddings\n", + "* _head_size_ ~ size of the embeddings for Q, K, V matrices after transformation\n", + "* _num_heads_ ~ number of heads\n", + "* _out_size_ ~ size of the output embeddings\n", + "* _query_in_size_ ~ size of the input embeddings\n", + "\n", + "**Forward:**\n", + "* query, key, value ~ 3 tensors (one for each Q, K, and V transformation - these are not yet the tensors of shape $\\text{batch_size} \\times seq \\times d_k$, but tensors of shape $\\text{batch_size} \\times seq \\times \\text{in_size}$)\n", + "* mask ~ boolean mask for Masked Multi-head Attention (in the decoder)\n", + "\n", + "$$ Attention(Q, K, V) = softmax\\Bigg(\\frac{QK^T}{\\sqrt{d_k}}\\Bigg) \\cdot V $$\n", + "$$ MultiHead(Q, K, V) = Concat(head_1, ..., head_H) \\cdot W^O \\quad ; \\quad head_i = Attention(Q W_i^Q, K W_i^K, V W_i^V)$$" + ], + "metadata": { + "id": "ygtUy2dGZAC8" + } + }, + { + "cell_type": "code", + "source": [ + "class MultiHeadAttention(nn.Module):\n", + " \"\"\"\n", + " Class to calculate Multi-head attention (or Masked Multi-head attention for the decoder) operation\n", + " \"\"\"\n", + " def __init__(self, in_size, head_size, num_heads, out_size, query_in_size=None):\n", + " \"\"\"\n", + " Args:\n", + " in_size: embedding size of input\n", + " head_size: hidden size of Q, K, V matrices\n", + " num_heads: number of heads\n", + " out_size: output embedding size\n", + " query_in_size: embedding size of input for query (if not provided - same as in_size)\n", + " \"\"\"\n", + " super(MultiHeadAttention, self).__init__()\n", + "\n", + " # Store all passed layer hyperparameters\n", + " self.in_size = in_size\n", + " self.head_size = head_size\n", + " self.num_heads = num_heads\n", + " self.out_size = out_size\n", + " self.query_in_size = self.in_size if query_in_size is None else query_in_size\n", + "\n", + " # Linear transformations for Q, K, V matrices (get all Q, K, V matrices directly)\n", + " self.query_matrix = nn.Linear(self.query_in_size, self.num_heads * self.head_size, bias=False)\n", + " self.key_matrix = nn.Linear(self.in_size, self.num_heads * self.head_size, bias=False)\n", + " self.value_matrix = nn.Linear(self.in_size, self.num_heads * self.head_size, bias=False)\n", + " # Linear transformation for concatenating heads\n", + " self.out = nn.Linear(self.head_size * self.num_heads, self.out_size)\n", + "\n", + " def forward(self, query, key, value, mask=None):\n", + " \"\"\"\n", + " Args:\n", + " query : tensor for query\n", + " key : tensor for key\n", + " value : tensor for value\n", + " mask: mask for the decoder\n", + "\n", + " Returns:\n", + " output vector from multihead attention\n", + " \"\"\"\n", + " # Tensors come with the shape batch_size x seq_len x in_size\n", + " batch_size = key.size(0)\n", + " seq_len = key.size(1)\n", + "\n", + " # The number of tokens in the query will differ for the decoder\n", + " query_seq_len = query.size(1)\n", + "\n", + " # Apply linear transformations to the input\n", + " q = self.query_matrix(query) # (batch_size, query_seq_len, head_size * num_heads)\n", + " k = self.key_matrix(key) # (batch_size, seq_len, head_size * num_heads)\n", + " v = self.value_matrix(value) # (batch_size, seq_len, head_size * num_heads)\n", + "\n", + " q = q.view(batch_size, query_seq_len, self.num_heads, self.head_size).transpose(1,2) # (batch_size, num_heads, query_seq_len, head_size)\n", + " k = k.view(batch_size, seq_len, self.num_heads, self.head_size).transpose(1,2) # (batch_size, num_heads, seq_len, head_size)\n", + " v = v.view(batch_size, seq_len, self.num_heads, self.head_size).transpose(1,2) # (batch_size, num_heads, seq_len, head_size)\n", + "\n", + " # Считаем релевантность\n", + " relevance = q @ k.transpose(2, 3) / math.sqrt(self.head_size) # (batch_size, num_heads, query_seq_len, seq_len)\n", + "\n", + " # Если есть маска (для декодера), то заполняем значения по маске как минус бесконечность (чтобы exp(r) = 0 в softmax)\n", + " if mask is not None:\n", + " relevance = relevance.masked_fill(mask, -torch.inf)\n", + "\n", + " # Получаем вероятности\n", + " relevance = F.softmax(relevance, dim=-1)\n", + "\n", + " # Считаем выходы из каждой головы\n", + " head_i = torch.matmul(relevance, v) # (batch_size, num_heads, query_seq_len, head_size)\n", + "\n", + " # Конкатенируем выходы\n", + " concat = head_i.transpose(1,2).reshape(batch_size, query_seq_len, self.head_size * self.num_heads) # (batch_size, query_seq_len, num_heads * head_size)\n", + "\n", + " return self.out(concat) # (batch_size, query_seq_len, out_size)" + ], + "metadata": { + "id": "U2vT-vwEY6_S" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing MultiHeadAttention for the encoder" + ], + "metadata": { + "id": "xyTV7cqXay9b" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = MultiHeadAttention(\n", + " in_size=10,\n", + " head_size=4,\n", + " num_heads=3,\n", + " out_size=15,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "8A6Af8DEazNa", + "outputId": "80961b75-44ba-4ef2-839d-31f34e4e5fb9" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (out): Linear(in_features=12, out_features=15, bias=True)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 3 + } + ] + }, + { + "cell_type": "code", + "source": [ + "# Check in normal forward pass from the encoder\n", + "tmp_input = torch.rand(2, 5, 10)\n", + "\n", + "print(\"Encoder-like input, no mask\")\n", + "print(f'Input shape: {tmp_input.shape}')\n", + "tmp_output = tmp_layer(tmp_input, tmp_input, tmp_input)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "3ilYxygIa33f", + "outputId": "6412f347-246a-47c8-d7bc-3368ee9743c2" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Encoder-like input, no mask\n", + "Input shape: torch.Size([2, 5, 10])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing MultiHeadAttention for a mixture of encoder and decoder" + ], + "metadata": { + "id": "if4DQzHFbAml" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = MultiHeadAttention(\n", + " in_size=10,\n", + " head_size=4,\n", + " num_heads=3,\n", + " out_size=15,\n", + " query_in_size=12,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "ATH6d79ibCn2", + "outputId": "295cfa64-4d6b-4ad6-839d-e8879728c6fc" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=12, out_features=12, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (out): Linear(in_features=12, out_features=15, bias=True)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 5 + } + ] + }, + { + "cell_type": "code", + "source": [ + "# Check forward pass in the decoder, where we mix information from the encoder and decoder\n", + "tmp_input_q = torch.rand(2, 5, 12)\n", + "tmp_input_kv = torch.rand(2, 7, 10)\n", + "\n", + "print(\"Encoder+Decoder-like input, no mask\")\n", + "print(f'Input Q shape: {tmp_input_q.shape}')\n", + "print(f'Input KV shape: {tmp_input_kv.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input_q, tmp_input_kv, tmp_input_kv)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input_q, tmp_input_kv, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "FDNk8KctbEZj", + "outputId": "e7ecebaf-e8aa-466e-b9d8-b3652ef25a91" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Encoder+Decoder-like input, no mask\n", + "Input Q shape: torch.Size([2, 5, 12])\n", + "Input KV shape: torch.Size([2, 7, 10])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Triangular Mask in the decoder" + ], + "metadata": { + "id": "4_a1I_XnbKSJ" + } + }, + { + "cell_type": "code", + "source": [ + "def make_decoder_mask(decoder_embed):\n", + " \"\"\"\n", + " Make mask for decoder Masked Multi-head Attention based on input sequence\n", + " Args:\n", + " decoder_embed: decoder sequence after embed\n", + " Returns:\n", + " mask: mask for Masked Multi-head Attention\n", + " \"\"\"\n", + " batch_size, decoder_seq_len, _ = decoder_embed.shape\n", + " mask = torch.tril(torch.ones((decoder_seq_len, decoder_seq_len))).expand(\n", + " batch_size, 1, decoder_seq_len, decoder_seq_len\n", + " ).bool()\n", + " return mask" + ], + "metadata": { + "id": "sEZC_D24bMSe" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing MultiHeadAttention for the decoder with a mask" + ], + "metadata": { + "id": "0-KrJQ0ObP3c" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_input = torch.rand(1, 10, 256)\n", + "tmp_mask = make_decoder_mask(tmp_input)\n", + "print(f\"Mask shape: {tmp_mask.shape}\")\n", + "\n", + "# Visualize the mask\n", + "fig, ax = plt.subplots(figsize=(10, 10))\n", + "plt.imshow(tmp_mask[0, 0, :, :])\n", + "\n", + "# Add text labels\n", + "for i in range(tmp_mask.shape[-2]):\n", + " for j in range(tmp_mask.shape[-1]):\n", + " text = plt.text(j, i, tmp_mask[0, 0, i, j].item(), ha=\"center\", va=\"center\", color=\"red\")\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 848 + }, + "id": "qQjcaCqrbR-J", + "outputId": "8c8e8714-31b9-4caa-8cab-9854d8c2cf90" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Mask shape: torch.Size([1, 1, 10, 10])\n" + ] + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAygAAAMtCAYAAAB0OVVCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABPOklEQVR4nO39f5CedX0v/j+vbMgSkmxCgEASfkiU2JoIgUYoMFPqyJCeKZ3j5+NYZTxTLB2/0x7iD6IM4Bz1OCKUUhmNcPw1ih2pR+gAYvXT02+/+AEEa8mBBEGNqDSEEn4ILLshkM2PfX//gJNkIRHeyGbf2X08ZjLket/Xdd+v+5lld5973de9XSmlBAAAoAGTxnoAAACA/0NBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQjMl7+wGHh4ezYcOGzJgxI13X7e2HBwAAxkApJRs3bsy8efMyadKez5Ps9YKyYcOGHHHEEXv7YQEAgAY89NBDOfzww/d4+14vKDNmzEiSPHj369I33SvMXqn/a+Gbx3oEAAB41bZla27P/7OjD+zJXi8o/+dlXX3TJ6VvhoLySk3u9hvrEQAA4NUrz//n5S7z0BAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQeFVOaOsy43lprEeY58is3oyqyezejKrJ7N6Mqsns3rjJbPJYz3Aa23S3F/+xtvLhw9M+chBe2ma9p1fVuWMPPiS9bPzR9nQTR+Didons3oyqyezejKrJ7N6Mqsns3oTPbNXVVCuuuqqXH755Xn00Udz3HHH5fOf/3xOPPHE13q2V2X4ntft3LjpmXSXP5Vy+5E716btctKolGR7ksnd3hqvSXfm0Pxt3jJibSC9YzTNvkFm9WRWT2b1ZFZPZvVkVk9m9SZyZtUF5dprr82KFSvyxS9+MSeddFI++9nPZtmyZfn5z3+eOXPmjMaMdebs8pT6JiXdLms/fDaT3rEhw9fMTXfZU8naoZRvzUt37cZkYDjl63N3HNp97NfJT4ZSbjj8+YXhklzZn+6aweTX25MF+6WcNzs5c99vsVvTk/5u/xFr7yj3Z1lZl8OyKRszJT/K3Hwlx2Zzt/sPmQXl6fxV7snC9KckeTjT87mckPu72UmSReWJ/EXuzcL0ZyC9uSPz87Us3uP9tU5m9WRWT2b1ZFZPZvVkVk9m9SZyZtWPfsUVV+R973tf/vzP/zxJ8sUvfjHf+9738rWvfS0XXnjhaz7gaOgueTLl4wcnR+2XzJyUZOPLH7SyP931G1MuOyRZMCX50XPplj+WMrsnOWXqqM+8tw2ny1VZkkczLXOzKe/P6rwvP87nc8Ju978wd+ZXmZWVOT7D6fL6DGTbC5c4zS3P5NL8IFdncT6TpZmZoSzPmizP6pf8ZGBfJrN6Mqsns3oyqyezejKrJ7N6EyWzqoKyZcuW3HXXXbnooot2rE2aNCmnn356/vVf/3W3xwwNDWVoaGjH9uDg4Ksc9bVTzp+dnHbAKz9gqKRb2Z9y3bxk6Qtl5Kj9kjs3p7tmIGUfLyi/n0fynXLjju1VOSyf6k7esf1YpuXrZVE+mLv3+D/AnDybf8jCPNT1JUkezowdt52Vtbk5R+bG7pgdt11VluQzuSWfKydka9czGk9rVMmsnszqyayezOrJrJ7M6sms3kTOrKqgPPHEE9m+fXsOPfTQEeuHHnpo1q5du9tjLr300nzyk5989ROOhuP2f/l9drVuS7rnSvKuDSPXt5Zk8b7/WsA1OSQrd/nA3pyeHF8ey1lZmyOyMQdkW3oynN4Mp7dsy9BuTvtdn2OyInfl9LI+d2dObsvheeSFi7gWZCBHZyBvK+tHHNOTZG42ZX36RvX5jQaZ1ZNZPZnVk1k9mdWTWT2Z1ZvImY36C8wuuuiirFixYsf24OBgjjjiiNF+2N/sgBddFL+7N1vetsvfN5UkSfnGvGTui9rklH3/AvvNmTziHSEOLZtyce7IP2ZBrs7iDGZKFueJfCR3ZXKGM7Sb+/hGtyjfL0fmpDySE/No/iw/zSXlpNzRzc/UbMv3siDfzhtectzjqTiT1RCZ1ZNZPZnVk1k9mdWTWT2Z1ZvImVUVlIMPPjg9PT157LHHRqw/9thjOeyww3Z7TG9vb3p7Gz/LcFBPsnbLyLWfDO1MZ+GUlN4ueXjruLze5MWOSX+6lHwpx6V0zxew08p/vOxxD3czckNm5IYszEfLv2VZ1uWOzM8vMitHZXBcvy2ezOrJrJ7M6smsnszqyayezOpNpMyqflHjlClT8nu/93u5+eabd6wNDw/n5ptvzsknn/wbjmxbOXVqcs9Qct1g8sCWdJc/ObKwTJ+U/OWsdJ944vl91m1Nfrw5+erTz2+PMxsyPful5O35ZQ4rz+T08mDOzAN73H9K2Z7lZXWOLY9nTtmUReWJLMxTWf/C6xyvzRvzpjyZ5WV1Xl+ezvyyMSeXDVleVu+tpzTqZFZPZvVkVk9m9WRWT2b1ZFZvImVW/RKvFStW5Oyzz87SpUtz4okn5rOf/Ww2bdq041299klvnZacd2C6i59Mhkry7hnJO2ckP9t5sqxcMDs5qCfdyv5k/eNJX0/y5t6UDxw4hoOPjge6WflCOTbvys9zTu7LvTk4X8viXJBVu91/OF36siUXZFVmZSiDmZLbMz9/l0VJkn/vZuXD5bSck/tyRW5Jl5INmZ5bc/jefFqjSmb1ZFZPZvVkVk9m9WRWT2b1JlJmXSml1B505ZVX7vhFjUuWLMnKlStz0kknvaJjBwcHM3PmzPTfvyB9M6pO4Exoy+YtGesRAADgVdtWtuaW3JSBgYH09e35IvxXVVB+GwrKq6OgAACwL3ulBUVDAAAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABoxuSxHoBX5p83rBnrEfY5y+YtGesRAACo5AwKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgrsJWeUdbmx3DTWY+xTZFZPZvVkVk9m9WRWT2b1xktmk8d6gNfapLm//I23lw8fmPKRg/bSNPsGmdU5v6zKGXnwJetn54+yoZs+BhO1T2b1ZFZPZvVkVk9m9WRWb6JnNu4KyvA9r9u5cdMz6S5/KuX2I3euTdvlpFEpyfYkk7u9NV6TZFbvzhyav81bRqwNpHeMptk3yKyezOrJrJ7M6smsnszqTeTMxl1ByZxdnlLfpKTbZe2Hz2bSOzZk+Jq56S57Klk7lPKteemu3ZgMDKd8fe6OQ7uP/Tr5yVDKDYc/vzBckiv7010zmPx6e7Jgv5TzZidnjoMWK7NqW9OT/m7/EWvvKPdnWVmXw7IpGzMlP8rcfCXHZnO3+//NFpSn81e5JwvTn5Lk4UzP53JC7u9mJ0kWlSfyF7k3C9OfgfTmjszP17J4j/fXOpnVk1k9mdWTWT2Z1ZNZvYmc2b75L/Zb6i55MuXjBydH7ZfMnJRk48sftLI/3fUbUy47JFkwJfnRc+mWP5Yyuyc5ZeqozzzWZPbyhtPlqizJo5mWudmU92d13pcf5/M5Ybf7X5g786vMysocn+F0eX0Gsu2Fy8LmlmdyaX6Qq7M4n8nSzMxQlmdNlmf1S36asi+TWT2Z1ZNZPZnVk1k9mdWbKJlNyIJSzp+dnHbAKz9gqKRb2Z9y3bxk6QvfWB+1X3Ln5nTXDKSMw2+2X0xmI/1+Hsl3yo07tlflsHyqO3nH9mOZlq+XRflg7t7jJ405eTb/kIV5qOtLkjycGTtuOytrc3OOzI3dMTtuu6osyWdySz5XTsjWrmc0ntaoklk9mdWTWT2Z1ZNZPZnVm8iZTciCkuP2f/l9drVuS7rnSvKuDSPXt5Zk8cR4LaDMRlqTQ7Jyl08Gm9OT48tjOStrc0Q25oBsS0+G05vh9JZtGdrNqdLrc0xW5K6cXtbn7szJbTk8j7xw4duCDOToDORtZf2IY3qSzM2mrE/fqD6/0SCzejKrJ7N6Mqsns3oyqzeRM5uYBeWAF13gvbs3W962y983lSRJ+ca8ZO6L2uSUCXKxuMxG2JzJI95F49CyKRfnjvxjFuTqLM5gpmRxnshHclcmZzhDu7mPb3SL8v1yZE7KIzkxj+bP8tNcUk7KHd38TM22fC8L8u284SXHPZ6KM1kNkVk9mdWTWT2Z1ZNZPZnVm8iZTcyC8mIH9SRrt4xc+8nQznQWTknp7ZKHt47LaydeFZmNcEz606XkSzkupXu+gJ1W/uNlj3u4m5EbMiM3ZGE+Wv4ty7Iud2R+fpFZOSqD4/qtBGVWT2b1ZFZPZvVkVk9m9SZSZn5RY5Jy6tTknqHkusHkgS3pLn9y5Dff0yclfzkr3SeeeH6fdVuTH29Ovvr089sTkMxG2pDp2S8lb88vc1h5JqeXB3NmHtjj/lPK9iwvq3NseTxzyqYsKk9kYZ7K+hdeG3pt3pg35cksL6vz+vJ05peNOblsyPKyem89pVEns3oyqyezejKrJ7N6Mqs3kTJzBiVJ3jotOe/AdBc/mQyV5N0zknfOSH6282RZuWB2clBPupX9yfrHk76e5M29KR84cAwHH0MyG+GBbla+UI7Nu/LznJP7cm8OzteyOBdk1W73H06XvmzJBVmVWRnKYKbk9szP32VRkuTfu1n5cDkt5+S+XJFb0qVkQ6bn1hy+N5/WqJJZPZnVk1k9mdWTWT2Z1ZtImXWllLI3H3BwcDAzZ85M//0L0jfDCRxGz7J5S8Z6BAAAXrCtbM0tuSkDAwPp69vzRfgaAgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGZPHegAYLf+8Yc1Yj7DPWTZvyViPAABMcM6gAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAzTqjrMuN5aaxHmOfIrN6Mqsns3oyqyezeuMls8ljPcBrbdLcX/7G28uHD0z5yEF7aZp9g8zqyazO+WVVzsiDL1k/O3+UDd30MZiofTKrJ7N6Mqsns3oyqzfRM6sqKJdeemluuOGGrF27NlOnTs0pp5ySyy67LG984xtHa75qw/e8bufGTc+ku/yplNuP3Lk2bZeTRqUk25NM7vbWeE2SWT2Z1bszh+Zv85YRawPpHaNp9g0yqyezejKrJ7N6Mqs3kTOrKii33nprzj333LzlLW/Jtm3b8tGPfjRnnHFGfvrTn2batGmjNWOdObs8pb5JSbfL2g+fzaR3bMjwNXPTXfZUsnYo5Vvz0l27MRkYTvn63B2Hdh/7dfKToZQbDn9+YbgkV/anu2Yw+fX2ZMF+KefNTs4cBy1WZvVkVm1retLf7T9i7R3l/iwr63JYNmVjpuRHmZuv5Nhs7nb/qWlBeTp/lXuyMP0pSR7O9HwuJ+T+bnaSZFF5In+Re7Mw/RlIb+7I/Hwti/d4f62TWT2Z1ZNZPZnVk1m9iZxZ1aP/r//1v0Zsf/3rX8+cOXNy11135Q/+4A9e08FGU3fJkykfPzg5ar9k5qQkG1/+oJX96a7fmHLZIcmCKcmPnku3/LGU2T3JKVNHfeaxJrN6Mnt5w+lyVZbk0UzL3GzK+7M678uP8/mcsNv9L8yd+VVmZWWOz3C6vD4D2fbCpXRzyzO5ND/I1Vmcz2RpZmYoy7Mmy7P6JT+B2pfJrJ7M6smsnszqyazeRMnst6pHAwMDSZLZs2fvcZ+hoaEMDQ3t2B4cHPxtHvI1Uc6fnZx2wCs/YKikW9mfct28ZOkL3yQetV9y5+Z01wykjMNvHF9MZvVkNtLv55F8p9y4Y3tVDsunupN3bD+Wafl6WZQP5u49fqKdk2fzD1mYh7q+JMnDmbHjtrOyNjfnyNzYHbPjtqvKknwmt+Rz5YRs7XpG42mNKpnVk1k9mdWTWT2Z1ZvImb3qgjI8PJwPfehDOfXUU7N48eI97nfppZfmk5/85Kt9mNFx3P4vv8+u1m1J91xJ3rVh5PrWkiyeGK8FlNmrILMR1uSQrNzlE+jm9OT48ljOytockY05INvSk+H0Zji9ZVuGdnN6+fockxW5K6eX9bk7c3JbDs8jL1wsuCADOToDeVtZP+KYniRzsynr0zeqz280yKyezOrJrJ7M6sms3kTO7FUXlHPPPTf33Xdfbr/99t+430UXXZQVK1bs2B4cHMwRRxzxah/2tXHAiy5W3t2bLW/b5e+bSpKkfGNeMvdFbXLKBLnwWWb1ZDbC5kwe8c4jh5ZNuTh35B+zIFdncQYzJYvzRD6SuzI5wxnazX18o1uU75cjc1IeyYl5NH+Wn+aSclLu6OZnarble1mQb+cNLznu8VScyWqIzOrJrJ7M6smsnszqTeTMXlVBWb58eb773e/mtttuy+GHH/4b9+3t7U1vb+M//T2oJ1m7ZeTaT4Z2prNwSkpvlzy8dVxeB/CqyKyezEY4Jv3pUvKlHJfSPV/ATiv/8bLHPdzNyA2ZkRuyMB8t/5ZlWZc7Mj+/yKwclcFx/faLMqsns3oyqyezejKrN5Eyq/pFjaWULF++PDfeeGO+//3v5+ijjx6tufaqcurU5J6h5LrB5IEt6S5/cuQ3ktMnJX85K90nnnh+n3Vbkx9vTr769PPbE5DM6slspA2Znv1S8vb8MoeVZ3J6eTBn5oE97j+lbM/ysjrHlsczp2zKovJEFuaprH/h9bTX5o15U57M8rI6ry9PZ37ZmJPLhiwvq/fWUxp1Mqsns3oyqyezejKrN5EyqzqDcu655+ab3/xmbrrppsyYMSOPPvpokmTmzJmZOnUf/onvW6cl5x2Y7uInk6GSvHtG8s4Zyc92niwrF8xODupJt7I/Wf940teTvLk35QMHjuHgY0hm9WQ2wgPdrHyhHJt35ec5J/fl3hycr2VxLsiq3e4/nC592ZILsiqzMpTBTMntmZ+/y6Ikyb93s/LhclrOyX25IrekS8mGTM+t+c1nefclMqsns3oyqyezejKrN5Ey60op5RXv3O3+dfBXX3113vve976i+xgcHMzMmTPTf/+C9M2oOoEDjLJl85aM9QgAwDi1rWzNLbkpAwMD6evb80X4VWdQKroMAABANacwAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaMXmsBwDa8c8b1oz1CPucZfOWjPUIADCuOIMCAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIwjpxR1uXGctNYj7FPkVk9mdWTWT2Z1RsvmU0e6wFea5Pm/vI33l4+fGDKRw7aS9PsG2RWT2b1ZFbn/LIqZ+TBl6yfnT/Khm76GEzUPpnVk1k9mdWTWb2Jntm4KyjD97xu58ZNz6S7/KmU24/cuTZtl5NGpSTbk0zu9tZ4TZJZPZnVk1m9O3No/jZvGbE2kN4xmmbfILN6Mqsns3oyqzeRMxt3BSVzdnlKfZOSbpe1Hz6bSe/YkOFr5qa77Klk7VDKt+alu3ZjMjCc8vW5Ow7tPvbr5CdDKTcc/vzCcEmu7E93zWDy6+3Jgv1SzpudnDkOWqzM6smsnsyqbU1P+rv9R6y9o9yfZWVdDsumbMyU/Chz85Ucm83d7j+dLyhP569yTxamPyXJw5mez+WE3N/NTpIsKk/kL3JvFqY/A+nNHZmfr2XxHu+vdTKrJ7N6Mqsns3oTObN981/st9Rd8mTKxw9OjtovmTkpycaXP2hlf7rrN6ZcdkiyYEryo+fSLX8sZXZPcsrUUZ95rMmsnszqyezlDafLVVmSRzMtc7Mp78/qvC8/zudzwm73vzB35leZlZU5PsPp8voMZNsLlx/OLc/k0vwgV2dxPpOlmZmhLM+aLM/ql/zUbl8ms3oyqyezejKrN1Eym5AFpZw/OzntgFd+wFBJt7I/5bp5ydIXvuE5ar/kzs3prhlIGYffBL2YzOrJrJ7MRvr9PJLvlBt3bK/KYflUd/KO7ccyLV8vi/LB3L3HL05z8mz+IQvzUNeXJHk4M3bcdlbW5uYcmRu7Y3bcdlVZks/klnyunJCtXc9oPK1RJbN6Mqsns3oyqzeRM5uQBSXH7f/y++xq3ZZ0z5XkXRtGrm8tyeKJ8VpAmb0KMqsnsxHW5JCs3OWLzub05PjyWM7K2hyRjTkg29KT4fRmOL1lW4Z2c0r++hyTFbkrp5f1uTtzclsOzyMvXGC5IAM5OgN5W1k/4pieJHOzKevTN6rPbzTIrJ7M6smsnszqTeTMJmZBOeBFF97u7s2Wt+3y900lSVK+MS+Z+6I2OWWCXMQrs3oyqyezETZn8oh3azm0bMrFuSP/mAW5OoszmClZnCfykdyVyRnO0G7u4xvdony/HJmT8khOzKP5s/w0l5STckc3P1OzLd/Lgnw7b3jJcY+n4kxWQ2RWT2b1ZFZPZvUmcmYTs6C82EE9ydotI9d+MrQznYVTUnq75OGt4/I17a+KzOrJrJ7MRjgm/elS8qUcl9I9X8BOK//xssc93M3IDZmRG7IwHy3/lmVZlzsyP7/IrByVwXH9lpUyqyezejKrJ7N6Eykzv6gxSTl1anLPUHLdYPLAlnSXPznym6Lpk5K/nJXuE088v8+6rcmPNydfffr57QlIZvVkVk9mI23I9OyXkrfnlzmsPJPTy4M5Mw/scf8pZXuWl9U5tjyeOWVTFpUnsjBPZf0Lr0G+Nm/Mm/JklpfVeX15OvPLxpxcNmR5Wb23ntKok1k9mdWTWT2Z1ZtImTmDkiRvnZacd2C6i59Mhkry7hnJO2ckP9t5sqxcMDs5qCfdyv5k/eNJX0/y5t6UDxw4hoOPIZnVk1k9mY3wQDcrXyjH5l35ec7Jfbk3B+drWZwLsmq3+w+nS1+25IKsyqwMZTBTcnvm5++yKEny792sfLiclnNyX67ILelSsiHTc2sO35tPa1TJrJ7M6smsnszqTaTMulJK2ZsPODg4mJkzZ6b//gXpm+EEDrBvWzZvyViPAAD7hG1la27JTRkYGEhf354vwtcQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNmDzWAwDsy/55w5qxHmGfs2zekrEeAYCGOYMCAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggLAhHZGWZcby01jPcY+RWb1ZFZPZvXGS2aTx3qA19qkub/8jbeXDx+Y8pGD9tI0+waZ1ZNZPZnVk1md88uqnJEHX7J+dv4oG7rpYzBR+2RWT2b1ZFZvomf2WxWUv/7rv85FF12UD37wg/nsZz/7Go302xm+53U7N256Jt3lT6XcfuTOtWm7nDQqJdmeZHK3t8ZrkszqyayezOrJrN6dOTR/m7eMWBtI7xhNs2+QWT2Z1ZNZvYmc2asuKKtWrcqXvvSlHHvssa/lPL+9Obs8pb5JSbfL2g+fzaR3bMjwNXPTXfZUsnYo5Vvz0l27MRkYTvn63B2Hdh/7dfKToZQbDn9+YbgkV/anu2Yw+fX2ZMF+KefNTs4cBy1WZvVkVk9m9WRWbWt60t/tP2LtHeX+LCvrclg2ZWOm5EeZm6/k2Gzudv8lcEF5On+Ve7Iw/SlJHs70fC4n5P5udpJkUXkif5F7szD9GUhv7sj8fC2L93h/rZNZPZnVk1m9iZzZq3r0Z555Ju95z3vyla98JRdffPFrPdOo6y55MuXjBydH7ZfMnJRk48sftLI/3fUbUy47JFkwJfnRc+mWP5Yyuyc5ZeqozzzWZFZPZvVkVk9mL284Xa7KkjyaaZmbTXl/Vud9+XE+nxN2u/+FuTO/yqyszPEZTpfXZyDbXrhkc255JpfmB7k6i/OZLM3MDGV51mR5Vr/kJ537MpnVk1k9mdWbKJm9qoJy7rnn5o//+I9z+umnv2xBGRoaytDQ0I7twcHBV/OQr6ly/uzktANe+QFDJd3K/pTr5iVLX/jifdR+yZ2b010zkDIOv6C/mMzqyayezOrJbKTfzyP5Trlxx/aqHJZPdSfv2H4s0/L1sigfzN17/II+J8/mH7IwD3V9SZKHM2PHbWdlbW7OkbmxO2bHbVeVJflMbsnnygnZ2vWMxtMaVTKrJ7N6Mqs3kTOrLijf+ta3cvfdd2fVqlWvaP9LL700n/zkJ6sHG1XH7f/y++xq3ZZ0z5XkXRtGrm8tyeKJ8VpAmb0KMqsns3oyG2FNDsnKXb5Qb05Pji+P5ayszRHZmAOyLT0ZTm+G01u2ZWg3L2O4PsdkRe7K6WV97s6c3JbD88gLF6UuyECOzkDeVtaPOKYnydxsyvr0jerzGw0yqyezejKrN5EzqyooDz30UD74wQ/mX/7lX7L//q/si+JFF12UFStW7NgeHBzMEUccUTfla+2AF11Eurs3W962y983lSRJ+ca8ZO6L2uSUCXJBqszqyayezOrJbITNmTziHW4OLZtyce7IP2ZBrs7iDGZKFueJfCR3ZXKGM7Sb+/hGtyjfL0fmpDySE/No/iw/zSXlpNzRzc/UbMv3siDfzhtectzjqTiT1RCZ1ZNZPZnVm8iZVRWUu+66K48//nhOOGFnm9u+fXtuu+22XHnllRkaGkpPz8gveL29ventbfyncgf1JGu3jFz7ydDOdBZOSentkoe3jsvXZ78qMqsns3oyqyezEY5Jf7qUfCnHpXTPF7DTyn+87HEPdzNyQ2bkhizMR8u/ZVnW5Y7Mzy8yK0dlcFy/zafM6smsnszqTaTMqn5R49ve9rbce++9WbNmzY4/S5cuzXve856sWbPmJeVkX1FOnZrcM5RcN5g8sCXd5U+O/AI/fVLyl7PSfeKJ5/dZtzX58ebkq08/vz0ByayezOrJrJ7MRtqQ6dkvJW/PL3NYeSanlwdzZh7Y4/5TyvYsL6tzbHk8c8qmLCpPZGGeyvoXXrd9bd6YN+XJLC+r8/rydOaXjTm5bMjysnpvPaVRJ7N6Mqsns3oTKbOqMygzZszI4sWLR6xNmzYtBx100EvW9ylvnZacd2C6i59Mhkry7hnJO2ckP9t5sqxcMDs5qCfdyv5k/eNJX0/y5t6UDxw4hoOPIZnVk1k9mdWT2QgPdLPyhXJs3pWf55zcl3tzcL6Wxbkgu7+Ocjhd+rIlF2RVZmUog5mS2zM/f5dFSZJ/72blw+W0nJP7ckVuSZeSDZmeW3P43nxao0pm9WRWT2b1JlJmXSml/DZ38Id/+IdZsmTJK/5FjYODg5k5c2b671+QvhlVJ3AAGAeWzVsy1iMAMAa2la25JTdlYGAgfX17vgj/t/4tLLfccstvexcAAABJKq9BAQAAGE0KCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZkwe6wEAmFj+ecOasR5hn7Ns3pKxHgFgr3EGBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAKhyRlmXG8tNYz3GPkVm9WRWb7xkNnmsB3itTZr7y994e/nwgSkfOWgvTbNvkFk9mdWTWT2Z1ZNZnfPLqpyRB1+yfnb+KBu66WMwUftkVk9m9SZ6ZuOuoAzf87qdGzc9k+7yp1JuP3Ln2rRdThqVkmxPMrnbW+M1SWb1ZFZPZvVkVk9m9e7MofnbvGXE2kB6x2iafYPM6sms3kTObNwVlMzZ5Sn1TUq6XdZ++GwmvWNDhq+Zm+6yp5K1Qynfmpfu2o3JwHDK1+fuOLT72K+Tnwyl3HD48wvDJbmyP901g8mvtycL9ks5b3Zy5jhosTKrJ7N6Mqsns3oyq7Y1Penv9h+x9o5yf5aVdTksm7IxU/KjzM1Xcmw2d7v/tmFBeTp/lXuyMP0pSR7O9HwuJ+T+bnaSZFF5In+Re7Mw/RlIb+7I/Hwti/d4f62TWT2Z1ZvIme2b/2K/pe6SJ1M+fnBy1H7JzElJNr78QSv7012/MeWyQ5IFU5IfPZdu+WMps3uSU6aO+sxjTWb1ZFZPZvVkVk9mL284Xa7KkjyaaZmbTXl/Vud9+XE+nxN2u/+FuTO/yqyszPEZTpfXZyDbXrjMdW55JpfmB7k6i/OZLM3MDGV51mR5Vr/kp8P7MpnVk1m9iZLZhCwo5fzZyWkHvPIDhkq6lf0p181Llr7wheio/ZI7N6e7ZiBlHH5xejGZ1ZNZPZnVk1k9mY30+3kk3yk37thelcPyqe7kHduPZVq+Xhblg7l7j98Ezcmz+YcszENdX5Lk4czYcdtZWZubc2Ru7I7ZcdtVZUk+k1vyuXJCtnY9o/G0RpXM6sms3kTObEIWlBy3/8vvs6t1W9I9V5J3bRi5vrUkiyfGawFl9irIrJ7M6smsnsxGWJNDsnKXb242pyfHl8dyVtbmiGzMAdmWngynN8PpLdsytJuXflyfY7Iid+X0sj53Z05uy+F55IULeRdkIEdnIG8r60cc05NkbjZlffpG9fmNBpnVk1m9iZzZxCwoB7zogsjdvdnytl3+vqkkSco35iVzX9Qmp0yQiytlVk9m9WRWT2b1ZDbC5kwe8a5Ah5ZNuTh35B+zIFdncQYzJYvzRD6SuzI5wxnazX18o1uU75cjc1IeyYl5NH+Wn+aSclLu6OZnarble1mQb+cNLznu8VScyWqIzOrJrN5EzmxiFpQXO6gnWbtl5NpPhnams3BKSm+XPLx1XL7W+FWRWT2Z1ZNZPZnVk9kIx6Q/XUq+lONSuucL2GnlP172uIe7GbkhM3JDFuaj5d+yLOtyR+bnF5mVozI4rt8aVWb1ZFZvImXmFzUmKadOTe4ZSq4bTB7Yku7yJ0d+sZo+KfnLWek+8cTz+6zbmvx4c/LVp5/fnoBkVk9m9WRWT2b1ZDbShkzPfil5e36Zw8ozOb08mDPzwB73n1K2Z3lZnWPL45lTNmVReSIL81TWv/Ba92vzxrwpT2Z5WZ3Xl6czv2zMyWVDlpfVe+spjTqZ1ZNZvYmUmTMoSfLWacl5B6a7+MlkqCTvnpG8c0bys50ny8oFs5ODetKt7E/WP5709SRv7k35wIFjOPgYklk9mdWTWT2Z1ZPZCA90s/KFcmzelZ/nnNyXe3NwvpbFuSCrdrv/cLr0ZUsuyKrMylAGMyW3Z37+LouSJP/ezcqHy2k5J/flitySLiUbMj235vC9+bRGlczqyazeRMqsK6WUvfmAg4ODmTlzZvrvX5C+GU7gAMDLWTZvyViPAPBb21a25pbclIGBgfT17fkifA0BAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANCMyWM9AADwm/3zhjVjPcI+Z9m8JWM9AvAqOYMCAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAMMrOKOtyY7lprMfYp8is3njJbPJYD/BamzT3l7/x9vLhA1M+ctBemmbfILN6Mqsns3oyqyezejKrc35ZlTPy4EvWz84fZUM3fQwmap/M6k30zKoLysMPP5wLLrgg//RP/5Rnn302b3jDG3L11Vdn6dKlozFfteF7Xrdz46Zn0l3+VMrtR+5cm7bLSaNSku1JJnd7a7wmyayezOrJrJ7M6smsnszq3ZlD87d5y4i1gfSO0TT7BpnVm8iZVRWU/v7+nHrqqXnrW9+af/qnf8ohhxySX/ziFznwwANHa756c3Z5Sn2Tkm6XtR8+m0nv2JDha+amu+ypZO1Qyrfmpbt2YzIwnPL1uTsO7T726+QnQyk3HP78wnBJruxPd81g8uvtyYL9Us6bnZw5DlqszOrJrJ7M6smsnszqyaza1vSkv9t/xNo7yv1ZVtblsGzKxkzJjzI3X8mx2dzt/lutBeXp/FXuycL0pyR5ONPzuZyQ+7vZSZJF5Yn8Re7NwvRnIL25I/PztSze4/21Tmb1JnJmVY9+2WWX5YgjjsjVV1+9Y+3oo49+zYcabd0lT6Z8/ODkqP2SmZOSbHz5g1b2p7t+Y8plhyQLpiQ/ei7d8sdSZvckp0wd9ZnHmszqyayezOrJrJ7M6sns5Q2ny1VZkkczLXOzKe/P6rwvP87nc8Ju978wd+ZXmZWVOT7D6fL6DGTbC5cGzy3P5NL8IFdncT6TpZmZoSzPmizP6pf8RH1fJrN6EyWzqoLyne98J8uWLcs73/nO3HrrrZk/f37+63/9r3nf+963x2OGhoYyNDS0Y3twcPDVT/saKefPTk474JUfMFTSrexPuW5esvSFT6pH7ZfcuTndNQMp4/AT7YvJrJ7M6smsnszqyayezEb6/TyS75Qbd2yvymH5VHfyju3HMi1fL4vywdy9x28c5+TZ/EMW5qGuL0nycGbsuO2srM3NOTI3dsfsuO2qsiSfyS35XDkhW7ue0Xhao0pm9SZyZlUF5YEHHsgXvvCFrFixIh/96EezatWqfOADH8iUKVNy9tln7/aYSy+9NJ/85Cdfk2FfM8ft//L77GrdlnTPleRdG0auby3J4onxWkCZvQoyqyezejKrJ7N6MhthTQ7Jyl2+IdycnhxfHstZWZsjsjEHZFt6MpzeDKe3bMvQbl4uc32OyYrcldPL+tydObkth+eRFy5+XpCBHJ2BvK2sH3FMT5K52ZT16RvV5zcaZFZvImdWVVCGh4ezdOnSXHLJJUmS448/Pvfdd1+++MUv7rGgXHTRRVmxYsWO7cHBwRxxxBG/xcivgQNedHHf7t5sedsuf99UkiTlG/OSuS9qk1MmyIWCMqsns3oyqyezejKrJ7MRNmfyiHdSOrRsysW5I/+YBbk6izOYKVmcJ/KR3JXJGc7Qbu7jG92ifL8cmZPySE7Mo/mz/DSXlJNyRzc/U7Mt38uCfDtveMlxj6fiTFZDZFZvImdWVVDmzp2bN73pTSPWfvd3fzfXX3/9Ho/p7e1Nb2/jPy05qCdZu2Xk2k+GdqazcEpKb5c8vHVcvm72VZFZPZnVk1k9mdWTWT2ZjXBM+tOl5Es5LqV7voCdVv7jZY97uJuRGzIjN2RhPlr+LcuyLndkfn6RWTkqg+P67WRlVm8iZVb1ixpPPfXU/PznPx+xdv/99+eoo456TYfa28qpU5N7hpLrBpMHtqS7/MmRn3inT0r+cla6Tzzx/D7rtiY/3px89enntycgmdWTWT2Z1ZNZPZnVk9lIGzI9+6Xk7fllDivP5PTyYM7MA3vcf0rZnuVldY4tj2dO2ZRF5YkszFNZ/8L1AdfmjXlTnszysjqvL09nftmYk8uGLC+r99ZTGnUyqzeRMqs6g3LeeefllFNOySWXXJI//dM/zZ133pkvf/nL+fKXvzxa8+0db52WnHdguoufTIZK8u4ZyTtnJD/bebKsXDA7Oagn3cr+ZP3jSV9P8ubelA809BbLe5PM6smsnszqyayezOrJbIQHuln5Qjk278rPc07uy705OF/L4lyQVbvdfzhd+rIlF2RVZmUog5mS2zM/f5dFSZJ/72blw+W0nJP7ckVuSZeSDZmeW3P43nxao0pm9SZSZl0ppdQc8N3vfjcXXXRRfvGLX+Too4/OihUrfuO7eL3Y4OBgZs6cmf77F6RvRtUJHACAV2TZvCVjPQLwItvK1tySmzIwMJC+vj1fhF/9W1jOPPPMnHnmmb/VcAAAALvjFAYAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMmj/UAAACvtX/esGasR9jnLJu3ZKxHgCTOoAAAAA1RUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAADQnDPKutxYbhrrMfYp4yWzyWM9wGtt0txf/sbby4cPTPnIQXtpmn2DzOrJrJ7M6smsnszqyayezOqcX1bljDz4kvWz80fZ0E0fg4naN9EzG3cFZfie1+3cuOmZdJc/lXL7kTvXpu1y0qiUZHuSyd3eGq9JMqsns3oyqyezejKrJ7N6Mqt3Zw7N3+YtI9YG0jtG0+wbJnJm466gZM4uT6lvUtLtsvbDZzPpHRsyfM3cdJc9lawdSvnWvHTXbkwGhlO+PnfHod3Hfp38ZCjlhsOfXxguyZX96a4ZTH69PVmwX8p5s5Mzx0GLlVk9mdWTWT2Z1ZNZPZnVk1m1relJf7f/iLV3lPuzrKzLYdmUjZmSH2VuvpJjs7nb/benC8rT+avck4XpT0nycKbnczkh93ezkySLyhP5i9ybhenPQHpzR+bna1m8x/tr3UTObN/8F/stdZc8mfLxg5Oj9ktmTkqy8eUPWtmf7vqNKZcdkiyYkvzouXTLH0uZ3ZOcMnXUZx5rMqsns3oyqyezejKrJ7N6Mnt5w+lyVZbk0UzL3GzK+7M678uP8/mcsNv9L8yd+VVmZWWOz3C6vD4D2fbC5dRzyzO5ND/I1Vmcz2RpZmYoy7Mmy7P6JWch9mUTJbMJWVDK+bOT0w545QcMlXQr+1Oum5csfeETxFH7JXduTnfNQMo4/KTxYjKrJ7N6Mqsns3oyqyezejIb6ffzSL5TbtyxvSqH5VPdyTu2H8u0fL0sygdz9x6/2Z6TZ/MPWZiHur4kycOZseO2s7I2N+fI3Ngds+O2q8qSfCa35HPlhGztekbjaY2qiZzZhCwoOW7/l99nV+u2pHuuJO/aMHJ9a0kWT4zXAsrsVZBZPZnVk1k9mdWTWT2ZjbAmh2TlLt9Eb05Pji+P5ayszRHZmAOyLT0ZTm+G01u2ZWg3LzG6PsdkRe7K6WV97s6c3JbD88gLF4wvyECOzkDeVtaPOKYnydxsyvr0jerzGw0TObOJWVAOeNGFart7s+Vtu/x9U0mSlG/MS+a+qE1OmSAXvcmsnszqyayezOrJrJ7M6slshM2ZPOLdpw4tm3Jx7sg/ZkGuzuIMZkoW54l8JHdlcoYztJv7+Ea3KN8vR+akPJIT82j+LD/NJeWk3NHNz9Rsy/eyIN/OG15y3OOpOJPVkImc2cQsKC92UE+ydsvItZ8M7Uxn4ZSU3i55eOu4fA3oqyKzejKrJ7N6Mqsns3oyqyezEY5Jf7qUfCnHpXTPF7DTyn+87HEPdzNyQ2bkhizMR8u/ZVnW5Y7Mzy8yK0dlcFy/Be9EyswvakxSTp2a3DOUXDeYPLAl3eVPjvwkMn1S8pez0n3iief3Wbc1+fHm5KtPP789AcmsnszqyayezOrJrJ7M6slspA2Znv1S8vb8MoeVZ3J6eTBn5oE97j+lbM/ysjrHlsczp2zKovJEFuaprH/hmopr88a8KU9meVmd15enM79szMllQ5aX1XvrKY26iZSZMyhJ8tZpyXkHprv4yWSoJO+ekbxzRvKznSfLygWzk4N60q3sT9Y/nvT1JG/uTfnAgWM4+BiSWT2Z1ZNZPZnVk1k9mdWT2QgPdLPyhXJs3pWf55zcl3tzcL6Wxbkgq3a7/3C69GVLLsiqzMpQBjMlt2d+/i6LkiT/3s3Kh8tpOSf35Yrcki4lGzI9t+bwvfm0RtVEyqwrpZS9+YCDg4OZOXNm+u9fkL4ZTuAAALRg2bwlYz0C49y2sjW35KYMDAykr2/PF+FrCAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZkwe6wEAABh7/7xhzViPsM9ZNm/JWI8wLjmDAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAADjwBllXW4sN431GL+1yWM9wGtt0txf/sbby4cPTPnIQXtpmn2DzOrJrJ7M6smsnszqyayezOrJrM75ZVXOyIMvWT87f5QN3fQxmGjvqioo27dvz3//7/8911xzTR599NHMmzcv733ve/Pf/tt/S9d1ozVjleF7Xrdz46Zn0l3+VMrtR+5cm7bLSaNSku1JJrcx+1iRWT2Z1ZNZPZnVk1k9mdWTWT2Z1bszh+Zv85YRawPpHaNp9q6ql3hddtll+cIXvpArr7wyP/vZz3LZZZflb/7mb/L5z39+tOarN2fyzj99k5Jul7VfbsmkNzyQ3Lwp3RkPpTvqV8mdz6X74GPp3vvIiLvpPvbrdP/3f+xcGC7JyqfSnbgu3dG/Sve29cl3n9m7z220yKyezOrJrJ7M6smsnszqyayezKptTU/6u/1H/Pm/8ot8ufx/851yY/6+fC/vL3dn/7Jtj/exoDydy8utual8O98u385V5f+XheWpHbcvKk/kivL/5rvlhvx9+V7+a1nzG+9vb6k6g/LDH/4w//k//+f88R//cZLkda97Xf7n//yfufPOO0dluNHSXfJkyscPTo7aL5k5KcnGlz9oZX+66zemXHZIsmBK8qPn0i1/LGV2T3LK1FGfeazJrJ7M6smsnszqyayezOrJrJ7MXt5wulyVJXk00zI3m/L+rM778uN8Pifsdv8Lc2d+lVlZmeMznC6vz0C2vXB+Ym55JpfmB7k6i/OZLM3MDGV51mR5Vr/kzM3eVlVQTjnllHz5y1/O/fffn4ULF+aee+7J7bffniuuuGKPxwwNDWVoaGjH9uDg4Kuf9jVSzp+dnHbAKz9gqKRb2Z9y3bxk6Qsf7Eftl9y5Od01Aynj8H+AF5NZPZnVk1k9mdWTWT2Z1ZNZPZmN9Pt5JN8pN+7YXpXD8qnu5B3bj2Vavl4W5YO5e48FZU6ezT9kYR7q+pIkD2fGjtvOytrcnCNzY3fMjtuuKkvymdySz5UTsrXrGY2n9YpUFZQLL7wwg4OD+Z3f+Z309PRk+/bt+fSnP533vOc9ezzm0ksvzSc/+cnfetDX1HH71+2/bku650ryrg0j17eWZPHEeC2gzF4FmdWTWT2Z1ZNZPZnVk1k9mY2wJodk5S7FY3N6cnx5LGdlbY7IxhyQbenJcHoznN6yLUPdS7+tvz7HZEXuyullfe7OnNyWw/PICxfZL8hAjs5A3lbWjzimJ8ncbMr69I3q8/tNqgrKddddl7//+7/PN7/5zSxatChr1qzJhz70ocybNy9nn332bo+56KKLsmLFih3bg4ODOeKII367qX9bB7zooqvdXYmz68vvNpUkSfnGvGTui9rklAlyAZfM6smsnszqyayezOrJrJ7M6slshM2ZPOIduw4tm3Jx7sg/ZkGuzuIMZkoW54l8JHdlcoYztJv7+Ea3KN8vR+akPJIT82j+LD/NJeWk3NHNz9Rsy/eyIN/OG15y3OOpOJM1CqoKyvnnn58LL7ww7373u5Mkb37zm/Pggw/m0ksv3WNB6e3tTW9v4y32oJ5k7ZaRaz8Z2pnOwikpvV3y8NZx+XrGV0Vm9WRWT2b1ZFZPZvVkVk9m9WQ2wjHpT5eSL+W4lBfePfe08h8vc1TycDcjN2RGbsjCfLT8W5ZlXe7I/Pwis3JUBpt82+KqgvLss89m0qSRdbanpyfDw8Ov6VB7Wzl1arr/8XRy3WCydP901298/n+IxVOe32H6pOQvZ6X7xBMpJcmJU5PB7cmqzcmMScmfjt0psLEis3oyqyezejKrJ7N6Mqsns3oyG2lDpme/lLw9v8y/lrlZnCdzZh7Y4/5Tyvb8f/Lj3Jb5eTTTckiey8I8ldszP0lybd6Ylfl/s7yszj/l6GxOT47MxvxeHsuV3fF762ntVlVB+ZM/+ZN8+tOfzpFHHplFixZl9erVueKKK3LOOeeM1nx7x1unJecdmO7iJ5Ohkrx7RvLOGcnPdp4sKxfMTg7qSbeyP1n/eNLXk7y5N+UDB47h4GNIZvVkVk9m9WRWT2b1ZFZPZvVkNsID3ax8oRybd+XnOSf35d4cnK9lcS7Iqt3uP5wufdmSC7IqszKUwUzJ7Zmfv8uiJMm/d7Py4XJazsl9uSK3pEvJhkzPrTl8bz6t3epKKeWV7rxx48Z87GMfy4033pjHH3888+bNy1lnnZWPf/zjmTJlyiu6j8HBwcycOTP99y9I34yqX8MCAADNWDZvyViPsE/ZVrbmltyUgYGB9PXt+QxXVUF5LSgoAACMBwpKnVdaUDQEAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmTB7rAQAAYF/0zxvWjPUI+5TBjcM5cOHL7+cMCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQjMljPcBrbdLcX/7G28uHD0z5yEF7aZp9g8zqyayezOrJrJ7M6smsnszqyazeRM6sK6WUvfmAg4ODmTlzZvrvX5C+GaNwAufxbTv/ftMz6S5/KuX2I3euTZv0/J8kKSXZnmRy99rPsS+RWT2Z1ZNZPZnVk1k9mdWTWT2Z1RuHmQ1uHM6BCx/IwMBA+vr69rjf+HuJ15zJO//0TUq6XdZ+uSWT3vBAcvOmdGc8lO6oXyV3Ppfug4+le+8jI+6m+9iv0/3f/7FzYbgkK59Kd+K6dEf/Kt3b1ifffWbvPrfRIrN6Mqsns3oyqyezejKrJ7N6Mqs3gTMbdy/xeiW6S55M+fjByVH7JTMnJdn48get7E93/caUyw5JFkxJfvRcuuWPpczuSU6ZOuozjzWZ1ZNZPZnVk1k9mdWTWT2Z1ZNZvfGa2YQsKOX82clpB7zyA4ZKupX9KdfNS5a+8A931H7JnZvTXTOQ0sg/5miSWT2Z1ZNZPZnVk1k9mdWTWT2Z1RuvmU3IgpLj9q/bf92WdM+V5F0bRq5vLcni3tdurpbJrJ7M6smsnszqyayezOrJrJ7M6o3TzCZmQTngRRcQ7e5KnF2uS8qm599HoHxjXjK3Z+R+U9q+GOk1I7N6Mqsns3oyqyezejKrJ7N6Mqs3TjObmAXlxQ7qSdZuGbn2k6Gd6SycktLbJQ9vbea1eWNOZvVkVk9m9WRWT2b1ZFZPZvVkVm+cZKagJCmnTk33P55OrhtMlu6f7vqNz//jLp7y/A7TJyV/OSvdJ55IKUlOnJoMbk9WbU5mTEr+dM9vkzZeyayezOrJrJ7M6smsnszqyayezOqNl8wUlCR567TkvAPTXfxkMlSSd89I3jkj+dnQjl3KBbOTg3rSrexP1j+e9PUkb+5N+cCBYzj4GJJZPZnVk1k9mdWTWT2Z1ZNZPZnVGyeZjb9f1AgAADRn4v6iRgAAYJ+loAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA0Q0EBAACaoaAAAADNUFAAAIBmKCgAAEAzFBQAAKAZCgoAANAMBQUAAGiGggIAADRDQQEAAJqhoAAAAM1QUAAAgGYoKAAAQDMUFAAAoBkKCgAA0AwFBQAAaMbkvf2ApZQkyeAzw3v7oQEAgDHyf77//z99YE/2ekHZuHFjkuSoE9bt7YcGAADG2MaNGzNz5sw93t6Vl6swr7Hh4eFs2LAhM2bMSNd1e/Ohf6PBwcEcccQReeihh9LX1zfW4zBO+Thjb/Bxxt7g44y9wcfZ+FJKycaNGzNv3rxMmrTnK032+hmUSZMm5fDDD9/bD/uK9fX1+R+AUefjjL3Bxxl7g48z9gYfZ+PHbzpz8n+4SB4AAGiGggIAADRDQXlBb29vPvGJT6S3t3esR2Ec83HG3uDjjL3Bxxl7g4+ziWmvXyQPAACwJ86gAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAXlBVdddVVe97rXZf/9989JJ52UO++8c6xHYhy59NJL85a3vCUzZszInDlz8va3vz0///nPx3osxrG//uu/Ttd1+dCHPjTWozAOPfzww/kv/+W/5KCDDsrUqVPz5je/Of/7f//vsR6LcWT79u352Mc+lqOPPjpTp07N61//+nzqU5+KN5+dGBSUJNdee21WrFiRT3ziE7n77rtz3HHHZdmyZXn88cfHejTGiVtvvTXnnntufvSjH+Vf/uVfsnXr1pxxxhnZtGnTWI/GOLRq1ap86UtfyrHHHjvWozAO9ff359RTT81+++2Xf/qnf8pPf/rTfOYzn8mBBx441qMxjlx22WX5whe+kCuvvDI/+9nPctlll+Vv/uZv8vnPf36sR2Mv8HtQkpx00kl5y1vekiuvvDJJMjw8nCOOOCLvf//7c+GFF47xdIxHv/71rzNnzpzceuut+YM/+IOxHodx5JlnnskJJ5yQ//E//kcuvvjiLFmyJJ/97GfHeizGkQsvvDB33HFHfvCDH4z1KIxjZ555Zg499NB89atf3bH2jne8I1OnTs0111wzhpOxN0z4MyhbtmzJXXfdldNPP33H2qRJk3L66afnX//1X8dwMsazgYGBJMns2bPHeBLGm3PPPTd//Md/POJzGryWvvOd72Tp0qV55zvfmTlz5uT444/PV77ylbEei3HmlFNOyc0335z7778/SXLPPffk9ttvz3/6T/9pjCdjb5g81gOMtSeeeCLbt2/PoYceOmL90EMPzdq1a8doKsaz4eHhfOhDH8qpp56axYsXj/U4jCPf+ta3cvfdd2fVqlVjPQrj2AMPPJAvfOELWbFiRT760Y9m1apV+cAHPpApU6bk7LPPHuvxGCcuvPDCDA4O5nd+53fS09OT7du359Of/nTe8573jPVo7AUTvqDA3nbuuefmvvvuy+233z7WozCOPPTQQ/ngBz+Yf/mXf8n+++8/1uMwjg0PD2fp0qW55JJLkiTHH3987rvvvnzxi19UUHjNXHfddfn7v//7fPOb38yiRYuyZs2afOhDH8q8efN8nE0AE76gHHzwwenp6cljjz02Yv2xxx7LYYcdNkZTMV4tX7483/3ud3Pbbbfl8MMPH+txGEfuuuuuPP744znhhBN2rG3fvj233XZbrrzyygwNDaWnp2cMJ2S8mDt3bt70pjeNWPvd3/3dXH/99WM0EePR+eefnwsvvDDvfve7kyRvfvOb8+CDD+bSSy9VUCaACX8NypQpU/J7v/d7ufnmm3esDQ8P5+abb87JJ588hpMxnpRSsnz58tx44435/ve/n6OPPnqsR2Kcedvb3pZ77703a9as2fFn6dKlec973pM1a9YoJ7xmTj311Je8Tfr999+fo446aowmYjx69tlnM2nSyG9Te3p6Mjw8PEYTsTdN+DMoSbJixYqcffbZWbp0aU488cR89rOfzaZNm/Lnf/7nYz0a48S5556bb37zm7npppsyY8aMPProo0mSmTNnZurUqWM8HePBjBkzXnJN07Rp03LQQQe51onX1HnnnZdTTjkll1xySf70T/80d955Z7785S/ny1/+8liPxjjyJ3/yJ/n0pz+dI488MosWLcrq1atzxRVX5Jxzzhnr0dgLvM3wC6688spcfvnlefTRR7NkyZKsXLkyJ5100liPxTjRdd1u16+++uq8973v3bvDMGH84R/+obcZZlR897vfzUUXXZRf/OIXOfroo7NixYq8733vG+uxGEc2btyYj33sY7nxxhvz+OOPZ968eTnrrLPy8Y9/PFOmTBnr8RhlCgoAANCMCX8NCgAA0A4FBQAAaIaCAgAANENBAQAAmqGgAAAAzVBQAACAZigoAABAMxQUAACgGQoKAADQDAUFAABohoICAAA04/8Pp2TqsxxSD9QAAAAASUVORK5CYII=\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = MultiHeadAttention(\n", + " in_size=10,\n", + " head_size=4,\n", + " num_heads=3,\n", + " out_size=15,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "cu4gFFYBbWMa", + "outputId": "58aba7a8-a103-4f17-b658-ff445ab6ac8d" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=12, bias=False)\n", + " (out): Linear(in_features=12, out_features=15, bias=True)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 9 + } + ] + }, + { + "cell_type": "code", + "source": [ + "tmp_input = torch.rand(2, 5, 10)\n", + "tmp_mask = make_decoder_mask(tmp_input)\n", + "\n", + "print(\"Decoder-like input, with mask\")\n", + "print(f'Input shape: {tmp_input.shape}')\n", + "print(f'Mask shape: {tmp_mask.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input, tmp_input, tmp_input, tmp_mask)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input, tmp_mask, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "gPA5Ba6UbYFP", + "outputId": "7790ce5c-c428-42e0-8d84-da9994d01c1e" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Decoder-like input, with mask\n", + "Input shape: torch.Size([2, 5, 10])\n", + "Mask shape: torch.Size([2, 1, 5, 5])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.2 Positional Encoding\n", + "\n", + "#### Main class - PositionalEncoding\n", + "\n", + "**Initialization:**\n", + "* max_seq_len ~ the maximum token length of the sequence\n", + "* emb_size ~ the embedding size of the input\n", + "\n", + "**Forward:**\n", + "* _decoder_emb_ ~ embeddings of tokens from the decoder input\n", + "\n", + "$$\\text{PE}_{(\\text{pos}, 2i)} = sin\\Bigg( \\frac{\\text{pos}}{10000^{\\frac{2i}{\\text{emb_size}}}} \\Bigg) \\quad ; \\quad \\text{PE}_{(\\text{pos}, 2i + 1)} = cos\\Bigg( \\frac{\\text{pos}}{10000^{\\frac{2i}{\\text{emb_size}}}} \\Bigg)$$" + ], + "metadata": { + "id": "nqpfRJPlbVpf" + } + }, + { + "cell_type": "code", + "source": [ + "class PositionalEncoding(nn.Module):\n", + " \"\"\"\n", + " Class to calculate Positional Encodings, suggested in `Attention is all you need [Vaswaniet al., 2017]`\n", + " \"\"\"\n", + " def __init__(self, max_seq_len, emb_size):\n", + " \"\"\"\n", + " Args:\n", + " max_seq_len: max length of input sequence\n", + " emb_size: demension of embedding\n", + " \"\"\"\n", + " super(PositionalEncoding, self).__init__()\n", + "\n", + " # Запишем все переданые гиперпараметры слоя\n", + " self.max_seq_len = max_seq_len\n", + " self.emb_size = emb_size\n", + "\n", + " # Посчитаем позиционные эмбеддинги в тензорном виде\n", + " pos = torch.arange(max_seq_len)[:, None]\n", + " inds = torch.arange(emb_size)[None, ::2]\n", + "\n", + " pe = torch.zeros(max_seq_len, self.emb_size)\n", + " pe[:, ::2] = torch.sin(pos / (10000 ** ((2 * inds) / self.emb_size)))\n", + " pe[:, 1::2] = torch.cos(pos / (10000 ** ((2 * inds) / self.emb_size)))\n", + " pe = pe.unsqueeze(0)\n", + "\n", + " # Добавляем полученный тензор как параметр, который будет сохранятся вместе с моделью, но не будет обучаться\n", + " self.register_buffer('pe', pe)\n", + "\n", + "\n", + " def forward(self, decoder_emb):\n", + " \"\"\"\n", + " Args:\n", + " decoder_emb: decoder sequence after embed\n", + " Returns:\n", + " output: input with positional encodings\n", + " \"\"\"\n", + " # Тензоры приходят размера batch_size x seq_len x emb_size\n", + " seq_len = decoder_emb.size(1)\n", + "\n", + " # Прибавляем позиционные эмбеддинги\n", + " return decoder_emb + self.pe[:, :seq_len]" + ], + "metadata": { + "id": "oo22_W7gbql2" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing PositionalEncoding" + ], + "metadata": { + "id": "CUlw9fUJbvW0" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = PositionalEncoding(\n", + " max_seq_len=5,\n", + " emb_size=10,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Qq1zwI4dbysA", + "outputId": "d4f63ad9-9541-4cfb-c9f3-e80528fba9d8" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "PositionalEncoding()" + ] + }, + "metadata": {}, + "execution_count": 15 + } + ] + }, + { + "cell_type": "code", + "source": [ + "tmp_input = torch.rand(2, 5, 10)\n", + "\n", + "print(f'Input shape: {tmp_input.shape}')\n", + "tmp_output = tmp_layer(tmp_input)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "xUDvRmxLb1RM", + "outputId": "5eae2649-e3de-4500-b55e-0d3471d8b2d5" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Input shape: torch.Size([2, 5, 10])\n", + "Output shape: torch.Size([2, 5, 10])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let’s examine the positional encodings." + ], + "metadata": { + "id": "b1SMX9aWb4-a" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer.pe.shape" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "zLpl1jWzb9As", + "outputId": "68c293f3-800c-45eb-bec3-86db51edbe24" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "torch.Size([1, 5, 10])" + ] + }, + "metadata": {}, + "execution_count": 17 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "the article substantiates [Attention is all you need [Vaswaniet al., 2017]](https://www.semanticscholar.org/reader/204e3073870fae3d05bcbc2f6a8e263d9b72e776):\n", + "\n", + "We chose this function because we hypothesized it would allow the model to easily learn to attend by\n", + "relative positions, since for any fixed offset k, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$." + ], + "metadata": { + "id": "iMmyRsQ1b_rF" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = PositionalEncoding(\n", + " max_seq_len=200,\n", + " emb_size=100,\n", + ")\n", + "\n", + "fig, ax = plt.subplots(figsize=(10, 10))\n", + "plt.imshow(tmp_layer.pe[0, :, :], aspect=\"auto\")\n", + "plt.xlabel(\"emb_size\")\n", + "plt.ylabel(\"max_seq_len\")\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 853 + }, + "id": "G5gnKgeGcQPR", + "outputId": "ffa750f7-0827-4793-a10a-1ced840035db" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1IAAANECAYAAAC6qR6BAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAChSElEQVR4nOzdeZgdZZn//7uqztLb6e50Z+lOJ2QjC0sCCUuIgIKgkCiIoCKoLCKLsiiZGZ38xnGEcQyCIioRhvmi6FcQVBYRFAWURQhhCSGsgSxk7yzd6fR+lqr6/cE3sTOc535SXel0d/J+XVeuS3LnrlN9uuOV59TzeW4nDMNQAAAAAAC7ze3vGwAAAACAwYaFFAAAAABExEIKAAAAACJiIQUAAAAAEbGQAgAAAICIWEgBAAAAQEQspAAAAAAgIhZSAAAAABARCykAAAAAiIiFFAAAAABEtM8spBYsWCBjx46VkpISmTlzpjz//PP9fUsAAAAA9lFOGIZhf99EXPfcc4+cd955cuutt8rMmTPlpptukt/+9reybNkyGT58uLU/CALZsGGDZDIZcRxnL9wxAAAAgIEoDENpa2uTkSNHiuuanzvtEwupmTNnylFHHSU333yziLy3MBo9erRceeWV8q//+q/W/nXr1sno0aP7+jYBAAAADBJr166VUaNGGeuJvXgvfSKXy8lLL70k8+bN2/l7ruvKySefLAsXLizak81mJZvN7vzvHWvJUd/+prglJUV7XjnzZ8Z7OOy+L6r3qPXG7ee1eW1em9fmtXltXpvX5rV5bV57z/W2tgcyZsa7kslk1GsM+oXU1q1bxfd9GTFixC6/P2LECHnrrbeK9syfP1+uueaa9/2+W1JiXEhVZsyP9Uw9u9Mbt5/X5rV5bV6b1+a1eW1em9fmtXntPfvaImKN/Owzh01EMW/ePNm+ffvOX2vXru3vWwIAAAAwiAz6J1JDhw4Vz/Nk06ZNu/z+pk2bpK6urmhPOp2WdDr9vt+/42O3SoVhdXrF+uON93DJyY+r9/hwp74inn7kcrW+It9urNVOalJ7twddaj3Z0KHWs2HeWAtrc2qvHwZqPcgU1LraW6pf2yZM9T4aGCbixQpDL8Zrx/3oI85ZKpzDAgAAsNOgfyKVSqXkiCOOkMcf/8diJggCefzxx2XWrFn9eGcAAAAA9lWD/omUiMjcuXPl/PPPlyOPPFKOPvpouemmm6Sjo0MuvPDC/r41AAAAAPugfWIhdfbZZ8uWLVvkW9/6ljQ2Nsrhhx8ujzzyyPsOoAAAAACAPWGfWEiJiFxxxRVyxRVXxLrGcC8nGa/4bscXb5pu7Pvh9QvU6x70xJfU+i9m3a7Wb20y57M+Nup1tXdRd6VanzJis1rfUMgaa0OGmLNbIiLtoblXRCRZoWestHyWU6rnq2z5rDDV+4xVmIw5es2L0evGzGfF6A/jZqTIZwEAgH3IoM9IAQAAAMDexkIKAAAAACJiIQUAAAAAEe0zGak9YfZTXxa3tPjMp4l3LTL2rfput3rduvtSav2YE/T7+vwrM4y1351wi9r7863mfJWIyKyalWr9jfxQY21cdbPau9X31Xp1Rp9x1R2ac1DJEj0jVRD9tZ20XlczVjHmQInEmyMV+6OPWDmlmF93f+acyFgBAIA9jCdSAAAAABARCykAAAAAiIitfT1MuqldEl7xI7cLs6YZ+7701lj1uplHXlPrrYG+NbD2WfPWwEM/qu9Z+uvqiWr9J9N/rdafap9irE2q0I9OX+9XqPXh5frx6dsD8/a78lL9aPV8qG/dSyT1eiDmbWxOUj863Xb0epwjzOMcX/7ea8fo7cftcf26LRAAAKAInkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakeglVrJXCSRWub7hlv7PMeqFOvW+5vVOv3tOk5pmGLthlracP97uAvy6j1yUdvV+s3NI011s4daT4SXkTknaz+vtSXtqr1lsD841lZomekOi0ZqVRaPz5dy1i5loyUlq8SEZGEXlczVjEzUmGcI8zj5pRiHp/eb8hnAQCAIngiBQAAAAARsZACAAAAgIhYSAEAAABARGSketh0weHipUuK1p458kZj39mXn6let+OkqWp9wTJz/kpEpOHN5cbaxoI+i6n6LbUsI7xStb68cZixNmGsPkfqwe0z1PqoEnP2S0Rki19urFWnu9TejkDP46STxeeF7ZAXJSPl2WZQ6Rkqx7PMmdLE/eijP+dIxemPma/q1zlUZKwAANgn8UQKAAAAACJiIQUAAAAAEbGQAgAAAICIyEj18LmL/iIlFcXfkse6hhr7Chv0OVGr/3OkWq9YWK3WQ9+cyXmya7TaW71Mz1AlHU+tO+vMGaqRXqfau6x1hFqfM/xVtd5YqDbWhqT11+4M9a+rNGmbI2XOMSWTekbKD/U8j+PFyPvEnCMVR6wZVCJkhQAAwD6FJ1IAAAAAEBELKQAAAACIiIUUAAAAAERERqqHS6velcpM8bXl1FuvMPYdcGSbet35x96r1m//2SfUujdxnLH2wNYqvXdVo1rvDHJqvWy9OdgyzNN/fNZsr1brDQ3Nav01Jf9Vk+xQe9uClFqvSGXVereSc0pY5kDZ5ki5lpxTIErd0usr2a7d6VcxiwkAAGAnnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakePv3OqZIoTxetjf3J68a+5f96sHrdz2a2qfVfLl2p1pvOOMRYe2tFpdo7cevLan2dn1frmfXmmUmljp5DammuUOvDPD1btj5bbazVpVr11w7K1HpFUs9IZZUoUdoyg8rXMk4i4sbIWDlx50jFyRrFzSn1a8aq9+9bSDYMAAAUwRMpAAAAAIiIhRQAAAAARMTWvh66f1wviWRJ0Vp5xQZj36dOeUa97lPd+uv6rfo2tc0fMG+vK3ur+P3urjdyI9R62XrzzXuOvg53m5JqvdbVt9dt6DIf7X5o+Xq1t8nXtxVmLFv7ukPPWEt55u+HiEjecgS5Z9na5ytHrztezK19cbcGxhDG2F7HFjcAADDQ8EQKAAAAACJiIQUAAAAAEbGQAgAAAICIyEj1kP7LYkk4xXM9b/3wGGPffcMfUK97+LNfVOvjG5rU+kdnvGqsvf7oNLXXGzpUrS9s13MryY3mo9t9SxYo3aSv06tcPfjS2GE+2r16RIfauylfrdbLvZxa1zJS6YR+/HleyTiJ2DNSGtvx54Hl6HVb1kj9nsbJOO3GawMAAAwmPJECAAAAgIhYSAEAAABARCykAAAAACAiMlI9ZE+ZIb5hjtS9Z/zI2LfJz6vXrb6vXK23HVmq1i8ffp+x9o1lY9Ref1ydWn9+a5laL2lqNtbaQ30WU0mTnqkpM+TRdtjWYX5far12tffNrga1XpnoUusdYcpYK03o3299ypRI0jKHKhBzTsmNmVOyZaz05lgvHU/c1+7Hew/JhgEAsE/iiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkeqh7KoNkihPF60N9cy5mEtWfEa97pA/vanWl//rwWp9Wqp4bktExFmzQe1tOeMQtb5lo55Tmti+xljb5OvzkEqbLHOmHP3Hr6vN/HVn3G61tymv59LqUq1qvTMo/nMgIlKi/CyIiOQtMaSEZY6Ur8yCss+RssyoipOxGsQ5pVjizs/qT4P1PQcAYBDgiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkerhngP/LJWZ4mvLA//yVWNf5RJznkZEpL7jRbU+7qi1an1dwTwzyW/Vsz7bpqhlSazX711Ccz5kQyGjtpY06Vkiz7Gs49vMP54ZR792c07PSE0s3azWWwNzPqvEK6i93aH+dVnnSCnvuRtnDpTEnSPVf1mhcDDnlAAAwD6JJ1IAAAAAEBELKQAAAACIiIUUAAAAAERERqqHW1rGS0mh+Fty0PXmLJLTlVWv2/WhqWr962N+pdbvaDnS/NrJlNqbnKxnqFJ/rVLrTtqcoXo7V6e/dnOXWrdJtJnX+RlXn5fUnC1T6xlPvzdtjlSpbY6U5fOJpOXetTlSrq1XyVeJiHWuUKC8dvw5Uv2Yc2KeEgAA2MN4IgUAAAAAEbGQAgAAAICIWEgBAAAAQERkpHq4+2cni5cqPj9oxMqXjH1hVs9IrfpXPUt0UmmnWr/i+WOMtcl1W/Vrj3lbrS9ZP12te0OqjbVlnXpWyG0xz78SEcmH+jylZJs52FLi6KGX7VnzHCgRkYzbrdY35IcYa6VeTu3Nx5wjlVdyTp6nZ6Rs4s2R0st+GO/e4rw2AADA3sYTKQAAAACIiIUUAAAAAETE1r4ehv/sZUk4yaK1xsvMR5DXP7VNve68Dz2k1rOhvkWuYqH5KO/sgcPV3k8M+ZNaf3v9FLUeDDdvcXunXd9vFbbqW/tsX3eqzVwrcTy1t63bfHy5iEi5q2/HbPPNWwNtx593h/pfq6Srb+3Tqp7lCPFA9O11cXbIxdoWGPfF+9NgvW8AANCneCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6cCccIK5XPFtzwWV/NPbdOnSOet1Lqjao9TvbGtR63cJWY23zkRm198i0nlPyGvV8V+ch9cbahm16eKS+c6VabwsKaj3Vas7kpA1Zth26u1JqvcySkWr3zRmrEteWkdLvLeHqOSat6lp6fdFzTI6lX2/ux4xU3JxSf+acYrxvYb/edz++NgAAgwBPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVA9vX10ubmnx+UF/qDbnfZbPeUm97kvZnFr/0fIPq/Xa15Yba9s/f7jaW+WWqvWgWc9ItTWMNtZam8vV3rqc/nW3BPo6PtVuzvO4lgBHvkvPKZU7+r21Fszv2+iSZrXXlpFKuXo2LK9EaryYs5wcS+5FnUNFZgYAAGAnnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGake/nT8rZLJFF9bfmzZ2ca+3026V73u8S9doNazL9ao9SFZc0Zq2KStau82v1OtB516vWOUORjjNelZIAn1PE9zUDyPtkOy1Zwl8hz9MwCny1PrZY6eU2otmOdIpW1zpAJ9hlXK9dW6r4SRPNsMKst77sbIWDmx50jF7I8h7MfXBgAA+yaeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR42+ynp9IuvLbtvGGns67xNz7wk7x+i1mvWZtV6YuwBxtrHR72m9r6QrVLrthxTd4M5D1S2ypKRcvWcUmOhWq0n2/UskvrSXfrQoxJHzxq1580ZqTJXn0FlmyOV9mxzpMyfb3iWrI8vet2Wc/K1n4e4c6QG6xyqwXrfAACgT/FECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9XDhHy4Tt6T4bKMJf3rO2HfF6tPV6w596G39hS05pW0nTzLWZmd+q/bevvWDat3VRznJsIYWYy14Yajl2uackYjIhryeHfPa9OyY2mvNSOn9HUpGqsSxZKQCPSOVcnufkUrY5kip1bhzpPR6YMln2bJGfqjcPXOgAADAAMMTKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1cPEm9dKwi2ejen42FHGvjf+oGdiGpoX6S8c6HOoNs0y1w5N6cGTv707Ua2Pr21S69OHrTPWXm+uUXud8nK1vi6nZ6Scji5jTc3TiEiiU39fkpbAT0c+ZayVuPp8q+15/etOWmZY+UqYyLNkpNQ5UGKfI6U3D+KcErOgAADAHsYTKQAAAACIiIUUAAAAAETE1r4ewq5uCQ3brir+xbzFreySMvW6+eOnqfXkG+Zri4gcdZT5+PS0o28rDJZVqHV/uL5VbFbli8baiqaD1F6pzqjl9V36Frmwq9tYK4i+HTJhbhURkaSjf4bQlTf/1Shx9PvuDvW/VmnL1sB86BlrCcu2QBvXsj0vUA5Qj707Lta2wrgv3o8G870DAAAjnkgBAAAAQEQDfiE1f/58OeqooySTycjw4cPljDPOkGXLlu3yZ0444QRxHGeXX5dddlk/3TEAAACAfd2AX0g9+eSTcvnll8tzzz0njz76qOTzefnoRz8qHR0du/y5iy++WDZu3Ljz1/XXX99PdwwAAABgXzfgM1KPPPLILv99xx13yPDhw+Wll16SD37wgzt/v6ysTOrq6mK91ruXThavpKRo7bWJNxv75rw9Q73uisuPUetjSsao9W/V/7extq5gPiJcRGTIW3oupatBP6r78PRaY+2e5k61N6jWr72xs6DW010txlp3qPd6XfrXnRRzDklEpDtnzp7ZMlKdfvEj9HdIu/q955R7sx5/rlbtGSlNrKPTRcRy4vzANVjvW0TCQXzvAAAMdAP+idT/tn37dhERqanZdYbRnXfeKUOHDpVDDz1U5s2bJ52d5n/kZ7NZaW1t3eUXAAAAAOyuAf9EqqcgCORrX/uaHHvssXLooYfu/P1zzz1XxowZIyNHjpSlS5fKN77xDVm2bJncd999Ra8zf/58ueaaa/bWbQMAAADYxwyqhdTll18ur732mvz973/f5fcvueSSnf976tSpUl9fLyeddJKsWLFCJkyY8L7rzJs3T+bOnbvzv1tbW2X06NF9d+MAAAAA9imDZiF1xRVXyEMPPSRPPfWUjBo1Sv2zM2fOFBGR5cuXF11IpdNpSaffn2O59uw7pSxTPJ9y47aJxtfzDjLXRESuOOkvav1/tp2q1o8pMWdm7m7TF4DVyzrU+pYZ+pypcUlzJsfd1q72dh5Sr9ab2/WsUH12k/nagZ4GSurxLUk6ekYqn1PmSFnmQGUD/a9V0tXvXZ0jZclI2aZMudaMlZKDipmR6tc5UoM1KxT3Pe9Pg/U9BwBgNw34hVQYhnLllVfK/fffL0888YSMGzfO2rNkyRIREamv1/8hDwAAAAC9MeAXUpdffrncdddd8vvf/14ymYw0NjaKiEhVVZWUlpbKihUr5K677pI5c+ZIbW2tLF26VK6++mr54Ac/KNOmTevnuwcAAACwLxrwC6lbbrlFRN4butvTz3/+c7ngggsklUrJY489JjfddJN0dHTI6NGj5ayzzpJvfvOb/XC3AAAAAPYHA34hFYZ6RmD06NHy5JNP7pHX+mDpNqksLX4i/H/cdIGxL/dJ/bpfHbJcrd97TKNa3+qbc073bTlJ7XVXbVDr7adPUutVbqmxFra1qb1dtXqWraOj+MyundfPmzNUnZboSKJb/wOuJcBRyJlzStY5UkFKrQ/39PctH5r/WiYcPV/lW94XN0ZuxTYHKrAmtAAAAPYdg26OFAAAAAD0NxZSAAAAABARCykAAAAAiGjAZ6T2ppMXf0G8svfPlxIRabjtRWPf8Kf0rM9b+axa/+r4x9X6r1sPNtZeXDFG7Z3Y9LJazzXoeR8/NOdego4utbe7Vl+n+21JtS7KrKi2QO9NdOp5Hc/R7y3MmjNSKUsWqMvX7y2Z1udn5fpwjpRn61cyiU7sOVJxegfxPCUAALBP4okUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqoe6mhCQSxfMt7qRxxr4bR/1Mve4nXv+CWn/00LvV+lGLPm6slb6l57PEkgVqGNms1rcF5hxUmM+pvd21alkSreYskE1rWDzLtvPaXfq8JRsnb37f0pZZTraMVImjZ6T0OVJ6xsm3BJHi5Jxsc6Ts/YMz5xQO0vsGAAB9iydSAAAAABARCykAAAAAiIitfT04L7wujlN8W9Zbd0w39nmW7VTd941Q695Uvd9dWGWsVa/Ut5l5w/T9dUcOW6PWVxZSal2Tq9XvrWSTZWufspesxS9TW73umFv7subXTlm213Xbjj+3bu3Tjj/Xv658qH824lm2qfnSh8efxxFzW2G/Gsz3DgAAjHgiBQAAAAARsZACAAAAgIhYSAEAAABARGSketh+9lHipYofJ/7MiTcY+76+4RT1unV/WKXW//wv5gyUiEj9wk5jLdFiPp5cRMQfV6fWj6l4Qa0v7hprLrp6xsmryar15HI95+QYjqIXEWn2K9RetzOv1m28nDnYkrRkXuwZKT3nlFMyUsmYx5+7sY4/t+SrQsu1Le9boOSzAAAABhqeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR6O/PLLkqoonm9Z66eNfc88dJh63dGNC9X6D9/9iFovWbrCXFRmLYmIbPvkIWp9anqDWv/eRnP+yy3Vsz4jalrVendbqVp3UkpGqmDJSGXjZaTcnLmmJ6BEugr6nyhx9XvrCMw/a7Y5UkGo/zx4rp6x0qqWHzWrWP2WXj/Uvy7pzxlYAABgn8QTKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1cP19YulMlN8bXngr6809k26r0m9rj9zqlrfsKhcrY9tW63WNS1T9PqYhB4+eXXzSGOtvqJZ7R1ftVWtr2gbrtad0uIzvUREmgv6e+Z0KyEnsWdq3Kw2R0p/z7IF/a9V0imo9Xxonq81kOdI2S8wSHNKMbNhAABg38QTKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1cNX1x8lqYpU0drkG9cY+wrr9VlMK783S603PKVnZhIN5pySv1nPIaUm67OcKlxzDklEpGVDpbFWX6XPQ5pSrme71rRNVutOqXnOVHNez0hJVs9IBaLndTyl3bOEZnK+p9ZTjj4LKh+a+21zpOJmpPzQXLdlpAJ1ClVMgzVfBQAA9lk8kQIAAACAiFhIAQAAAEBELKQAAAAAICIyUj28fstU8ZLFM0NDOpcZ+7wDx6nXPffUp9T6iz8cpdbbjhljrGVe0tfCJ48x37eIfZ5S6Trzj4hfo+eUDixpVOtPtOoZq7DMnN/amtVfO8xm1Xo+1LNGrpKRSjr6e57Nx5sjlQ2SSq9+30Go35tnnUNlFnecUr+OYxqss6AG630DALAf4IkUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqIfPbFyXhFM+nrPqWeRZUokO/7v1Df6PWT280Z2JERDYeM95YS20frvaeUf1Htd4UdKn1inXm+T3ZmrTaOyG5Ra177XqOKciYM1Lbs+YZUyIiXk6fn5VX00AiXtb8dbuWzx/yljlSSctra3OkbHOgcqK/tq2/r3pF7HOo9N5YL92/Bum9h4P0vgEA2Ft4IgUAAAAAEbGQAgAAAICI2NrXQzjzUAkTxbeTfe/zdxj77th4rHrdDQV9C5tXXaXWJ85cbaw1rjYfjS4iMj2t7zt8LadvkatYbz4HvLNO35I4wlPOEBcRp6NbredHDTHW2rPmbX8iIkMKzfq1Lce+a7fuWfaZ5fP69rqU5Qjz7j48/jzh6l+3VrVtzfPFsnUv5tbAWOJsU2OLGwAAKIInUgAAAAAQEQspAAAAAIiIhRQAAAAARERGqoetX8uKV1a89uFSc+bmqLH3q9e94J3PqnX/cP0I86+P/r/G2mVTLlJ7q1w9A/VC11i1XrKhzVhrOnSo2lvjptS606EfvZ7PDDPW2rr0o9er8wW13h3qeR0v1/vjz31frycd/d6ygfmvZYWn5+18S6DHteSYfKUc5/jy3RFoCa3+zFcNZoP1fSOXBgAYBHgiBQAAAAARsZACAAAAgIhYSAEAAABARGSkevjL4b+WykzxteXMF75o7Ft01B3qdTc9cIBaz8/S7+vEUvO8pRGTtqi92/xOtf5cy3j9xbdsM5ayNbVqa5klIxV26RmpnOF7ISKS7bZc29fnLWUt0RFtjpRrCXAEeVtGSr+3fGieQ2Xv1f9Ku44+R0rv7XWriIhYxm8BAAAMKjyRAgAAAICIWEgBAAAAQEQspAAAAAAgIjJSPTzaWStlXvF8ysjvmd+qf19wjHrdhgfXqvUNPy5X69kwb6yd1vCa2rsoO0Stv7apXq2PannHfF9D9byOTditz0TKZcyhGr/T8qMb6PfWreSQRES8nDlL5Dn65w9hQa+ntHlJIpJT5kh5loyTbY5UwtX7taptjlRgmc0Vaw5V3HzVYJ2nBAAABiyeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR6+8+uzxUuXFK2NXrTQ2PfQn/SM1NjV5l4RkSsmNar1+9pHGWtzMkvV3tu2fkitd66vUOth3jxQKVljnm8lIpIP9ZxSkDNnv0REcpXmYIzTHe8zAFtGyrUNmtJY5ki5lrxO1jf/tYw/R0p/bS1jZesFAADYn/BECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9TDmv9+QhJMqWmv6vDkHNe7eVv3C0w9Ry5/K6Bmqk5ecb6z9ffqdau9fV01U62Xr9KyQuOZ63ZA2tXV7oGeobLOe8hlzJsfrjPcZQEdY/Pu88/rKHCkbp6APPUpa5kjlQ/PXZstI+UqviIgrvc85xZoDJSJOjFlQcV+7P4WD+N4BAIAZT6QAAAAAICIWUgAAAAAQEQspAAAAAIiIjFQPTmVGHDddtDbray8Y+948oqBe993/nKXWK5zir7lDx8Khxlp6RlLtDZfpc6Iq1ut5Ha/S3D+haqvau8mPt07XMlKp7fGu3Rno77mb1bNIGltGyrNkZnKB+a+l69jyVXrmLeHaMla9nyPlW/JX/ZpzipHP6leD9b4BANgP8EQKAAAAACJiIQUAAAAAEbG1r4dlVzaIW1pStPZw/UPGvo+NOU297uw55m2BIiJPd+vfhvqFWWNt3aXtau+Qt/TtVBUbcmpdhtYYS1PKX1Nb3y0M0a9tEWTMWya9Tfrx5dqx7SIirUHx7/PO9lyMrX152/Hnlq19vvnercefWz4bsW0rDJS9ZHF3mdm29vmhUmeLGwAAGGB4IgUAAAAAEbGQAgAAAICIWEgBAAAAQERkpHr41ZxbpCJTfG156boTjH3rTx+tXveeEb9V6+euOEOtp19eaaw92TVG7a1epmeo3LZutZ6vqzLWJpdsVHvfydapdSeh//ilMub8VqJTz0g5np6Rsh5/HiMj5eqn4Ytnyftox5+nHP3ituPPXUs+y1fCSLbjz/tTYPm6AAAA9jSeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR6GeHnJeMXXlkt/eJixb9iX1qrXbQv0XMsbz49T6xO2LTTW7t88Xe1139VzTKLN7hGRjulDjbWxiSa19y8th6p1x/LTV1XRZawFXeX6tZP6xdv8Ur0/H2OOlCUjlbT055U5Uq4Eem+of90JV/+6grD3GSn9zkScGLOg4vS+dwG97IfK3Q/gbBgAAOg/PJECAAAAgIhYSAEAAABARCykAAAAACAiMlI9zPnbV8QtLSlam/Tr54x9t173d/W6V64+Q63XPaenS7xhw4y1xSsOUHsnNi1W67Z5S22jzGvtEV5e7X2n1XzfIiLJUn3G1fByc31rV63a66T0JNJ2S0ZK8uagk5qnERE3rwdyPEvgJxeYvycpR8842eZjeTHyPk7MrFDcfgAAgIGEJ1IAAAAAEBELKQAAAACIiIUUAAAAAERERqqHyTdtl4TXXbSWP+5wY9/IxAvqdZf9fpJaP+D5NWq948ixxlrpMstUIkdfK4cFfehRZ4M5D1Tj6Xmc9duq1Pq4Ej1jVV/aaqy1dFnyNsmUWm7zi2fhdnCUjJSNa2m1fXqhzpFy4s2RckV/33xl4JJtjpRvmUkWy2DOV8WdgQUAAAYknkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakegjXrJXCKZ46abhhr7Ltu62HqdUc/sFGtF9auU+uNF4821oa9rM8V8obp85aCpma1nmzoMNbShvdqh64mfVZTWK7XR5WYs2PvdOlZISetZ6Q6Cnq+S5sjFVhyRraMlGcJzRQC8+cbtjlSWsZJRCTh9r7flpGysc2RCkT/nsZ78UGcsQIAAAMST6QAAAAAIKIBv5D69re/LY7j7PJrypQpO+vd3d1y+eWXS21trVRUVMhZZ50lmzZt6sc7BgAAALCvGxRb+w455BB57LHHdv53IvGP27766qvl4Ycflt/+9rdSVVUlV1xxhZx55pnyzDPPRH6dxi8eLl66+LHYC2fcZOw77K6vqtcdv3yhWnfLy9X68JmNxlrF3fr2usL4erWesBx/PmXEZrWuSTbpP15hhWVrX8q87TDRpW9RC0v0rX2tlq19Yd58NLttC5pjO/7c0bff5X3z5xuu5bXzofnodBERz9IfhNprx9za10+9sQ3m48sH870DADDADYqFVCKRkLq6uvf9/vbt2+X222+Xu+66Sz784Q+LiMjPf/5zOeigg+S5556TY445puj1stmsZLPZnf/d2mqeVwQAAAAA/9uA39onIvLOO+/IyJEjZfz48fK5z31O1qx57xCCl156SfL5vJx88sk7/+yUKVPkgAMOkIULzU+B5s+fL1VVVTt/jR5tPswBAAAAAP63Ab+Qmjlzptxxxx3yyCOPyC233CKrVq2S448/Xtra2qSxsVFSqZRUV1fv0jNixAhpbDRvh5s3b55s375956+1a9f28VcBAAAAYF8y4Lf2zZ49e+f/njZtmsycOVPGjBkjv/nNb6S0VM/YmKTTaUmn35+ROf/CP0tJRfG35KHOYcbrjbu/U30975DJar1QWTyXtcPc8fcaa/+9drra23L0oWp9aHONWp9V84qxlg3NOSIRkZImyzHfFXqOqS7ZYqx5XZYgkuX48/a85fhzJTuWD/V8lqu/Lfbjz31zzsl2/Hk+0P9K244w144/tx1frt9Z/OPTAQAABpIB/0Tqf6uurpZJkybJ8uXLpa6uTnK5nLS0tOzyZzZt2lQ0UwUAAAAAe8KgW0i1t7fLihUrpL6+Xo444ghJJpPy+OOP76wvW7ZM1qxZI7NmzerHuwQAAACwLxvwW/v++Z//WU477TQZM2aMbNiwQf7jP/5DPM+Tc845R6qqquSiiy6SuXPnSk1NjVRWVsqVV14ps2bNMp7YBwAAAABxDfiF1Lp16+Scc86RpqYmGTZsmBx33HHy3HPPybBh72WWfvjDH4rrunLWWWdJNpuVU045RX7605/26rW+VLVaKjPFH9JN/ekVxr7Rz+lzot79N/3pWEKPWMnssm3G2i1tbWpvyxS1LJWrK9X6kWUrjbXtQU7tLWnSMzH5jD4Da5hn/trcLj2IFJTq1+6wZKS8vPlIfN8yT8ktWOqWB8F+0Ps5UoElf2XPSCmv3Y8ZJ1s+y96/h25kbxus9y0i4SC+dwAAdseAX0jdfffdar2kpEQWLFggCxYs2Et3BAAAAGB/N+gyUgAAAADQ31hIAQAAAEBEA35r39505ttzJFFePDsz9ievGfucuhHqdWd9fKlaf2bNOPvNmV47qc9LSk/ertbbX9MzUpOT5v5GX//xKW2yzDzKmOcliYgMc7PGmtOtZ6QKGT0D1ZXX37eMb773ILRkoGxzpCyBHd83f77hOXpGKh/q72nSMocqCHufkdLvLH7OCQAAYCDhiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkeqh8KMRIsmSorVUlTnf0Th7tHrdXzX8QK3/vOJQtf6nziHGmjdSz2d99IC31PqfR+mDi0d4pcba4mxG7U0363OmOqaYry0iknHNWSInq1/bL9N/tDtz+pwpLSOVt86RUsv2OVK++eu2ZZzygSUj5er9vjK4KO4cKVtGSp3P1Z/5qrjzkAbrPCUybQAAqHgiBQAAAAARsZACAAAAgIhYSAEAAABARGSkekg9tlgSTvHszJs/MmeJ6idvivW6n696Va1/5s3PGWvehFq197Tqh9X6/Q1HqfWkY87cLM/Wqb2J5g61nq8sU+tlhu+FiIhYMlKFUv0zgmxe/9EPffNUpJxtjlQhXrYkCJQ5UpZpTbY5UmmxBLgUriUb5vdjpCawTbEi77N/Gay5NADAoMITKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1UP37CMkYZgj9cAnbjL2eZbsyNz1p6j1nx/wtFrfuKjeWCuboocBjkx3qvWyhna13hmYs0hvdpjvS0TEaWlT6/mKYWo97Zh/PMOcJSNVon9GkMvqc6RCbY6UJW7j5i11S4AjKGgZKf3Fg1C/tufoWSI/NL+2a+m1UcaCWTlkXgAAwADDEykAAAAAiIiFFAAAAABExNa+HiqvWifJ8lTRWrVrPja6wdOP8V70h6lq/aUvPa7W6xeaX3vNqfpauMItvlVxh0NHbFTr63zzPrXlrUPV3pL2ZrWez+jb1DzH/LWFOX3/XKFU3wvm5y2fIYTmbWx5y9a8uMefh775+knHvOVQxH78ua0/UD5bcS1HiNs2/jn9eQQ5WwMBAMAexhMpAAAAAIiIhRQAAAAARMRCCgAAAAAiIiPVwz0THpPKTPG15bg/fc3Yd+XMv6rXHXv/VrX+w499VK2XvbTaWKu/qkLt3ep3qPVjqleq9TdyI4y1jdsq1d6xXevVeqFSz+uo8raMlN4eZvUskYTmPE9eOSJcxJ6R0rJfIiKiZKRsx58XLBkp2xHmvhImsmWk4gqU97x/81X9+NoAAGDA4okUAAAAAETEQgoAAAAAImIhBQAAAAARkZHq4YfN46UklyxaO+j6FmPfzV/RM04T33hOrT/70ky9f9MiY+30hlVq76JsrVo/qlTPSD24fYaxlm3Wg0hhwTz/SkTEy+g5p3xozlDZrm2bI+Xkev8ZQrcth5SPmalRMlK2nFI+0O/NlrHylfyX7bW1fNXu9AMAAAwmPJECAAAAgIhYSAEAAABARCykAAAAACAiMlI93Pd/ThQvVVK0NmL1y8a+Cfdl1Ot6E8er9REL9WyJN2SIsXZKhTk/JSJyy5YT1Pp/1j+m1p/fOsZYSzVZZjFZZCq61Ho2NGeoQl+fQWWbI+Xk9Pdck7dlpCxzpKyCOHOkLDOuLHOkAuWzFdfy2jZxZkE5vf92xX7t/hQO0vsGAGB/wBMpAAAAAIiIhRQAAAAARMRCCgAAAAAiIiPVw7BfLJGEU3yO1IavHGnsq/uJnlNa+8/6nKjR921U67nDxxlrB6X0tfATqyaq9dqGZ9T66o3mOVQVTZZZTQn9x2toRYdabwuUWVGhZaZRmV73YmSkusPiPyM7ODEzUk5BmSNlySn16xypkDlSAABg/8ETKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1YMzcaw4Xrpo7ZJL/2Dse/jOSep1Dzp9mVrffkOzWt/4mXpjLW3IdO3gvFWh1r3j9bV0Yn3x90NEpHSrnnlxy8rU+vCy7Wq9Jej9Ot8v0e8t0d77a9syUm5en9Vk4/jKHClLzijuHClflHyWpTcuX8lv9escqJgzrPrVYL53AAAGOJ5IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpHpZ/rUTcspKitcuqVhv7fvGxj6vXvWPM99X6+emT1Hr6mCZjbU2hXe0dskzPtbQH3Wq9fJ05ZFHSrMx5EhEno+ez6ks2qPUtgZ6x0vhl+tedarZ8huCYv277HKm4GSlzzTYHqmDJlXmi31ugZKxs+azAEsjpy7iOb5krRlYIAADsaTyRAgAAAICIWEgBAAAAQERs7evhz8f9t2QyxdeWp751jrEvf+Y29bpljqfWw4MmqPWrJj1qrD3ROVbtrVrWptZXF/QtUZn15u176aas2htW6Vv7RqX0921LodJcVLbeiYg4pcr+OBHxcvqPvuOZv2fdgeX489hb+5QjyC29hUD/WUtq+wZFJBfq/XG4MY4w79fjzwEAAIrgiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkephbaFUygvF15aFG+qMff99y0/V6/5/jcer9U2zqtT6pyrWGGvnrTxN7XXf3ajWX82OVOtl67vM124110RE8sP0jNTIpJ6R2pAfYqxpGSYRkURJXq172bRaF8f8GYPt+HOJm5FS2j3LMd4F5fjy9/otx58rn624lqPX/UF8xnhg+doAAAD+N55IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpHi5+8BJxS0qK1iY8stDYd3Raz8yc98cj1Xr+A3rWqNRJGWsvrzhA7Z3UvFitP9euz7DyGpUcU17PIWWn1Kj1ukSLWn++Y7y5aMlIlZfm1LqT0zMxjmf+jKEj0PNVTl6f1WTjmEd37cYcKf1PuKJnpPKh+f8S4syB2p1+7c4sY8OsYvVbev3QkoljBhYAAPsknkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakeJt68WhJu8TxS+2lHG/tu396oXnfcfdvV+tG/eEWtP9FtzmCVL+v9PCQRkRe26BmryuZN+vUVXTV6jqnG7Vbr67uqjTUnoeezMiVZtd6lR6jUDFY20DNxjh8zI+WbQzn6OyriW+dI6XmdbqXftc2gCvUwkUNWCAAA7EN4IgUAAAAAEbGQAgAAAICIWEgBAAAAQERkpHoIczkxRUSG/PNqY993njxdve6kJc+r9YuGLFLrX1n5aWOt5i1l6JCIJIYPVesbNuiznio6VxlrjmWWU/dQPTNT7eqZm42dlcZaOtWmX7tEn82Vy1rmSCXNfzU6A/NcLxERKegZKdvcIUdp9ywDkWxzpDzLHCkt52TLV/Ul8lUAAGCg4YkUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqYdVlk8QrKSlae2viT419H/6vL6nXTYzVZzUdkKhQ68ueH2usTXy7Se3Nj69T66n1+kwkCc3ZlLCg57OytXquJePqP35b28uNtVFpfU5UdapTrTflLJmbhPneuq1zpPQcko2WkbJ98uFbMlK2WVC+8gqu6O+ZL3p+y+3PnNNgzVjpbykAAOhHPJECAAAAgIhYSAEAAABARGzt6+F7n/mllGWKH+n9vaaJxr7U35aq1133lSPV+ps5fRta3ULzdqxw7Ua1d9vMQ9V6xTq1LK5hq6OISNDdrfbma/Stf2lH3yLX3m5+bUnrR5APS21R6+/mLcefK1v7unzLdkhfP/48sGyR03bfeZa9Xr5yfPl7/fprB6bz/2V3tgXG29rnK9tI+/X488G6LRAAAPQpnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakejirZLpUlxdeW/98Pvmjsqyt/Q71u/SdWq/UFW05U65UvmINMhbY2tbdlslqWkX/X8zxubY2xFm7eqvaW1erZr6RTPI+2Q9BmziKFKT2nNCSpv7abteRekubrdwV6PksK+ntq4+rRMpXt+HMvRs6pX48vtwgk3pHzAAAAUfFECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9fCRF88XryxdtDb6/7xk7Gs5a4Z63bsP/L5a/9AjV6v1SeteMNacpJ7XKZ3Sotfv0dfS/vAhxprb1q72jqrertZtEq3mDFVYqn/dQxN6dszLWjI1yb6cI6W/tqO0u45ljpQlI+VaXjsfmt9zz9KrzaB677V7n7HSv+q+7x+8Lx7DYL1vAAD2Ep5IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpHhp+6EgiUXxt6Uweb+zbdlaHet0RXvHc1Q5DF+rfBjeTMddqzRkmEZFTDnhLrb+xuV6tt08fZaxVbDHfl4jIxMotat0P9cxNss0c0ghK9JxSTULPb7k5PccUxshIhb5lVlOoZ4W0jJRnCa74gV63zZHSck5x50g5lv5407cAAAD2Lp5IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpnha/KeIUz7+8/cvDjW23z/iFetn/3KLPmRr23Fa1XphqzmflSvVv4WnVD6r115r0/FZ7wxhjrXxVhdo7uew1tZ4NC2o9qcScAsvXXe12qvU4GanugmWOVJDXy5Z5TG6MsFDBMkfKs8xy8i2zoNTePhw8FDefBQAAsKfxRAoAAAAAIhrwC6mxY8eK4zjv+3X55ZeLiMgJJ5zwvtpll13Wz3cNAAAAYF824Lf2vfDCC+L7/9jr9Nprr8lHPvIR+fSnP73z9y6++GK59tprd/53WVnZXr1HAAAAAPuXAb+QGjZs2C7/fd1118mECRPkQx/60M7fKysrk7q6ut2+ZjablWw2u/O/W1tbRUSk5ZyjxEuVFO35+4duMF6vPqFnhS7+7XFqfcI7L6j1xq8ebax53WqrzEjpfyDo1uvt5jFSUlOjL1gnphvVemeoZ4lSreZcTMGWkfJiZqTSnrHW7Vvmfvn6e+pbckqOb667lofIgSUj5VryWYGSc7LllHzLvfVnzsk2w0rv3YM3srcN0nsPB+l9AwD2LwN+a19PuVxOfvWrX8kXv/hFcXr86+bOO++UoUOHyqGHHirz5s2Tzk79H9Hz58+Xqqqqnb9Gjx7d17cOAAAAYB8y4J9I9fTAAw9IS0uLXHDBBTt/79xzz5UxY8bIyJEjZenSpfKNb3xDli1bJvfdd5/xOvPmzZO5c+fu/O/W1lYWUwAAAAB226BaSN1+++0ye/ZsGTly5M7fu+SSS3b+76lTp0p9fb2cdNJJsmLFCpkwYULR66TTaUmn9WO/AQAAAMBk0CykVq9eLY899pj6pElEZObMmSIisnz5cuNCymTWZS9KuqL4jKDVhVJj39PdVep1x9+nDETaDYVZrcZa+8ZytbfCLZ752skSAMk1mHNM3UNTau+YxDa1vj3QcyupNnOep1Cq70rNOHr+ysnrGSm/1Py+5SxzpEp9Sw4ptGWkzDXP8v0KAr3uOfq9aXOkPEu+ysaWkdKuHifjBAAA0BcGTUbq5z//uQwfPlw+9rGPqX9uyZIlIiJSX1+/F+4KAAAAwP5oUDyRCoJAfv7zn8v5558vicQ/bnnFihVy1113yZw5c6S2tlaWLl0qV199tXzwgx+UadOm9eMdAwAAANiXDYqF1GOPPSZr1qyRL37xi7v8fiqVkscee0xuuukm6ejokNGjR8tZZ50l3/zmN3v1Ot+te0UqM8Uf0h145xXGPn9oTr3upBcWq3X34Elq/ZKD/m6s/a5ihtq71e/QX9syc6thZLOx1lUzQu0d5unbsVYW9JyatrWvu8Z8PLmISLlr2YaW07f+hSnz+5L19dcu9S3bBq3Hn6tllXVrn+W186H5a7NuzVO2BQIAAOxrBsVC6qMf/aiERXIlo0ePlieffLIf7ggAAADA/oyPkAEAAAAgIhZSAAAAABDRoNjat7dcvvYYSVUUP9J78o2rjH3tRxygXjcxUj9BcNMxNWr985WvG2v5UXpeZ1G2Vq27NUPU+pHD1hhrjw2tU3urLEevb/H1o9uTbeYcU1uD/qNbZjkm3MkX1LqfNH/GkM1b/toEluPP9W5xlYyUa/nsw5aRci3HnwehuT+p3ZiI+GJ77d4fYW47/tyWO5P+PD5df1v6rre/cWQ9AGAfxxMpAAAAAIiIhRQAAAAARMRCCgAAAAAiIiPVw7JbDpZEsniupzr3trGv7LGl6nU3XjBdrW87Up9pNNQzZ4lOqTDnp0REfrLpJLXu1+kZqWMqnjHWHq45Su1NOnp+a31ez4Z57VljzS/V51+VWF5bLBmpIG3+jCFvmSMV+noOyS9ylH9PceZIhYH+2YhnSWj5ymcradHfMxvXlmPqR4H2vpD12b8M5lwaAGCv4okUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqIfO7FyThJIvWVl7zAWPfuOtfUa9b8clGtf7BoWvV+vNZc4Zqeqr4/e7w5KoJan3EqFK1PjW9wVgr1OrZLpv1OT2f5XR0m19bv21JG76PO4QFyxwpJSNVKFjyV6GeQ7K9a67f+0yObY6UtV+ZI+VZZlD5YbzPZbQv2yW3AgAABhieSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR6CD0yTIFF8jtRNn7vd2Pdfr1ygXve2yT9U655lTs23133cWLtr3N/0a79VodbbR6plGZMwh1MqajvV3s4gp9Y3dFerdaejy1izZaRc2zAYW0YqZe4v5C1zpAL9+2mLQGlzpGxfV2jJSHmWWU750JL/isG1ZKz6kkPGCgAA7GE8kQIAAACAiFhIAQAAAEBELKQAAAAAICIyUj1sv7pTvLLiAZVZJS3Gvo1n6VmgA5NptZ509FzKcy9ONtbWjP6D2lu9TM+lNE3TwyMVbvHMmIjI6OoWtbfZkpFa31ml1qWrw1jyS/Wsj+dYPiPI9z4j5Rcs17bMkbIlheLMkRJLRsqWU9LmSLmWfFVg+VzGtWQBNU6MXgAAgL7AEykAAAAAiIiFFAAAAABExEIKAAAAACIiI9XDH6feI5WZ4mvLI5+72Nj3o5m/Vq9707ZJav1TmVfUet1Cc+2vp4xXe6vealPrm2br+S1fyftMqtys9m7w9Wtv6dBnXA3tbjbflyUjZRP6yrAmEfFTSq8tI2WRC/X+OOOW4s6RCpR78yw35ttmd1n05ZSpWBkrZlABAIAieCIFAAAAABGxkAIAAACAiNja18MfO4dLmVv8KPIDvmfe3/Ox33er173qDx9R69tPLFPrQ55vNNYe2DRd7XXXbFTr40bq2+uagi5jbWLpJrV3ZW64Wt/ebj5aXUSkJpc31oJSfWueTVjo/fHnkrfs9Qr1bWR5y+cXjnL8ufVYd8v+OOvWPmUfm+3odJs4x5/begPLe96vOLodAIB9Ek+kAAAAACAiFlIAAAAAEBELKQAAAACIiIxUD9ff+Snx0sVzO6NeNJ9B/kinfsz3+Hv1DNVdtUep9YmrXjbW3lhxhNo7qfkltX5krfmIcRGRt/Ll5vtKmbNbIiLPd05Q6/k2/X2TwJyDciwZKe3Y9veubcncaLfmxzzmO9T73UKMTI3l3mxZo3xQPCMoYs9X+ZZj3W2vHff4dAAAgL2JJ1IAAAAAEBELKQAAAACIiIUUAAAAAERERqqH0be9LgknVbTWdN4xxr4rnj9Mve6EZ19V69WHHq3W3bQ5sFO+rPj97mSZOzSrYplaX9w11lg7pfwNtfeX7XVq3W0z53Fs0qXmGVMiIgXRM1Shb8lYKW+rk4/3+UPOOkcqxsUt+SsbW35L7R3An8s4xK8AAMAeNnD/5QMAAAAAAxQLKQAAAACIiIUUAAAAAERERqoHd0iVuG7xPNLxVy0y9i36rj4HyqutUesjntuu1sNDzPOYat4qqL2J4UPV+sGpp9T63ZvN+a3zK/WM1IrttWo92db7dXxZSVatd4f6+yKWOVOBkpFy9XiWVT7Us2GOZcaVyjI+yzYLKlBmObmOfnHrHCnLa2scywyqvtSfrx1XOIjvHQCAgY4nUgAAAAAQEQspAAAAAIiIhRQAAAAARERGqodlV9aLW1pStPZw/R+MfbP/Uq5ed8unDlXrtXctVuubLj7CWKt/fIvamx+vz3Ia5SXV+quN9cZaxVjzfCsRkS3bMmq9tE0ti5Mw/3hWWjJSWUsGSkI9O+KnzHWnEG8oUT7U/9o5hRhZokC/N1tOKR+Y81u2fJWNa8nr+DFnYAEAAOxNPJECAAAAgIhYSAEAAABARCykAAAAACAiMlI93DP7p1KRKb62vGjNR8yNeX2wUHBmk1oPf6HPPGqb1WWs1f18g9q7bdY0tV7mKgOTRKRrQ4WxlnT0eUhBk56hSrXpmRknZb63IelOtbcjziwm6ds5Urn+nCNlySkFSk7JOkdKmUEVly1fZTOYZ0EBAICBiSdSAAAAABBRr55IdXR0yHXXXSePP/64bN68WYJg10+qV65cuUduDgAAAAAGol4tpL70pS/Jk08+KV/4whekvr5eHGffOLa41PGlzLAF6M0fmo8wT53kq9e97dAFav3fJ3xWrV8wdaGx9nRH8ePad2iZrJYlH+r3XrbOvA3Ntxwxnt6qb2FLten9Ttq8NbDWsrWvzXLEuE2QNm8FS3RaHuRa/j7kxbK1r2DZn6f1+vpr244wD5TteZ5l32AQ6u+LbWtgHH7Mo9n71GD9v8fBet8AAOwlvfrX5p/+9Cd5+OGH5dhjj93T9wMAAAAAA16vMlJDhgyRmpqaPX0vAAAAADAo9Goh9Z//+Z/yrW99Szo79e1VAAAAALAv6tXWvh/84AeyYsUKGTFihIwdO1aSyeQu9cWLF++Rm9vbTvvr5eKWFs8cTbrnOWPf27cfqV73iLR+xHjz0cPV+gXVvzTW/p4+Se0tm9yi1jf55qPVRUQq1ptzLV1hTu0t0U99l1SrJSNVVmqs1aS2qb1t2vnluyFIm+/NKcQ77DJvyW/FOf48bgypEPTdQZ7Wo9eVUE6/xnXICgEAgCJ6tZA644wz9vBtAAAAAMDg0auF1H/8x3/s6fsAAAAAgEGj1/t4Wlpa5P/8n/8j8+bNk+bmZhF5b0vf+vXr99jNAQAAAMBA1KsnUkuXLpWTTz5Zqqqq5N1335WLL75Yampq5L777pM1a9bIL39pzvQMZJN/tE0SXvHZRfnjpxv75h93r3rdW1sa1PrmD+jZkVGJCmPNqx+h9p465k21vixfpdYr1plzUM1BQe0tadYDO8l2vT8sM8/IGppsU3tbgjK1bpU037ur37aIo38+kQv7bo6UBHqgx/bJSRCa+11Lxsnv/ecyfc6x3Xs4gOdQAQCAAalX//KZO3euXHDBBfLOO+9ISck//rE7Z84ceeqpp/bYzQEAAADAQNSrhdQLL7wgl1566ft+v6GhQRobG2PfFAAAAAAMZL1aSKXTaWltbX3f77/99tsybNiw2DcFAAAAAANZrzJSp59+ulx77bXym9/8RkREHMeRNWvWyDe+8Q0566yz9ugN7k3hug0SOsXnD7XcaM61fDajzzQa98Cn1foJR7yh1l/PmWc9dR2oL1xPr3pArf+9Y7JaT23YbqxtKJjnPImIlDTpYaJEW1ath+XmjFSN16H2NhXMuTIREXEsWaK0b65ZMlKOq1+7P+dIeZaZSAUlv+WJfnFfyVeJiLjS+6/Lls/qT0GMrwsAAAxevXoi9YMf/EDa29tl+PDh0tXVJR/60IfkwAMPlEwmI//1X/+1p+8RAAAAAAaUXj2RqqqqkkcffVT+/ve/y9KlS6W9vV1mzJghJ5988p6+PwAAAAAYcHq1kNrhuOOOk+OOO25P3QsAAAAADAq7vZD68Y9/vNsXveqqq3p1M/1t40WHi5cunst5fvqPjH1PdSfV646/15y3ERH56uzH1PqCLScaa9umFJ97tcNhKfMcKBGRG9aOV+vSZM5/rcgPV1vTTd1q3W3XM1L5YeacU7XXqfauzdeodcfTZzklUkpGKq+2WudI5S1zpMTvu4yUjTZHyrNcPIg5R8oXS4ALAABgANnthdQPf/jD3fpzjuMM2oUUAAAAAOyO3V5IrVq1qi/vAwAAAAAGjXh7cSwqKytl5cqVffkSAAAAALDXxTpswiYMB9d8lYsueFhKK4q/Jfe3m/NA1776MfW6Bzz5ilo/PK3nnP700jRjzZmi568qXPMsJhGRNxtHqPVx280zrt7urlN73RZ91pPTree38uOHGGu1Xrvau7RrtFq35ZhSKfOwKMc2R8rTr90d6Jk6x9e/p2qvJSNl++REy0jFFWcWlK3XFg2zjA3rs97d4YfK3Q/g+VkAAOzv+vSJFAAAAADsi1hIAQAAAEBELKQAAAAAIKI+zUg5fR0u2MPOq1wnlZnia8vDb77S2Fe1Wk9ouJmMWt/s61miYQvNc4dKzm1Ue7darl1YX6bWw4I5ELSsXc9XyXY9xxQGehYoV2n+ujOuPqOqKWeeQSUi4nj6z2ZJ0vx1h3lLbsW1zZHS/9o5MeZI2cJClglW4gfme3ctF7d9Xa5tDlUf5rP61T76ZQEAsL/r0ydSg+2wCQAAAADYHX26kPrTn/4kDQ0NffkSAAAAALDX9Wpr39y5c3f7zx533HG9eYl+ccZbp0mivPhR5Afc/Kqxz0nob+O22ZPV+i+2r1HrQ5/bYqyd+C9L1d5nu4ep9fL1+lpa+9re3V6j9g7p2KDWbedK5yrM9YybV3u35sr1107q/eUp89HsHZbjz8XTN9B1W7bASWA7zNvMCeLtIwuUfWie5Sju7rDvPpdxOAYcAAAMML1aSL388svy8ssvSz6fl8mT31skvP322+J5nsyYMWPnnxtsGSkAAAAA2B29Wkiddtppkslk5Be/+IUMGfLe0NRt27bJhRdeKMcff7z80z/90x69SQAAAAAYSHq1F+cHP/iBzJ8/f+ciSkRkyJAh8p3vfEd+8IMf7LGbAwAAAICBqFdPpFpbW2XLlvfndrZs2SJtbW2xb6q/OD8eKk6ipGjNrTVnNPz1+hHkrWfp78n/vHasWh+/4g1jbU7Fa2rvjY0fUesV6y1Ht1dVGmtbmvVj3au6utS6Y8kS5SvNW0PLLJmZlmypWnc9/ej1sqQ5I9VlyUjZtrTmA8tfO9/8PfFD/ftlOWFcPMu9FZTjzz3L8ee248ttGau+RMYKAADsab16IvXJT35SLrzwQrnvvvtk3bp1sm7dOrn33nvloosukjPPPHNP3yMAAAAADCi9eiJ16623yj//8z/LueeeK/n8e6efJRIJueiii+SGG27YozcIAAAAAANNrxZSZWVl8tOf/lRuuOEGWbFihYiITJgwQcrLLUdOAwAAAMA+oFcLqR02btwoGzdulA9+8INSWloqYRgO6iPPk4+/LAknWbT25s0zjX1jH9BnNd0y/Xa1fuWPv6LfmGPegTkpmVJbn141Qa2PWdetv/ZQ86yosKn4zK1//AE9lxL6ek4pp0SwypX3RERke7Z41m2HIQn9665IZo21lrwlb2PJfmX7co6U/pZahZacUxyu6O+br8ywcvsz40S+CgAAFNGrjFRTU5OcdNJJMmnSJJkzZ45s3LhRREQuuugijj4HAAAAsM/r1ULq6quvlmQyKWvWrJGysrKdv3/22WfLI488stvXeeqpp+S0006TkSNHiuM48sADD+xSD8NQvvWtb0l9fb2UlpbKySefLO+8884uf6a5uVk+97nPSWVlpVRXV8tFF10k7e3tvfmyAAAAAGC39Goh9Ze//EW+973vyahRo3b5/YkTJ8rq1at3+zodHR1y2GGHyYIFC4rWr7/+evnxj38st956qyxatEjKy8vllFNOke7uf2zL+tznPievv/66PProo/LQQw/JU089JZdccklvviwAAAAA2C29ykh1dHTs8iRqh+bmZkmnLbmZHmbPni2zZ88uWgvDUG666Sb55je/KZ/4xCdEROSXv/yljBgxQh544AH57Gc/K2+++aY88sgj8sILL8iRRx4pIiI/+clPZM6cOfL9739fRo4cGenr6vr4EZJIFs/W/OHjNxn7Pu59Vb3uB/W4jtQ9q8+ZciaPN9ZceU7t9d7SDwBJNm5U67nRQ4y1dJNlHW7Ly1kyVPmMuZ529B/dti79Ta9J6P2ZRIex5hYsmRnLtfOBnqFylDlSNrY5UrZPTrRZUK7l4n7vPpfZI3zLz9J+a/BGVgEAGPB69S+f448/Xn75y1/u/G/HcSQIArn++uvlxBNP3CM3tmrVKmlsbJSTTz555+9VVVXJzJkzZeHChSIisnDhQqmurt65iBIROfnkk8V1XVm0aJHx2tlsVlpbW3f5BQAAAAC7q1dPpK6//no56aST5MUXX5RcLidf//rX5fXXX5fm5mZ55pln9siNNTY2iojIiBEjdvn9ESNG7Kw1NjbK8OHDd6knEgmpqanZ+WeKmT9/vlxzzTV75D4BAAAA7H969UTq0EMPlbfffluOO+44+cQnPiEdHR1y5plnyssvvywTJujHbQ8E8+bNk+3bt+/8tXbt2v6+JQAAAACDSK/nSFVVVcm//du/7cl72UVdXZ2IiGzatEnq6+t3/v6mTZvk8MMP3/lnNm/evEtfoVCQ5ubmnf3FpNPpolmu2itWS7K8+FymMmVAz7wTHjLWRETubKtV6+5rK9T6lnMPM9bWFDrV3iHLLKGZrc1quf2YEcZaSZN+aSelz7gKs+ZZTSIifsb8nicdPWfU3V18Htg/LqDXM0nznClbRsrx9M8nskHv50gFlllMtrKNr2SkPMvFg1D/uu0Zq76bI+UwCwoAAOxhvXoi9cgjj8jf//73nf+9YMECOfzww+Xcc8+Vbdu27ZEbGzdunNTV1cnjjz++8/daW1tl0aJFMmvWLBERmTVrlrS0tMhLL72088/89a9/lSAIZOZM8wBdAAAAAIijVwupf/mXf9l5QMOrr74qc+fOlTlz5siqVatk7ty5u32d9vZ2WbJkiSxZskRE3jtgYsmSJbJmzRpxHEe+9rWvyXe+8x158MEH5dVXX5XzzjtPRo4cKWeccYaIiBx00EFy6qmnysUXXyzPP/+8PPPMM3LFFVfIZz/72cgn9gEAAADA7urV1r5Vq1bJwQcfLCIi9957r5x22mny3e9+VxYvXixz5szZ7eu8+OKLu5zyt2MRdv7558sdd9whX//616Wjo0MuueQSaWlpkeOOO04eeeQRKSn5x9HWd955p1xxxRVy0kknieu6ctZZZ8mPf/zj3nxZAAAAALBberWQSqVS0tn5Xjbnsccek/POO09ERGpqaiIdJX7CCSdIqMx/cRxHrr32Wrn22muNf6ampkbuuuuu3X5NzZ3j/yaVmeIP6cY9fLWxb9XH/ke97rg/fUmtT+p6Sa03fSBvrD3ZZZ4xJSJSuUyfUeW3tqv19lHmh5aV7+qZF7dCn2Hl5wtq3cuYv25bRsrvtPxoJ/V6hWfOb7l5S97G0+8tH+p16cM5Up5lsJA2R8qLkXEa6ALp/XtuRT4LAIB9Uq8WUscdd5zMnTtXjj32WHn++eflnnvuERGRt99+W0aNGrVHbxAAAAAABppeZaRuvvlmSSQS8rvf/U5uueUWaWhoEBGRP/3pT3Lqqafu0RsEAAAAgIGmV0+kDjjgAHnoofcf+f3DH/5wl/++7rrr5LLLLpPq6upe3RwAAAAADES9niO1O7773e/KZz7zmUGzkLq+aaKUZIvPFzroevPQpJdOzqnXHXOvnh1JHKBvhzzzsMXG2n2bZqi97rsb1LofmGc1iYh0Npjrw1/UM05OpkKvt3eo9UxFl1pXr92t55BCS0Yq45nnSDmWOVKS0F/bOkdKyQ3asjy2jJSNH/TqIbWI7M4cqd5nhdy4A7IGq8EbOwMAYJ/X+3817QbtIAkAAAAAGKz6dCEFAAAAAPuiPt3aN9g89D8fFC9VUrQ2fO3Lxr6LX/2Cet3hjy9V69tPO1ytf2Xor421k5d8Te2dtO1Fte4kU2q9rMF8PHpqm9oqQbW+tc9tblHrQyv0rX8ar1P/jMC2ta9C2drn2rb2ufpr52xb+3x9u6XGtrXPdfS9YqFy/Lkb84hwz9Jv2xoYR5wdcuyu6yXeOADAPo4nUgAAAAAQEQspAAAAAIioTxdSxx9/vJSWlvblSwAAAADAXterjNQdd9whF1xwwft+v1AoyL//+7/L/PnzRUTkj3/8Y6yb29uG/nKxJJzix5+vv+poY1/JfXpmxknpR5A3ztLDBBOS5qxRxTI94ySufhS3W12l1g8dsdFYa9tWqfZ2j61V6+mNxfNoO9SXbVfrGq/LkgVK6+9LxjUfve4W9KxPaDv+3Nf/2oW++fq+5SRMp/fxqveuH5jfN88SwPL7MRQT88sGAACIrFdPpK666ir59Kc/Ldu2/eO0gWXLlsnMmTPl1782H4wAAAAAAPuCXi2kXn75ZVm3bp1MnTpVHn30UVmwYIHMmDFDpkyZIq+88sqevkcAAAAAGFB6tbVvwoQJ8swzz8jXvvY1OfXUU8XzPPnFL34h55xzzp6+PwAAAAAYcHo9R+rhhx+Wu+++W2bNmiVvv/223H777fKhD31IRo4cuSfvb69ypowXx0sXrX3lS7839j34oYPV67aeotenHbVCra/Km2c51bxZUHsTI4ap9XCInnM6pvp5Y+0vrRPU3u6hdWq9pEw/iGRUSaOxlg/1VIwyBkpERIKU/qNfqVzAyVvmKVkyUrlAr0vY+3lNtjlSniXHpM2RsrHNgXIdPd+lZawcSy8AAMDe1qutfZdeeql8+tOflm984xvy9NNPy9KlSyWVSsnUqVPlN7/5zZ6+RwAAAAAYUHr1ROqZZ56RRYsWyWGHHSYiInV1dfLHP/5RFixYIF/84hflM5/5zB69SQAAAAAYSHq1kHrppZcknX7/FrjLL79cTj755Ng3BQAAAAADWa8WUsUWUTtMnjy51zfT31Z8rUTcsuKzjS6qWmPsu79JzwJtOrNBrX931KNq/dbm44y18reb1d78eP3eCqV6Xueo0pXG2p/b9Wt31eh5m8pyPSPVkN5mrNkyUolOtSxhSt/VWuZkjTXHNkfK09/Tgi0j5eeMpUD017ZlpGx8JSPliSXjZMlI9SdbPgsAACCqXh828bvf/U5+85vfyJo1aySX2/UffosXL459YwAAAAAwUPXqI+Qf//jHcuGFF8qIESPk5ZdflqOPPlpqa2tl5cqVMnv27D19jwAAAAAwoPRqIfXTn/5UbrvtNvnJT34iqVRKvv71r8ujjz4qV111lWzfvn1P3yMAAAAADCi92tq3Zs0a+cAHPiAiIqWlpdLW1iYiIl/4whfkmGOOkZtvvnnP3eFe9Oixt0kmU3xt+dE3zjX2JY6tUa/7k6PvUuvHpvVgywWvHGGsTV73ltrbfNw0te7qUSOZmOwy1oKsOUckItJdq2ek/Iw5ayciUpcwL8qzoWV+Vpclz5PWc0rlrpKR8m1zpPTPJ/pyjpQlQiWu5bMTbY6Ua7l4YJlRFSen1J8Zp7gzrJzej+bqX4P1vgEA2Et69USqrq5OmpvfO+TggAMOkOeee05ERFatWiVhSKgbAAAAwL6tVwupD3/4w/Lggw+KiMiFF14oV199tXzkIx+Rs88+Wz75yU/u0RsEAAAAgIGmV1v7brvtNgmC97b5XH755TJ06FB55pln5PTTT5fLLrtsj94gAAAAAAw0vVpIua4ruVxOFi9eLJs3b5bS0tKdg3gfeeQROe200/boTe4tK/KlUp4v/pDOuWGYsW/lWfrbeGqZniXa5ner9SHPpoy1oKND7d1uGeuV3K4HIWpdZdaTZRtntlbP1BQySbVel2gx1rotOaKEOdolIiJBUv+6y528sebk9WBZmNa/LtscqWRgfl99yywnR+ndHUHQ+2CMbY6UZ8lY+b17QA68jxL1AwBgj+nVQuqRRx6RL3zhC9LU1PS+muM44vuWEwwAAAAAYBDr1UfAV155pXzmM5+RjRs3ShAEu/xiEQUAAABgX9erJ1KbNm2SuXPnyogRI/b0/fSrLz9wsbglJUVr4/+y0Nh32Y1t6nUf7ChT6xvyDWp9+KIWYy1M60eIV0zeptZbNlSqdc9R1tqWc53D2pxaz1m29g3zzPvz2ix7d5Jdlm1kaf0zhLRj/kDAKViuXa5fO+/rW/sSypbJwLKd0rEcf+7ZvmfK++rZLt6HbMef2+4s7hHmAAAA/1uvnkh96lOfkieeeGIP3woAAAAADA69eiJ18803y6c//Wl5+umnZerUqZJM7vpk4aqrrtojNwcAAAAAA1GvFlK//vWv5S9/+YuUlJTIE088IU6P7UKO47CQAgAAALBP69VC6t/+7d/kmmuukX/9138V1913jiw+cMG7knCLHzXedsbRxr65Q25Vrzvpbxep9fIK/fjzkW+sMNa8UfVq7+wD3lDrf5KD1Xp7YL4315LPGlLTrtZzFbVqvVr50dpQ0H90E5aMVC6j55RKtDxQwXL8ecKSkQr0eqlyYIv9+HO1bKUdf+5ZXjsQPX9lzTlZjk8HAAAYSHr1L5dcLidnn332PrWIAgAAAIDd1auV0Pnnny/33HPPnr4XAAAAABgUerW1z/d9uf766+XPf/6zTJs27X2HTdx444175OYAAAAAYCDq1ULq1VdflenTp4uIyGuvvbZLzbHMqRnIwkJBQsN2xeH/tNLY93beknG6t3juaodtk0r1+1IyM10Th6m9H6u6X61vHFGl1lcXzLkWp1yfjzWuulmtr8gMVetljjnH1BLo75lnmyNVq//olyg/xo41I6X/HbDNkZKg90GnuBkpbY6Ujd+HGSdbviouNXvWnzOo4v7f6eD9v2MAAAa8Xi2k/va3v+3p+wAAAACAQYPTIgAAAAAgIhZSAAAAABBRr7b27atWfflAcUtKitaWTbjF2Hfc0s+r18385TW1XrJ1klr3DhxrrG2arOevDkvl1PrMSnP2S0Tk1exIY82pzKi9kypWqfU3M/rXXeqYv7aWQM9neV0Fte6n9PetxFE+Y7BkmIKk/vlEwdfrYWjO5NgiUI4e3xLX8tlJqMyRci0BrMCSr0q6+s35fRjocfoz54ToBvP3i1waAOw3eCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6uP5Tv5DyTPEZP9/ZOsXY13X/CPW6GX+DWk++skKtN33iEGOtdYqeO6lwi2e+dphR+q5a/1XTB4w1v7pC7Z1U2qjW85V6DsJTckpNBf21va68WvfTljyPlpGyzJEKLHOkCoHl84vA/L74Sn5KpJ/nSFk+l0mLnlvTuNqcp34W2JJrgznvAwAAjHgiBQAAAAARsZACAAAAgIhYSAEAAABARGSkejiypE0qS4qvLefdcKKxr+5BfV5S+0emqfWSPzyv1jd/wJzJGTNhs97rd6j18ZafgJe2jjbW0rV6/mp8Sr+3Qqb3gZ5mv1ytO1k9IxXoY6Qk6RTPyomIiG/JSCX1nJFvmSOl9lrqjpKv2h1aRsqzZIFsc6Rs/DDG+0IMCQAA7GU8kQIAAACAiFhIAQAAAEBELKQAAAAAICIyUj2c9PwF4pUVz/2Muf0FY1/BkplZe1aDWj/oxTq1fvKM18295RvV3me79RlXp5W1qvUNG2qMtfpa/cdntNeu1iVjmfUUmjM5W/MZtdfpzunXTqtlcZXPGMK4c6QKSv5KRET5um2pMtscKVf0e9PGVHmWWU62jJQXY8iVG3NAlhsjvuXEi34BAIB9FE+kAAAAACAiFlIAAAAAEBFb+3o44KZQEl7x7UvOIRONfWFafxuv+8Dv1PqCI89W61eO+LGxZttu9f3Gj6r1M8qfVuup9UljratW3/M0zNPfl7JMVq1nw4Kx1pzTjz+Xbv3agfnLEhHLFrgg3ta+wHL8eagcYW475jvmDjgJA/axAQAA7A6eSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR7CJW9L6BQPzyz/1VRjn7O2VL3uZyq2q/VvfkA/DntaqviR7CIi+VDP6zy9coJa3z7qL2q9Yp251jVcbZVSJ6XWa8o71XpnaD4efVtOf8/DvOVo9bQeNvIc5TMGy3H3geVvVRAjh2Q//twSorJR2m1HkPuWz2Vc2/HpfK4DAAAGEf7lAgAAAAARsZACAAAAgIhYSAEAAABARGSketh27hHiGfJIzx7/fWPfguaj1es+0aWvV8ccrQSRRGRjod1YG+6Vqb3Jt/T6qln6vVWsN+eBWg7Se9WckYjUlbeq9e1K3mdbVv+6Utltaj3Q41uq0NezQkFSz0CFviUjFZqv72vzraR/50gFYd/NoHIdW75K51j6AQAAouKJFAAAAABExEIKAAAAACJiIQUAAAAAEZGR6uGDlz0v6Yric6TeyZvnFv1/Q5eo1z3smS+q9Ztm/Eat/7zlCGPtnKqX1N4hy/T0yJLsaLVeur7DXKzVg0a+kvURERlZqs/XagrSxlprt3m2lojIUMscqSAVIzMTWDJSlr9Vod/7zy98Sw4pdhRIub5nmwMV2jJztjlUfZexiiN2vmpgflkAACAmnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakerh3+qlRmiq8tJ/7ycmPfX865Qb1uzX3lav2kYzvV+lUvHGOsjThczxlVLdNnNS1snaDWvc3meUy1NfrX1R5m1XpDWp/11FioMtY6uvV8Vm2hoNaDdIyBS755tpaIPSMl1jlS5kxOYJ0jped5bLO9tIFM1oyU5d7cGEOubHOk4gqU97xfMf8KAIABiydSAAAAABARCykAAAAAiIiFFAAAAABEREaqh0vXfkCS5cWzN5NuWmns+/LRZ6vXrX7kTbWe/76euSlbaM4i3V8/Xe11Vm9U64s363OkhjatMdbGVukZqK2WLNHIZItabyxUG2vZruLzvnYILa8tqd7ndUJLniZIWjJQtoyU1mrLSFm+bCvLnCqNbcaVvb/vPtfp64wVAADY//BECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9bBywRRJJEuK1qoK7xj7Gh/QZzHVdbyo1h/sGKH3L2wz1l6f1qD2Ttqmv/bWDQeq9ZrubvO1KzrU3rWFSrVel2hR60+1TzHWgi7Lj64lx+Sm9DCRHyoZKkv+KrTcmhMjI5W35Ihsc6SslC/bljMKLPdmm0Olsb22LTsGAACwp/FECgAAAAAiYiEFAAAAABGxta+HivtekIRT/FjtFd+ZZew78H/Wqtft/PA0tX7TigPU+pDXVxhrFcsOV3udhP4tLlmvHyMujnnL1EGlG9TWt3N1av2YUvOR8iIi67qGmG+ry1N7bRKWrX0FMddDy/a5wPKWKpe2sm5h6/2p7u+JcYR5EPP484DPdQAAwCDCv1wAAAAAIKJ+XUg99dRTctppp8nIkSPFcRx54IEHdtby+bx84xvfkKlTp0p5ebmMHDlSzjvvPNmwYdenIGPHjhXHcXb5dd111+3lrwQAAADA/qRfF1IdHR1y2GGHyYIFC95X6+zslMWLF8u///u/y+LFi+W+++6TZcuWyemnn/6+P3vttdfKxo0bd/668sor98btAwAAANhP9WtGavbs2TJ79uyitaqqKnn00Ud3+b2bb75Zjj76aFmzZo0ccMA/ckWZTEbq6vQ8zu7wjz9MnETx489/8tn/Y+z7wTcPUa/77rx6tV7y3HC1XtVlzkjVvFVQe71hQ9V6+TrLMeFlZcba2OQWtfeBliPU+pzyt9X6xi7z8emJrnh5nFRKf9987fh07Wh0EQks8a04x5/bjhiPffy50m47vjyw5Ldcp/cBLjfG0ekiIo7l+HS9N9ZLAwCAfdSgykht375dHMeR6urqXX7/uuuuk9raWpk+fbrccMMNUijo/0jOZrPS2tq6yy8AAAAA2F2D5tS+7u5u+cY3viHnnHOOVFb+40nFVVddJTNmzJCamhp59tlnZd68ebJx40a58cYbjdeaP3++XHPNNXvjtgEAAADsgwbFQiqfz8tnPvMZCcNQbrnlll1qc+fO3fm/p02bJqlUSi699FKZP3++pNPpotebN2/eLn2tra0yevTovrl5AAAAAPucAb+Q2rGIWr16tfz1r3/d5WlUMTNnzpRCoSDvvvuuTJ48ueifSafTRRdZHV9tE688V7TnqPR242u6hx+s3tM3P/SgWr/zyo+p9cQY8yKv/O1mtTc3Qc+OZdbr2yDdGvMsp9GJTrV3WdsI/bWH62GiLR0VxprXaQmuWIItpam8Ws/HGPZkmyPl9OEcKcfvu4yUjR9zjpRvyX8NVnHyWf0pHKT3DQDA3jKgF1I7FlHvvPOO/O1vf5Pa2lprz5IlS8R1XRk+XD/AAQAAAAB6q18XUu3t7bJ8+fKd/71q1SpZsmSJ1NTUSH19vXzqU5+SxYsXy0MPPSS+70tjY6OIiNTU1EgqlZKFCxfKokWL5MQTT5RMJiMLFy6Uq6++Wj7/+c/LkCHmJykAAAAAEEe/LqRefPFFOfHEE3f+947c0vnnny/f/va35cEH39sSd/jhh+/S97e//U1OOOEESafTcvfdd8u3v/1tyWazMm7cOLn66qt3yT8BAAAAwJ7WrwupE044QUJlXo9WExGZMWOGPPfcc3vsfv449XdSmSme0zjsmUvNjWeaszwiIhdVNar1e5e8q9ZbPjzRWKt8+FW1d9vx09T6sOe3qXW/zvxkr8bVf3xWb9OfCpY5KbW+va3UWEt3q60ijp63KUvqGamsNivK8nMZJPW6U+h9ligf6u95jFFN/6/ffG+2WU62GVe2OVRxxM1nAQAARLVvprsBAAAAoA+xkAIAAACAiFhIAQAAAEBEA/r4873t9x3DpdQp/paMvd6c75jy3y+o130+q+dx/K1Nar1xljkjVfGbDrW3pfgorZ1GPKRnpFpnjTXWSi0Zp9ZtZWo96ehzpPLt5uuXdamt4iT1H+2yZPF5YTvkLDkojSXGFCvHZJ0jFeO+RSTWHKm4tK/NjTnTKG4/AADA/8YTKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1cMPfnWWeOmSorWGxYuMfd8Z8Yx63WNfvFCtNwzdrNaPOOodY601nVZ7Kyc3q3V/W4tab28w55g8y6wmb6ueobJx28yvnei0zGpK6D/aFcmsWs/GiNQElr9Vnm0GlsK3zGpygphZIGWOlGfJGQWWWU5u3CFX/cTpz3wV47EAABiweCIFAAAAABGxkAIAAACAiNja18Oo/35VEoYjvbeef7Sxry14Ur1uyX3Var17Rrla/7eRtxlr3x35SbX31NFvqvWXsvpauqPBvK0pH/pqb7o53r6kZJv53pJd+jYxJ6VvK8wk9K19HbYzzBVhQt8K5vq9//wiL/qR8eLH24YWZxdbYNmH5on+PQuUbYt9vS3Q789z3/dVbEsEAOzjeCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6cIfViusWP078w1ctNPZdufoM9bq1f3xbra+6fLJaP0Y54bxz0jC19/Sq+9T6S86Rat0fZT6ruz3Qc0YlTXruJBvm1XqyzVxLdFsyLamkWs5oFxeR7tCSRVKEScvR7Hq0TMQxh0v6/vhzc8mz5Ihsx5/3JVs+i7gOAADY03giBQAAAAARsZACAAAAgIhYSAEAAABARGSkelh25QhxS0uK1h4e8Xtj39S7vqJed2TzIrVefcwmtd4amHNK2ybr85KmpfRAjltRodbH1jcZa1sseZzSJn32T2egZ6RSrebrJzrjzZGq8PR8V2egBNMsbHOknEKvLy1523yrmBGpOOOabBkp1zKkylc+1/HiDLjqY35ouTcCWgAA7JN4IgUAAAAAEbGQAgAAAICIWEgBAAAAQERkpHr47SkLpCJTfG153upTjX2jH2hUr5v74GFq/eoJ96r1u9rMc6a2T9EDN2WunhVyh9ao9SNrVxtrawuVam9Jk56Bagv1QE6qzZw98br17FdYon/dVYlOtd4R6v2qhCW/ZcshOebPN3zbvCQ/RshJRETJOdk+denPOVJ9yRnA+SwAANB/eCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6SDmBpAwxjxU3HmTsq3hHnxO18sqZav3jZVvU+tHPf9xYGztRn0G12e9Q6/m6arU+q+JxY+2dbJ3am2zWc0jNflKtp9rMeR+vU89fhWk945RxzbO5RETa/FJz0bHklJKWjJQe7xLHNV/fNkcqbpwn3hwp/XMZT/SL+2o+K94XZpthBQAAEBVPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVA+nP3a5uKUlRWuTfmvOQXkHT1Kve9VJf1brScdT685zVcbaJ85/Ru19tnuEWu9sKP717nBwypzBeqJ1itrrbtfzWZv9CrWeajPnoNxuS0aqVM9flblZtd4RpM1FZc6TiIhryUi5loyURssRiYhIEHeOlLnk2V7aMuOqL9nmaw1kQcz8FwAA6B88kQIAAACAiFhIAQAAAEBELKQAAAAAICIyUj1M/lGTJLzi2ZjcCTOMfWuP1XNGV1avVOuPdJWr9bpnzfOYTvnKG2rv/I2nqvW2UXo+a5Rnzhotbxum9oat7Wq9sWDOfomIeO05Y82xZKQKVcocKBGp9PQ5Uu/mhppfW5nzJCLiefHmSGkZLOscKT9e3ibOuKXAkt/yLEOqAj7XAQAAgwj/cgEAAACAiFhIAQAAAEBEbO3rIWzcLKGTKlpr/7F5C9ysmrfV676V14/avvHdM9R6eql5a+CBSeWYbhF5duUEtS4N+l6uMrf4+yEisqalWu0d2b5CrW/MD1Hrbpt5+52TNW/7ExEplOlbFssc/XuyvVBmLlqOP08m9b171q19ytZB3/bZRxjzKG2l3fapi21rXxxunD2Hu9GvbTp0Bu/J6vH2agIAABVPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVA8bLpomXrr4UeYvHPYjY19noB/FPee1L6j1ppeHq/VxrWuMNVf0AEfqLf0Y8PxB5qPVRUTyoTnQ09akH9se5vUc09ruGrXudCpHlBcKaq9fqn9GUO7qGal2X8meWY4/T3q2jJSeW3GUUE4+1LNfEvf4c/2EcpUtI+WqSST70e6DVayMlaXXD2N8wwAAQCw8kQIAAACAiFhIAQAAAEBELKQAAAAAIKJ9M5TQS18+/0EprSj+ltzTVm/sO63cnGESEcnfp2eg6jfqmZpEw0hj7d3C39XeIcv0DEXVhzer9U1+l/m+mpJqr82Grkq1HnaZX1sCPQuUt2WkHD3X1loonpUTEXE8PbiSTur5rYJ1jpT53m0ZKacP50hZ0lkS9ukcKf3nuC9nWAEAABTDEykAAAAAiIiFFAAAAABExEIKAAAAACIiI9XDOZmNUpkpvrac/uMrjX1PfnKpet0Rf1gZ677aZo4x1h7vnKT2Vi7brtZn1ryr1pflq4y1dJMll+LqqZpNnRm1Xtm9Sb++olBqyTE5elCpo6DMkbLkq1KWOVK+LSOlDB4KQstnH0G8uUJO0PusUWAZeuQ5en6r2/a1xeBYXhsAACAqnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakeTnvjE5IoL56NGX3zK8a+hYnD1OuO3rRQf2HL7J/GWeONtQcaD1d7ndUb1PrM8hVq/cVO82uXNOn37ZaaZzGJiGxrL1PrmWzWXPT0/JUtI1Xu6lmi9rw5I+V4em86oc+R6rZkpBzla7PNkbLN17KJEyXqy1lOtnwVAADA3sYTKQAAAACIiIUUAAAAAETEQgoAAAAAIiIj1UPyR0MkkSie63GHm3MxY+/fql63cMw0tZ548121Pv7oNcbam8sb1N5JLS+o9cnJJrV+6/oTjLXSZj0r5FSUq/WuNmVWk4iEysAlWxqnUKrXk5b+jkLK3Ovl1N60p2ekXN+S93HNX509I6V/T/zQMmdKKXvKfCsRe0bK0y6+G/39hRlUAACgGJ5IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpHhJPLJGEUzw988ZPjzb2TfrK8+p1V94wS62Pe8g8q0lE5F8O+JWx9k9/vFjtdRL6t3hkQs8pvdk4wlg7oEnPCkllhVp22iw/fsp8LS0/JSLiWzJSJY7+GUJ7zpyRGuLpvWUJ/X1xLHOkRLk3P7R89mGZSWYTJw4U9mPGybek5lxyTgAAYA/jiRQAAAAARMRCCgAAAAAiYmtfD52nHymJZPHjz/8454fGvn+a+Hn1ul/46JNq/bdNJ6j1E0u7jbWat/Sjtr0Rw9V62rCVcYfC+jJjLdHUovb6Q/Tjz5NtMdbxli1shVK9nrRs7evOKe+Lpx9BXuKZv18iIq5ta5+yddB2/LkTxNzCppxQbvtu2Y4vdx3L0ezKK7gyiLfmDdZthQPzNHoAAAYMnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGake6q5YKcny4sdep5R8x9oz6tTr/n7ob9X6fcccptazYd5YK3+7Se3NjTcfX267tohI+Xolt7K9Xe3tGl2v1pNtlhCGq+SBAtvx57aMlJ410jJStiPlSz39PXV8S2ZGyW/ZMlIS6DkkmzhxHltGyt7ff5/r+DGPjQcAAPsfnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakefjH2SanMFF9bjntwrrHviNPfVq+7ptCl1i+frM+Z+n1Hg7EWrtuo9m77kD5HakMhq9Yr1pszN2Fbm9rbXTtKrSf1dnGS5h/PMKtnpIJSPSvkWj5DyOeVLJKr99oyUq4lI+Uoc6QKgeWzD0vWJ7DNY4oRFbK1ejEubp9BZZth1fvXdgbrHCgAANCneCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6+O7WKZLuLj4/6KDvbzH2/eSvD6jXPe+dc9T67yb/Rq1/5JXzjLUhne+ovS2T9XzHG/mhar18XbexFnTo2a+uWn2dXtKs35ubThtrfi6n9jqlBf3alkyNn1UyUgl9llPa1V/b0csijvne7HOk4uV5tDiQZ3nP4s6RsuWc9kvkswAAGLB4IgUAAAAAEbGQAgAAAICIWEgBAAAAQERkpHr4823HipcqKVobtuEVY1/G1d/GrfePVusV88xZIBGR9oXDjLXakrVqb9XkZrX+fMcEtZ5s3G6s+Xk9p5St1fMdlWv02UBSWvx7ISLidHaqralSfZaT51g+Q8iZ66Ey50lEpMy15LdsOSbPnIMKwnhzpGws45osL61nnLwYF48zB6qvBRLjTQMAAIMWT6QAAAAAICIWUgAAAAAQEQspAAAAAIiIjFQPtb9aLAmn+BypdV872tj3zU16Jmbk79eo9Wfm6uvZ+oVZY80dWaf2zh79plpf1DRWrbtb9YyVJlerZ0dSrZZ5S2Wl5mKLPk+prET/ntg4SkZKyzCJiJR55u+XiIjj2zJS5tfOx8xIWfM8StlV5luJxJ8jZc1/xeDKwM1Y9SlGcwEA0Gf69YnUU089JaeddpqMHDlSHMeRBx54YJf6BRdcII7j7PLr1FNP3eXPNDc3y+c+9zmprKyU6upqueiii6S9vX0vfhUAAAAA9jf9upDq6OiQww47TBYsWGD8M6eeeqps3Lhx569f//rXu9Q/97nPyeuvvy6PPvqoPPTQQ/LUU0/JJZdc0te3DgAAAGA/1q9b+2bPni2zZ89W/0w6nZa6uuLb195880155JFH5IUXXpAjjzxSRER+8pOfyJw5c+T73/++jBw5MtL9OAcfKI5X/Cjyr110n7Hvht9+Ur3umLUL1fpN6z6i1tNLVhlrnUfpx5d/vPJetX73G0eo9Qmtjeaiq29xS9R06/U2fZtZUG7e2uekUmpvZYm+vc7GzSp7ohL6113i6FsW3YJlm5lr/nyjEOivLb6v1y3inDJuO/7c7cNjwvtyW2Dc3XHsrusF3jQAwCAw4A+beOKJJ2T48OEyefJk+fKXvyxNTU07awsXLpTq6uqdiygRkZNPPllc15VFixYZr5nNZqW1tXWXXwAAAACwuwb0QurUU0+VX/7yl/L444/L9773PXnyySdl9uzZ4v+/T90bGxtl+PDhu/QkEgmpqamRxkbzk5T58+dLVVXVzl+jR+sDcwEAAACgpwF9at9nP/vZnf976tSpMm3aNJkwYYI88cQTctJJJ/X6uvPmzZO5c+fu/O/W1lYWUwAAAAB224BeSP1v48ePl6FDh8ry5cvlpJNOkrq6Otm8efMuf6ZQKEhzc7MxVyXyXu4qnX5/FmrlV5PilhXP3pxXud54vd/dt1297/CIQ9T60uczan1C03PGWvPkKWrvtJSemXHWKUeMi4gE5n63vFxtravRt0x67cWPmt+hUGW+t0RK761Od6l1GzdnDmmEScvx567l+PNADyKFMY4/Dy3Hn9s4MWJMli/LyleCMV4f5qsAAAB6Y0Bv7fvf1q1bJ01NTVJfXy8iIrNmzZKWlhZ56aWXdv6Zv/71rxIEgcycObO/bhMAAADAPq5fn0i1t7fL8uXLd/73qlWrZMmSJVJTUyM1NTVyzTXXyFlnnSV1dXWyYsUK+frXvy4HHnignHLKKSIictBBB8mpp54qF198sdx6662Sz+fliiuukM9+9rORT+wDAAAAgN3Vr0+kXnzxRZk+fbpMnz5dRETmzp0r06dPl29961vieZ4sXbpUTj/9dJk0aZJcdNFFcsQRR8jTTz+9y7a8O++8U6ZMmSInnXSSzJkzR4477ji57bbb+utLAgAAALAf6NcnUieccIKa6fjzn/9svUZNTY3cdddde+R+HvvA/0gmU3xteeKrXzD2lS95Q73uu/81S63XPafnP7xhw4y11in6zKIyV5+3VLZeH9jiJM39bqZC7Z1QtVWtb+4cotYLoyqNtWRSz0jVpjvUuo2nZaQS+ucPJW5erTt9OUfKElTybRkqpexZhvvY5kh5cQJYMTmWAVnxpm8BAID90aDKSAEAAADAQMBCCgAAAAAiYiEFAAAAABENqjlSfe2tfLmU54uvLdM3mPM8iTH6MN+Pn7pIf93bGtR65xFjjbXxExvV3s2+nhXKrNfTIW51lbEWVunzr6aUr1TrmzpK1HouY84DlZa8fw5YTzWpzWrdD/W8jjYKypqRciwZKdvAJWWOVMEyR0osX5dVjFlQtoyUTaB8ba414xTvtQEAAKLiiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkerhyvsvErekeG5n3OMLjX3rr/qAet3fjrhXrX9y9dFqvfFCcwbrK/WvqL1Pd9Wr9bL13WpdhpmzYYWqUrX1wPQmtf5Ed51az1WY1/lhSp8jNTTZrtYLlslB2iiowDpHKqfWHd+SY1LnSNkyUvrXFYj+2nFGPQW2OVKWAJZvy3/1E1s+CwAA7J8G5r9cAAAAAGAAYyEFAAAAABGxkAIAAACAiMhI9TB+wSpJuKmitbYzZxr76s5YrV63OdAzM255uVqvPcY8K+rk8jfV3v/aMEetJzduU+tdBw4z1oK0vg4fm9yq1sOc/r7kKs2Zm7C0+PdphyEJfX5W3pIl8vpyjpSvZ25Cz/x1F0LzbC0REQkKet0iThyoL5NE+2tOyWE8FgAAAxZPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVE9BKKakR8M/vWNsmz/qQfWyV6w6S3/ZqSPU+tXjHzDWDkym1d6FK8ep9UlNy9V6+wkNxppt5tBIT89A2TJS+QpzLSjV50jVevocqXyo37yXM2dygoQeXLFlpCToyzlSMQZBiYg2Zsq1fO4SWuZIuZYZVoH0XSCoPzNWzn6a7+rDbycAAAMCT6QAAAAAICIWUgAAAAAQEVv7elj1lfHilpQUrS0bd4uxLxvq2+tWPjhBrTvH6Pc1u8x8jLhr+Ram3ypV60G7vgWuvcG8PyfZqbZKjae/LxLqW57ylea6X6J/3dWe5fhzy2Hd2vHnQVL//CHp6EerOwV9i1uYNB9xHli29nmB/nX5lq/bsXxPNLatfX3J5zMh7CH9+GMMABhk+NcHAAAAAETEQgoAAAAAImIhBQAAAAARkZHq4Qdn/VzKM8XzKf+x5ZBeX3f0/RvU+vL5lWrdc8yb9t8t6EGlIcv0vI4tp9Q1qmCs+U3mLI+ISNrRjyi3KWTMWaJCqf7a1W6XWu+2fN3a8eeh9fhzy3vu668dps3XL4T6Zx9ujIyTiP1Ie03MlxZf+do8y9HpAAAAextPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVA+Hp9olkyq+tvzGzR809m0/UL/u+JUL1fqXDmlT63/urDLWNuSHqL2Vy7ar9TCtz3qqHtlqrLWInu2yUrJfIiKSyRtLhTI9f5Vxzb0iIt2WYTFaRipI6r1JS57HCSxzpDzz5xu+ZY6UTWALMillLasnYp8j5cUJYMXkOvrXrd2ZY+kFAAD7J55IAQAAAEBELKQAAAAAICIWUgAAAAAQERmpHk5a9EVxy0qK1sb9/AVjX+2Mg9TruodOUevnV92u1s9+61xjLe2Z5zyJiLhrNur1IdVq/fAR6421Z7tTam821HNKTkLPOZVlssZaoUR/7TJLrqU50H/0vay5v1Cqf/6QsmWBfFtGSpkjZctIWfJXNvHmSFkybxaBmPutGSfLfC0AAIA9jX99AAAAAEBELKQAAAAAICIWUgAAAAAQERmpHsb8KJCE5xetOYdOMvaFL7ymXnf1N2ep9eFeuVrfsGiksZYbrmekJrWYs10iIu60erU+s9L8ta0aUqv2bg9yat1J6RmpmvJOY62zVJ9hVe7onxGsC/TXdvPKHKmM2iqWMVP2HJOSkfJtOSTLnChfGxQlos6RsrGNqPIsF/fJOQEAgEGEf7kAAAAAQEQspAAAAAAgIhZSAAAAABARGakewlfeltApnp1Zeeehxr6Jc0eo1z3246+o9ZeyepaofqE5B9U8Rc/6OAn9W9zVoAd+ZpS+a6y9VDVW7W30PbXulBaf2bVDXXmrsfZ2qZ7tSjv6190d6u+bly2elRMRCSzvqX5lEccyRyrwzJ9v+PvwHKm+ZJtDFYc1d9aHr20V51sycL+dAAAMCDyRAgAAAICIWEgBAAAAQEQspAAAAAAgIjJSPTR//gjxUsVzO88e931j3ymn/7N63bsablTrV649Ra2Xv7RGqR6g9np1en6rvUH/ERifMOe3JpU3qr3v5mvUulNaqtZHlpqv/0aZ2ipJR89ntQZ6PsvNmcNCviUElXQs4RJLjkkbp1Tw4332YYtAaXEe1/K5i22OlGsJYAVKxirpmjNrIiJ+HwZ6nP7MOKF3+J4BAPYCnkgBAAAAQEQspAAAAAAgIrb29XDypQslXVF839ZrOfMx4aWf3KReNx/qW5r+/tJBan1i4yJjreztcrU3N2G4Wu9o0LdEDXHN2+8mpvWve3m2Tq2HFfrWvob0NmPN13fmiWvZ6tUZpPX+nHkrWWj5W5N0LJ9PWI4/DxO9P/48tOyv82377+LsiBrAx58PWmxR27/wVwgABhWeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR7+fdgbUpkpvrac9IsvG/t+f84P1OvOXTdbrQ9fqG+M96qrjLVwvX4EefOJ+vHn3Q15/bWVvM/YRJPa+6dtU9V6UKEHnUYmW4y1QqmeHdHuW0SkNdDzWU6uYKwFScv3yxZ0sOSUQuXkdu2I8P/3B/S6heWEclVouTfPcvi6r3yukxbz92N3uLHCX30rsB5KDwAABiKeSAEAAABARCykAAAAACAiFlIAAAAAEBEZqR4uWn28JMtTRWsH3rjc2DfpPD3r89KDh6r1MYs2qvXc4ROMNe+JxWrv9sl6NmRYQ4veH3QZayMTerZjeeswte4YZnbt0JBQ5kiVxcuVtFsGUTl58xypwPK3xrV9PuGbry0iErrmrFEQxBs0o7+yiGObM6WIm0Ky5r/6kD9wI1QAAGCA4okUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqYe2CAyWRLJ6dqZSVxr4bt01UrzvmgS1q3V+5Rq1v/HS9sXbAc3rWZ8jkZrU+fdg6tb4qb15rH5TSf3zWN5vnX4mIDK/U+2u8TmMtLI2XkWqzZKQkZ56vFejRLvEcPesTBvq9BwktI2X57CO0XFvvjjVHSvo149R3nwkpkTUAALAf44kUAAAAAETEQgoAAAAAImIhBQAAAAARkZHqofyBFyXhFA/ALP/uLGPfssfMc55ERA588zm17qTTaj11jDnn5Iwy56dEROaMfl2tj0vr+a0l2dHG2uHpzWpvtrlUrecq9HV8jVsw1txSc01EJB/qE5Paff09d/Lm68eeIxXoQ4tCT2mNOUfKSrk1VyzZL8ssJs8yaUqbI+XFCm+JuDH747BE5gAAwCDFEykAAAAAiIiFFAAAAABExEIKAAAAACIiI9WD/8HDxEkUny9029n/bez7rwvOV6/rTdIzVGGZnte5YtJfjbX/O/E0tfdjlb9R6ynLZKFbtpxgrF1QqWekUk1K2EdEchm1LBnX3F9SmlN7bRmp1oJljlTB3B9aM1KWUEyg31vgmftDS0YqtOSvfEuOybHU9ReP0QsAADDI8EQKAAAAACJiIQUAAAAAEbG1r4eur22XRHl30dphqXZjn/vMUvW6a/5lplpPtep7oj6VWWWs/Xhy8ePadzg0qV87b9nat3iz+fjzbMNTam+6Sd+Glq9Uy1LmpIy18hLL1j7Rt8+15fWtfWEub6zZjj/3nL48/jzeZx+2Q8DjnBJu23ZoO4LcVz7XcW1Hp/OZEAAA2Mv41wcAAAAARMRCCgAAAAAiYiEFAAAAABGRkerhj4c8IJWZ4mvLQ56+zNh3YO1G9bqHnv6WWn9+xVi1XuGYj0dvnVJQe8tcc85od2zdUGWsNU/Lqr0lTXquZfuB+msnHXNYqKqkeJZth07LEePtBf3IeSmYv7bAkjuz8uMcf265tvUP2PpjfG2h5dh3iyBmv8aNda47AADA+/FECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9fCb9hFSanhLxn/PnGvZ+jE97HPHAd9X6z8uP0atP91t/jYdOFHPZ20smOdfiYgM98rUesl685yqTb4+w6q0Sc8CNU3v/Tp+SLpTrXdaIjEdeT07lvbN1w9j/q0JLTkkbY6UbVaTjS96f6w5UoM4hqR92Q75KgAAUARPpAAAAAAgIhZSAAAAABARCykAAAAAiIiMVA8/+b9niJcuKVob+coiY1/hGj0jVabMQxIRuaRmoVq/YtWnjLUz6peovU93N6j1D5duUOvl68z5kHcLtWpvulmfMxVk9IyVZkhKz0i1Bfq1O/N6PVUwz+cKEjEzM4EeRAr6MiNlmdUUKw5kubYn+sWD0Py5jmcJb9myX4PWPvplAQCwL+CJFAAAAABExEIKAAAAACJiIQUAAAAAEZGR6qHhf16VhFN8vtCWLx5t7PufqT9Sr/uvGz+k1m9uMOevRETeeH6csfb9z9yr9n573cfV+tiRW9V6Zr05K/ROdoTam2jqUOvpTKlaz4Z5Y602qV+7NUyr9c6sPkeqyjfPwAqTMTNSMeZIiS0jFXeYk9LuOZbPXQbwuCW3H2dB9escKmZgAQDQZ3giBQAAAAAR9etC6qmnnpLTTjtNRo4cKY7jyAMPPLBL3XGcor9uuOGGnX9m7Nix76tfd911e/krAQAAALA/6deFVEdHhxx22GGyYMGCovWNGzfu8utnP/uZOI4jZ5111i5/7tprr93lz1155ZV74/YBAAAA7Kf6NSM1e/ZsmT17trFeV1e3y3///ve/lxNPPFHGjx+/y+9nMpn3/VlNNpuVbPYfM45aW1tFRMQdPlRct3i25tQr/m683hFpPW/z+YePUOtvXvA3tV6/0DxDZ8K5es5o0cqxan3xEL1esqHNWHurvV7tdVrb1Xp1hVqWzsCckRqa1K/d4pep9Vze8qPflxkpyxwpNSPlxxssFFgGEzlBjK/N0mqdI6Xcm2uZI9Wfgri5NAAAMCgNmozUpk2b5OGHH5aLLrrofbXrrrtOamtrZfr06XLDDTdIQRmmKiIyf/58qaqq2vlr9OjRfXXbAAAAAPZBg+bUvl/84heSyWTkzDPP3OX3r7rqKpkxY4bU1NTIs88+K/PmzZONGzfKjTfeaLzWvHnzZO7cuTv/u7W1lcUUAAAAgN02aBZSP/vZz+Rzn/uclJSU7PL7PRdE06ZNk1QqJZdeeqnMnz9f0uni2/TS6bSxBgAAAAA2g2Ih9fTTT8uyZcvknnvusf7ZmTNnSqFQkHfffVcmT54c6XXe+uowcUtLitYeHv6Ase+27SPV6469f5tav+Xj+pypyhfWq3VN+i09Q/XsgRPUurvZfO8rWoeqvaXtTWp9eLlalrbQnIupSegZqWZfD2Dlc/qPfqhlhbx4mZjQOkdKyTHZ5khZ+NaMVIyLhzHvLUa/H/bdLuX+nEEFAAAGrkGRkbr99tvliCOOkMMOO8z6Z5csWSKu68rw4cP3wp0BAAAA2B/16xOp9vZ2Wb58+c7/XrVqlSxZskRqamrkgAMOEJH38ku//e1v5Qc/+MH7+hcuXCiLFi2SE088UTKZjCxcuFCuvvpq+fznPy9DhgzZa18HAAAAgP1Lvy6kXnzxRTnxxBN3/veOvNP5558vd9xxh4iI3H333RKGoZxzzjnv60+n03L33XfLt7/9bclmszJu3Di5+uqrd8lNRXH/RxZIRab4Q7pzV33c2Ldw6UT1upNeeV6t/+Hlo/T+tS8YaysKXWpvzTLzMd4iIq8c1aDW67etNNY2bhuh9o7rWqdfu1Q/XbHZTxprNZ6+tW95Vj8O389ZHsYq2wqdpL7/zVd6RUTEcsR4oB1/Hvfkddv2uTjXZwccAADYj/TrQuqEE06w5kUuueQSueSSS4rWZsyYIc8991xf3BoAAAAAGA2KjBQAAAAADCQspAAAAAAgokFx/PlAsObGScbauO161icxboxaH7pQ/za45eZzwv/aYb4vEZHM29vV+vqNlWq9Lps11nLNxY+K3yEs6O/L6BI957RZOcK82utUe7fm9ePPJasFkURE2XLqJPQMVGALC1kyVKFya05fH39u2WqrsrTajhEPlCPMvZgBLNtr294XAACA/40nUgAAAAAQEQspAAAAAIiIhRQAAAAARERGqodPPnq5uKXFcz+T7l1k7HOSKfW66y4/Uq03/GWrWi9MO9BYu3+jPnjYW7NRrZetrVHr4pizI6mtlpyRRX2yRa03FqqMtWnp9WpvU96cKxMRcXK9z8R4CX02l01omSOlZaQk3kvbs0CWEVgq24wqC+uMK62Xz4QAAMBexr8+AAAAACAiFlIAAAAAEBELKQAAAACIiIxUD1N+tEUSbrporfvDRxj70i+vVK87+oxVar1w8xa13njV0cZa23I9nzWpZZ1aL1+v53XcCvM8ppImy0wiS3asIblNrS/uHGusHVv6rtq7LVem1t0YGamEZY5UPoyZoerDOVI2lnFLscSZBeU6ccJb8Tgx3xQlZggAAAYxnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakegs1bJXCK53q6f2pec3b+crJ63XvGf1+tXyInqPX8rDZjLbMwo/Y6Cf1bXLE+p9bdoeY5UyVNlnyVYSbXDsM889clIrIhW22slVlyJ9uyfZeRSlrmSAW2YUyhXg+1jzdiRoUC9eIiEsbIA8XMVwW2GVcxuHFvrp/EzWcBAIC+wxMpAAAAAIiIhRQAAAAARMRCCgAAAAAiIiPVw/qLp4qXLp7rWTz1J8a+w848X73uCK/4bKod3Enj1fqlBz9trD3w84+ovV7dCLVeskHPKeXrqo210qaC2utU6vmtGlfPZ23oqjTWylxl2JKIbO/W81le1pLHUYb/pCwZKT9mHidQ/lbGHafkW3JIjt9/GSk/7H1Gyrdlv7DXheS7AAD7OP71AQAAAAARsZACAAAAgIhYSAEAAABARGSkerjyCw9IaUXxt+TOtnpj360zfqVe95rNR6j1LcfUqvXPV75urP3lnRlqb/ZAPSOVem21Wu88+EBjrXxdl9obVlWo9WpXX8dv6jRnrJKiZ6Q6uovPA9vBEs8Sccz3lk7o2bC8ZU6UbVZTqHxpThBv1pJv+ewkTqzFsWSc4sxy8gbpHCgAALDv4okUAAAAAETEQgoAAAAAImJrXw+fqdgklZnia8sZN11p7Hv16p+q1/3Sb45V6/lZ+laxoV65sRaub1R7t31Y39o37Bn9+PO2UeZ9ZpVLO9Te/Ajz8eUiImVuUq1vay8z1pKOvrWvu0vf2ldu2drnuOZtaratfTFPKJfQM29ji3v8eWA7JjzODrqYu++s9xaDa9mzGOfodQAAsH/iiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkephzutnSKI8XbQ2asESY9/dXxqiXnf8ve1qfdItb6n157N5Yy3o7FR7Wybr2ZCheT0s1NGg5HVa9a8re7B+rHvCcoR5V1vx74WIPSMVdOk/2tbjzz3z9dOenpHKWY43t9GOP48bwPJFzwI5ce69H08ot31dcdjyVQAAYP/EEykAAAAAiIiFFAAAAABExEIKAAAAACIiI9VD6U1VkkiUFK25deZwyry/H6xed9ILL6r1Lw99Vq3/14Y5xppb0q321k5uUuvi6lkjp6HLWAva9IxUZ61+bc/R1/FOW+9/PJ0u/dpeVs+9OEpGqsQzZ9ZERPIxIzXaOCVXf2kr3zKryQn6LiPlWbJGgTLLyY07QKsfOWSsAADYJ/FECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9eA99Yp4TrJo7Y1bjzb2jblPz0AkRjWo9YNSZWr92ZcmG2tTRm1Wez826nW1vqhymFqfWG++vt+l57OytfFm+yTber/O9zotGakYc6TKEnpQqduSQ7IJE8rsrmy89zSwfXYSI4rkBHHvrS9nQfVdxsrvzwFaNn33lgIAsN/jiRQAAAAARMRCCgAAAAAiYiEFAAAAABGRkeqh44wjJZEsPkfqz7NvNPZ99WsnqdfdeOF0tb6moM9jGvGsOejQOXGo2vvxynvU+qKhZ6v1mTXLjLVng5Ta212jZ0fyoa/Wk229D3gkuvVeL6dnZpyE+a9GqSVglY/5+USojN9yfMt74uh13xaaCftwjlSMLJFnCW8FMXNpGGDIdgEABgH+9QEAAAAAEbGQAgAAAICIWEgBAAAAQERkpHoYfflySZbruZ+iksVnT+1Q+cmNav225llqvWZRo7G27hMj1d5DU3rYIF9frdZnlq8w1p6Vg9TeQm1BrWdDfR5TstVcs+WrvC61LF7OktdRM1J6pi0fd46Up8yRijkOybfcmxNnJFLMcUpBODCDMQPzrgAAQH/jiRQAAAAARMRCCgAAAAAiYmtfD7ePeVoqM8XXluN+f7Wxr+Gj+uafn002H50uIjL7qSvU+oGrlhhrrZOHq71pR9922NGQVuuTk03GmpPUt0GW13aq9bZA3/qXajPvFbNt7UvYtvZl9X1oTsJ8Bnna1e+7O4z310rb2mc5BdwqsH12EuP481jbAqV/t/YFA3UD3wC9LQAAwBMpAAAAAIiMhRQAAAAARMRCCgAAAAAiIiPVw39uOVjSXcUzRQffsNnYt+w71ep1xyVK1HrVs3rdTZtzTJMmblB7Nxb0o7rbRulr6ZEJ82u75aVq76jqFrXeHJhzSCIiqXZzICgb6jmlRKce2HFzlrBRypwtK/Nyamt3qOfSrJRvifX4c0f/ftqOP491hLmlN86nNq4lgOVbru7FDXD1IT9GLg0AAPQfnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakeHvvvWeKliueVhm1aauy7+eg/qde9sXmKWh/+XKtaDw+aYKydXv+U2vt0d4Na72zQQzfaHConk1F7J1VuUutNgZ6xSrWac1AdoX7fiS49d+JZMlJh0vxXo8zt44xUwnxvcaM+vmUwkePHGFQVcw5Uf86R6kvOAM5nAQCA3uOJFAAAAABExEIKAAAAACJiIQUAAAAAEZGR6qHmVy9JwpAJWvfPM419p5Y9o1738gdPUusT33hZrW/60hHG2kfK31J7v7XuNLWebOhQ69kwb6wFVRVq7+SyV9R6Y6FKrSfazFmkTkueJtEdc46UkpFKu+b3RESkO0jp13YsWSDPfO/WOVIWgWWOVJw4T9wokO3e+otthhUAANg/Dcx/uQAAAADAAMZCCgAAAAAiYiEFAAAAABGRkerBOWySOF66aO1fvvgbY9/DncVnT+0w4d5utR7mzfOSRERaZ3WZr53QZzE9v2KsWp82Zr1a31DIGmuFWv21J6Ya1fobWX3Glddufu22QJ/VlOjUw0RuzlfrYdIz1vp6jpSjZaT02xbH1fNXvu2zkzBGHsjS6sUYE+WJ/v30LZk513ZzAAAAEfFECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9bD6q664ZcWzMWdnNhr7DnniEvW6ExYuVeve+APU+nlTF6l1Telben5r1vSVav2N/FBjrbtWn5c0OrFdrT/aeqhadzrN2bKWQM9nJbr1MJGT03NpYcr8V6PEsWSkLPktcfTPL9w+nCNlyxL1ZUbKJpAYIaqY/H587TgCsl8AAPQbnkgBAAAAQEQspAAAAAAgIhZSAAAAABARGakeHp/5M8lkiq8tT1j6BWNf/b16VsgbVqvWtx09Qq1fNOT/GmsrCnq2Y8gyPSt0ZJmekXqqfYqx1lWjr8PrzKOYRERkfVe1Wg87zPOzWvxytdfr0jNQtoyUX27+npa4ebW3raDnt2xcVwlC2TJSlvxVYPvsJEYGK27KKFDyW54lHGb9ugAAAPYw/vUBAAAAABGxkAIAAACAiNja18OSXIWU54rvRyu/vsrYl3z+NfW6W86eptabZujblkYlKoy1BS2j1d7MMv0I8slJvX5D01hjrXuovpmrwk2r9Y2dlWq9pLvZWGvyze+JiIjbpW+/k7y+tS9Imfclljj6tTsDfaun4+rvm+uZfx6sx59bru2H+mcnTj8ef96XXGcA3xwAABiUeCIFAAAAABGxkAIAAACAiFhIAQAAAEBEZKR6+Kd7LxS3pKRobezfFpobDT07ndmklo+rbVTrb+Y6jbUHNh6u9ibXbFDrIzz9qO7ljcOMNadGz50kHf3886aOMrU+Mmd+X7YV9OPPnW49x+RYMlJ+2vwZgzUj5evZMNsR5YlEjIyUhW/77KQPM1K2T22048/7ky1fZT2RfmB+WXbkygAAUPFECgAAAAAiYiEFAAAAABGxkAIAAACAiMhI9TDupysl4RafAdT6qZnGPq9bzxL8zyE/Ueu+6CGKBVtONNaWv1Ov9k5qXavWbTkmZ505Q1Wotcxqsuho07NlYS5nrG3N63OknKy5V0RECr5aDpJKRsrVv+5sYPlrZZn15LlaRkr/WXMsgRxrDilGRipupEa7N9eSRMqH++b/lcXOV1n6/TBm6A4AgP0YT6QAAAAAICIWUgAAAAAQ0b65Hyai8P9tZyoE5u1ghXy3uT+v72lqb9O3z9i29uXazVvJgi7zfYmIFEJ9G1qr5d6CbvP1g66Y1+7s/b1n29VWKQRZ/Q8E+vHn2ve7w/J1ZZXvl4hIwbINze9U7j1ne8/0LY1d7Zav2ze/dpyfFRGRNkt/ocP82rb7tvwVlFy7/r60l5nvTbsvEfvXpX4/RX9fbX9HrN8Ty/8/qK8do3e/fm3L3wP1tWP08tq8Nq/Na/Pae663tf29WmiJPDih7U/sB9atWyejR4/u79sAAAAAMECsXbtWRo0aZayzkBKRIAhkw4YNksn8/+3de1BU5R/H8c9yWxblIqiLVBAUE94DCQVNLTAta8ZbTpMmecksTC5T3hpE84La6Jhomv1RTaaljY2XsvKSpKaCkk6mYWkNpiLTRVERL7vn98cv9+fmpd8acYDer5mdYZ/nOTvf3fkwy5dz9tlAWSwWVVZW6o477tDRo0cVFBRkdnlowMgaagtZQ20ha6gtZA3/FMMwdObMGUVERMjL68afhOLSPkleXl7X7TaDgoL4xUStIGuoLWQNtYWsobaQNfwTgoOD/3INm00AAAAAgIdopAAAAADAQzRS12G1WpWXlyer1Wp2KWjgyBpqC1lDbSFrqC1kDWZjswkAAAAA8BBnpAAAAADAQzRSAAAAAOAhGikAAAAA8BCNFAAAAAB4iEbqTxYuXKg777xT/v7+6tixo4qKiswuCfVcfn6+7rvvPgUGBqp58+bq06ePSktL3dZUV1crIyNDYWFhaty4sfr376+TJ0+aVDEaipkzZ8pisSgrK8s1RtZQU44dO6bBgwcrLCxMNptNbdu21e7du13zhmFo0qRJatGihWw2m9LS0vT999+bWDHqI4fDodzcXEVHR8tms+muu+7S1KlTdfVeaWQNZqGRusoHH3ygnJwc5eXlqaSkRO3bt1fPnj1VUVFhdmmoxwoLC5WRkaGdO3dqw4YNunTpkh566CGdO3fOtSY7O1tr167VypUrVVhYqOPHj6tfv34mVo36rri4WG+88YbatWvnNk7WUBN+//13de7cWb6+vlq/fr0OHDigOXPmqEmTJq41s2fP1vz587V48WLt2rVLjRo1Us+ePVVdXW1i5ahvZs2apUWLFmnBggU6ePCgZs2apdmzZ6ugoMC1hqzBNAZckpKSjIyMDNd9h8NhREREGPn5+SZWhYamoqLCkGQUFhYahmEYp06dMnx9fY2VK1e61hw8eNCQZOzYscOsMlGPnTlzxoiNjTU2bNhgdOvWzcjMzDQMg6yh5owbN87o0qXLDeedTqcRHh5uvPrqq66xU6dOGVar1Vi+fHltlIgGonfv3sawYcPcxvr162cMGjTIMAyyBnNxRuoPFy9e1J49e5SWluYa8/LyUlpamnbs2GFiZWhoTp8+LUkKDQ2VJO3Zs0eXLl1yy15cXJwiIyPJHm5JRkaGevfu7ZYpiayh5qxZs0aJiYl6/PHH1bx5c8XHx+vNN990zf/4448qLy93y1pwcLA6duxI1uCRlJQUbdq0SYcOHZIk7du3T9u2bdPDDz8siazBXD5mF1BX/PLLL3I4HLLb7W7jdrtd3333nUlVoaFxOp3KyspS586d1aZNG0lSeXm5/Pz8FBIS4rbWbrervLzchCpRn73//vsqKSlRcXHxNXNkDTXlyJEjWrRokXJycjRx4kQVFxdrzJgx8vPzU3p6uitP13tPJWvwxPjx41VZWam4uDh5e3vL4XBo+vTpGjRokCSRNZiKRgqoRRkZGdq/f7+2bdtmdilogI4eParMzExt2LBB/v7+ZpeDBszpdCoxMVEzZsyQJMXHx2v//v1avHix0tPTTa4ODcmKFSv03nvvadmyZWrdurX27t2rrKwsRUREkDWYjkv7/tC0aVN5e3tfs3vVyZMnFR4eblJVaEhGjx6tdevW6YsvvtDtt9/uGg8PD9fFixd16tQpt/VkD57as2ePKioqlJCQIB8fH/n4+KiwsFDz58+Xj4+P7HY7WUONaNGihVq1auU21rJlS5WVlUmSK0+8p+LveumllzR+/Hg98cQTatu2rZ566illZ2crPz9fElmDuWik/uDn56cOHTpo06ZNrjGn06lNmzYpOTnZxMpQ3xmGodGjR+ujjz7S5s2bFR0d7TbfoUMH+fr6umWvtLRUZWVlZA8eSU1N1TfffKO9e/e6bomJiRo0aJDrZ7KGmtC5c+drvsbh0KFDioqKkiRFR0crPDzcLWuVlZXatWsXWYNHqqqq5OXl/ueqt7e3nE6nJLIGc3Fp31VycnKUnp6uxMREJSUlad68eTp37pyGDh1qdmmoxzIyMrRs2TKtXr1agYGBrmu2g4ODZbPZFBwcrOHDhysnJ0ehoaEKCgrSCy+8oOTkZHXq1Mnk6lGfBAYGuj57d0WjRo0UFhbmGidrqAnZ2dlKSUnRjBkzNHDgQBUVFWnJkiVasmSJJLm+v2zatGmKjY1VdHS0cnNzFRERoT59+phbPOqVxx57TNOnT1dkZKRat26tr7/+WnPnztWwYcMkkTWYzOxtA+uagoICIzIy0vDz8zOSkpKMnTt3ml0S6jlJ17299dZbrjXnz583nn/+eaNJkyZGQECA0bdvX+PEiRPmFY0G4+rtzw2DrKHmrF271mjTpo1htVqNuLg4Y8mSJW7zTqfTyM3NNex2u2G1Wo3U1FSjtLTUpGpRX1VWVhqZmZlGZGSk4e/vb8TExBgvv/yyceHCBdcasgazWAzjqq+GBgAAAAD8JT4jBQAAAAAeopECAAAAAA/RSAEAAACAh2ikAAAAAMBDNFIAAAAA4CEaKQAAAADwEI0UAAAAAHiIRgoAAAAAPEQjBQD4V+jevbuysrLq3WMDAOomH7MLAACgvlu1apV8fX3NLgMAUItopAAA+JtCQ0PNLgEAUMu4tA8AUOc4nU7l5+crOjpaNptN7du314cffihJ2rJliywWiz777DPFx8fLZrPpwQcfVEVFhdavX6+WLVsqKChITz75pKqqqtwe9/Llyxo9erSCg4PVtGlT5ebmyjCM/6um119/XbGxsfL395fdbteAAQNcc1df2nelvj/fnn76adf61atXKyEhQf7+/oqJidGUKVN0+fLlv/eiAQBqFWekAAB1Tn5+vpYuXarFixcrNjZWX375pQYPHqxmzZq51kyePFkLFixQQECABg4cqIEDB8pqtWrZsmU6e/as+vbtq4KCAo0bN851zDvvvKPhw4erqKhIu3fv1siRIxUZGalnnnnmpvXs3r1bY8aM0bvvvquUlBT99ttv2rp163XXpqSk6MSJE677Bw8e1COPPKKuXbtKkrZu3aohQ4Zo/vz5uv/++3X48GGNHDlSkpSXl3fLrxkAoHZZjP/3X3EAANSCCxcuKDQ0VBs3blRycrJrfMSIEaqqqtLIkSP1wAMPaOPGjUpNTZUkzZw5UxMmTNDhw4cVExMjSRo1apR++uknffrpp5L+e9aooqJC3377rSwWiyRp/PjxWrNmjQ4cOHDTmlatWqWhQ4fq559/VmBg4DXz3bt317333qt58+a5jf/6669KSkpSr169tHDhQklSWlqaUlNTNWHCBNe6pUuXauzYsTp+/LiHrxYAwCyckQIA1Ck//PCDqqqq1KNHD7fxixcvKj4+3nW/Xbt2rp/tdrsCAgJcTdSVsaKiIrfH6NSpk6uJkqTk5GTNmTNHDodD3t7eN6ypR48eioqKUkxMjHr16qVevXqpb9++CggIuOExly5dUv/+/RUVFaXXXnvNNb5v3z5t375d06dPd405HA5VV1erqqrqpo8JAKg7aKQAAHXK2bNnJUkff/yxbrvtNrc5q9Wqw4cPS5LbLnkWi+WaXfMsFoucTmeN1BQYGKiSkhJt2bJFn3/+uSZNmqTJkyeruLhYISEh1z3mueee09GjR1VUVCQfn/+93Z49e1ZTpkxRv379rjnG39+/RuoFAPzzaKQAAHVKq1atZLVaVVZWpm7dul0zf6WRuhW7du1yu79z507Fxsbe9GzUFT4+PkpLS1NaWpry8vIUEhKizZs3X7chmjt3rlasWKGvvvpKYWFhbnMJCQkqLS3V3XfffcvPAwBgPhopAECdEhgYqBdffFHZ2dlyOp3q0qWLTp8+re3btysoKEhRUVG3/NhlZWXKycnRs88+q5KSEhUUFGjOnDl/edy6det05MgRde3aVU2aNNEnn3wip9Ope+6555q1Gzdu1NixY7Vw4UI1bdpU5eXlkiSbzabg4GBNmjRJjz76qCIjIzVgwAB5eXlp37592r9/v6ZNm3bLzw0AULtopAAAdc7UqVPVrFkz5efn68iRIwoJCVFCQoImTpz4ty7XGzJkiM6fP6+kpCR5e3srMzPTtWPezYSEhGjVqlWaPHmyqqurFRsbq+XLl6t169bXrN22bZscDodGjRqlUaNGucbT09P19ttvq2fPnlq3bp1eeeUVzZo1S76+voqLi9OIESNu+XkBAGofu/YBAAAAgIf4Ql4AAAAA8BCNFADgX2/r1q1q3LjxDW8AAPwZl/YBAP71zp8/r2PHjt1wnh32AAB/RiMFAAAAAB7i0j4AAAAA8BCNFAAAAAB4iEYKAAAAADxEIwUAAAAAHqKRAgAAAAAP0UgBAAAAgIdopAAAAADAQ/8Bh9wLd2HKtcoAAAAASUVORK5CYII=\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "code", + "source": [ + "fig, ax = plt.subplots(figsize=(10, 10))\n", + "plt.imshow(tmp_layer.pe[0, :, 50:], aspect=\"auto\")\n", + "plt.xlabel(\"emb_size\")\n", + "plt.ylabel(\"max_seq_len\")\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 853 + }, + "id": "ovX65ZQUcVSJ", + "outputId": "cae6d154-c719-4272-d1de-19de4b4f43d9" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1IAAANECAYAAAC6qR6BAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA4kUlEQVR4nO3de7hXdZ33/9cXkQ3aBtwopwRPmdYoKJpEoWFgAnNbJqWpJRqKzgWa7Ltb457yNM1AOpajot5d02jdpZaNWWOJoSaoIQpIHu6GFCl05NBIgmBy2vv3h5f7146DfLabfYDH47q+18V3rfVd6725Vuqz9V1rV+rr6+sDAADAduvQ2gMAAAC0N0IKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoNBOE1LTpk3L/vvvn86dO2fw4MF54oknWnskAABgJ1Wpr6+vb+0h3q0f/vCHOeuss3LLLbdk8ODBue6663LXXXdl4cKF6dmz5zt+vq6uLq+88kqqq6tTqVRaYGIAAKAtqq+vz+uvv56+ffumQ4etX3faKUJq8ODB+dCHPpQbb7wxyVth1K9fv1x44YX5yle+8o6ff/nll9OvX78dPSYAANBOvPTSS9l33323ur5jC86yQ6xfvz7z5s3L5MmTG5Z16NAhI0aMyOzZs7f4mXXr1mXdunUN799uyaEZnY7ZvXiGn/zumeLP/KVPv//wJn/WsR3bsR3bsR3bsR3bsR3bsZvv2KvX1GW/Qb9PdXX1Nrdr9yH13//939m0aVN69erVaHmvXr3yn//5n1v8zJQpU3LllVdutrxjdk/HSnlIda1+d7eaNeWYju3Yju3Yju3Yju3Yju3Yjr1jjp3kHW/52WkeNlFi8uTJWbVqVcPrpZdeau2RAACAdqTdX5Hae++9s9tuu2X58uWNli9fvjy9e/fe4meqqqpSVVXVEuMBAAA7oXZ/RapTp0456qij8uCDDzYsq6ury4MPPpghQ4a04mQAAMDOqt1fkUqS2trajB07NkcffXSOOeaYXHfddVm7dm3OOeec1h4NAADYCe0UIXXaaaflj3/8Yy677LIsW7YsRxxxRKZPn77ZAygAAACaw04RUkkyceLETJw4sbXHAAAAdgHt/h4pAACAliakAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKNTmQ2rKlCn50Ic+lOrq6vTs2TMnn3xyFi5c2GibYcOGpVKpNHpdcMEFrTQxAACws2vzITVz5sxMmDAhjz/+eGbMmJENGzbkE5/4RNauXdtou/POOy9Lly5teF199dWtNDEAALCz69jaA7yT6dOnN3p/2223pWfPnpk3b16OO+64huV77LFHevfu3dLjAQAAu6A2f0Xqr61atSpJUlNT02j5D37wg+y999457LDDMnny5Lzxxhtb3ce6deuyevXqRi8AAIDt1eavSP2lurq6XHzxxfnoRz+aww47rGH5GWeckf322y99+/bN008/nUsvvTQLFy7M3XffvcX9TJkyJVdeeWVLjQ0AAOxk2lVITZgwIc8++2weffTRRsvHjx/f8OfDDz88ffr0yfDhw7No0aIcdNBBm+1n8uTJqa2tbXi/evXq9OvXb8cNDgAA7FTaTUhNnDgx9957b2bNmpV99913m9sOHjw4SfLCCy9sMaSqqqpSVVW1Q+YEAAB2fm0+pOrr63PhhRfmJz/5SR5++OEccMAB7/iZBQsWJEn69Omzg6cDAAB2RW0+pCZMmJDbb789P/3pT1NdXZ1ly5YlSbp165YuXbpk0aJFuf322zN69Oj06NEjTz/9dCZNmpTjjjsuAwYMaOXpAQCAnVGbD6mbb745yVu/dPcv3XrrrTn77LPTqVOnPPDAA7nuuuuydu3a9OvXL2PGjMlXv/rVVpgWAADYFbT5kKqvr9/m+n79+mXmzJktNA0AAEA7/D1SAAAArU1IAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUKhjaw/QplQqb70AAAC2wRUpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACrX5kLriiitSqVQavQ499NCG9W+++WYmTJiQHj165D3veU/GjBmT5cuXt+LEAADAzq7Nh1SS/M3f/E2WLl3a8Hr00Ucb1k2aNCn/8R//kbvuuiszZ87MK6+8klNOOaUVpwUAAHZ2HVt7gO3RsWPH9O7de7Plq1atyne+853cfvvt+fjHP54kufXWW/OBD3wgjz/+eD784Q9vcX/r1q3LunXrGt6vXr16xwwOAADslNrFFannn38+ffv2zYEHHpgzzzwzS5YsSZLMmzcvGzZsyIgRIxq2PfTQQ9O/f//Mnj17q/ubMmVKunXr1vDq16/fDv8ZAACAnUebD6nBgwfntttuy/Tp03PzzTdn8eLFOfbYY/P6669n2bJl6dSpU7p3797oM7169cqyZcu2us/Jkydn1apVDa+XXnppB/8UAADAzqTNf7Vv1KhRDX8eMGBABg8enP322y8/+tGP0qVLlybts6qqKlVVVc01IgAAsItp81ek/lr37t3z/ve/Py+88EJ69+6d9evX57XXXmu0zfLly7d4TxUAAEBzaHchtWbNmixatCh9+vTJUUcdld133z0PPvhgw/qFCxdmyZIlGTJkSCtOCQAA7Mza/Ff7vvzlL+ekk07Kfvvtl1deeSWXX355dtttt5x++unp1q1bxo0bl9ra2tTU1KRr16658MILM2TIkK0+sQ8AAODdavMh9fLLL+f000/Pq6++mn322SdDhw7N448/nn322SdJ8q1vfSsdOnTImDFjsm7dupx44om56aabWnlqAABgZ9bmQ+rOO+/c5vrOnTtn2rRpmTZtWgtNBAAA7Ora3T1SAAAArU1IAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQqGNrD0CSSqW1JwAAAAq4IgUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhdp8SO2///6pVCqbvSZMmJAkGTZs2GbrLrjgglaeGgAA2Jl1bO0B3smTTz6ZTZs2Nbx/9tlnc8IJJ+Szn/1sw7LzzjsvV111VcP7PfbYo0VnBAAAdi1tPqT22WefRu+nTp2agw46KB/72Mcalu2xxx7p3bv3du9z3bp1WbduXcP71atXv/tBAQCAXUab/2rfX1q/fn2+//3v54tf/GIqlUrD8h/84AfZe++9c9hhh2Xy5Ml54403trmfKVOmpFu3bg2vfv367ejRAQCAnUibvyL1l+6555689tprOfvssxuWnXHGGdlvv/3St2/fPP3007n00kuzcOHC3H333Vvdz+TJk1NbW9vwfvXq1WIKAADYbu0qpL7zne9k1KhR6du3b8Oy8ePHN/z58MMPT58+fTJ8+PAsWrQoBx100Bb3U1VVlaqqqh0+LwAAsHNqN1/t+8Mf/pAHHngg55577ja3Gzx4cJLkhRdeaImxAACAXVC7Calbb701PXv2zN/+7d9uc7sFCxYkSfr06dMCUwEAALuidvHVvrq6utx6660ZO3ZsOnb8/0detGhRbr/99owePTo9evTI008/nUmTJuW4447LgAEDWnFiAABgZ9YuQuqBBx7IkiVL8sUvfrHR8k6dOuWBBx7Iddddl7Vr16Zfv34ZM2ZMvvrVr7bSpAAAwK6gXYTUJz7xidTX12+2vF+/fpk5c2YrTAQAAOzK2s09UgAAAG2FkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKNQuHn/eYiod3noBAABsg2oAAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQh1bewBaWaXS2hMAAEC744oUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFOjblQ2vXrs3UqVPz4IMPZsWKFamrq2u0/sUXX2yW4QAAANqiJoXUueeem5kzZ+YLX/hC+vTpk0ql0txzAQAAtFlNCqn77rsvP//5z/PRj360uecBAABo85p0j9Ree+2Vmpqa5p4FAACgXWhSSP3DP/xDLrvssrzxxhvNPQ8AAECb16Sv9l177bVZtGhRevXqlf333z+77757o/Xz589vluEAAADaoiaF1Mknn9zMYwAAALQfTQqpyy+/vLnnAAAAaDea/At5X3vttfzrv/5rJk+enJUrVyZ56yt9//Vf/9VswwEAALRFTboi9fTTT2fEiBHp1q1bfv/73+e8885LTU1N7r777ixZsiTf+973mntOAACANqNJV6Rqa2tz9tln5/nnn0/nzp0blo8ePTqzZs1qtuEAAADaoiaF1JNPPpnzzz9/s+Xvfe97s2zZsnc9FAAAQFvWpJCqqqrK6tWrN1v+u9/9Lvvss8+7HgoAAKAta1JIffKTn8xVV12VDRs2JEkqlUqWLFmSSy+9NGPGjGnWAQEAANqaJoXUtddemzVr1qRnz57585//nI997GN53/vel+rq6vzjP/5jc88IAADQpjTpqX3dunXLjBkz8uijj+bpp5/OmjVrMmjQoIwYMaK55wMAAGhzmhRSbxs6dGiGDh3aXLMAAAC0C9sdUtdff/127/Siiy5q0jAAAADtwXaH1Le+9a3t2q5SqQgpAABgp7bdIbV48eIdOQcAAEC70aSn9m2vrl275sUXX9yRhwAAAGhxOzSk6uvrd+TuAQAAWsUODSkAAICdkZACAAAoJKQAAAAK7dCQqlQqO3L3AAAArcLDJgAAAArt0JC677778t73vndHHgIAAKDFbfcv5P1LtbW1273t0KFDm3IIAACANqtJIfXUU0/lqaeeyoYNG3LIIYckSX73u99lt912y6BBgxq2c48UAACwM2pSSJ100kmprq7Od7/73ey1115Jkj/96U8555xzcuyxx+Z//s//2axDAgAAtCVNukfq2muvzZQpUxoiKkn22muvfP3rX8+1117bbMMBAAC0RU0KqdWrV+ePf/zjZsv/+Mc/5vXXX3/XQwEAALRlTQqpT3/60znnnHNy99135+WXX87LL7+cf//3f8+4ceNyyimnNPeMAAAAbUqT7pG65ZZb8uUvfzlnnHFGNmzY8NaOOnbMuHHjcs011zTrgAAAAG1Nk0Jqjz32yE033ZRrrrkmixYtSpIcdNBB2XPPPZt1OAAAgLboXf1C3qVLl2bp0qU5+OCDs+eee6a+vr655gIAAGizmhRSr776aoYPH573v//9GT16dJYuXZokGTdunEefAwAAO70mhdSkSZOy++67Z8mSJdljjz0alp922mmZPn36du9n1qxZOemkk9K3b99UKpXcc889jdbX19fnsssuS58+fdKlS5eMGDEizz//fKNtVq5cmTPPPDNdu3ZN9+7dM27cuKxZs6YpPxYAAMB2aVJI/fKXv8w3vvGN7Lvvvo2WH3zwwfnDH/6w3ftZu3ZtBg4cmGnTpm1x/dVXX53rr78+t9xyS+bMmZM999wzJ554Yt58882Gbc4888w899xzmTFjRu69997MmjUr48ePb8qPBQAAsF2a9LCJtWvXNroS9baVK1emqqpqu/czatSojBo1aovr6uvrc9111+WrX/1qPvWpTyVJvve976VXr16555578rnPfS6//e1vM3369Dz55JM5+uijkyQ33HBDRo8enX/+539O3759m/DTAQAAbFuTrkgde+yx+d73vtfwvlKppK6uLldffXWOP/74Zhls8eLFWbZsWUaMGNGwrFu3bhk8eHBmz56dJJk9e3a6d+/eEFFJMmLEiHTo0CFz5szZ6r7XrVuX1atXN3oBAABsryZdkbr66qszfPjwzJ07N+vXr88ll1yS5557LitXrsxjjz3WLIMtW7YsSdKrV69Gy3v16tWwbtmyZenZs2ej9R07dkxNTU3DNlsyZcqUXHnllc0yJwAAsOtp0hWpww47LL/73e8ydOjQfOpTn8ratWtzyimn5KmnnspBBx3U3DM2u8mTJ2fVqlUNr5deeqm1RwIAANqRJl2RSt76mt3f//3fN+csjfTu3TtJsnz58vTp06dh+fLly3PEEUc0bLNixYpGn9u4cWNWrlzZ8PktqaqqKrqXCwAA4C816YrU9OnT8+ijjza8nzZtWo444oicccYZ+dOf/tQsgx1wwAHp3bt3HnzwwYZlq1evzpw5czJkyJAkyZAhQ/Laa69l3rx5Dds89NBDqaury+DBg5tlDgAAgL/WpJD6X//rfzU8oOGZZ55JbW1tRo8encWLF6e2tna797NmzZosWLAgCxYsSPLWAyYWLFiQJUuWpFKp5OKLL87Xv/71/OxnP8szzzyTs846K3379s3JJ5+cJPnABz6QkSNH5rzzzssTTzyRxx57LBMnTsznPvc5T+wDAAB2mCZ9tW/x4sX54Ac/mCT593//95x00kn5p3/6p8yfPz+jR4/e7v3MnTu30VP+3o6wsWPH5rbbbssll1yStWvXZvz48XnttdcydOjQTJ8+PZ07d274zA9+8INMnDgxw4cPT4cOHTJmzJhcf/31TfmxAAAAtkuTQqpTp0554403kiQPPPBAzjrrrCRJTU1N0aPEhw0blvr6+q2ur1Qqueqqq3LVVVdtdZuamprcfvvt231MAACAd6tJITV06NDU1tbmox/9aJ544on88Ic/TJL87ne/y7777tusAwIAALQ1TbpH6sYbb0zHjh3z4x//ODfffHPe+973Jknuu+++jBw5slkHBAAAaGuadEWqf//+uffeezdb/q1vfavR+6lTp+aCCy5I9+7dmzQcAABAW9SkK1Lb65/+6Z+ycuXKHXkIAACAFrdDQ2pbD5IAAABor3ZoSAEAAOyMhBQAAEAhIQUAAFBISAEAABTaoSF17LHHpkuXLjvyEAAAAC2uSSF12223bXH5xo0bM3ny5Ib3v/jFL9KnT58mDQYAANBWNSmkLrroonz2s5/Nn/70p4ZlCxcuzODBg3PHHXc023AAAABtUZNC6qmnnsrLL7+cww8/PDNmzMi0adMyaNCgHHroofnNb37T3DMCAAC0KR2b8qGDDjoojz32WC6++OKMHDkyu+22W7773e/m9NNPb+75AAAA2pwmP2zi5z//ee68884MGTIk3bt3z3e+85288sorzTkbAABAm9SkkDr//PPz2c9+NpdeemkeeeSRPP300+nUqVMOP/zw/OhHP2ruGQEAANqUJn2177HHHsucOXMycODAJEnv3r3zi1/8ItOmTcsXv/jFnHrqqc06JAAAQFvSpJCaN29eqqqqNls+YcKEjBgx4l0PBQAA0JY16at9W4qotx1yyCFNHgYAAKA9aNIVqST58Y9/nB/96EdZsmRJ1q9f32jd/Pnz3/VgAAAAbVWTrkhdf/31Oeecc9KrV6889dRTOeaYY9KjR4+8+OKLGTVqVHPPCAAA0KY0KaRuuummfPvb384NN9yQTp065ZJLLsmMGTNy0UUXZdWqVc09IwAAQJvSpJBasmRJPvKRjyRJunTpktdffz1J8oUvfCF33HFH800HAADQBjUppHr37p2VK1cmSfr375/HH388SbJ48eLU19c333QAAABtUJNC6uMf/3h+9rOfJUnOOeecTJo0KSeccEJOO+20fPrTn27WAQEAANqaJj2179vf/nbq6uqSvPW7o/bee+889thj+eQnP5kLLrigWQcEAABoa5oUUh06dMj69eszf/78rFixIl26dGn4RbzTp0/PSSed1KxDAgAAtCVNCqnp06fnC1/4Ql599dXN1lUqlWzatOldDwYAANBWNekeqQsvvDCnnnpqli5dmrq6ukYvEQUAAOzsmhRSy5cvT21tbXr16tXc8wAAALR5TQqpz3zmM3n44YebeRQAAID2oUn3SN1444357Gc/m0ceeSSHH354dt9990brL7roomYZDgAAoC1qUkjdcccd+eUvf5nOnTvn4YcfTqVSaVhXqVSEFAAAsFNrUkj9/d//fa688sp85StfSYcOTfp2IAAAQLvVpApav359TjvtNBEFAADskppUQmPHjs0Pf/jD5p4FAACgXWjSV/s2bdqUq6++Ovfff38GDBiw2cMmvvnNbzbLcAAAAG1Rk0LqmWeeyZFHHpkkefbZZxut+8sHTwAAAOyMmhRSv/rVr5p7DgAAgHbD0yIAAAAKCSkAAIBCTfpqH82somcBAKA98V/wAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEChjq09QFtS6VBJpVJp7TF2Hf6uAQBop1yRAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoFCrhtSsWbNy0kknpW/fvqlUKrnnnnsa1m3YsCGXXnppDj/88Oy5557p27dvzjrrrLzyyiuN9rH//vunUqk0ek2dOrWFfxIAAGBX0qohtXbt2gwcODDTpk3bbN0bb7yR+fPn52tf+1rmz5+fu+++OwsXLswnP/nJzba96qqrsnTp0obXhRde2BLjAwAAu6iOrXnwUaNGZdSoUVtc161bt8yYMaPRshtvvDHHHHNMlixZkv79+zcsr66uTu/evXforAAAAG9rV/dIrVq1KpVKJd27d2+0fOrUqenRo0eOPPLIXHPNNdm4ceM297Nu3bqsXr260QsAAGB7teoVqRJvvvlmLr300px++unp2rVrw/KLLroogwYNSk1NTX79619n8uTJWbp0ab75zW9udV9TpkzJlVde2RJjAwAAO6F2EVIbNmzIqaeemvr6+tx8882N1tXW1jb8ecCAAenUqVPOP//8TJkyJVVVVVvc3+TJkxt9bvXq1enXr9+OGR4AANjptPmQejui/vCHP+Shhx5qdDVqSwYPHpyNGzfm97//fQ455JAtblNVVbXVyAIAAHgnbTqk3o6o559/Pr/61a/So0ePd/zMggUL0qFDh/Ts2bMFJgQAAHZFrRpSa9asyQsvvNDwfvHixVmwYEFqamrSp0+ffOYzn8n8+fNz7733ZtOmTVm2bFmSpKamJp06dcrs2bMzZ86cHH/88amurs7s2bMzadKkfP7zn89ee+3VWj8WAACwk2vVkJo7d26OP/74hvdv37c0duzYXHHFFfnZz36WJDniiCMafe5Xv/pVhg0blqqqqtx555254oorsm7duhxwwAGZNGlSo/ufAAAAmlurhtSwYcNSX1+/1fXbWpckgwYNyuOPP97cYwEAAGxTu/o9UgAAAG2BkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKBQq4bUrFmzctJJJ6Vv376pVCq55557Gq0/++yzU6lUGr1GjhzZaJuVK1fmzDPPTNeuXdO9e/eMGzcua9asacGfAgAA2NW0akitXbs2AwcOzLRp07a6zciRI7N06dKG1x133NFo/ZlnnpnnnnsuM2bMyL333ptZs2Zl/PjxO3p0AABgF9axNQ8+atSojBo1apvbVFVVpXfv3ltc99vf/jbTp0/Pk08+maOPPjpJcsMNN2T06NH553/+5/Tt27fZZwYAAGjz90g9/PDD6dmzZw455JD83d/9XV599dWGdbNnz0737t0bIipJRowYkQ4dOmTOnDlb3ee6deuyevXqRi8AAIDt1aZDauTIkfne976XBx98MN/4xjcyc+bMjBo1Kps2bUqSLFu2LD179mz0mY4dO6ampibLli3b6n6nTJmSbt26Nbz69eu3Q38OAABg59KqX+17J5/73Oca/nz44YdnwIABOeigg/Lwww9n+PDhTd7v5MmTU1tb2/B+9erVYgoAANhubfqK1F878MADs/fee+eFF15IkvTu3TsrVqxotM3GjRuzcuXKrd5Xlbx131XXrl0bvQAAALZXuwqpl19+Oa+++mr69OmTJBkyZEhee+21zJs3r2Gbhx56KHV1dRk8eHBrjQkAAOzkWvWrfWvWrGm4upQkixcvzoIFC1JTU5OamppceeWVGTNmTHr37p1Fixblkksuyfve976ceOKJSZIPfOADGTlyZM4777zccsst2bBhQyZOnJjPfe5zntgHAADsMK16RWru3Lk58sgjc+SRRyZJamtrc+SRR+ayyy7Lbrvtlqeffjqf/OQn8/73vz/jxo3LUUcdlUceeSRVVVUN+/jBD36QQw89NMOHD8/o0aMzdOjQfPvb326tHwkAANgFtOoVqWHDhqW+vn6r6++///533EdNTU1uv/325hwLAABgm9rVPVIAAABtgZACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACjUsbUHaFMqHd56AQAAbINqAAAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoJCQAgAAKCSkAAAACnVs7QFoZRUtDQAApfxXNAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFCoY2sPQFLpUGntEQAAgAKuSAEAABQSUgAAAIWEFAAAQCH3SLFrqrgvDQCApnNFCgAAoJCQAgAAKCSkAAAACgkpAACAQkIKAACgkJACAAAoJKQAAAAKCSkAAIBCQgoAAKCQkAIAACgkpAAAAAoJKQAAgEJCCgAAoFCrhtSsWbNy0kknpW/fvqlUKrnnnnsara9UKlt8XXPNNQ3b7L///putnzp1agv/JAAAwK6kVUNq7dq1GThwYKZNm7bF9UuXLm30+rd/+7dUKpWMGTOm0XZXXXVVo+0uvPDClhgfAADYRXVszYOPGjUqo0aN2ur63r17N3r/05/+NMcff3wOPPDARsurq6s323Zb1q1bl3Xr1jW8X7169XZ/FgAAoN3cI7V8+fL8/Oc/z7hx4zZbN3Xq1PTo0SNHHnlkrrnmmmzcuHGb+5oyZUq6devW8OrXr9+OGhsAANgJteoVqRLf/e53U11dnVNOOaXR8osuuiiDBg1KTU1Nfv3rX2fy5MlZunRpvvnNb251X5MnT05tbW3D+9WrV4spAABgu7WbkPq3f/u3nHnmmencuXOj5X8ZRAMGDEinTp1y/vnnZ8qUKamqqtrivqqqqra6DgAA4J20i6/2PfLII1m4cGHOPffcd9x28ODB2bhxY37/+9/v+MEAAIBdUrsIqe985zs56qijMnDgwHfcdsGCBenQoUN69uzZApMBAAC7olb9at+aNWvywgsvNLxfvHhxFixYkJqamvTv3z/JW/cv3XXXXbn22ms3+/zs2bMzZ86cHH/88amurs7s2bMzadKkfP7zn89ee+3VYj8HAACwa2nVkJo7d26OP/74hvdv3+80duzY3HbbbUmSO++8M/X19Tn99NM3+3xVVVXuvPPOXHHFFVm3bl0OOOCATJo0qdF9UwAAAM2tVUNq2LBhqa+v3+Y248ePz/jx47e4btCgQXn88cd3xGgAAABb1S7ukQIAAGhLhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAU6tjaA7QpHSpJpdLaUwAAAG2cK1IAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIWEFAAAQCEhBQAAUEhIAQAAFBJSAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFOrb2AG1BfX19kmRj/YYmfX7163Xv6vhNPa5jO7ZjO7ZjO7ZjO7ZjO7ZjN++xV69567NvN8LWVOrfaYtdwMsvv5x+/fq19hgAAEAb8dJLL2Xffffd6nohlaSuri6vvPJKqqurU6lUNlu/evXq9OvXLy+99FK6du3aChOyq3Cu0VKca7QU5xotxblGc6mvr8/rr7+evn37pkOHrd8J5at9STp06LDN2nxb165d/Q+TFuFco6U412gpzjVainON5tCtW7d33MbDJgAAAAoJKQAAgEJCajtUVVXl8ssvT1VVVWuPwk7OuUZLca7RUpxrtBTnGi3NwyYAAAAKuSIFAABQSEgBAAAUElIAAACFhBQAAEAhIfUOpk2blv333z+dO3fO4MGD88QTT7T2SLRzs2bNykknnZS+ffumUqnknnvuabS+vr4+l112Wfr06ZMuXbpkxIgRef7551tnWNq1KVOm5EMf+lCqq6vTs2fPnHzyyVm4cGGjbd58881MmDAhPXr0yHve856MGTMmy5cvb6WJaa9uvvnmDBgwoOEXoQ4ZMiT33Xdfw3rnGTvC1KlTU6lUcvHFFzcsc67RkoTUNvzwhz9MbW1tLr/88syfPz8DBw7MiSeemBUrVrT2aLRja9euzcCBAzNt2rQtrr/66qtz/fXX55ZbbsmcOXOy55575sQTT8ybb77ZwpPS3s2cOTMTJkzI448/nhkzZmTDhg35xCc+kbVr1zZsM2nSpPzHf/xH7rrrrsycOTOvvPJKTjnllFacmvZo3333zdSpUzNv3rzMnTs3H//4x/OpT30qzz33XBLnGc3vySefzP/5P/8nAwYMaLTcuUaLqmerjjnmmPoJEyY0vN+0aVN9375966dMmdKKU7EzSVL/k5/8pOF9XV1dfe/eveuvueaahmWvvfZafVVVVf0dd9zRChOyM1mxYkV9kvqZM2fW19e/dW7tvvvu9XfddVfDNr/97W/rk9TPnj27tcZkJ7HXXnvV/+u//qvzjGb3+uuv1x988MH1M2bMqP/Yxz5W/6Uvfam+vt4/02h5rkhtxfr16zNv3ryMGDGiYVmHDh0yYsSIzJ49uxUnY2e2ePHiLFu2rNF5161btwwePNh5x7u2atWqJElNTU2SZN68edmwYUOj8+3QQw9N//79nW802aZNm3LnnXdm7dq1GTJkiPOMZjdhwoT87d/+baNzKvHPNFpex9YeoK367//+72zatCm9evVqtLxXr175z//8z1aaip3dsmXLkmSL593b66Ap6urqcvHFF+ejH/1oDjvssCRvnW+dOnVK9+7dG23rfKMpnnnmmQwZMiRvvvlm3vOe9+QnP/lJPvjBD2bBggXOM5rNnXfemfnz5+fJJ5/cbJ1/ptHShBTALmDChAl59tln8+ijj7b2KOykDjnkkCxYsCCrVq3Kj3/844wdOzYzZ85s7bHYibz00kv50pe+lBkzZqRz586tPQ542MTW7L333tltt902e9LL8uXL07t371aaip3d2+eW847mNHHixNx777351a9+lX333bdhee/evbN+/fq89tprjbZ3vtEUnTp1yvve974cddRRmTJlSgYOHJh/+Zd/cZ7RbObNm5cVK1Zk0KBB6dixYzp27JiZM2fm+uuvT8eOHdOrVy/nGi1KSG1Fp06dctRRR+XBBx9sWFZXV5cHH3wwQ4YMacXJ2JkdcMAB6d27d6PzbvXq1ZkzZ47zjmL19fWZOHFifvKTn+Shhx7KAQcc0Gj9UUcdld13373R+bZw4cIsWbLE+ca7VldXl3Xr1jnPaDbDhw/PM888kwULFjS8jj766Jx55pkNf3au0ZJ8tW8bamtrM3bs2Bx99NE55phjct1112Xt2rU555xzWns02rE1a9bkhRdeaHi/ePHiLFiwIDU1Nenfv38uvvjifP3rX8/BBx+cAw44IF/72tfSt2/fnHzyya03NO3ShAkTcvvtt+enP/1pqqurG+4R6NatW7p06ZJu3bpl3Lhxqa2tTU1NTbp27ZoLL7wwQ4YMyYc//OFWnp72ZPLkyRk1alT69++f119/Pbfffnsefvjh3H///c4zmk11dXXDPZ5v23PPPdOjR4+G5c41WpKQ2obTTjstf/zjH3PZZZdl2bJlOeKIIzJ9+vTNHgQAJebOnZvjjz++4X1tbW2SZOzYsbnttttyySWXZO3atRk/fnxee+21DB06NNOnT/d9cIrdfPPNSZJhw4Y1Wn7rrbfm7LPPTpJ861vfSocOHTJmzJisW7cuJ554Ym666aYWnpT2bsWKFTnrrLOydOnSdOvWLQMGDMj999+fE044IYnzjJbjXKMlVerr6+tbewgAAID2xD1SAAAAhYQUAABAISEFAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgDsEoYNG5aLL7643e0bgLapY2sPAADt3d13353dd9+9tccAoAUJKQB4l2pqalp7BABamK/2AdDm1NXVZcqUKTnggAPSpUuXDBw4MD/+8Y+TJA8//HAqlUruv//+HHnkkenSpUs+/vGPZ8WKFbnvvvvygQ98IF27ds0ZZ5yRN954o9F+N27cmIkTJ6Zbt27Ze++987WvfS319fXbNdNNN92Ugw8+OJ07d06vXr3ymc98pmHdX3617+35/vp19tlnN2z/05/+NIMGDUrnzp1z4IEH5sorr8zGjRvf3V8aAC3KFSkA2pwpU6bk+9//fm655ZYcfPDBmTVrVj7/+c9nn332adjmiiuuyI033pg99tgjp556ak499dRUVVXl9ttvz5o1a/LpT386N9xwQy699NKGz3z3u9/NuHHj8sQTT2Tu3LkZP358+vfvn/POO2+b88ydOzcXXXRR/u///b/5yEc+kpUrV+aRRx7Z4rYf+chHsnTp0ob3v/3tbzN69Ogcd9xxSZJHHnkkZ511Vq6//voce+yxWbRoUcaPH58kufzyy5v8dwZAy6rUb+//FQcALWDdunWpqanJAw88kCFDhjQsP/fcc/PGG29k/PjxOf744/PAAw9k+PDhSZKpU6dm8uTJWbRoUQ488MAkyQUXXJDf//73mT59epK3rhqtWLEizz33XCqVSpLkK1/5Sn72s5/l//2//7fNme6+++6cc845efnll1NdXb3Z+mHDhuWII47Idddd12j5q6++mmOOOSYjR47MtGnTkiQjRozI8OHDM3ny5Ibtvv/97+eSSy7JK6+8Uvi3BUBrcUUKgDblhRdeyBtvvJETTjih0fL169fnyCOPbHg/YMCAhj/36tUre+yxR0NEvb3siSeeaLSPD3/4ww0RlSRDhgzJtddem02bNmW33Xbb6kwnnHBC9ttvvxx44IEZOXJkRo4cmU9/+tPZY489tvqZDRs2ZMyYMdlvv/3yL//yLw3Lf/Ob3+Sxxx7LP/7jPzYs27RpU95888288cYb29wnAG2HkAKgTVmzZk2S5Oc//3ne+973NlpXVVWVRYsWJUmjp+RVKpXNnppXqVRSV1fXLDNVV1dn/vz5efjhh/PLX/4yl112Wa644oo8+eST6d69+xY/83d/93d56aWX8sQTT6Rjx///X7dr1qzJlVdemVNOOWWzz3Tu3LlZ5gVgxxNSALQpH/zgB1NVVZUlS5bkYx/72Gbr3w6pppgzZ06j948//ngOPvjgbV6NelvHjh0zYsSIjBgxIpdffnm6d++ehx56aItB9M1vfjM/+tGP8utf/zo9evRotG7QoEFZuHBh3ve+9zX55wCg9QkpANqU6urqfPnLX86kSZNSV1eXoUOHZtWqVXnsscfStWvX7Lfffk3e95IlS1JbW5vzzz8/8+fPzw033JBrr732HT9377335sUXX8xxxx2XvfbaK7/4xS9SV1eXQw45ZLNtH3jggVxyySWZNm1a9t577yxbtixJ0qVLl3Tr1i2XXXZZ/sf/+B/p379/PvOZz6RDhw75zW9+k2effTZf//rXm/yzAdCyhBQAbc4//MM/ZJ999smUKVPy4osvpnv37hk0aFD+9//+3+/q63pnnXVW/vznP+eYY47Jbrvtli996UsNT8zblu7du+fuu+/OFVdckTfffDMHH3xw7rjjjvzN3/zNZts++uij2bRpUy644IJccMEFDcvHjh2b2267LSeeeGLuvffeXHXVVfnGN76R3XffPYceemjOPffcJv9cALQ8T+0DAAAo5BfyAgAAFBJSAOzyHnnkkbznPe/Z6gsA/pqv9gGwy/vzn/+c//qv/9rqek/YA+CvCSkAAIBCvtoHAABQSEgBAAAUElIAAACFhBQAAEAhIQUAAFBISAEAABQSUgAAAIX+Px+eCHVQDvPgAAAAAElFTkSuQmCC\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "code", + "source": [ + "fig, ax = plt.subplots(figsize=(10, 10))\n", + "plt.imshow(tmp_layer.pe[0, :, 50:51], aspect=\"auto\")\n", + "plt.xlabel(\"emb_size\")\n", + "plt.ylabel(\"max_seq_len\")\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 853 + }, + "id": "Emdh9j6acXH5", + "outputId": "516ee9f3-2db2-4fb0-a0f9-51054bd52c84" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1IAAANECAYAAAC6qR6BAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABA4klEQVR4nO3de5hVdb0/8M/aAwzYgcFRAVG8ZlrHK5YcDE2DEuhQJqapeYu89Hj5CaefyjnmrTqQl/KolE/nWNovr3XUfDQpwBAlxAuSlwoTMTQBS5LhkgPMrN8f6LRHQPd3y8yemf16nef7PO611l7z2T2rOu++6/P9Znme5wEAAEDJCpUuAAAAoLMRpAAAABIJUgAAAIkEKQAAgESCFAAAQCJBCgAAIJEgBQAAkEiQAgAASCRIAQAAJBKkAAAAEnWZIDVlypTYZZddomfPnjFkyJB47LHHKl0SAADQRWV5nueVLuL9uuOOO+Kkk06KG264IYYMGRLXXHNN/PSnP40FCxZEv3793vP7zc3N8eqrr0bv3r0jy7J2qBgAAOiI8jyPlStXxsCBA6NQ2Py8U5cIUkOGDImPfexjcf3110fEhmA0aNCgOOecc+LCCy98z++/8sorMWjQoLYuEwAA6CRefvnl2HHHHTd7vls71tIm1q5dG08++WRMnDix5VihUIgRI0bEnDlzNvmdxsbGaGxsbPn8dpYcFqOjW3Rv24IBAIAOa32si0fiF9G7d+93va7TB6m//vWv0dTUFP379291vH///vGHP/xhk9+ZNGlSXHbZZRsd7xbdo1smSAEAQNV6632992r56TKLTaSYOHFirFixomW8/PLLlS4JAADoRDr9jNS2224bNTU1sWzZslbHly1bFgMGDNjkd2pra6O2tnbjE1m2YQAAAFUqa5mVejedfkaqR48eceCBB8aMGTNajjU3N8eMGTNi6NChFawMAADoqjr9jFRExIQJE+Lkk0+Oj370o3HQQQfFNddcE6tXr45TTz210qUBAABdUJcIUscee2z85S9/iYsvvjiWLl0a+++/f0ydOnWjBSgAAAC2hC6xj9T71dDQEHV1dXFY4Sir9gEAQBVbn6+Lmc13xYoVK6JPnz6bva7T90gBAAC0N0EKAAAgkSAFAACQqEssNrGlZIXsPXcwBgAAuq4szyKa3/s6M1IAAACJBCkAAIBEXu0rVlMTkdVUugoAAKBS8uaI9e99mRkpAACARIIUAABAIkEKAAAgkR6pIlkWlj8HAIAqVmoaMCMFAACQSJACAABIJEgBAAAk0iNVrFCIyGRLAACoWnlpeUBqAAAASCRIAQAAJBKkAAAAEumRKlZTE5HVVLoKAACgUvLS8oAZKQAAgESCFAAAQCJBCgAAIJEeqSJZlkWWZZUuAwAAqJAsSssDZqQAAAASCVIAAACJBCkAAIBEeqSKFbIIPVIAAFC9cj1SAAAAbUKQAgAASOTVvmJZYcMAAACqVGl5QGoAAABIJEgBAAAkEqQAAAAS6ZEqVihsGAAAQHVq1iMFAADQJgQpAACARIIUAABAIj1SxewjBQAA1a3EPCA1AAAAJBKkAAAAEglSAAAAifRIFStkGwYAAFClSssDZqQAAAASCVIAAACJBCkAAIBEeqSKZIVCZAXZEgAAqlVW4lyT1AAAAJBIkAIAAEgkSAEAACTSI1WskEXokQIAgCpmHykAAIA2IUgBAAAkEqQAAAAS6ZEqlmUbBgAAUJ1KzANmpAAAABIJUgAAAIm82lfMq30AAFDdvNoHAADQNgQpAACARIIUAABAIj1SxWoKEQXZEgAAqlZWWh6QGgAAABIJUgAAAIkEKQAAgER6pIrZRwoAAKqbfaQAAADahiAFAACQSJACAABIpEeqmB4pAACobnqkAAAA2oYgBQAAkEiQAgAASKRHqkheKEReI1sCAEC1ykuca5IaAAAAEglSAAAAiQQpAACARHqkitlHCgAAqpt9pAAAANqGIAUAAJBIkAIAAEikR6qYHikAAKhueqQAAADahiAFAACQyKt9xQrZhgEAAFSn3Kt9AAAAbaLDB6lJkybFxz72sejdu3f069cvjjzyyFiwYEGraw477LDIsqzVOPPMMytUMQAA0NV1+CD10EMPxVlnnRWPPvpoTJs2LdatWxef/vSnY/Xq1a2uO+2002LJkiUt44orrqhQxQAAQFfX4Xukpk6d2urzTTfdFP369Ysnn3wyDj300JbjW221VQwYMOD9/bFCYcMAAACqU15aHuh0qWHFihUREVFfX9/q+C233BLbbrtt7L333jFx4sRYs2bNZu/R2NgYDQ0NrQYAAECpOvyMVLHm5uY477zz4uMf/3jsvffeLcePP/742HnnnWPgwIHx9NNPxwUXXBALFiyIu+66a5P3mTRpUlx22WXtVTYAANDFZHme55UuolRf/epX44EHHohHHnkkdtxxx81e9+CDD8bw4cPjhRdeiN13332j842NjdHY2NjyuaGhIQYNGhTDP/y16FZT2ya1AwAAHd/6psaY8furYsWKFdGnT5/NXtdpZqTOPvvsuO+++2LWrFnvGqIiIoYMGRIRsdkgVVtbG7W1GwemPMsiz+wjBQAA1arUPNDhg1Se53HOOefE3XffHTNnzoxdd931Pb8zf/78iIjYfvvt27g6AACgGnX4IHXWWWfFrbfeGj//+c+jd+/esXTp0oiIqKuri169esXChQvj1ltvjdGjR8c222wTTz/9dIwfPz4OPfTQ2HfffStcPQAA0BV1+CD1/e9/PyI2bLpb7Ec/+lGccsop0aNHj5g+fXpcc801sXr16hg0aFCMHTs2LrroogpUCwAAVIMOH6Teay2MQYMGxUMPPbRl/lghOuGC8AAAwBZT4lJ8YgMAAEAiQQoAACCRIAUAAJCow/dItatCYcMAAACqU15aHpAaAAAAEglSAAAAiQQpAACARHqkiuRZFnmWVboMAACgQkrNA2akAAAAEglSAAAAibzaV6wQoiUAAFSzvLTLxAYAAIBEghQAAEAiQQoAACCRHqliWURY/hwAAKpXiXHAjBQAAEAiQQoAACCRIAUAAJBIj1SRPMsiL+iRAgCAapU3l5YHzEgBAAAkEqQAAAASCVIAAACJ9EgVK2QbBgAAUJ1KzANmpAAAABIJUgAAAIkEKQAAgER6pIrkWRZ5pkcKAACqVal5wIwUAABAIkEKAAAgkSAFAACQSI9UMftIAQBAdbOPFAAAQNsQpAAAABIJUgAAAIn0SBXJsw0DAACoTqXmATNSAAAAiQQpAACARF7tK5ZlGwYAAFCdSswDZqQAAAASCVIAAACJBCkAAIBEeqSKFUK0BACAalZiHhAbAAAAEglSAAAAiQQpAACARHqkiuSFLPKCfaQAAKBalZoHzEgBAAAkEqQAAAASCVIAAACJ9EgVybMs8kyPFAAAVKtS84AZKQAAgESCFAAAQCJBCgAAIJEeqWLZWwMAAKhOJeYBM1IAAACJBCkAAIBEghQAAEAiPVJF8pos8hpNUgAAUK3y3D5SAAAAbUKQAgAASCRIAQAAJNIjVSTPNgwAAKA6lZoHzEgBAAAkEqQAAAASebWvlSwi824fAABUL8ufAwAAtAlBCgAAIJEgBQAAkEiPVBHLnwMAQHWz/DkAAEAbEaQAAAASCVIAAACJ9EgVK4RoCQAA1azEPCA2AAAAJBKkAAAAEglSAAAAifRIFcmzLPLMRlIAAFCtSs0DZqQAAAASCVIAAACJBCkAAIBEeqSKZW8NAACgOpWYB8xIAQAAJBKkAAAAEglSAAAAifRIFclrNgwAAKA65c2lXWdGCgAAIFGHD1KXXnppZFnWauy1114t5998880466yzYptttol/+qd/irFjx8ayZcsqWDEAANDVdYpX+/75n/85pk+f3vK5W7d/lD1+/Pi4//7746c//WnU1dXF2WefHUcddVTMnj07+e/kWRZ5Zv1zAACoVqXmgU4RpLp16xYDBgzY6PiKFSvixhtvjFtvvTU++clPRkTEj370o/jwhz8cjz76aPzLv/zLJu/X2NgYjY2NLZ8bGhrapnAAAKBL6vCv9kVE/PGPf4yBAwfGbrvtFieccEIsXrw4IiKefPLJWLduXYwYMaLl2r322it22mmnmDNnzmbvN2nSpKirq2sZgwYNavPfAAAAdB0dPkgNGTIkbrrpppg6dWp8//vfj0WLFsUhhxwSK1eujKVLl0aPHj2ib9++rb7Tv3//WLp06WbvOXHixFixYkXLePnll9v4VwAAAF1Jh3+1b9SoUS3/vO+++8aQIUNi5513jjvvvDN69epV1j1ra2ujtrZ24xPZWwMAAKhOJeaBDj8j9U59+/aND33oQ/HCCy/EgAEDYu3atfHGG2+0umbZsmWb7KkCAADYEjpdkFq1alUsXLgwtt9++zjwwAOje/fuMWPGjJbzCxYsiMWLF8fQoUMrWCUAANCVdfhX+772ta/FmDFjYuedd45XX301LrnkkqipqYnjjjsu6urqYty4cTFhwoSor6+PPn36xDnnnBNDhw7d7Ip9AAAA71eHD1KvvPJKHHfccfH666/HdtttF8OGDYtHH300tttuu4iI+O53vxuFQiHGjh0bjY2NccQRR8T3vve9sv5Wnm0YAABAdSo1D2R5nudtW0rH19DQEHV1dfGxz30junXvWelyAACAClm/7s14/OdfjxUrVkSfPn02e12n65ECAACoNEEKAAAgUYfvkWpPeWHDAAAAqlOpeUBsAAAASCRIAQAAJBKkAAAAEumRKpZlGwYAAFCdSswDZqQAAAASCVIAAACJBCkAAIBEeqSK5NmGAQAAVKdS84AZKQAAgESCFAAAQCJBCgAAIJEeqWKFiFy0BACA6lViHhAbAAAAEglSAAAAibzaVyx7awAAANXJ8ucAAABtQ5ACAABIJEgBAAAk0iNVJI+IXI8UAABUrbzE68xIAQAAJBKkAAAAEglSAAAAifRIFbOPFAAAVDf7SAEAALQNQQoAACCRIAUAAJBIj1SRvJBFXtAkBQAA1arUPGBGCgAAIJEgBQAAkEiQAgAASKRHqkiebRgAAEB1KjUPmJECAABIJEgBAAAkEqQAAAAS6ZEqlr01AACA6qRHCgAAoG0IUgAAAIkEKQAAgER6pIrkhQ0DAACoTqXmAbEBAAAgkSAFAACQyKt9RfJswwAAAKpTqXnAjBQAAEAiQQoAACCRIAUAAJBIj1Sx7K0BAABUJz1SAAAAbUOQAgAASCRIAQAAJNIjVUyPFAAAVDc9UgAAAG1DkAIAAEgkSAEAACTSI1UkL2wYAABAdSo1D4gNAAAAiQQpAACARIIUAABAIj1SRfJswwAAAKpTqXnAjBQAAEAiQQoAACCRIAUAAJBIj1Sx7K0BAABUJz1SAAAAbUOQAgAASOTVviJ5IYu84N0+AACoVqXmATNSAAAAiQQpAACARIIUAABAIj1SxSx/DgAA1c3y5wAAAG1DkAIAAEgkSAEAACTSI1Ukj4hcjxQAAFStvMTrzEgBAAAkEqQAAAASCVIAAACJ9EgVs48UAABUN/tIAQAAtA1BCgAAIJEgBQAAkEiPVJG8sGEAAADVqdQ8IDYAAAAk6vBBapdddoksyzYaZ511VkREHHbYYRudO/PMMytcNQAA0JV1+Ff7Hn/88Whqamr5/Oyzz8anPvWp+MIXvtBy7LTTTovLL7+85fNWW23VrjUCAADVpcMHqe22267V58mTJ8fuu+8en/jEJ1qObbXVVjFgwICS79nY2BiNjY0tnxsaGjb8g32kAACgunXFfaTWrl0bP/nJT+LLX/5yZNk/fuEtt9wS2267bey9994xceLEWLNmzbveZ9KkSVFXV9cyBg0a1NalAwAAXUiHn5Eqds8998Qbb7wRp5xySsux448/PnbeeecYOHBgPP3003HBBRfEggUL4q677trsfSZOnBgTJkxo+dzQ0CBMAQAAJetUQerGG2+MUaNGxcCBA1uOnX766S3/vM8++8T2228fw4cPj4ULF8buu+++yfvU1tZGbW1tm9cLAAB0TZ0mSP3pT3+K6dOnv+tMU0TEkCFDIiLihRde2GyQ2pw82zAAAIDqVGoe6DQ9Uj/60Y+iX79+8ZnPfOZdr5s/f35ERGy//fbtUBUAAFCNOsWMVHNzc/zoRz+Kk08+Obp1+0fJCxcujFtvvTVGjx4d22yzTTz99NMxfvz4OPTQQ2PfffetYMUAAEBX1imC1PTp02Px4sXx5S9/udXxHj16xPTp0+Oaa66J1atXx6BBg2Ls2LFx0UUXlfV38sKGAQAAVKdS80CnCFKf/vSnI8/zjY4PGjQoHnrooQpUBAAAVDPzLwAAAIkEKQAAgESd4tW+dpO9NQAAgOrU1ZY/BwAA6CgEKQAAgESCFAAAQCI9UsX0SAEAQHXTIwUAANA2BCkAAIBEghQAAEAiPVJF8mzDAAAAqlOpecCMFAAAQCJBCgAAIJEgBQAAkEiPVJG8sGEAAADVqdQ8IDYAAAAkEqQAAAASCVIAAACJBCkAAIBEghQAAEAiQQoAACCRIAUAAJDIPlLFsrcGAABQnUrMA2akAAAAEglSAAAAibzaVyQvbBgAAEB1KjUPiA0AAACJBCkAAIBEghQAAEAiPVLFLH8OAADVzfLnAAAAbUOQAgAASCRIAQAAJNIjVSTPNgwAAKA6lZoHzEgBAAAkEqQAAAASCVIAAACJ9Ei9kx4pAADgPZiRAgAASCRIAQAAJBKkAAAAEumRKpIXNgwAAKA6lZoHxAYAAIBEghQAAEAiQQoAACCRHqliWb5hAAAA1anEPGBGCgAAIFFZM1KrV6+OyZMnx4wZM+K1116L5ubmVudffPHFLVIcAABAR1RWkPrKV74SDz30UJx44omx/fbbR5ZlW7quysgi8i7yUwAAgDKUmAfKClIPPPBA3H///fHxj3+8nK8DAAB0amX1SG299dZRX1+/pWsBAADoFMoKUt/4xjfi4osvjjVr1mzpegAAADq8sl7tu/rqq2PhwoXRv3//2GWXXaJ79+6tzs+bN2+LFNfuCmEdQwAAqGYl5oGygtSRRx5ZztcAAAC6hLKC1CWXXLKl6wAAAOg0yn6R7Y033oj/+Z//iYkTJ8by5csjYsMrfX/+85+3WHEAAAAdUVkzUk8//XSMGDEi6urq4qWXXorTTjst6uvr46677orFixfHj3/84y1dZ7vI7SMFAABVrdQ8UNaM1IQJE+KUU06JP/7xj9GzZ8+W46NHj45Zs2aVc0sAAIBOo6wg9fjjj8cZZ5yx0fEddtghli5d+r6LAgAA6MjKClK1tbXR0NCw0fHnn38+tttuu/ddFAAAQEdWVo/UZz/72bj88svjzjvvjIiILMti8eLFccEFF8TYsWO3aIHtKntrAAAA1akte6SuvvrqWLVqVfTr1y/+/ve/xyc+8Yn44Ac/GL17945vfetb5dwSAACg0yhrRqquri6mTZsWjzzySDz99NOxatWqGDx4cIwYMWJL1wcAANDhlBWk3jZs2LAYNmzYlqoFAACgUyg5SF177bUl3/Tcc88tq5iK0yMFAADVrcQ8UHKQ+u53v1va382yzhukAAAASlBykFq0aFFb1gEAANBplLVqX6n69OkTL774Ylv+CQAAgHb3vhabeC95nrfl7be4PNswAACA6lRqHmjTGSkAAICuSJACAABIJEgBAAAkatMeqSzrZA1H9pECAIDq1hF6pDrbYhMAAAClaNMg9cADD8QOO+zQln8CAACg3ZX1at+ECRNKvnbYsGHl/ImKsPw5AABUt1LzQFlB6qmnnoqnnnoq1q1bF3vuuWdERDz//PNRU1MTgwcPbrmu0/VIAQAAlKCsIDVmzJjo3bt33HzzzbH11ltHRMTf/va3OPXUU+OQQw6Jf/u3f9uiRQIAAHQkZfVIXX311TFp0qSWEBURsfXWW8c3v/nNuPrqq7dYcQAAAB1RWTNSDQ0N8Ze//GWj43/5y19i5cqV77uoiinkGwYAAFCdSswDZc1Iff7zn49TTz017rrrrnjllVfilVdeif/93/+NcePGxVFHHVXOLQEAADqNsmakbrjhhvja174Wxx9/fKxbt27Djbp1i3HjxsWVV165RQsEAADoaMoKUltttVV873vfiyuvvDIWLlwYERG77757fOADH9iixQEAAHREZQWpty1ZsiSWLFkShx56aPTq1SvyPO/US57bRwoAAKpbqXmgrB6p119/PYYPHx4f+tCHYvTo0bFkyZKIiBg3bpylzwEAgC6vrCA1fvz46N69eyxevDi22mqrluPHHntsTJ06teT7zJo1K8aMGRMDBw6MLMvinnvuaXU+z/O4+OKLY/vtt49evXrFiBEj4o9//GOra5YvXx4nnHBC9OnTJ/r27Rvjxo2LVatWlfOzAAAASlJWkPrVr34V3/72t2PHHXdsdXyPPfaIP/3pTyXfZ/Xq1bHffvvFlClTNnn+iiuuiGuvvTZuuOGGmDt3bnzgAx+II444It58882Wa0444YR47rnnYtq0aXHffffFrFmz4vTTTy/nZwEAAJSkrB6p1atXt5qJetvy5cujtra25PuMGjUqRo0atclzeZ7HNddcExdddFF87nOfi4iIH//4x9G/f/+455574otf/GL8/ve/j6lTp8bjjz8eH/3oRyMi4rrrrovRo0fHVVddFQMHDkz7YdlbAwAAqE5t2SN1yCGHxI9//ON//K0si+bm5rjiiivi8MMPL+eWG1m0aFEsXbo0RowY0XKsrq4uhgwZEnPmzImIiDlz5kTfvn1bQlRExIgRI6JQKMTcuXM3e+/GxsZoaGhoNQAAAEpV1ozUFVdcEcOHD48nnngi1q5dG+eff34899xzsXz58pg9e/YWKWzp0qUREdG/f/9Wx/v3799ybunSpdGvX79W57t16xb19fUt12zKpEmT4rLLLtsidQIAANWnrBmpvffeO55//vkYNmxYfO5zn4vVq1fHUUcdFU899VTsvvvuW7rGLW7ixImxYsWKlvHyyy9XuiQAAKATKXsfqbq6uviP//iPLVlLKwMGDIiIiGXLlsX222/fcnzZsmWx//77t1zz2muvtfre+vXrY/ny5S3f35Ta2tpN93LpkQIAgOrWlj1SU6dOjUceeaTl85QpU2L//feP448/Pv72t7+Vc8uN7LrrrjFgwICYMWNGy7GGhoaYO3duDB06NCIihg4dGm+88UY8+eSTLdc8+OCD0dzcHEOGDNkidQAAALxTWUHq//7f/9uyQMMzzzwTEyZMiNGjR8eiRYtiwoQJJd9n1apVMX/+/Jg/f35EbFhgYv78+bF48eLIsizOO++8+OY3vxn33ntvPPPMM3HSSSfFwIED48gjj4yIiA9/+MMxcuTIOO200+Kxxx6L2bNnx9lnnx1f/OIX01fsAwAAKFFZr/YtWrQoPvKRj0RExP/+7//GmDFj4j//8z9j3rx5MXr06JLv88QTT7Ra5e/tEHbyySfHTTfdFOeff36sXr06Tj/99HjjjTdi2LBhMXXq1OjZs2fLd2655ZY4++yzY/jw4VEoFGLs2LFx7bXXlvOzAAAASlJWkOrRo0esWbMmIiKmT58eJ510UkRE1NfXJy0lfthhh0We55s9n2VZXH755XH55Zdv9pr6+vq49dZbS/6b7ybP8sgLm68HAADo2vKstDxQVpAaNmxYTJgwIT7+8Y/HY489FnfccUdERDz//POx4447lnNLAACATqOsHqnrr78+unXrFj/72c/i+9//fuywww4REfHAAw/EyJEjt2iBAAAAHU1ZM1I77bRT3HfffRsd/+53v9vq8+TJk+PMM8+Mvn37llUcAABAR1T2PlKl+M///M845phjOk+Qso8UAABUt7bcR6pU77aQBAAAQGfVpkEKAACgK2rTV/s6Ha/2AQBAdesIr/YBAAB0RYIUAABAojYNUoccckj06tWrLf8EAABAuyurR+qmm26KU045ZaPj69evj69//esxadKkiIj4xS9+8b6Ka295IY+8YKVBAACoVqXmgbJmpM4999z4whe+EH/7299aji1YsCCGDBkSt912Wzm3BAAA6DTKClJPPfVUvPLKK7HPPvvEtGnTYsqUKTF48ODYa6+94re//e2WrhEAAKBDKevVvt133z1mz54d5513XowcOTJqamri5ptvjuOOO25L1wcAANDhlL2P1P333x+33357DB06NJ5//vm48cYb4xOf+EQMHDhwS9bXvuwjBQAA1a0t95E644wz4gtf+EJccMEF8fDDD8fTTz8dPXr0iH322SfuvPPOcm4JAADQaZQ1IzV79uyYO3du7LfffhERMWDAgPjFL34RU6ZMiS9/+ctxzDHHbNEiAQAAOpKygtSTTz4ZtbW1Gx0/66yzYsSIEe+7KAAAgI6srCC1qRD1tj333LPsYipOjxQAAFS3EvNA2YtN/OxnP4s777wzFi9eHGvXrm11bt68eeXeFgAAoMMra7GJa6+9Nk499dTo379/PPXUU3HQQQfFNttsEy+++GKMGjVqS9cIAADQoZQVpL73ve/FD37wg7juuuuiR48ecf7558e0adPi3HPPjRUrVmzpGgEAADqUsoLU4sWL4+CDD46IiF69esXKlSsjIuLEE0+M2267bctVBwAA0AGVFaQGDBgQy5cvj4iInXbaKR599NGIiFi0aFHkeb7lqgMAAOiAygpSn/zkJ+Pee++NiIhTTz01xo8fH5/61Kfi2GOPjc9//vNbtEAAAICOpqxV+37wgx9Ec3NzRGzYO2rbbbeN2bNnx2c/+9k488wzt2iBAAAAHU1ZQapQKMTatWtj3rx58dprr0WvXr1aNuKdOnVqjBkzZosW2W4K+YYBAABUpxLzQFlBaurUqXHiiSfG66+/vtG5LMuiqampnNsCAAB0CmX1SJ1zzjlxzDHHxJIlS6K5ubnVEKIAAICurqwZqWXLlsWECROif//+W7qeysreGgAAQHUqMQ+UNSN19NFHx8yZM8v5KgAAQKdX1ozU9ddfH1/4whfi4Ycfjn322Se6d+/e6vy55567RYoDAADoiMoKUrfddlv86le/ip49e8bMmTMjy/4x/5VlmSAFAAB0aWUFqf/4j/+Iyy67LC688MIoFMp6O7BjyvINAwAAqE4l5oGyUtDatWvj2GOP7VohCgAAoERlJaGTTz457rjjji1dCwAAQKdQ1qt9TU1NccUVV8Qvf/nL2HfffTdabOI73/nOFikOAACgIyorSD3zzDNxwAEHRETEs88+2+pc8cITnU4hypyjAwAAuoQS80BZQerXv/51OV8DAADoEsy/AAAAJBKkAAAAEpX1al+XZR8pAACobm25jxQAAEA1E6QAAAASCVIAAACJ9EgVybINAwAAqE6l5gEzUgAAAIkEKQAAgESCFAAAQCI9UsXsIwUAANXNPlIAAABtQ5ACAABIJEgBAAAk0iNVJMvyyAp6pAAAoFpleqQAAADahiAFAACQyKt9xbK3BgAAUJ1KzANmpAAAABIJUgAAAIkEKQAAgER6pIpk2YYBAABUp1LzgBkpAACARIIUAABAIkEKAAAgkR6pIlmWR5bllS4DAACokFLzgBkpAACARIIUAABAIkEKAAAgkR6pInqkAACguumRAgAAaCOCFAAAQCJBCgAAIJEeqWJZvmEAAADVSY8UAABA2xCkAAAAEglSAAAAifRIFcmyDQMAAKhOpeYBM1IAAACJBCkAAIBEghQAAEAiPVJFskIehYJ9pAAAoFrlJeYBM1IAAACJBCkAAIBEXu0rkmV5ZJlX+wAAoFqVmgfMSAEAACSqaJCaNWtWjBkzJgYOHBhZlsU999zTcm7dunVxwQUXxD777BMf+MAHYuDAgXHSSSfFq6++2uoeu+yyS2RZ1mpMnjy5nX8JAABQTSoapFavXh377bdfTJkyZaNza9asiXnz5sXXv/71mDdvXtx1112xYMGC+OxnP7vRtZdffnksWbKkZZxzzjntUT4AAFClKtojNWrUqBg1atQmz9XV1cW0adNaHbv++uvjoIMOisWLF8dOO+3Ucrx3794xYMCA911PlpX+TiQAAND1ZFlp13WqHqkVK1ZElmXRt2/fVscnT54c22yzTRxwwAFx5ZVXxvr169/1Po2NjdHQ0NBqAAAAlKrTrNr35ptvxgUXXBDHHXdc9OnTp+X4ueeeG4MHD476+vr4zW9+ExMnTowlS5bEd77znc3ea9KkSXHZZZe1R9kAAEAXlOV53iHeZcuyLO6+++448sgjNzq3bt26GDt2bLzyyisxc+bMVkHqnX74wx/GGWecEatWrYra2tpNXtPY2BiNjY0tnxsaGmLQoEGxx08ujJqtNv0dAACg62ta0xh//NLkWLFixbvmjg4/I7Vu3bo45phj4k9/+lM8+OCD7/pjIiKGDBkS69evj5deein23HPPTV5TW1u7yZBVKORRKHSIXAkAAFRAXmIe6NBB6u0Q9cc//jF+/etfxzbbbPOe35k/f34UCoXo169fO1QIAABUo4oGqVWrVsULL7zQ8nnRokUxf/78qK+vj+233z6OPvromDdvXtx3333R1NQUS5cujYiI+vr66NGjR8yZMyfmzp0bhx9+ePTu3TvmzJkT48ePjy996Uux9dZbV+pnAQAAXVxFg9QTTzwRhx9+eMvnCRMmRETEySefHJdeemnce++9ERGx//77t/rer3/96zjssMOitrY2br/99rj00kujsbExdt111xg/fnzLfQAAANpCRYPUYYcdFu+21sV7rYMxePDgePTRR7dYPVmW20cKAACqWKl5oFPtIwUAANARCFIAAACJBCkAAIBEHXr58/aWvTUAAIDqVGoeMCMFAACQSJACAABIJEgBAAAk0iNVpJDlUbCPFAAAVK3cPlIAAABtQ5ACAABI5NW+IoVCcxQKzZUuAwAAqJC8xDxgRgoAACCRIAUAAJBIkAIAAEikR6pIIdswAACA6pSXmAfMSAEAACQSpAAAABIJUgAAAIn0SBXJsjyyLK90GQAAQIWUmgfMSAEAACQSpAAAABIJUgAAAIn0SBWpyZqjptBc6TIAAIBKyUrLA2akAAAAEglSAAAAiQQpAACARHqkimRZHgX7SAEAQNVqto8UAABA2xCkAAAAEglSAAAAifRIFcneGgAAQHUqNQ+YkQIAAEgkSAEAACQSpAAAABLpkSpSsI8UAABUtVLzgBkpAACARIIUAABAIq/2FakpNEdNobnSZQAAABWSl5gHzEgBAAAkEqQAAAASCVIAAACJ9EgVsfw5AABUN8ufAwAAtBFBCgAAIJEgBQAAkEiPVBE9UgAAUN30SAEAALQRQQoAACCRIAUAAJBIj1SRmqw5arLmSpcBAABUSF5iHjAjBQAAkEiQAgAASCRIAQAAJNIjVcQ+UgAAUN3sIwUAANBGBCkAAIBEghQAAEAiPVJFsqz0dyIBAICuJ8tKu86MFAAAQCJBCgAAIJEgBQAAkEiPVBH7SAEAQHWzjxQAAEAbEaQAAAASebWvSLesKbplTZUuAwAAqJC8xDxgRgoAACCRIAUAAJBIkAIAAEikR6qI5c8BAKC6Wf4cAACgjQhSAAAAiQQpAACARHqkiuiRAgCA6qZHCgAAoI0IUgAAAIkEKQAAgER6pIp0y5qjW9Zc6TIAAIAKyUvMA2akAAAAEglSAAAAiQQpAACARHqkithHCgAAqpt9pAAAANqIIAUAAJBIkAIAAEikR6pIIWuOgn2kAACgapWaByo6IzVr1qwYM2ZMDBw4MLIsi3vuuafV+VNOOSWyLGs1Ro4c2eqa5cuXxwknnBB9+vSJvn37xrhx42LVqlXt+CsAAIBqU9EgtXr16thvv/1iypQpm71m5MiRsWTJkpZx2223tTp/wgknxHPPPRfTpk2L++67L2bNmhWnn356W5cOAABUsYq+2jdq1KgYNWrUu15TW1sbAwYM2OS53//+9zF16tR4/PHH46Mf/WhERFx33XUxevTouOqqq2LgwIFJ9RQijxrLnwMAQNVqii6y/PnMmTOjX79+seeee8ZXv/rVeP3111vOzZkzJ/r27dsSoiIiRowYEYVCIebOnbvZezY2NkZDQ0OrAQAAUKoOHaRGjhwZP/7xj2PGjBnx7W9/Ox566KEYNWpUNDU1RUTE0qVLo1+/fq2+061bt6ivr4+lS5du9r6TJk2Kurq6ljFo0KA2/R0AAEDX0qFX7fviF7/Y8s/77LNP7LvvvrH77rvHzJkzY/jw4WXfd+LEiTFhwoSWzw0NDcIUAABQsg4dpN5pt912i2233TZeeOGFGD58eAwYMCBee+21VtesX78+li9fvtm+qogNfVe1tbUbHe+WNUc3y58DAEDVau4My5+neuWVV+L111+P7bffPiIihg4dGm+88UY8+eSTLdc8+OCD0dzcHEOGDKlUmQAAQBdX0RmpVatWxQsvvNDyedGiRTF//vyor6+P+vr6uOyyy2Ls2LExYMCAWLhwYZx//vnxwQ9+MI444oiIiPjwhz8cI0eOjNNOOy1uuOGGWLduXZx99tnxxS9+MXnFPgAAgFJVdEbqiSeeiAMOOCAOOOCAiIiYMGFCHHDAAXHxxRdHTU1NPP300/HZz342PvShD8W4cePiwAMPjIcffrjVa3m33HJL7LXXXjF8+PAYPXp0DBs2LH7wgx9U6icBAABVIMvzvOo3TmpoaIi6uro46dfHRo9/6lHpcgAAgApZu2pt/PjwO2LFihXRp0+fzV7XqXqkAAAAOgJBCgAAIJEgBQAAkKhT7SPV1mqyPGqyqm8ZAwCAqlVqHjAjBQAAkEiQAgAASCRIAQAAJNIjVaRb1hzdsqZKlwEAAFRIc9Zc0nVmpAAAABIJUgAAAIkEKQAAgER6pIoUsjwK9pECAICqVWoeMCMFAACQSJACAABIJEgBAAAk0iNVpBDNUROlrRsPAAB0PYUS84AZKQAAgESCFAAAQCKv9hWx/DkAAFQ3y58DAAC0EUEKAAAgkSAFAACQSI9Uke5ZU3TPZEsAAKhWzVlTSddJDQAAAIkEKQAAgESCFAAAQCI9UkXsIwUAANXNPlIAAABtRJACAABIJEgBAAAk0iNVpCZrjpqsudJlAAAAFVJqHjAjBQAAkEiQAgAASCRIAQAAJNIjVaRb1hTdM9kSAACqVVPWVNJ1UgMAAEAiQQoAACCRIAUAAJBIj1SRQuRRiLzSZQAAABVSah4wIwUAAJBIkAIAAEgkSAEAACTSI1WkJmuOmqy50mUAAAAVUmoeMCMFAACQSJACAABI5NW+IoUsj4JX+wAAoGoVMsufAwAAtAlBCgAAIJEgBQAAkEiPVJHuWVN0z7JKlwEAAFRIU9ZU0nVmpAAAABIJUgAAAIkEKQAAgER6pIrURB41Udq68QAAQNdTah4wIwUAAJBIkAIAAEgkSAEAACTSI1Uky5qjkDVXugwAAKBCshLzgBkpAACARIIUAABAIkEKAAAgkR6pIt2zpuieZZUuAwAAqJD1WVNJ15mRAgAASCRIAQAAJBKkAAAAEumRKlITzVET9pECAIBqVWoeMCMFAACQSJACAABI5NW+IjVZc9RkXu0DAIBqVWoeMCMFAACQSJACAABIJEgBAAAk0iNVpBDNUbD8OQAAVK1S84AZKQAAgESCFAAAQCJBCgAAIJEeqSLds+bokTVVugwAAKBC1ttHCgAAoG0IUgAAAIkEKQAAgER6pIoUojkKWVbpMgAAgAqxjxQAAEAbEaQAAAASCVIAAACJ9EgVqYk8aiKvdBkAAECFlJoHzEgBAAAkqmiQmjVrVowZMyYGDhwYWZbFPffc0+p8lmWbHFdeeWXLNbvssstG5ydPntzOvwQAAKgmFQ1Sq1evjv322y+mTJmyyfNLlixpNX74wx9GlmUxduzYVtddfvnlra4755xz2qN8AACgSlW0R2rUqFExatSozZ4fMGBAq88///nP4/DDD4/ddtut1fHevXtvdO27aWxsjMbGxpbPDQ0NERHRPVsf3TM9UgAAUK26Z00lXddpeqSWLVsW999/f4wbN26jc5MnT45tttkmDjjggLjyyitj/fr173qvSZMmRV1dXcsYNGhQW5UNAAB0QZ1m1b6bb745evfuHUcddVSr4+eee24MHjw46uvr4ze/+U1MnDgxlixZEt/5znc2e6+JEyfGhAkTWj43NDQIUwAAQMk6TZD64Q9/GCeccEL07Nmz1fHiQLTvvvtGjx494owzzohJkyZFbW3tJu9VW1u72XMAAADvpVMEqYcffjgWLFgQd9xxx3teO2TIkFi/fn289NJLseeeeyb9nZqsOWqyrNwyAQCATq4may7puk7RI3XjjTfGgQceGPvtt997Xjt//vwoFArRr1+/dqgMAACoRhWdkVq1alW88MILLZ8XLVoU8+fPj/r6+thpp50iYkP/0k9/+tO4+uqrN/r+nDlzYu7cuXH44YdH7969Y86cOTF+/Pj40pe+FFtvvXW7/Q4AAKC6VDRIPfHEE3H44Ye3fH673+nkk0+Om266KSIibr/99sjzPI477riNvl9bWxu33357XHrppdHY2Bi77rprjB8/vlXfVIqayKMmLH8OAADVqtQ8kOV5XvXJoaGhIerq6uKn8/eKrXrXVLocAACgQtasbIov7P+HWLFiRfTp02ez13WKHikAAICORJACAABI1CmWP28vWTRHISx/DgAA1SqLLrT8OQAAQEciSAEAACQSpAAAABLpkSrSI2uKHlqkAACgaq3Lmkq6zowUAABAIkEKAAAgkSAFAACQSI9UkUKWRyErbd14AACg6ylkeWnXtXEdAAAAXY4gBQAAkEiQAgAASKRHqkhNNEdNpYsAAAAqpiZKWzPBjBQAAEAiQQoAACCRIAUAAJBIj1SR7llTdC9x3XgAAKDr6V7ivrJmpAAAABIJUgAAAIkEKQAAgER6pIoUIo+a0CMFAADVqlBiHjAjBQAAkEiQAgAASOTVviI1Xu0DAICqVmoeMCMFAACQSJACAABIJEgBAAAk0iNVpJDlUcj0SAEAQLUqNQ+YkQIAAEgkSAEAACQSpAAAABLpkSrSPZqje6WLAAAAKqZ7NJd0nRkpAACARIIUAABAIkEKAAAgkR6pIjVZHjX2kQIAgKpVah4wIwUAAJBIkAIAAEgkSAEAACTSI1WkJvKoCT1SAABQrUrNA2akAAAAEglSAAAAiQQpAACARHqkinTLIrpnla4CAAColG4l5gEzUgAAAIkEKQAAgERe7StSCMkSAACqWal5QG4AAABIJEgBAAAkEqQAAAAS6ZEqUvPWAAAAqlOpecCMFAAAQCJBCgAAIJEgBQAAkEiPVJGaLIuaLKt0GQAAQIWUmgfMSAEAACQSpAAAABIJUgAAAIn0SBXpHll0Dz1SAABQrUrNA2akAAAAEglSAAAAiQQpAACARHqkimRZFgX7SAEAQNXK7CMFAADQNgQpAACARIIUAABAIj1SRWoiixr7SAEAQNUqNQ+YkQIAAEgkSAEAACQSpAAAABLpkSrSPauJ7plsCQAA1aq7faQAAADahiAFAACQyKt9RQpv/R8AAFCdSk0DUgMAAEAiQQoAACCRIAUAAJBIj1SRQmRRiNKWOwQAALqeUvOAGSkAAIBEghQAAEAiQQoAACCRHqkiNVkhajLZEgAAqlVNiUsmSA0AAACJBCkAAIBEXu2LiDzPIyKiYVVzhSsBAAAq6e1M8HZG2BxBKiJWrlwZERE7D36psoUAAAAdwsqVK6Ourm6z57P8vaJWFWhubo5XX301evfuHVlmQ95SNDQ0xKBBg+Lll1+OPn36VLocujDPGu3Fs0Z78azRXjxr5cnzPFauXBkDBw6MQmHznVBmpCKiUCjEjjvuWOkyOqU+ffr4NybtwrNGe/Gs0V48a7QXz1q6d5uJepvFJgAAABIJUgAAAIkEKcpSW1sbl1xySdTW1la6FLo4zxrtxbNGe/Gs0V48a23LYhMAAACJzEgBAAAkEqQAAAASCVIAAACJBCkAAIBEghQlW758eZxwwgnRp0+f6Nu3b4wbNy5WrVpV0nfzPI9Ro0ZFlmVxzz33tG2hdHqpz9ry5cvjnHPOiT333DN69eoVO+20U5x77rmxYsWKdqyazmDKlCmxyy67RM+ePWPIkCHx2GOPvev1P/3pT2OvvfaKnj17xj777BO/+MUv2qlSOruUZ+2///u/45BDDomtt946tt566xgxYsR7PpvwttT/XHvb7bffHlmWxZFHHtm2BXZhghQlO+GEE+K5556LadOmxX333RezZs2K008/vaTvXnPNNZFlWRtXSFeR+qy9+uqr8eqrr8ZVV10Vzz77bNx0000xderUGDduXDtWTUd3xx13xIQJE+KSSy6JefPmxX777RdHHHFEvPbaa5u8/je/+U0cd9xxMW7cuHjqqafiyCOPjCOPPDKeffbZdq6czib1WZs5c2Ycd9xx8etf/zrmzJkTgwYNik9/+tPx5z//uZ0rp7NJfdbe9tJLL8XXvva1OOSQQ9qp0i4qhxL87ne/yyMif/zxx1uOPfDAA3mWZfmf//znd/3uU089le+www75kiVL8ojI77777jauls7s/Txrxe688868R48e+bp169qiTDqhgw46KD/rrLNaPjc1NeUDBw7MJ02atMnrjznmmPwzn/lMq2NDhgzJzzjjjDatk84v9Vl7p/Xr1+e9e/fOb7755rYqkS6inGdt/fr1+cEHH5z/z//8T37yySfnn/vc59qh0q7JjBQlmTNnTvTt2zc++tGPthwbMWJEFAqFmDt37ma/t2bNmjj++ONjypQpMWDAgPYolU6u3GftnVasWBF9+vSJbt26tUWZdDJr166NJ598MkaMGNFyrFAoxIgRI2LOnDmb/M6cOXNaXR8RccQRR2z2eogo71l7pzVr1sS6deuivr6+rcqkCyj3Wbv88sujX79+3trYAvx/GJRk6dKl0a9fv1bHunXrFvX19bF06dLNfm/8+PFx8MEHx+c+97m2LpEuotxnrdhf//rX+MY3vlHyq6d0fX/961+jqakp+vfv3+p4//794w9/+MMmv7N06dJNXl/qc0h1KudZe6cLLrggBg4cuFGQh2LlPGuPPPJI3HjjjTF//vx2qLDrMyNV5S688MLIsuxdR6n/wf9O9957bzz44INxzTXXbNmi6ZTa8lkr1tDQEJ/5zGfiIx/5SFx66aXvv3CAdjR58uS4/fbb4+67746ePXtWuhy6kJUrV8aJJ54Y//3f/x3bbrttpcvpEsxIVbl/+7d/i1NOOeVdr9ltt91iwIABGzUurl+/PpYvX77ZV/YefPDBWLhwYfTt27fV8bFjx8YhhxwSM2fOfB+V09m05bP2tpUrV8bIkSOjd+/ecffdd0f37t3fb9l0Edtuu23U1NTEsmXLWh1ftmzZZp+rAQMGJF0PEeU9a2+76qqrYvLkyTF9+vTYd99927JMuoDUZ23hwoXx0ksvxZgxY1qONTc3R8SGNz8WLFgQu+++e9sW3cUIUlVuu+22i+222+49rxs6dGi88cYb8eSTT8aBBx4YERuCUnNzcwwZMmST37nwwgvjK1/5Sqtj++yzT3z3u99t9W9iqkNbPmsRG2aijjjiiKitrY17773X/5JLKz169IgDDzwwZsyY0bLUb3Nzc8yYMSPOPvvsTX5n6NChMWPGjDjvvPNajk2bNi2GDh3aDhXTWZXzrEVEXHHFFfGtb30rfvnLX7bqEYXNSX3W9tprr3jmmWdaHbvoooti5cqV8V//9V8xaNCg9ii7a6n0ahd0HiNHjswPOOCAfO7cufkjjzyS77HHHvlxxx3Xcv6VV17J99xzz3zu3LmbvUdYtY8SpD5rK1asyIcMGZLvs88++QsvvJAvWbKkZaxfv75SP4MO5vbbb89ra2vzm266Kf/d736Xn3766Xnfvn3zpUuX5nme5yeeeGJ+4YUXtlw/e/bsvFu3bvlVV12V//73v88vueSSvHv37vkzzzxTqZ9AJ5H6rE2ePDnv0aNH/rOf/azVf36tXLmyUj+BTiL1WXsnq/a9P2akKNktt9wSZ599dgwfPjwKhUKMHTs2rr322pbz69atiwULFsSaNWsqWCVdQeqzNm/evJYV/T74wQ+2uteiRYtil112abfa6biOPfbY+Mtf/hIXX3xxLF26NPbff/+YOnVqS6P24sWLo1D4R+vwwQcfHLfeemtcdNFF8e///u+xxx57xD333BN77713pX4CnUTqs/b9738/1q5dG0cffXSr+1xyySV6PXlXqc8aW1aW53le6SIAAAA6ExEVAAAgkSAFAACQSJACAABIJEgBAAAkEqQAAAASCVIAAACJBCkAAIBEghQAAEAiQQqAqnDYYYfFeeed1+nuDUDH1K3SBQBAZ3fXXXdF9+7dK10GAO1IkAKA96m+vr7SJQDQzrzaB0CH09zcHJMmTYpdd901evXqFfvtt1/87Gc/i4iImTNnRpZl8ctf/jIOOOCA6NWrV3zyk5+M1157LR544IH48Ic/HH369Injjz8+1qxZ0+q+69evj7PPPjvq6upi2223ja9//euR53lJNX3ve9+LPfbYI3r27Bn9+/ePo48+uuVc8at9b9f3znHKKae0XP/zn/88Bg8eHD179ozddtstLrvssli/fv37+xcNgHZlRgqADmfSpEnxk5/8JG644YbYY489YtasWfGlL30ptttuu5ZrLr300rj++utjq622imOOOSaOOeaYqK2tjVtvvTVWrVoVn//85+O6666LCy64oOU7N998c4wbNy4ee+yxeOKJJ+L000+PnXbaKU477bR3reeJJ56Ic889N/7f//t/cfDBB8fy5cvj4Ycf3uS1Bx98cCxZsqTl8+9///sYPXp0HHrooRER8fDDD8dJJ50U1157bRxyyCGxcOHCOP300yMi4pJLLin7XzMA2leWl/o/xQFAO2hsbIz6+vqYPn16DB06tOX4V77ylVizZk2cfvrpcfjhh8f06dNj+PDhERExefLkmDhxYixcuDB22223iIg488wz46WXXoqpU6dGxIZZo9deey2ee+65yLIsIiIuvPDCuPfee+N3v/vdu9Z01113xamnnhqvvPJK9O7de6Pzhx12WOy///5xzTXXtDr++uuvx0EHHRQjR46MKVOmRETEiBEjYvjw4TFx4sSW637yk5/E+eefH6+++mriv1oAVIoZKQA6lBdeeCHWrFkTn/rUp1odX7t2bRxwwAEtn/fdd9+Wf+7fv39stdVWLSHq7WOPPfZYq3v8y7/8S0uIiogYOnRoXH311dHU1BQ1NTWbrelTn/pU7LzzzrHbbrvFyJEjY+TIkfH5z38+ttpqq81+Z926dTF27NjYeeed47/+679ajv/2t7+N2bNnx7e+9a2WY01NTfHmm2/GmjVr3vWeAHQcghQAHcqqVasiIuL++++PHXbYodW52traWLhwYUREq1XysizbaNW8LMuiubl5i9TUu3fvmDdvXsycOTN+9atfxcUXXxyXXnppPP7449G3b99NfuerX/1qvPzyy/HYY49Ft27/+K/bVatWxWWXXRZHHXXURt/p2bPnFqkXgLYnSAHQoXzkIx+J2traWLx4cXziE5/Y6PzbQaocc+fObfX50UcfjT322ONdZ6Pe1q1btxgxYkSMGDEiLrnkkujbt288+OCDmwxE3/nOd+LOO++M3/zmN7HNNtu0Ojd48OBYsGBBfPCDHyz7dwBQeYIUAB1K796942tf+1qMHz8+mpubY9iwYbFixYqYPXt29OnTJ3beeeey77148eKYMGFCnHHGGTFv3ry47rrr4uqrr37P7913333x4osvxqGHHhpbb711/OIXv4jm5ubYc889N7p2+vTpcf7558eUKVNi2223jaVLl0ZERK9evaKuri4uvvji+Nd//dfYaaed4uijj45CoRC//e1v49lnn41vfvObZf82ANqXIAVAh/ONb3wjtttuu5g0aVK8+OKL0bdv3xg8eHD8+7//+/t6Xe+kk06Kv//973HQQQdFTU1N/J//839aVsx7N3379o277rorLr300njzzTdjjz32iNtuuy3++Z//eaNrH3nkkWhqaoozzzwzzjzzzJbjJ598ctx0001xxBFHxH333ReXX355fPvb347u3bvHXnvtFV/5ylfK/l0AtD+r9gEAACSyIS8AAEAiQQqAqvfwww/HP/3TP212AMA7ebUPgKr397//Pf785z9v9rwV9gB4J0EKAAAgkVf7AAAAEglSAAAAiQQpAACARIIUAABAIkEKAAAgkSAFAACQSJACAABI9P8BsQHM0k+0a/gAAAAASUVORK5CYII=\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "code", + "source": [ + "fig, ax = plt.subplots(figsize=(10, 10))\n", + "plt.imshow(tmp_layer.pe[0, 50:51, :], aspect=\"auto\")\n", + "plt.xlabel(\"emb_size\")\n", + "plt.ylabel(\"max_seq_len\")\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 850 + }, + "id": "NDEk6GqycZRX", + "outputId": "fdf0ca04-be31-4076-818f-eedbdc176aaf" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1kAAANBCAYAAAAShHTFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA1v0lEQVR4nO3deZjWdb3/8dcMywwoi7iAC4gaJzRX5IcHl9PCpEQ/Oy6ppbkQaXYkl2lRTj/BMkVNzFTSn5462knNMjt5LEnEMPWQKKhpx6XjcnADMtJhUUDm/v3R5fyaZAYcPjDe8nhc11yX93e5v+/7vr6IT7/3/Z2aSqVSCQAAAEXUdvYAAAAA7yUiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgrp29gDvds3NzXnppZfSq1ev1NTUdPY4AABAJ6lUKlm8eHG22Wab1Na2fb1KZK3BSy+9lIEDB3b2GAAAwLvE888/n+22267N9SJrDXr16pUk+btxE9Ole/1qt7nztO+2uf9+P/lCu88/+OzZ7a6v3aRnu+uX/nDLNtdtOqFbu/u+2Xv1r+ctXf/7xXbXd7u+7ee/cacZ7e477Ppx7a7f4WdN7a7/x+/f3ea6mxsPbHff7i+82u76za9Z1O76P3267fft+nt+1e6+H76s/fPh9i9e3u76g393dJvrzhoyrd19/7BiQLvrFyzv3e76Q/vObXPd+f8zpt19r9zpp+2uP+yRE9pdf8deP2xz3QH3fbbdfWfv/2/trh92V/vHfvgj17W5bs9fjW1330dHt71vkuz2i/aP/ejHr21z3e63tn/s333iX9tdv/vP17D/P7a9/+4/a/89/92h3293/R5r2P+RdvZfl32TZI9b1rD/Ye0cex32dewqPPZP17Dv4Ws49jrs79iO7djv3mM3LWnO9sOea2mEtoisNXjrI4JdutenS93q/+O6d6+2LxXW1q8hZGraD6Hamu7t779JXdvrurT/3Om6htlq2z92t03afv723pNkLd6XLsvbXd9j07ZP3a5rel1d2n7PkqTbJmt4z9t5X9b0uts6h9Z6/55tz96zV5d2961f3v4f97pu7Z8vm7QzW3vnYZL0WofXlazhz1jPdXtPa3t0fP912XejPvYa/vyvy79THduxHduxHdux1/exk6zxa0RufAEAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFVU1kLVq0KMccc0x69+6dvn37Zty4cVmyZMla7VupVPKxj30sNTU1+fd///f1OygAALBRq5rIOuaYY/L73/8+06dPz2233Zbf/OY3Oemkk9Zq30svvTQ1NTXreUIAAICka2cPsDYef/zxTJs2LQ888ECGDx+eJLn88sszZsyYXHzxxdlmm23a3Pfhhx/OlClT8uCDD2brrbfeUCMDAAAbqaq4kjVr1qz07du3JbCSpKGhIbW1tbn//vvb3G/ZsmU5+uijM3Xq1AwYMGBDjAoAAGzkquJK1vz587PVVlu1Wta1a9f069cv8+fPb3O/M844I/vuu2/+8R//ca2PtXz58ixfvrzlcVNT0zsfGAAA2Gh16pWss846KzU1Ne3+PPHEEx167ltvvTV33XVXLr300ne03+TJk9OnT5+Wn4EDB3bo+AAAwMapU69kfelLX8oJJ5zQ7jY77rhjBgwYkIULF7Za/uabb2bRokVtfgzwrrvuytNPP52+ffu2Wn744YfngAMOyMyZM1e734QJE9LY2NjyuKmpSWgBAABrrVMja8stt8yWW265xu1GjhyZV199NXPmzMnee++d5C8R1dzcnH322We1+5x11ln53Oc+12rZbrvtlm9/+9s5+OCD2zxWXV1d6urq3sGrAAAA+P+q4jtZO++8c0aPHp0TTzwxV111VVauXJnx48fnU5/6VMudBV988cWMGjUqP/jBDzJixIgMGDBgtVe5Bg0alB122GFDvwQAAGAjURV3F0yS66+/PkOHDs2oUaMyZsyY7L///rn66qtb1q9cuTJPPvlkli1b1olTAgAAG7uquJKVJP369csNN9zQ5vrBgwenUqm0+xxrWg8AALCuquZKFgAAQDUQWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAACioaiJr0aJFOeaYY9K7d+/07ds348aNy5IlS9rd/otf/GLe//73p0ePHhk0aFBOPfXUvPbaaxtwagAAYGNTNZF1zDHH5Pe//32mT5+e2267Lb/5zW9y0kkntbn9Sy+9lJdeeikXX3xxHnvssVx77bWZNm1axo0btwGnBgAANjZdO3uAtfH4449n2rRpeeCBBzJ8+PAkyeWXX54xY8bk4osvzjbbbPO2fXbdddf89Kc/bXm800475bzzzstnPvOZvPnmm+natSpeOgAAUGWq4krWrFmz0rdv35bASpKGhobU1tbm/vvvX+vnee2119K7d2+BBQAArDdVURvz58/PVltt1WpZ165d069fv8yfP3+tnuOVV17Jueee2+5HDJNk+fLlWb58ecvjpqamdz4wAACw0erUK1lnnXVWampq2v154okn1vk4TU1N+fjHP55ddtkl55xzTrvbTp48OX369Gn5GThw4DofHwAA2Hh06pWsL33pSznhhBPa3WbHHXfMgAEDsnDhwlbL33zzzSxatCgDBgxod//Fixdn9OjR6dWrV372s5+lW7du7W4/YcKENDY2tjxuamoSWgAAwFrr1Mjacssts+WWW65xu5EjR+bVV1/NnDlzsvfeeydJ7rrrrjQ3N2efffZpc7+mpqYcdNBBqaury6233pr6+vo1Hquuri51dXVr/yIAAAD+SlXc+GLnnXfO6NGjc+KJJ2b27Nm57777Mn78+HzqU59qubPgiy++mKFDh2b27NlJ/hJYBx54YJYuXZrvfe97aWpqyvz58zN//vysWrWqM18OAADwHlYVN75Ikuuvvz7jx4/PqFGjUltbm8MPPzyXXXZZy/qVK1fmySefzLJly5Ikc+fObbnz4Pve975Wz/Xss89m8ODBG2x2AABg41E1kdWvX7/ccMMNba4fPHhwKpVKy+MPfehDrR4DAABsCFXxcUEAAIBqIbIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAV17chOS5cuzQUXXJAZM2Zk4cKFaW5ubrX+mWeeKTIcAABAtelQZH3uc5/L3XffnWOPPTZbb711ampqSs8FAABQlToUWbfffnt+8YtfZL/99is9DwAAQFXr0HeyNttss/Tr16/0LAAAAFWvQ5F17rnnZuLEiVm2bFnpeQAAAKpahz4uOGXKlDz99NPp379/Bg8enG7durVaP3fu3CLDAQAAVJsORdYhhxxSeAwAAID3hg5F1qRJk0rPAQAA8J7Q4V9G/Oqrr+Zf/uVfMmHChCxatCjJXz4m+OKLLxYbDgAAoNp06ErW7373uzQ0NKRPnz557rnncuKJJ6Zfv3655ZZbMm/evPzgBz8oPScAAEBV6NCVrMbGxpxwwgn5wx/+kPr6+pblY8aMyW9+85tiwwEAAFSbDkXWAw88kM9//vNvW77ttttm/vz56zwUAABAtepQZNXV1aWpqelty5966qlsueWW6zwUAABAtepQZH3iE5/IN77xjaxcuTJJUlNTk3nz5uXMM8/M4YcfXnRAAACAatKhyJoyZUqWLFmSrbbaKq+//no++MEP5n3ve1969eqV8847r/SMAAAAVaNDdxfs06dPpk+fnnvvvTe/+93vsmTJkgwbNiwNDQ2l5wMAAKgqHYqst+y///7Zf//9S80CAABQ9dY6si677LK1ftJTTz21Q8MAAABUu7WOrG9/+9trtV1NTY3IAgAANlprHVnPPvvs+pwDAADgPaFDdxdcW717984zzzyzPg8BAADwrrJeI6tSqazPpwcAAHjXWa+RBQAAsLERWQAAAAWJLAAAgILWa2TV1NSsz6cHAAB413HjCwAAgILWa2Tdfvvt2XbbbdfnIQAAAN5V1vqXEf+1xsbGtd52//3378ghAAAAqlKHIuuhhx7KQw89lJUrV+b9739/kuSpp55Kly5dMmzYsJbtfCcLAADY2HQosg4++OD06tUr1113XTbbbLMkyZ///OeMHTs2BxxwQL70pS8VHRIAAKBadOg7WVOmTMnkyZNbAitJNttss3zzm9/MlClTig0HAABQbToUWU1NTfnjH//4tuV//OMfs3jx4nUeCgAAoFp1KLIOPfTQjB07NrfcckteeOGFvPDCC/npT3+acePG5bDDDis9IwAAQNXo0Heyrrrqqnz5y1/O0UcfnZUrV/7libp2zbhx4/Ktb32r6IAAAADVpEOR1bNnz3z3u9/Nt771rTz99NNJkp122imbbLJJ0eEAAACqzTr9MuKXX345L7/8coYMGZJNNtkklUql1FwAAABVqUOR9ac//SmjRo3K3/3d32XMmDF5+eWXkyTjxo1b77dvnzp1agYPHpz6+vrss88+mT17drvb/+QnP8nQoUNTX1+f3XbbLb/85S/X63wAAMDGrUORdcYZZ6Rbt26ZN29eevbs2bL8qKOOyrRp04oN97duuummNDY2ZtKkSZk7d2722GOPHHTQQVm4cOFqt//P//zPfPrTn864cePy0EMP5ZBDDskhhxySxx57bL3NCAAAbNw6FFl33HFHLrzwwmy33Xatlg8ZMiT/8z//U2Sw1bnkkkty4oknZuzYsdlll11y1VVXpWfPnvn+97+/2u2/853vZPTo0fnKV76SnXfeOeeee26GDRuWK664Yr3NCAAAbNw6FFlLly5tdQXrLYsWLUpdXd06D7U6K1asyJw5c9LQ0NCyrLa2Ng0NDZk1a9Zq95k1a1ar7ZPkoIMOanP7JFm+fHmamppa/QAAAKytDkXWAQcckB/84Actj2tqatLc3JyLLrooH/7wh4sN99deeeWVrFq1Kv3792+1vH///pk/f/5q95k/f/472j5JJk+enD59+rT8DBw4cN2HBwAANhoduoX7RRddlFGjRuXBBx/MihUr8tWvfjW///3vs2jRotx3332lZ9ygJkyYkMbGxpbHTU1NQgsAAFhrHYqsXXfdNU899VSuuOKK9OrVK0uWLMlhhx2WU045JVtvvXXpGZMkW2yxRbp06ZIFCxa0Wr5gwYIMGDBgtfsMGDDgHW2fJHV1devtI48AAMB7X4ciK0n69OmTr33tayVnaVf37t2z9957Z8aMGTnkkEOSJM3NzZkxY0bGjx+/2n1GjhyZGTNm5PTTT29ZNn369IwcOXIDTAwAAGyMOvSdrGnTpuXee+9teTx16tTsueeeOfroo/PnP/+52HB/q7GxMddcc02uu+66PP744/nCF76QpUuXZuzYsUmS4447LhMmTGjZ/rTTTsu0adMyZcqUPPHEEznnnHPy4IMPthllAAAA66pDkfWVr3yl5a57jz76aBobGzNmzJg8++yzrb7PVNpRRx2Viy++OBMnTsyee+6Zhx9+ONOmTWu5ucW8efNafjFykuy777654YYbcvXVV2ePPfbIzTffnH//93/Prrvuut5mBAAANm4d+rjgs88+m1122SVJ8tOf/jQHH3xwzj///MydOzdjxowpOuDfGj9+fJtXombOnPm2ZUcccUSOOOKI9ToTAADAWzp0Jat79+5ZtmxZkuTOO+/MgQcemCTp16+f3ysFAABs1Dp0JWv//fdPY2Nj9ttvv8yePTs33XRTkuSpp57KdtttV3RAAACAatKhK1lXXHFFunbtmptvvjlXXnlltt122yTJ7bffntGjRxcdEAAAoJp06ErWoEGDctttt71t+be//e1Wjy+44IKcfPLJ6du3b4eGAwAAqDYdupK1ts4///wsWrRofR4CAADgXWW9RlalUlmfTw8AAPCus14jCwAAYGMjsgAAAAoSWQAAAAWJLAAAgILWa2QdcMAB6dGjx/o8BAAAwLtKhyLr2muvXe3yN998MxMmTGh5/Mtf/jJbb711hwYDAACoRh2KrFNPPTVHHHFE/vznP7cse/LJJ7PPPvvkxhtvLDYcAABAtelQZD300EN54YUXsttuu2X69OmZOnVqhg0blqFDh+aRRx4pPSMAAEDV6NqRnXbaaafcd999Of300zN69Oh06dIl1113XT796U+Xng8AAKCqdPjGF7/4xS/yox/9KCNHjkzfvn3zve99Ly+99FLJ2QAAAKpOhyLr85//fI444oiceeaZueeee/K73/0u3bt3z2677ZYf//jHpWcEAACoGh36uOB9992X+++/P3vssUeSZMCAAfnlL3+ZqVOn5rOf/WyOPPLIokMCAABUiw5F1pw5c1JXV/e25aecckoaGhrWeSgAAIBq1aGPC64usN7y/ve/v8PDAAAAVLsOXclKkptvvjk//vGPM2/evKxYsaLVurlz567zYAAAANWoQ1eyLrvssowdOzb9+/fPQw89lBEjRmTzzTfPM888k4997GOlZwQAAKgaHYqs7373u7n66qtz+eWXp3v37vnqV7+a6dOn59RTT81rr71WekYAAICq0aHImjdvXvbdd98kSY8ePbJ48eIkybHHHpsbb7yx3HQAAABVpkORNWDAgCxatChJMmjQoPz2t79Nkjz77LOpVCrlpgMAAKgyHYqsj3zkI7n11luTJGPHjs0ZZ5yRj370oznqqKNy6KGHFh0QAACgmnTo7oJXX311mpubk/zld2NtscUWue+++/KJT3wiJ598ctEBAQAAqkmHIqu2tjYrVqzI3Llzs3DhwvTo0aPllxBPmzYtBx98cNEhAQAAqkWHImvatGk59thj86c//elt62pqarJq1ap1HgwAAKAadeg7WV/84hdz5JFH5uWXX05zc3OrH4EFAABszDoUWQsWLEhjY2P69+9feh4AAICq1qHI+uQnP5mZM2cWHgUAAKD6deg7WVdccUWOOOKI3HPPPdltt93SrVu3VutPPfXUIsMBAABUmw5F1o033pg77rgj9fX1mTlzZmpqalrW1dTUiCwAAGCj1aHI+trXvpavf/3rOeuss1Jb26FPHAIAALwndaiQVqxYkaOOOkpgAQAA/I0OVdLxxx+fm266qfQsAAAAVa9DHxdctWpVLrroovzqV7/K7rvv/rYbX1xyySVFhgMAAKg2HYqsRx99NHvttVeS5LHHHmu17q9vggEAALCx6VBk/frXvy49BwAAwHuCO1cAAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKCgqousqVOnZvDgwamvr88+++yT2bNnt7ntNddckwMOOCCbbbZZNttsszQ0NLS7PQAAwLqqqsi66aab0tjYmEmTJmXu3LnZY489ctBBB2XhwoWr3X7mzJn59Kc/nV//+teZNWtWBg4cmAMPPDAvvvjiBp4cAADYWFRVZF1yySU58cQTM3bs2Oyyyy656qqr0rNnz3z/+99f7fbXX399/umf/il77rlnhg4dmn/5l39Jc3NzZsyYsYEnBwAANhZVE1krVqzInDlz0tDQ0LKstrY2DQ0NmTVr1lo9x7Jly7Jy5cr069dvfY0JAABs5Lp29gBr65VXXsmqVavSv3//Vsv79++fJ554Yq2e48wzz8w222zTKtT+1vLly7N8+fKWx01NTR0bGAAA2ChVzZWsdXXBBRfkRz/6UX72s5+lvr6+ze0mT56cPn36tPwMHDhwA04JAABUu6qJrC222CJdunTJggULWi1fsGBBBgwY0O6+F198cS644ILccccd2X333dvddsKECXnttddafp5//vl1nh0AANh4VE1kde/ePXvvvXerm1a8dROLkSNHtrnfRRddlHPPPTfTpk3L8OHD13icurq69O7du9UPAADA2qqa72QlSWNjY44//vgMHz48I0aMyKWXXpqlS5dm7NixSZLjjjsu2267bSZPnpwkufDCCzNx4sTccMMNGTx4cObPn58k2XTTTbPpppt22usAAADeu6oqso466qj88Y9/zMSJEzN//vzsueeemTZtWsvNMObNm5fa2v9/ce7KK6/MihUr8slPfrLV80yaNCnnnHPOhhwdAADYSFRVZCXJ+PHjM378+NWumzlzZqvHzz333PofCAAA4K9UzXeyAAAAqoHIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEFVF1lTp07N4MGDU19fn3322SezZ89eq/1+9KMfpaamJocccsj6HRAAANioVVVk3XTTTWlsbMykSZMyd+7c7LHHHjnooIOycOHCdvd77rnn8uUvfzkHHHDABpoUAADYWFVVZF1yySU58cQTM3bs2Oyyyy656qqr0rNnz3z/+99vc59Vq1blmGOOyde//vXsuOOOG3BaAABgY1Q1kbVixYrMmTMnDQ0NLctqa2vT0NCQWbNmtbnfN77xjWy11VYZN27chhgTAADYyHXt7AHW1iuvvJJVq1alf//+rZb3798/TzzxxGr3uffee/O9730vDz/88FofZ/ny5Vm+fHnL46ampg7NCwAAbJyq5krWO7V48eIce+yxueaaa7LFFlus9X6TJ09Onz59Wn4GDhy4HqcEAADea6rmStYWW2yRLl26ZMGCBa2WL1iwIAMGDHjb9k8//XSee+65HHzwwS3LmpubkyRdu3bNk08+mZ122ult+02YMCGNjY0tj5uamoQWAACw1qomsrp375699947M2bMaLkNe3Nzc2bMmJHx48e/bfuhQ4fm0UcfbbXs//yf/5PFixfnO9/5TpvhVFdXl7q6uuLzAwAAG4eqiawkaWxszPHHH5/hw4dnxIgRufTSS7N06dKMHTs2SXLcccdl2223zeTJk1NfX59dd9211f59+/ZNkrctBwAAKKWqIuuoo47KH//4x0ycODHz58/PnnvumWnTprXcDGPevHmprX3Pfs0MAACoAlUVWUkyfvz41X48MElmzpzZ7r7XXntt+YEAAAD+iss+AAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKEhkAQAAFCSyAAAAChJZAAAABYksAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAFiSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEiCwAAoCCRBQAAUJDIAgAAKKhrZw/wblepVJIkq1a80eY2TYub21zX/Ebb+yXJm5WV7a6vraxof/+ly9tet6rtuZLkzTdr2l2f5vaPXbO00ua69t6TZC3el1Vtv64keX3Jm23v+2b7z127hudeuXQN73lz2/9vYk2ve9Xy9mdb4/7L2p592eJV7e77xoq237MkWb68/XNxade2Z2vvPEySxevwupI1/Blbtm7vafPrHd9/XfbdqI+9hj//6/LvVMd2bMd2bMd27PV57KYlf1n3ViO0paaypi02ci+88EIGDhzY2WMAAADvEs8//3y22267NteLrDVobm7OSy+9lF69eqWmpiZNTU0ZOHBgnn/++fTu3buzx+M9zLnGhuJcY0NxrrGhONdYXyqVShYvXpxtttkmtbVtf7rJxwXXoLa2drWV2rt3b39o2SCca2wozjU2FOcaG4pzjfWhT58+a9zGjS8AAAAKElkAAAAFiax3qK6uLpMmTUpdXV1nj8J7nHONDcW5xobiXGNDca7R2dz4AgAAoCBXsgAAAAoSWQAAAAWJLAAAgIJEFgAAQEEi6x2YOnVqBg8enPr6+uyzzz6ZPXt2Z49ElZs8eXL+1//6X+nVq1e22mqrHHLIIXnyySdbbfPGG2/klFNOyeabb55NN900hx9+eBYsWNBJE/NeccEFF6Smpiann356yzLnGqW8+OKL+cxnPpPNN988PXr0yG677ZYHH3ywZX2lUsnEiROz9dZbp0ePHmloaMgf/vCHTpyYarRq1aqcffbZ2WGHHdKjR4/stNNOOffcc/PX93RzrtFZRNZauummm9LY2JhJkyZl7ty52WOPPXLQQQdl4cKFnT0aVezuu+/OKaeckt/+9reZPn16Vq5cmQMPPDBLly5t2eaMM87If/zHf+QnP/lJ7r777rz00ks57LDDOnFqqt0DDzyQ//t//2923333Vsuda5Tw5z//Ofvtt1+6deuW22+/Pf/1X/+VKVOmZLPNNmvZ5qKLLspll12Wq666Kvfff3822WSTHHTQQXnjjTc6cXKqzYUXXpgrr7wyV1xxRR5//PFceOGFueiii3L55Ze3bONco9NUWCsjRoyonHLKKS2PV61aVdlmm20qkydP7sSpeK9ZuHBhJUnl7rvvrlQqlcqrr75a6datW+UnP/lJyzaPP/54JUll1qxZnTUmVWzx4sWVIUOGVKZPn1754Ac/WDnttNMqlYpzjXLOPPPMyv7779/m+ubm5sqAAQMq3/rWt1qWvfrqq5W6urrKjTfeuCFG5D3i4x//eOWzn/1sq2WHHXZY5ZhjjqlUKs41OpcrWWthxYoVmTNnThoaGlqW1dbWpqGhIbNmzerEyXivee2115Ik/fr1S5LMmTMnK1eubHXuDR06NIMGDXLu0SGnnHJKPv7xj7c6pxLnGuXceuutGT58eI444ohstdVW2WuvvXLNNde0rH/22Wczf/78Vudanz59ss8++zjXeEf23XffzJgxI0899VSS5JFHHsm9996bj33sY0mca3Surp09QDV45ZVXsmrVqvTv37/V8v79++eJJ57opKl4r2lubs7pp5+e/fbbL7vuumuSZP78+enevXv69u3batv+/ftn/vz5nTAl1exHP/pR5s6dmwceeOBt65xrlPLMM8/kyiuvTGNjY/75n/85DzzwQE499dR07949xx9/fMv5tLq/U51rvBNnnXVWmpqaMnTo0HTp0iWrVq3Keeedl2OOOSZJnGt0KpEF7xKnnHJKHnvssdx7772dPQrvQc8//3xOO+20TJ8+PfX19Z09Du9hzc3NGT58eM4///wkyV577ZXHHnssV111VY4//vhOno73kh//+Me5/vrrc8MNN+QDH/hAHn744Zx++unZZpttnGt0Oh8XXAtbbLFFunTp8ra7bC1YsCADBgzopKl4Lxk/fnxuu+22/PrXv852223XsnzAgAFZsWJFXn311VbbO/d4p+bMmZOFCxdm2LBh6dq1a7p27Zq77747l112Wbp27Zr+/fs71yhi6623zi677NJq2c4775x58+YlScv55O9U1tVXvvKVnHXWWfnUpz6V3XbbLccee2zOOOOMTJ48OYlzjc4lstZC9+7ds/fee2fGjBkty5qbmzNjxoyMHDmyEyej2lUqlYwfPz4/+9nPctddd2WHHXZotX7vvfdOt27dWp17Tz75ZObNm+fc4x0ZNWpUHn300Tz88MMtP8OHD88xxxzT8s/ONUrYb7/93varKJ566qlsv/32SZIddtghAwYMaHWuNTU15f7773eu8Y4sW7YstbWt/1O2S5cuaW5uTuJco3P5uOBaamxszPHHH5/hw4dnxIgRufTSS7N06dKMHTu2s0ejip1yyim54YYb8vOf/zy9evVq+Yx4nz590qNHj/Tp0yfjxo1LY2Nj+vXrl969e+eLX/xiRo4cmb//+7/v5OmpJr169Wr5rt9bNtlkk2y++eYty51rlHDGGWdk3333zfnnn58jjzwys2fPztVXX52rr746SVp+P9s3v/nNDBkyJDvssEPOPvvsbLPNNjnkkEM6d3iqysEHH5zzzjsvgwYNygc+8IE89NBDueSSS/LZz342iXONTtbZtzesJpdffnll0KBBle7du1dGjBhR+e1vf9vZI1Hlkqz251//9V9btnn99dcr//RP/1TZbLPNKj179qwceuihlZdffrnzhuY9469v4V6pONco5z/+4z8qu+66a6Wurq4ydOjQytVXX91qfXNzc+Xss8+u9O/fv1JXV1cZNWpU5cknn+ykaalWTU1NldNOO60yaNCgSn19fWXHHXesfO1rX6ssX768ZRvnGp2lplL5q1+LDQAAwDrxnSwAAICCRBYAAEBBIgsAAKAgkQUAAFCQyAIAAChIZAEAABQksgAAAAoSWQBs9D70oQ/l9NNPr7rnBuDdqWtnDwAA72W33HJLunXr1tljALABiSwAWI/69evX2SMAsIH5uCAAVaW5uTmTJ0/ODjvskB49emSPPfbIzTffnCSZOXNmampq8qtf/Sp77bVXevTokY985CNZuHBhbr/99uy8887p3bt3jj766CxbtqzV87755psZP358+vTpky222CJnn312KpXKWs303e9+N0OGDEl9fX369++fT37yky3r/vrjgm/N97c/J5xwQsv2P//5zzNs2LDU19dnxx13zNe//vW8+eab6/amAbBBuZIFQFWZPHlyfvjDH+aqq67KkCFD8pvf/Caf+cxnsuWWW7Zsc8455+SKK65Iz549c+SRR+bII49MXV1dbrjhhixZsiSHHnpoLr/88px55pkt+1x33XUZN25cZs+enQcffDAnnXRSBg0alBNPPLHdeR588MGceuqp+bd/+7fsu+++WbRoUe65557Vbrvvvvvm5Zdfbnn8+OOPZ8yYMfmHf/iHJMk999yT4447LpdddlkOOOCAPP300znppJOSJJMmTerwewbAhlVTWdv/TQcAnWz58uXp169f7rzzzowcObJl+ec+97ksW7YsJ510Uj784Q/nzjvvzKhRo5IkF1xwQSZMmJCnn346O+64Y5Lk5JNPznPPPZdp06Yl+cvVpoULF+b3v/99ampqkiRnnXVWbr311vzXf/1XuzPdcsstGTt2bF544YX06tXrbes/9KEPZc8998yll17aavmf/vSnjBgxIqNHj87UqVOTJA0NDRk1alQmTJjQst0Pf/jDfPWrX81LL730Dt8tADqLK1kAVI3//u//zrJly/LRj3601fIVK1Zkr732anm8++67t/xz//7907Nnz5bAemvZ7NmzWz3H3//937cEVpKMHDkyU6ZMyapVq9KlS5c2Z/roRz+a7bffPjvuuGNGjx6d0aNH59BDD03Pnj3b3GflypU5/PDDs/322+c73/lOy/JHHnkk9913X84777yWZatWrcobb7yRZcuWtfucALx7iCwAqsaSJUuSJL/4xS+y7bbbtlpXV1eXp59+Okla3c2vpqbmbXf3q6mpSXNzc5GZevXqlblz52bmzJm54447MnHixJxzzjl54IEH0rdv39Xu84UvfCHPP/98Zs+ena5d//9fxUuWLMnXv/71HHbYYW/bp76+vsi8AKx/IguAqrHLLrukrq4u8+bNywc/+MG3rX8rsjri/vvvb/X4t7/9bYYMGdLuVay3dO3aNQ0NDWloaMikSZPSt2/f3HXXXauNpUsuuSQ//vGP85//+Z/ZfPPNW60bNmxYnnzyybzvfe/r8OsAoPOJLACqRq9evfLlL385Z5xxRpqbm7P//vvntddey3333ZfevXtn++237/Bzz5s3L42Njfn85z+fuXPn5vLLL8+UKVPWuN9tt92WZ555Jv/wD/+QzTbbLL/85S/T3Nyc97///W/b9s4778xXv/rVTJ06NVtssUXmz5+fJOnRo0f69OmTiRMn5n//7/+dQYMG5ZOf/GRqa2vzyCOP5LHHHss3v/nNDr82ADYskQVAVTn33HOz5ZZbZvLkyXnmmWfSt2/fDBs2LP/8z/+8Th8BPO644/L6669nxIgR6dKlS0477bSWO/u1p2/fvrnllltyzjnn5I033siQIUNy44035gMf+MDbtr333nuzatWqnHzyyTn55JNblh9//PG59tprc9BBB+W2227LN77xjVx44YXp1q1bhg4dms997nMdfl0AbHjuLggAAFCQX0YMAABQkMgCgHbcc8892XTTTdv8AYC/5eOCANCO119/PS+++GKb690JEIC/JbIAAAAK8nFBAACAgkQWAABAQSILAACgIJEFAABQkMgCAAAoSGQBAAAUJLIAAAAKElkAAAAF/T++xgIS+CWqCAAAAABJRU5ErkJggg==\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.3 Encoder\n", + "\n", + "#### Picture\n", + "\n", + "" + ], + "metadata": { + "id": "n9MLJLySca14" + } + }, + { + "cell_type": "markdown", + "source": [ + "#### TransformerEncoderBlock\n", + "\n", + "**Initialization:**\n", + "\n", + "* in_size ~ input embedding size\n", + "* head_size ~ size of the Q, K, V matrices embeddings after transformation\n", + "* num_heads ~ number of attention heads\n", + "* out_size ~ output embedding size for attention and the block\n", + "* ff_hidden_size ~ hidden size for feed-forward layers\n", + "* dropout_p ~ dropout probability\n", + "* query_in_size ~ input embedding size for the query (if None, defaults to in_size)\n", + "\n", + "Forward:\n", + "\n", + "*query, key, value ~ 3 tensors (one for each Q, K, and V transformation - these are not yet the tensors $\\text{batch_size} \\times seq \\times d_k$, but tensors of shape $\\text{batch_size} \\times seq \\times \\text{in_size}$)" + ], + "metadata": { + "id": "B24kUNvlckeC" + } + }, + { + "cell_type": "code", + "source": [ + "class TransformerEncoderBlock(nn.Module):\n", + " \"\"\"\n", + " Class with one full block within transformer's encoder\n", + " \"\"\"\n", + " def __init__(self, in_size, head_size, num_heads, out_size, ff_hidden_size, dropout_p=0.2, query_in_size=None):\n", + " \"\"\"\n", + " Args:\n", + " in_size: input embedding size\n", + " head_size: size of each attention head\n", + " num_heads: number of attention heads\n", + " out_size: output embedding size\n", + " ff_hidden_size: hidden size for feed forward net\n", + " dropout_p: probability for dropout\n", + " query_in_size: embedding size of input for query (if not provided - same as in_size)\n", + " \"\"\"\n", + " super(TransformerEncoderBlock, self).__init__()\n", + "\n", + " # Запишем все переданые гиперпараметры слоя\n", + " self.in_size = in_size\n", + " self.head_size = head_size\n", + " self.num_heads = num_heads\n", + " self.out_size = out_size\n", + " self.ff_hidden_size = ff_hidden_size\n", + " self.dropout_p = dropout_p\n", + " self.query_in_size = in_size if query_in_size is None else query_in_size\n", + "\n", + " self.attention = MultiHeadAttention(self.in_size, self.head_size, self.num_heads, self.out_size, self.query_in_size)\n", + " # Если выход и вход attention-а имеют разный размер, то используем линейный слой на residual connection-е\n", + " self.adapt_residual = nn.Linear(self.query_in_size, self.out_size) if self.query_in_size != self.out_size else nn.Identity()\n", + "\n", + " self.norm_1 = nn.LayerNorm(self.out_size)\n", + " self.dropout_1 = nn.Dropout(self.dropout_p)\n", + "\n", + " self.feed_forward = nn.Sequential(OrderedDict([\n", + " (\"lin_1\", nn.Linear(self.out_size, self.ff_hidden_size)),\n", + " (\"act\", nn.ReLU()),\n", + " (\"lin_2\", nn.Linear(self.ff_hidden_size, self.out_size)),\n", + " ]))\n", + "\n", + " self.norm_2 = nn.LayerNorm(self.out_size)\n", + " self.dropout_2 = nn.Dropout(self.dropout_p)\n", + "\n", + "\n", + " def forward(self, query, key, value):\n", + " \"\"\"\n", + " Args:\n", + " block_input: input to corresponding block\n", + " \"\"\"\n", + " # Получаем на вход 3 тензора batch_size x seq_len x in_size\n", + " attention_out = self.attention(query, key, value) # (batch_size, seq_len, out_size)\n", + " attention_residual_out = attention_out + self.adapt_residual(query)\n", + " norm_1_out = self.dropout_1(self.norm_1(attention_residual_out))\n", + "\n", + " # (batch_size, seq_len, out_size) -> (batch_size, seq_len, ff_hidden_size) -> (batch_size, seq_len, out_size)\n", + " ff_out = self.feed_forward(norm_1_out)\n", + " ff_residual_out = ff_out + norm_1_out\n", + " return self.dropout_2(self.norm_2(ff_residual_out))" + ], + "metadata": { + "id": "vAsmZjqEceXe" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing TransformerEncoderBlock for the encoder" + ], + "metadata": { + "id": "yTrJWG_bdA9f" + } + }, + { + "cell_type": "code", + "source": [ + "# We check the standard forward pass from the encoder\n", + "tmp_layer = TransformerEncoderBlock(\n", + " in_size=10,\n", + " head_size=7,\n", + " num_heads=2,\n", + " out_size=15,\n", + " ff_hidden_size=20,\n", + " dropout_p=0.1,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Wne9dA-BdDEI", + "outputId": "dbabfef9-790e-40fb-dc14-a372179c88d0" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "TransformerEncoderBlock(\n", + " (attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Linear(in_features=10, out_features=15, bias=True)\n", + " (norm_1): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_1): Dropout(p=0.1, inplace=False)\n", + " (feed_forward): Sequential(\n", + " (lin_1): Linear(in_features=15, out_features=20, bias=True)\n", + " (act): ReLU()\n", + " (lin_2): Linear(in_features=20, out_features=15, bias=True)\n", + " )\n", + " (norm_2): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_2): Dropout(p=0.1, inplace=False)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 23 + } + ] + }, + { + "cell_type": "code", + "source": [ + "tmp_input = torch.rand(2, 5, 10)\n", + "\n", + "print(\"Encoder-like input\")\n", + "print(f'Input shape: {tmp_input.shape}')\n", + "tmp_output = tmp_layer(tmp_input, tmp_input, tmp_input)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "yFAF2RRGdHGu", + "outputId": "dba6ef98-a7ec-4d34-ec49-c6831f278590" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Encoder-like input\n", + "Input shape: torch.Size([2, 5, 10])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing TransformerEncoderBlock for the decoder" + ], + "metadata": { + "id": "enhFxwBtdIlQ" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = TransformerEncoderBlock(\n", + " in_size=10,\n", + " head_size=7,\n", + " num_heads=2,\n", + " out_size=15,\n", + " ff_hidden_size=20,\n", + " dropout_p=0.1,\n", + " query_in_size=12,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "NFSh5Kzbdas5", + "outputId": "bb29185c-a9a4-4426-860f-e87467c50ae3" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "TransformerEncoderBlock(\n", + " (attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Linear(in_features=12, out_features=15, bias=True)\n", + " (norm_1): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_1): Dropout(p=0.1, inplace=False)\n", + " (feed_forward): Sequential(\n", + " (lin_1): Linear(in_features=15, out_features=20, bias=True)\n", + " (act): ReLU()\n", + " (lin_2): Linear(in_features=20, out_features=15, bias=True)\n", + " )\n", + " (norm_2): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_2): Dropout(p=0.1, inplace=False)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 25 + } + ] + }, + { + "cell_type": "code", + "source": [ + "# We check the forward pass from the decoder, where we mix information from the encoder and decoder." + ], + "metadata": { + "id": "ASr5ZUnWdbuw" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "tmp_input_q = torch.rand(2, 5, 12)\n", + "tmp_input_kv = torch.rand(2, 7, 10)\n", + "\n", + "print(\"Encoder+Decoder-like input\")\n", + "print(f'Input Q shape: {tmp_input_q.shape}')\n", + "print(f'Input KV shape: {tmp_input_kv.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input_q, tmp_input_kv, tmp_input_kv)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input_q, tmp_input_kv, tmp_output" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "zpId_xxbdfGb", + "outputId": "45b71cd9-3c28-4b4c-ebba-ee3d2ff3d084" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Encoder+Decoder-like input\n", + "Input Q shape: torch.Size([2, 5, 12])\n", + "Input KV shape: torch.Size([2, 7, 10])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### TransformerEncoder\n", + "\n", + "**Initialization:**\n", + "\n", + "* max_seq_len ~ maximum sequence length in tokens\n", + "* vocab_size ~ vocabulary size\n", + "* emb_size ~ input embedding size\n", + "* num_layers ~ number of TransformerEncoderBlocks\n", + "* att_out_size ~ output embedding size from attention and the block\n", + "* att_head_size ~ embedding size of Q, K, V matrices after transformation\n", + "* num_heads ~ number of attention heads\n", + "* ff_hidden_size ~ hidden size for the feed-forward layers\n", + "* dropout_p ~ dropout probability\n", + "\n", + "**Forward:**\n", + "\n", + "* encoder_input ~ tokens input to the encoder before embedding" + ], + "metadata": { + "id": "bAAU2hKcdh4z" + } + }, + { + "cell_type": "code", + "source": [ + "class TransformerEncoderBlock(nn.Module):\n", + " \"\"\"\n", + " Class with one full block within transformer's encoder\n", + " \"\"\"\n", + " def __init__(self, in_size, head_size, num_heads, out_size, ff_hidden_size, dropout_p=0.2, query_in_size=None):\n", + " \"\"\"\n", + " Args:\n", + " in_size: input embedding size\n", + " head_size: size of each attention head\n", + " num_heads: number of attention heads\n", + " out_size: output embedding size\n", + " ff_hidden_size: hidden size for feed-forward net\n", + " dropout_p: probability for dropout\n", + " query_in_size: embedding size for the query input (if not provided, use in_size)\n", + " \"\"\"\n", + " super(TransformerEncoderBlock, self).__init__()\n", + "\n", + " # Store all passed layer hyperparameters\n", + " self.in_size = in_size\n", + " self.head_size = head_size\n", + " self.num_heads = num_heads\n", + " self.out_size = out_size\n", + " self.ff_hidden_size = ff_hidden_size\n", + " self.dropout_p = dropout_p\n", + " self.query_in_size = in_size if query_in_size is None else query_in_size\n", + "\n", + " self.attention = ...\n", + " self.adapt_residual = ...\n", + "\n", + " self.norm_1 = ...\n", + " self.dropout_1 = ...\n", + "\n", + " self.feed_forward = nn.Sequential(OrderedDict([\n", + " (\"lin_1\", ...),\n", + " (\"act\", ...),\n", + " (\"lin_2\", ...),\n", + " ]))\n", + "\n", + " self.norm_2 = ...\n", + " self.dropout_2 = ...\n", + "\n", + "\n", + " def forward(self, query, key, value):\n", + " \"\"\"\n", + " Args:\n", + " block_input: input to corresponding block\n", + " \"\"\"\n", + " # Input of 3 tensors batch_size x seq_len x in_size\n", + " attention_out = ...\n", + " attention_residual_out = ...\n", + " norm_1_out = ...\n", + "\n", + " ff_out = ...\n", + " ff_residual_out = ...\n", + " norm_2_out = ...\n", + " return norm_2_out" + ], + "metadata": { + "id": "vg8IP5CqdwAb" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing TransformerEncoder" + ], + "metadata": { + "id": "1GO52uVYd-Qb" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = TransformerEncoder(\n", + " max_seq_len=20,\n", + " vocab_size=10000,\n", + " emb_size=10,\n", + " num_layers=2,\n", + " att_head_size=7,\n", + " num_heads=2,\n", + " att_out_size=15,\n", + " ff_hidden_size=20,\n", + " dropout_p=0.1,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "id": "BBlrE6acd9fP" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "tmp_input = torch.randint(10000, (2, 5))\n", + "\n", + "print(f'Input shape: {tmp_input.shape}')\n", + "tmp_output = tmp_layer(tmp_input)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input, tmp_output" + ], + "metadata": { + "id": "6HJUmeR8eFe8" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.4 Decoder\n", + "\n", + "#### Picture\n", + "\n", + "\n", + "\n", + "#### TransformerDecoderBlock\n", + "\n", + "**Initialization:**\n", + "\n", + "* in_size ~ input embedding size\n", + "* head_size ~ size of Q, K, V matrix embeddings after transformation\n", + "* num_heads ~ number of attention heads\n", + "* out_size ~ output embedding size for attention and the block\n", + "* ff_hidden_size ~ hidden size for feed-forward layers\n", + "* dropout_p ~ dropout probability\n", + "* encoder_out_size ~ encoder output embedding size (if None, defaults to in_size)\n", + "\n", + "**Forward:**\n", + "\n", + "* decoder_emb ~ tensor from the previous block or embeddings with positional encodings\n", + "* encoder_output ~ output tensor from the corresponding encoder" + ], + "metadata": { + "id": "RYI4RNxyeGR-" + } + }, + { + "cell_type": "code", + "source": [ + "class TransformerDecoderBlock(nn.Module):\n", + " \"\"\"\n", + " Class with one full block within transformer's decoder\n", + " \"\"\"\n", + " def __init__(self, in_size, head_size, num_heads, out_size, ff_hidden_size, dropout_p=0.2, encoder_out_size=None):\n", + " \"\"\"\n", + " Args:\n", + " in_size: input embedding size\n", + " head_size: size of each attention head\n", + " num_heads: number of attention heads\n", + " out_size: output embedding size\n", + " ff_hidden_size: hidden size for feed forward net\n", + " dropout_p: probability for dropout\n", + " encoder_out_size: embedding size of outputs from encoder (if not provided - same as in_size)\n", + " \"\"\"\n", + " super(TransformerDecoderBlock, self).__init__()\n", + "\n", + " # Запишем все переданые гиперпараметры слоя\n", + " self.in_size = in_size\n", + " self.head_size = head_size\n", + " self.num_heads = num_heads\n", + " self.out_size = out_size\n", + " self.ff_hidden_size = ff_hidden_size\n", + " self.dropout_p = dropout_p\n", + " self.encoder_out_size = in_size if encoder_out_size is None else encoder_out_size\n", + "\n", + "\n", + " self.masked_attention = MultiHeadAttention(self.in_size, self.head_size, self.num_heads, self.out_size)\n", + " # Если выход и вход attention-а имеют разный размер, то используем линейный слой на residual connection-е\n", + " self.adapt_residual = nn.Linear(self.in_size, self.out_size) if self.in_size != self.out_size else nn.Identity()\n", + " self.norm = nn.LayerNorm(self.out_size)\n", + " self.dropout = nn.Dropout(self.dropout_p)\n", + " self.encoder_block = TransformerEncoderBlock(self.encoder_out_size, self.head_size, self.num_heads, self.out_size, self.ff_hidden_size, self.dropout_p, self.out_size)\n", + "\n", + "\n", + " def forward(self, decoder_emb, encoder_output):\n", + " \"\"\"\n", + " Args:\n", + " decoder_emb: decoder sequence after embed\n", + " encoder_output: output from encoder\n", + " \"\"\"\n", + " # Получаем на вход тензор batch_size x seq_len x in_size и тензор batch_size x encoder_seq_len x encoder_out_size\n", + " mask = make_decoder_mask(decoder_emb) # batch_size x 1 x seq_len x seq_len\n", + " attention = self.masked_attention(decoder_emb, decoder_emb, decoder_emb, mask=mask) # batch_size x seq_len x out_size\n", + " mmha_out = self.dropout(self.norm(attention + self.adapt_residual(decoder_emb)))\n", + "\n", + " return self.encoder_block(mmha_out, encoder_output, encoder_output) # batch_size x seq_len x out_size" + ], + "metadata": { + "id": "WI673PnTeOn8" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing TransformerDecoderBlock" + ], + "metadata": { + "id": "EBJy06RwfHeY" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = TransformerDecoderBlock(\n", + " in_size=10,\n", + " head_size=7,\n", + " num_heads=2,\n", + " out_size=15,\n", + " ff_hidden_size=20,\n", + " dropout_p=0.1,\n", + " encoder_out_size=12,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "zsXTUhlIfAkd", + "outputId": "1c2ea4d0-e0c1-4c05-e2a0-1ec30768c896" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "TransformerDecoderBlock(\n", + " (masked_attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Linear(in_features=10, out_features=15, bias=True)\n", + " (norm): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (encoder_block): TransformerEncoderBlock(\n", + " (attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Identity()\n", + " (norm_1): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_1): Dropout(p=0.1, inplace=False)\n", + " (feed_forward): Sequential(\n", + " (lin_1): Linear(in_features=15, out_features=20, bias=True)\n", + " (act): ReLU()\n", + " (lin_2): Linear(in_features=20, out_features=15, bias=True)\n", + " )\n", + " (norm_2): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_2): Dropout(p=0.1, inplace=False)\n", + " )\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 29 + } + ] + }, + { + "cell_type": "code", + "source": [ + "# Testing the forward pass in the decoder, where we mix information from the encoder and decoder\n", + "tmp_input_decoder = torch.rand(2, 5, 10)\n", + "tmp_output_encoder = torch.rand(2, 7, 12)\n", + "\n", + "print(\"Encoder+Decoder-like input\")\n", + "print(f'Decoder input shape: {tmp_input_decoder.shape}')\n", + "print(f'Encoder output shape: {tmp_output_encoder.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input_decoder, tmp_output_encoder)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input_decoder, tmp_output_encoder" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "zulYoZ6GfKVP", + "outputId": "1c43ddb6-0bee-4885-8516-b023c3ec874f" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Encoder+Decoder-like input\n", + "Decoder input shape: torch.Size([2, 5, 10])\n", + "Encoder output shape: torch.Size([2, 7, 12])\n", + "Output shape: torch.Size([2, 5, 15])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### TransformerDecoder\n", + "\n", + "**Initialization:**\n", + "\n", + "* max_seq_len ~ maximum token length of the sequence\n", + "* vocab_size ~ size of the vocabulary\n", + "*\temb_size ~ input embedding size\n", + "* num_layers ~ number of TransformerEncoderBlocks\n", + "*\tatt_out_size ~ output embedding size for attention and the block\n", + "*\tatt_head_size ~ embedding size of the Q, K, V matrices after transformation\n", + "*\tnum_heads ~ number of attention heads\n", + "*\tff_hidden_size ~ hidden size for the feed-forward layers\n", + "*\tdropout_p ~ dropout probability\n", + "*\tencoder_out_size ~ encoder output embedding size (if None, defaults to in_size)\n", + "\n", + "**Forward:**\n", + "\n", + "*\tdecoder_input ~ input tokens to the decoder before embeddings\n", + "*\tencoder_output ~ output tensor from the corresponding encoder" + ], + "metadata": { + "id": "xRLiJy5FfOUq" + } + }, + { + "cell_type": "code", + "source": [ + "class TransformerDecoder(nn.Module):\n", + " \"\"\"\n", + " Class for decoder within transformer.\n", + " \"\"\"\n", + " def __init__(self, max_seq_len, vocab_size, emb_size, num_layers, att_out_size, att_head_size, num_heads, ff_hidden_size, dropout_p, encoder_out_size=None):\n", + " \"\"\"\n", + " Args:\n", + " max_seq_len : maximum length of input sequence\n", + " vocab_size: size of the vocabulary\n", + " emb_size: embeddings size\n", + " num_layers: number of encoder layers\n", + " att_out_size: output size for attention and each encoder block\n", + " att_head_size: size of each attention head\n", + " num_heads: number of heads in multihead attention\n", + " ff_hidden_size: hidden size for feed forward net\n", + " dropout_p: probability for dropout\n", + " encoder_out_size: embedding size of outputs from encoder (if not provided - same as in_size)\n", + " \"\"\"\n", + " super(TransformerDecoder, self).__init__()\n", + "\n", + " # Запишем все переданые гиперпараметры слоя\n", + " self.max_seq_len = max_seq_len\n", + " self.vocab_size = vocab_size\n", + " self.emb_size = emb_size\n", + " self.num_layers = num_layers\n", + " self.att_out_size = att_out_size\n", + " self.att_head_size = att_head_size\n", + " self.num_heads = num_heads\n", + " self.ff_hidden_size = ff_hidden_size\n", + " self.dropout_p = dropout_p\n", + " self.encoder_out_size = in_size if encoder_out_size is None else encoder_out_size\n", + "\n", + " self.embedding_layer = nn.Embedding(self.vocab_size, self.emb_size)\n", + " self.positional_encoder = PositionalEncoding(self.max_seq_len, self.emb_size)\n", + " self.dropout = nn.Dropout(self.dropout_p)\n", + "\n", + " self.decoder_blocks = nn.ModuleDict({\n", + " f\"decoder_block_{i}\": TransformerDecoderBlock(\n", + " in_size=self.emb_size if i==0 else self.att_out_size,\n", + " head_size=self.att_head_size,\n", + " num_heads=self.num_heads,\n", + " out_size=self.att_out_size,\n", + " ff_hidden_size=self.ff_hidden_size,\n", + " dropout_p=self.dropout_p,\n", + " encoder_out_size=self.encoder_out_size,\n", + " ) for i in range(self.num_layers)\n", + " })\n", + "\n", + " self.fc = nn.Linear(self.att_out_size, self.vocab_size)\n", + "\n", + " def forward(self, decoder_input, encoder_output):\n", + " \"\"\"\n", + " Args:\n", + " decoder_input:\n", + " encoder_output:\n", + " Returns:\n", + " out: output vector\n", + " \"\"\"\n", + " # Получаем на вход batch_size x seq_len и batch_size x encoder_seq_len x encoder_out_size\n", + " decoder_emb = self.embedding_layer(decoder_input) # batch_size x seq_len x emb_size\n", + " decoder_emb = self.positional_encoder(decoder_emb)\n", + "\n", + " out = self.dropout(decoder_emb)\n", + "\n", + " for block in self.decoder_blocks.values():\n", + " out = block(out, encoder_output) # batch_size x seq_len x att_out_size\n", + "\n", + " return self.fc(out)" + ], + "metadata": { + "id": "aLuT6JaQfby_" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Testing TransformerDecoder" + ], + "metadata": { + "id": "3vBYyZrXzQHW" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = TransformerDecoder(\n", + " max_seq_len=20,\n", + " vocab_size=10000,\n", + " emb_size=10,\n", + " num_layers=2,\n", + " att_head_size=7,\n", + " num_heads=2,\n", + " att_out_size=15,\n", + " ff_hidden_size=20,\n", + " dropout_p=0.1,\n", + " encoder_out_size=12,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "WOnV1Y20zV9G", + "outputId": "2375326f-a3f6-4e95-dea0-aba9ca82b975" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "TransformerDecoder(\n", + " (embedding_layer): Embedding(10000, 10)\n", + " (positional_encoder): PositionalEncoding()\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (decoder_blocks): ModuleDict(\n", + " (decoder_block_0): TransformerDecoderBlock(\n", + " (masked_attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=10, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Linear(in_features=10, out_features=15, bias=True)\n", + " (norm): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (encoder_block): TransformerEncoderBlock(\n", + " (attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Identity()\n", + " (norm_1): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_1): Dropout(p=0.1, inplace=False)\n", + " (feed_forward): Sequential(\n", + " (lin_1): Linear(in_features=15, out_features=20, bias=True)\n", + " (act): ReLU()\n", + " (lin_2): Linear(in_features=20, out_features=15, bias=True)\n", + " )\n", + " (norm_2): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_2): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " (decoder_block_1): TransformerDecoderBlock(\n", + " (masked_attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Identity()\n", + " (norm): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout): Dropout(p=0.1, inplace=False)\n", + " (encoder_block): TransformerEncoderBlock(\n", + " (attention): MultiHeadAttention(\n", + " (query_matrix): Linear(in_features=15, out_features=14, bias=False)\n", + " (key_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (value_matrix): Linear(in_features=12, out_features=14, bias=False)\n", + " (out): Linear(in_features=14, out_features=15, bias=True)\n", + " )\n", + " (adapt_residual): Identity()\n", + " (norm_1): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_1): Dropout(p=0.1, inplace=False)\n", + " (feed_forward): Sequential(\n", + " (lin_1): Linear(in_features=15, out_features=20, bias=True)\n", + " (act): ReLU()\n", + " (lin_2): Linear(in_features=20, out_features=15, bias=True)\n", + " )\n", + " (norm_2): LayerNorm((15,), eps=1e-05, elementwise_affine=True)\n", + " (dropout_2): Dropout(p=0.1, inplace=False)\n", + " )\n", + " )\n", + " )\n", + " (fc): Linear(in_features=15, out_features=10000, bias=True)\n", + ")" + ] + }, + "metadata": {}, + "execution_count": 44 + } + ] + }, + { + "cell_type": "code", + "source": [ + "# We will test the Transformer model by passing through both the encoder and decoder, ensuring that information from the encoder is correctly used in the decoder for sequence generation.\n", + "tmp_input_decoder = torch.randint(10000, (2, 5))\n", + "tmp_output_encoder = torch.rand(2, 7, 12)\n", + "\n", + "print(\"Encoder+Decoder-like input\")\n", + "print(f'Decoder input shape: {tmp_input_decoder.shape}')\n", + "print(f'Encoder output shape: {tmp_output_encoder.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input_decoder, tmp_output_encoder)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input_decoder, tmp_output_encoder" + ], + "metadata": { + "id": "WU6z4-uczXom" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.5 Transformer" + ], + "metadata": { + "id": "X7PUNDaT0OBW" + } + }, + { + "cell_type": "code", + "source": [ + "class Transformer(nn.Module):\n", + " \"\"\"\n", + " Class for full encoder-decoder transformer\n", + " \"\"\"\n", + " def __init__(\n", + " self,\n", + " max_seq_len,\n", + " vocab_size,\n", + " emb_size,\n", + "\n", + " num_encoder_layers,\n", + " enc_att_out_size,\n", + " enc_att_head_size,\n", + " enc_num_heads,\n", + " enc_ff_hidden_size,\n", + " enc_dropout_p,\n", + "\n", + " num_decoder_layers,\n", + " dec_att_out_size,\n", + " dec_att_head_size,\n", + " dec_num_heads,\n", + " dec_ff_hidden_size,\n", + " dec_dropout_p,\n", + " ):\n", + " super(Transformer, self).__init__()\n", + "\n", + " # Store all the passed hyperparameters of the model\n", + " self.max_seq_len = max_seq_len\n", + " self.vocab_size = vocab_size\n", + " self.emb_size = emb_size\n", + "\n", + " self.num_encoder_layers = num_encoder_layers\n", + " self.enc_att_out_size = enc_att_out_size\n", + " self.enc_att_head_size = enc_att_head_size\n", + " self.enc_num_heads = enc_num_heads\n", + " self.enc_ff_hidden_size = enc_ff_hidden_size\n", + " self.enc_dropout_p = enc_dropout_p\n", + "\n", + " self.num_decoder_layers = num_decoder_layers\n", + " self.dec_att_out_size = dec_att_out_size\n", + " self.dec_att_head_size = dec_att_out_size\n", + " self.dec_num_heads = dec_num_heads\n", + " self.dec_ff_hidden_size = dec_ff_hidden_size\n", + " self.dec_dropout_p = dec_dropout_p\n", + "\n", + " # Encoder\n", + " self.encoder = TransformerEncoder(\n", + " max_seq_len=self.max_seq_len,\n", + " vocab_size=self.vocab_size,\n", + " emb_size=self.emb_size,\n", + " num_layers=self.num_encoder_layers,\n", + " att_head_size=self.enc_att_head_size,\n", + " num_heads=self.enc_num_heads,\n", + " att_out_size=self.enc_att_out_size,\n", + " ff_hidden_size=self.enc_ff_hidden_size,\n", + " dropout_p=self.enc_dropout_p,\n", + " )\n", + "\n", + " # Decoder\n", + " self.decoder = TransformerDecoder(\n", + " max_seq_len=self.max_seq_len,\n", + " vocab_size=self.vocab_size,\n", + " emb_size=self.emb_size,\n", + " num_layers=self.num_decoder_layers,\n", + " att_head_size=self.dec_att_head_size,\n", + " num_heads=self.dec_num_heads,\n", + " att_out_size=self.dec_att_out_size,\n", + " ff_hidden_size=self.dec_ff_hidden_size,\n", + " dropout_p=self.dec_dropout_p,\n", + " encoder_out_size=self.enc_att_out_size,\n", + " )\n", + "\n", + " def forward(self, encoder_input, decoder_input):\n", + " \"\"\"\n", + " Args:\n", + " encoder_input: input to encoder\n", + " decoder_input: input to decoder\n", + " out:\n", + " out: final tensor with logits of each word in vocab\n", + " \"\"\"\n", + " # Input has shape batch_size x enc_seq_len and batch_size x dec_seq_len\n", + " encoder_output = self.encoder(encoder_input) # (batch_size, enc_seq_len, enc_att_out_size)\n", + "\n", + " return self.decoder(decoder_input, encoder_output) # (batch_size, dec_seq_len, vocab_size)" + ], + "metadata": { + "id": "nkvAAfu20OYC" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "### 1.6 Testing" + ], + "metadata": { + "id": "6ZRK9UIn0Yvg" + } + }, + { + "cell_type": "code", + "source": [ + "tmp_layer = Transformer(\n", + " max_seq_len=20,\n", + " vocab_size=10000,\n", + " emb_size=10,\n", + "\n", + " num_encoder_layers=3,\n", + " enc_att_head_size=7,\n", + " enc_num_heads=3,\n", + " enc_att_out_size=20,\n", + " enc_ff_hidden_size=30,\n", + " enc_dropout_p=0.2,\n", + "\n", + " num_decoder_layers=2,\n", + " dec_att_head_size=7,\n", + " dec_num_heads=2,\n", + " dec_att_out_size=15,\n", + " dec_ff_hidden_size=20,\n", + " dec_dropout_p=0.1,\n", + ")\n", + "\n", + "tmp_layer" + ], + "metadata": { + "id": "365uJEiR0bNS" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "tmp_input_encoder = torch.randint(10000, (2, 9))\n", + "tmp_input_decoder = torch.randint(10000, (2, 5))\n", + "\n", + "print(f'Encoder input shape: {tmp_input_encoder.shape}')\n", + "print(f'Decoder input shape: {tmp_input_decoder.shape}')\n", + "\n", + "tmp_output = tmp_layer(tmp_input_encoder, tmp_input_decoder)\n", + "print(f'Output shape: {tmp_output.shape}')\n", + "\n", + "del tmp_input_decoder, tmp_input_encoder" + ], + "metadata": { + "id": "7eEBPO9F0y3r" + }, + "execution_count": null, + "outputs": [] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/week07_LLM_v1/final_llama_practice.ipynb b/week07_LLM_v1/final_llama_practice.ipynb new file mode 100644 index 0000000..c02d330 --- /dev/null +++ b/week07_LLM_v1/final_llama_practice.ipynb @@ -0,0 +1,585 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "8373863c", + "metadata": {}, + "source": [ + "# Fine-Tuning LLaMA Tutorial with Explanations" + ] + }, + { + "cell_type": "markdown", + "id": "1e21a994", + "metadata": {}, + "source": [ + "\n", + "In this practical seminar, we will go through the full pipeline for fine-tuning the LLaMA model.\n", + "Each section includes theoretical context and links to documentation to ensure a comprehensive understanding \n", + "of the implementation. **Note**: Ensure you have installed all required packages as listed below.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "998af1f6", + "metadata": {}, + "outputs": [], + "source": [ + "!pip install -q transformers accelerate bitsandbytes datasets" + ] + }, + { + "cell_type": "markdown", + "id": "aec82bf5", + "metadata": {}, + "source": [ + "\n", + "**Theory**: We begin by installing essential packages, including `transformers` for the model, `datasets` for handling our dataset, \n", + "and `accelerate` to efficiently distribute computations. `bitsandbytes` allows for lower-precision quantization, optimizing performance.\n", + "Refer to [Transformers documentation](https://huggingface.co/docs/transformers/index) for more details.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8cd881f3", + "metadata": {}, + "outputs": [], + "source": [ + "!nvidia-smi" + ] + }, + { + "cell_type": "markdown", + "id": "0f6c8736", + "metadata": {}, + "source": [ + "### Checking GPU Availability\n", + "`nvidia-smi` command is used to verify GPU status and memory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "298a9326", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments\n", + "from datasets import Dataset\n", + "import torch\n", + "import random\n", + "import logging\n", + "\n", + "logging.basicConfig(level=logging.INFO)\n", + "logger = logging.getLogger(__name__)\n", + "logger.info(\"Libraries imported successfully.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "01899fba", + "metadata": {}, + "source": [ + "\n", + "### Importing Required Libraries\n", + "\n", + "- `AutoModelForCausalLM` and `AutoTokenizer` from Hugging Face's `transformers` for loading a pre-trained language model.\n", + "- `Dataset` from `datasets` for creating and managing data efficiently.\n", + "- `Trainer` and `TrainingArguments` for configuring and training the model.\n", + "- `torch` for handling tensor operations.\n", + "- `logging` for enabling info-level logging to track progress.\n", + "\n", + "For more on each of these, refer to the [datasets library](https://huggingface.co/docs/datasets/) and [transformers library](https://huggingface.co/docs/transformers/) documentation.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1780d5af", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "def gen_dataset(size, digits=(2, 18), operation='addition'):\n", + " logger.info(\"Generating dataset with varied difficulty.\")\n", + " for _ in range(size):\n", + " a = random.randint(10**digits[0], 10**digits[1])\n", + " b = random.randint(10**digits[0], 10**digits[1])\n", + " if operation == 'addition':\n", + " c = a + b\n", + " prompt = f'Calculate the sum of {a} and {b}: {c}'\n", + " elif operation == 'multiplication':\n", + " c = a * b\n", + " prompt = f'Calculate the product of {a} and {b}: {c}'\n", + " yield {'prompt': prompt, 'response': str(c)}\n" + ] + }, + { + "cell_type": "markdown", + "id": "63d91a12", + "metadata": {}, + "source": [ + "\n", + "### Dataset Generation Function\n", + "\n", + "This function generates a dataset of simple arithmetic problems, with each example containing a prompt (arithmetic question) \n", + "and the corresponding answer. Here, we're generating synthetic data for fine-tuning purposes.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c4060a26", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "train_dataset = Dataset.from_generator(gen_dataset, gen_kwargs={\"size\": 400, \"digits\": (2, 18), \"operation\": \"addition\"})\n", + "test_dataset = Dataset.from_generator(gen_dataset, gen_kwargs={\"size\": 40, \"digits\": (8, 10), \"operation\": \"multiplication\"})\n", + "logger.info(f\"Generated train dataset size: {len(train_dataset)}, test dataset size: {len(test_dataset)}\")\n", + "train_dataset\n" + ] + }, + { + "cell_type": "markdown", + "id": "13164565", + "metadata": {}, + "source": [ + "\n", + "### Creating Train and Test Datasets\n", + "\n", + "Using `from_generator`, we generate two datasets (train and test) for addition and multiplication operations. \n", + "The [datasets.from_generator](https://huggingface.co/docs/datasets/v2.1.0/en/package_reference/main_classes#datasets.Dataset.from_generator) function creates a `Dataset` from a generator function.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b12ce55e", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "from huggingface_hub import login\n", + "login()\n" + ] + }, + { + "cell_type": "markdown", + "id": "5567b504", + "metadata": {}, + "source": [ + "\n", + "### Logging into Hugging Face Hub\n", + "\n", + "This cell logs into the Hugging Face Hub for accessing pretrained models and saving fine-tuned models. \n", + "See [huggingface_hub login documentation](https://huggingface.co/docs/huggingface_hub/quick_start#step-3-log-in-to-the-hugging-face-hub).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "93a728ef", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n", + "bnb_config = BitsAndBytesConfig(\n", + " load_in_4bit=True,\n", + " bnb_4bit_use_double_quant=True,\n", + " bnb_4bit_quant_type=\"nf4\",\n", + " bnb_4bit_compute_dtype=torch.bfloat16\n", + ")\n" + ] + }, + { + "cell_type": "markdown", + "id": "02dc767d", + "metadata": {}, + "source": [ + "\n", + "### Configuring BitsAndBytes Quantization\n", + "\n", + "We configure BitsAndBytes for 4-bit quantization to reduce memory usage, making model fine-tuning more efficient. \n", + "Read more about [BitsAndBytesConfig](https://huggingface.co/docs/transformers/main_classes/configuration#transformers.BitsAndBytesConfig).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b74f42ba", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "model_name = \"meta-llama/Llama-3.2-1B\"\n", + "tokenizer = AutoTokenizer.from_pretrained(model_name, padding_side=\"right\", add_eos_token=True, add_bos_token=True)\n", + "tokenizer.pad_token = tokenizer.eos_token\n", + "model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=bnb_config, device_map=\"auto\")\n", + "logger.info(f\"Model {model_name} loaded with {model.num_parameters()} parameters.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "218e5e92", + "metadata": {}, + "source": [ + "\n", + "### Loading the Pre-trained Model and Tokenizer\n", + "\n", + "We initialize the LLaMA model and tokenizer, setting `add_eos_token` and `add_bos_token` for proper tokenization handling.\n", + "The `device_map=\"auto\"` automatically distributes model parts across available devices, improving efficiency. \n", + "For more, see [AutoTokenizer documentation](https://huggingface.co/docs/transformers/main_classes/tokenizer).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8b41bd1d", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "def tokenize_data(prompt):\n", + " return tokenizer(prompt['prompt'])\n", + "\n", + "train_dataset = train_dataset.map(tokenize_data)\n", + "test_dataset = test_dataset.map(tokenize_data)\n" + ] + }, + { + "cell_type": "markdown", + "id": "a81907b8", + "metadata": {}, + "source": [ + "\n", + "### Tokenizing the Dataset\n", + "\n", + "This function tokenizes each prompt from the dataset, enabling the model to process them effectively. \n", + "The `map` function applies the tokenization across all dataset examples. For more, see [map documentation](https://huggingface.co/docs/datasets/process#map).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a71e0279", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "import matplotlib.pyplot as plt\n", + "\n", + "def plot_data_lengths(tokenized_train_dataset, tokenized_val_dataset):\n", + " lengths = [len(x['input_ids']) for x in tokenized_train_dataset]\n", + " lengths += [len(x['input_ids']) for x in tokenized_val_dataset]\n", + " plt.figure(figsize=(10, 6))\n", + " plt.hist(lengths, bins=20, alpha=0.7, color='blue')\n", + " plt.xlabel('Length of input_ids')\n", + " plt.ylabel('Frequency')\n", + " plt.title('Distribution of Lengths of input_ids')\n", + " plt.show()\n", + "\n", + "plot_data_lengths(train_dataset, test_dataset)\n" + ] + }, + { + "cell_type": "markdown", + "id": "6b359ff3", + "metadata": {}, + "source": [ + "\n", + "### Plotting Data Lengths\n", + "\n", + "This section provides a histogram of tokenized prompt lengths, visualizing the distribution of input lengths.\n", + "This can help in setting max length parameters for padding/truncation during training.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "538b1589", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "max_length = 32\n", + "\n", + "def tokenize_data(prompt):\n", + " result = tokenizer(\n", + " prompt['prompt'],\n", + " truncation=True,\n", + " max_length=max_length,\n", + " padding=\"max_length\",\n", + " )\n", + " result[\"labels\"] = result[\"input_ids\"].copy()\n", + " return result\n", + "\n", + "train_dataset = train_dataset.map(tokenize_data)\n", + "test_dataset = test_dataset.map(tokenize_data)\n", + "logger.info(\"Tokenization complete with diagnostic shape checks.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "24802608", + "metadata": {}, + "source": [ + "\n", + "### Applying Padding and Label Creation\n", + "\n", + "This code modifies tokenization to include truncation, padding, and copying input IDs to labels, which is necessary \n", + "for certain types of language model training. See [padding and truncation documentation](https://huggingface.co/docs/transformers/padding_and_truncation).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "40b64faa", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "from peft import prepare_model_for_kbit_training\n", + "\n", + "model.gradient_checkpointing_enable()\n", + "model = prepare_model_for_kbit_training(model)\n" + ] + }, + { + "cell_type": "markdown", + "id": "09a49f4a", + "metadata": {}, + "source": [ + "\n", + "### Preparing Model for Efficient Training with K-Bit Precision\n", + "\n", + "This section enables gradient checkpointing, reducing memory usage during training. \n", + "For more, see [gradient checkpointing](https://huggingface.co/docs/transformers/main_classes/accelerate#transformers.Accelerate.gradient_checkpointing_enable).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "da38a19d", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "def print_trainable_parameters(model):\n", + " trainable_params = 0\n", + " all_param = 0\n", + " for _, param in model.named_parameters():\n", + " all_param += param.numel()\n", + " if param.requires_grad:\n", + " trainable_params += param.numel()\n", + " print(f\"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "f7cbf314", + "metadata": {}, + "source": [ + "\n", + "### Printing Trainable Parameters\n", + "\n", + "This function calculates and displays the number of trainable parameters in the model, a critical metric for efficient fine-tuning.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1339ec0a", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "from peft import LoraConfig, get_peft_model\n", + "\n", + "config = LoraConfig(\n", + " r=32,\n", + " lora_alpha=64,\n", + " target_modules=[\n", + " \"q_proj\",\n", + " \"k_proj\",\n", + " \"v_proj\",\n", + " \"o_proj\",\n", + " \"gate_proj\",\n", + " \"up_proj\",\n", + " \"down_proj\",\n", + " \"lm_head\",\n", + " ],\n", + " bias=\"none\",\n", + " lora_dropout=0.05,\n", + " task_type=\"CAUSAL_LM\",\n", + ")\n", + "\n", + "model = get_peft_model(model, config)\n", + "print_trainable_parameters(model)\n" + ] + }, + { + "cell_type": "markdown", + "id": "0860579a", + "metadata": {}, + "source": [ + "\n", + "### Configuring and Applying LoRA\n", + "\n", + "LoRA (Low-Rank Adaptation) configuration reduces memory and computational requirements, focusing on specific layers. \n", + "For more, check [LoRA documentation](https://huggingface.co/docs/peft/api/lora).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "762d3268", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "training_args = TrainingArguments(\n", + " output_dir=\"./llama_finetuned\",\n", + " per_device_train_batch_size=1,\n", + " per_device_eval_batch_size=1,\n", + " gradient_accumulation_steps=4,\n", + " num_train_epochs=5,\n", + " learning_rate=3e-5,\n", + " fp16=True,\n", + " logging_steps=50,\n", + " evaluation_strategy=\"epoch\",\n", + " save_strategy=\"epoch\",\n", + " logging_dir='./logs',\n", + ")\n", + "logger.info(\"Training configuration set with advanced parameters.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "59ab402f", + "metadata": {}, + "source": [ + "\n", + "### Setting Training Arguments\n", + "\n", + "Here we define `TrainingArguments` for fine-tuning, such as batch size, learning rate, and logging. \n", + "For more details, refer to [TrainingArguments documentation](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "29b551c0", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "trainer = Trainer(\n", + " model=model,\n", + " args=training_args,\n", + " train_dataset=train_dataset,\n", + " eval_dataset=test_dataset,\n", + ")\n", + "logger.info(\"Trainer initialized with training and evaluation datasets.\")\n", + "trainer.train()\n" + ] + }, + { + "cell_type": "markdown", + "id": "fed8c6f0", + "metadata": {}, + "source": [ + "\n", + "### Initializing and Training the Model\n", + "\n", + "The `Trainer` class simplifies model training, managing the training loop, evaluation, and logging.\n", + "Check the [Trainer documentation](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) for more information.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "316803c7", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "eval_results = trainer.evaluate()\n", + "logger.info(f\"Evaluation Results: {eval_results}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "4a4dd7c6", + "metadata": {}, + "source": [ + "### Model Evaluation\n", + "Evaluates the fine-tuned model using the test dataset and logs results." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3db6c2ab", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "for i in range(5):\n", + " example = test_dataset[i]\n", + " input_ids = tokenizer(example['prompt'], return_tensors=\"pt\").input_ids\n", + " output_ids = model.generate(input_ids, max_new_tokens=32)\n", + " output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)\n", + " print(f\"Input: {example['prompt']})\n", + "Expected Output: {example['response']}\n", + "Model Output: {output_text}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "c42c2f00", + "metadata": {}, + "source": [ + "\n", + "### Testing Model Outputs\n", + "\n", + "Generates outputs for the test set to compare model predictions with expected responses.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4f112b5d", + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "sample_inputs = [\"Calculate the sum of 10 and 15:\", \"Calculate the sum of 4 and 6:\"]\n", + "for input_text in sample_inputs:\n", + " input_ids = tokenizer(input_text, return_tensors=\"pt\").input_ids\n", + " output_ids = model.generate(input_ids)\n", + " output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)\n", + " print(f\"Input: {input_text}\n", + "Output: {output_text}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "70f47ee6", + "metadata": {}, + "source": [ + "\n", + "### Additional Test Cases\n", + "\n", + "Here, we input new arithmetic prompts to see how the model generalizes to similar tasks beyond the test dataset.\n" + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 5 +}